Unless someone is hard of hearing then there is no need to be looking at the other person all the time! That can be intimidating.
Some kind of eye contact makes for much better communication but I guess it varies depending on the situation and your mood.
No eye contact with someone would lead me to think they were not worth my attention and as someone has already mentioned, no eye contact is often a symptom that someone is lying or has something to hide though in my experience the worst liars and con men/women can you look straight in the eye as they lie/con with no conscience at all.
Comfortable eye contact is good and makes for better communication. If I am not sure we are communicating I often go for more eye contact.