I sometimes find people's eyes to be distracting -- it's hard to stay focused on what I'm talking about because the "eye language" drowns it out. I'm not sure if this is similar to what you experience or not.
Best advice I can give you is to force yourself to look at their breasts instead, and over time this will become a regular habit. Try it!
