The eyes are where to look and I find you reach a natural moment where it's right to look away for a moment and then back to it doesn't become intimidating.
...obviously it depends on how 'involved' you are with the eye-contactee...
With a larger group I do the same but if more than one person is listening it is polite to give eye contact to those people as well even if only momentarily.
It really isn't a big deal and the more you make it one, the more pressure you put yourself under.