I do that sometimes. Usually I avoid contact or just look at something random. But sometimes someone else, or something in the back of my head goes you're supposed to look at the person you're talking to...then I proceed to stare awkwardly. But when you're not supposed to....I think it helps if you give yourself something to look at. Like your phone or whatever you carry around. Or just some random thing out there. May not be much less awkward though if you're staring very intensely at a fire hydrant.