I don't currently drive, but that is more because I never learned how to parallel park which is required for my driver's test more than anything.

I definitely felt anxious about how other drivers perceived my driving though, if I was going too slow, people beeping their horns, etc. In a sense you don't have to interact with people when you are driving because you're in your car, but there's still a social situation when you are driving on roads with traffic because there are other drivers involved. It just wasn't as bad for me as my main problem has always been more interpersonal avoidance.