I think this is a big deal, checking out your perception of his perceptions. I don't think too many people are good readers of other people, especially when it relates to reading others about us. And when we have an emotional reaction to our wrong perceptions, that's the cost. Most people are hard to read, and some (unlike your T) either can't or won't tell you what they think.

What has been helpful to me when I've checked out my guesses about T's emotional responding is that what I've read as anger or frustration is really his desire (similar I think to your T) to help me, to relieve my suffering (or as he sometimes says, "to rescue you from that x,y, or z"). I'm not sure if this is the same thing your T is saying, but it seems in the neighborhood. I think asking about whether my perceptions of him have been accurate has given me a strong sense of whoa, I'm way off there, especially because I'm reading his desire to help as something negative about me. I think it's also been useful in decreasing my anxiety about what other people think, because it seems pretty clear I can't know unless I check it out with them. And if I'm unwilling or unable to do so, I prefer to proceed on the most benign interpretation possible. It's helped me let go of a sense that people react negatively to me, or if they do, that I don't mind unless I'm unhappy with my responses to them.

I think it's great your T willing to be straight with you about this. It seems like he has said a couple of times that he wants to help you with your understanding of how people react to you. The understated part of this may be that you often distort what you think about other people's reaction as most of us do, over interpreting them as negative, which I think in turn drives your anxiety about their reactions, which may spin into a cycle because then your anxiety may drive behavior towards others that makes them want to back off, then that feels negative/bad, rinse, repeat, etc.

I think when therapy is a safe place to check our perceptions like this out, it's possible to make great progress in interrupting cycles of negativity.