Could you have someone nearby, with in earshot, but not necessarily right there. I kind of agree that saying something is a good idea. If he isn't told it made you uncomfortable, he may get the wrong idea.
My experience with AA, some people can be there for the wrong reasons, unfortunately. But some people also can read signals wrong. I've had some men read the fact that I gave them a hug as "interest" when I hug a lot of the people at the meeting.
It took me a while to get to the hugging point, for the same reasons as you. I do not like being touched and really can only tolerate it at AA, with people I trust. What he did would have made me freeze in place also.
You did all the right things, talking about. Take your time, figure out what you want to say and come up with a game plan. It will all work out how it is supposed to work out.