I have the same challenge. I know why I am this way. I cannot get physical with someone before I get emotional connection and until I feel somewhat emotionally safe about him and the relationship.
Do you feel emotionally safe with him? More importantly, do you feel emotionally safe with the relationship and the state of the relationship? Please let me clarify, I am not asking if you are having a great time and having fun. That, i hope is the case. What I am asking is the emotional connection. May be it is different for you but i need to feel at least a little emotional connection and safety in order to get physical.
Please do not feel guilty. You do not have a problem. I would actually think that he needs to understand your situation. Besides, once it happens, it will continue to happen. [emoji4]
It might be a red flag if he is getting upset over this.