It does sound like it’s more on him, and he does have some intimacy issues. You didn’t do anything so bad. Just leaving him alone now is your best course of action.

I had an ex with similar issues. He’s take two steps forward/ three steps back. He made a comment that I had ‘child bearing hips’ and cooled off to me. Mind you, I had a thin, great figure. His issues!