It sounds like your ex is using psychological projection. He is trying to avoid responsibility for his behavior by projecting it on you.

Next time you go to his house, can you ask someone to come with you? It's helpful to have a witness because of his tendency to twist stories.