Holy crap your brother sounds scary. o.o' It's probably good you don't remember much of that, and that's a defense mechanism we humans sometimes have. His abuse definitely could be the cause of how you feel now. Someone who should have protected you from the monstrosities of this world was in fact a monster himself.
Absolutely neglect can have that effect on a person. My mom was pretty "hands off" and verbally didn't say much in the way of encouragement and the only contact I had from my dad was neutral and negative, so while I fear and avoid contact from people (not as badly as you), I still want it very badly. I know what it feels like thanks to my grandma. She would give me cuddles until I was a teenager and was forced to stop because I was literally too big.