I've always felt as though I must have had a target on my back... or perhaps a sign that says: "kick him- he's vulnerable."
I sometimes think people can smell vulnerability on you.
Over the years I've become pretty-much thoroughly reclusive in real life though. So I don't have those problems anymore.