Well, without getting into particulars, would it be fair to say that your first time with that one girl wasn't with love, but just sex? Your latter part of the post makes it sound that way. It could be that you need to have the emotional satisfaction present before sexual attraction is present. I'm largely the same way, minus the "don't want sex" part.

Anyway, barring the first part, it sounds to me by and large that you could simply have a low sex drive, which isn't a bad thing per se. Just who you happen to be.