This may not be about men at all. It could be about an unfulfilled emotional need that you unconsciously associate with masculinity or masculine attributes, which could be experienced with a woman as well. The physical desire you feel may just be a mechanism for getting your attention to this matter. Anyway, something to think about.