Experimentation and Confusion

Hi im 26yo male, i consider myself straight as i enjoy company of women and have had many relationships with women. However, since puberty ive always had fantasies of being with a man, specifically being a bottom for and older man. Although these fantasies became less and less as i got older, i did act on these urges once when i was 19.
I was extremely uncomfortable and asked him to stop, i didnt enjoy it. I went home releived because i finally figured out im not gay or bi or whatever, but 7 years later i still have these urges! They have been very strong and almost hard to resist, a part of me says maybe i need to get to know a guy first and take it slow, maybe i didnt enjoy it the first time because i didnt know him, and it was my first time, [ plus it was in back of a car which was also uncomfortable ] but another part of me says I am a MAN and a FATHER and this is totally wrong..it toke me years after having sex with that guy to regain my self esteem.
i felt degraded, and felt like less of a man after what i did...what do i do? Do i act on these urges and take it slow this time but risk ruining my self esteem again? Or do i repress them and maybe they will go away since i am still young? I really need help and im terribly confused. It also doesnt help that i have major depression, anxiety, and a bit of a chemical inbalance which is what i blame these urges on..its like a switch, one min i want a girl, next i want a man. I dont get it! please help!
thx