I don't think you're having shameful fantasies technically because shame and guilt are two different things. Shame is thrown upon a person by a third party whether it be a some bully or a town. Guilt is something that one feels because of their conscience. I don't think you have crossed any lines with this person.

I work with a female and catch myself looking her way all the time. She's married though, so I dare not making one friggin move on her, that would be crossing a line and then other people might shame me, and then my conscience would make me ffeel guit.