Love that post Silvertrees! Started typing, then realised you said it better than I ever could. "You aren't ugly. You're attracted to someone; you're human!"
Plus you already wise to the fact that there are boundaries between your fantasy and him as a human being. I don't have any guilt whatsoever about sexual pleasure through fantasy. Perverts can't draw a line between their fantasies and other people's right to choose/ consent, that's very different.