Only you can define what's inappropriate. If someone touches you and you feel uncomfortable, then it's definitely inappropriate.

For example, I've been touched sexually by people (men and women) and I was ok with that because they were physically attractive. However, I find it inappropriate when someone I don't like touches me in any way.