I think that if you WANT to do it, it's concerning. But I have invasive thoughts sometimes, things I would never act on. When I get scared by them and try my hardest not to think of it, they bother me. But if I actually consider it I quickly realize how ridiculous the thought it and it helps me to dismiss it more permanently. I'm not sure if that's the same with thoughts of self injury. When I have those thoughts, it's something that I have done and want to do so it's more concerning for me.
So I guess what I'm saying is that maybe the reason you're having the thought repeatedly is because it alarmed you so much to have it at all, and your anxiety is related to having an unwanted thought and not because you actually want to hurt yourself.