It doesn't sound rude to me at all. I know what you mean. There's my sister who lives 3000 miles away who is married with two adult children. I haven't spent any holidays with them and I wouldn't be interested to. It would be too much drama for me. They have never come to visit me. If I had the choice between me visiting them or them visiting me, I would choose the latter. It would make me feel more in control. But I think it would be nice to see them go if they did.