If the person they asked happens to be too inassertive to decline. That's not necessarily on the person that asked for a favor. I don't think it's on anyone. It's unfortunate that an otherwise mundane thing for most is anxiety inducing for another. But there's so many potential variables to be taken into account with each behavior and action you take, especially spur of the moment ones such as this, that sometimes it's simply inefficient and probably impossible for most to deliberate on all the variables and decide which one is the most considerate action.

As for feeling obligated to stay and burden someone. Well, you're not. If for some reason for which there are many, you don't want to look after someones belongings. That is your right, im sure most would understand, and you could argue that declining might make the asker not want to insist further. Others would just move along to the next person without second thought.

So conclusion, people can't control how they make others feel from a otherwise harmless intent action. That goes for both the asker and the decliner.