I have the same problem.

That being said, when you eat less food your scale will move two ways (1) the actual weight of food in your body will be less, (2) your body will be burning off glycogen stores which weigh a lot because glycogen contains water and (3) you will be getting rid of water.

I personally believe that exercise, regardless of muscle mass, puts weight on you due to water. I believe that your body uses water to restore muscle and even if you don't do enough exercise to "build muscle" I believe your body prepares based on signs that you are exercising -- such as sweating. I mean it can be like .5 of a lb but to you it looks like you aren't losing.

I never find exercise to do anything to assist with weight loss. Even the alleged 20% doesn't work on me.

What I have found seems to work on me is like NEAT. Non-exercise activity thermogenesis. So standing for like 8 hours or walking on a treadmill for like two hours at low speed. It could be the way I am built.