I have the same issue! The only place that has a neutral toilet is my work place. Although if you can find single toilets those are great, because if anyone questions you coming out of one you can just say the other was broken.
But using a room with stalls is scary, I usually just hide until the bathroom is empty or go with one of my cis friends.