I don't think a major change in your schedule is an unfounded worry. I think that it's wise to consider how it will affect you and those around you. I had a major change about a year and a half ago and it really did change things. It changed my time and my energy despite bringing in a significant shift in income.
If a change in phone is your gift to yourself for making a milestone change, then so be it. He's lucky that you are even voicing your goals to him.
If he cannot be on board and supportive, I don't know what advise to give. Perhaps cross that path when it happens?