I don't have children and don't want them. Neither does my husband. I generally say, I never wanted them and try to change the subject.
Once a co-worker asked if I had kids. For some reason I blurted out "God no!" like it would be the worst thing ever. He laughed and said it was actually refreshing. He said he knows some other people that don't want kids and they often feel like they need to justify their reasons, or go on about how they like children but don't want any of their own.
I guess in your case, saying I can't and I'd prefer not to discuss the reasons could work too.