In my jurisdiction, parents can petition the court for guardianship of an incompetent adult child and in that petition they can ask the court to appoint a person other than themselves to be the guardian - there is a public official in each county whose job includes being in charge of wards. The law in my state cannot insist or force the parent to care for an incompetent adult offspring.
As I am one of the people appointed to represent the alleged incapacitated person (it is considered an adversarial proceeding so they have a right to counsel) I have seen how hard it is for parents to get to that point.
If you have not done so, and if you are in the united states, I would suggest consulting a family law/probate attorney (guardianship are often lumped into probate) about what you can do do be taken off as the responsible person or to get someone else named (like his father). It also might take (depending on the jurisdiction) something like you having the psychiatrist and therapist help you prepare info with reasons why you cannot continue to be the responsible person.