I don't gt annoyed, because 1 - 'you' is also a plural and 2 - most people don't know, so its not their fault. The current 2 people in my life who know refer to us in the plural, or at least acknowledge others inside might feel or believe differently about a given thing. For everyone else they don't know, so it aint their fault.