For me it depends what the subject matter is. If it's things I find interesting then I can sustain the small talk conversation but if it's things I find utterly boring and inconsequential, pointless, then yes, I find it draining and want the person to go away etc. Especially if they start talking about people negatively, gossiping. But I can do small talk for a short time.