You seem to have four options now. You can give in to your anxiety and avoid people for the rest of your life. You can keep fighting your anxiety and spend all your time being sweaty and scared. You can be on meds for the rest of your life. Or you can learn about your fear, and master it.
As you can probably tell, I am in favor of the fourth option. Your fear is due to your perspective. You are approaching situations in a certain way that causes you to feel afraid. I had the same problem as you, only with public speaking (wow, I'm all alone on that one

). Anyways, I would get sweaty and shaky and forgetful. But then I realized that it was only certain kinds of public speaking that made me feel this way. Prepared speeches were an unmitigated horror for me, but afterwards I was cool as a cucumber answering people's questions. So I COULD speak publicly, I was just doing it wrong. So, instead of looking at a presentation as something that had to be prepared and perfected, I looked at it as a bunch of questions. Instantly, I was dry and steady while speaking, even with MUCH less time spent preparing.
So are there any people that you can talk to without anxiety? Figure out what you're doing different with people you can talk to versus people that freak you out, and then apply that to the freak-inducers. Or else learn to enjoy sweat and stuttering