People who are afraid of their own quirks will make stupid comments to someone who is trying to be genuine. It's unfortunate but it happens. People pick on me at work for my accent and other things. It is annoying and has worked against my confidence from time to time. In the past if it has gotten too bad I have simply asked those people to stop saying crap like that to me.
It very well can destroy ones confidence to have people say negative things about you.
As for me, If I'm tired of someone razzing me about something I usually turn the table and give them a little dose of their own medicine.
Your clumsiness could be caused by your own frustration as to how you see yourself. I hope you can feel connected to yourself because at the end of the day only you know who you are as only I know who I am.
Confidence is knowing who you are. Next time someone speaks negatively toward you just look at them and know you are better and you don't care what they say. When it gets to you try to find some time to yourself and relax and spend time away from those people. If you have to be around them judge them as they judge you. I'm not saying be rude but don't be afraid to stand your ground. So many people mistake patience for weakness. That's something I never understand. Anyhow, Be well!