Self confidence is variable. A highly talented sports person (or team) can show a lack of confidence after a string of poor performances. However, from what you have said it appears your problem is more a lack of self esteem, undermined by the negativity of those around you.
In British Universities they seem very quick to offer psychological type tests to help identify students who have learning/developmental difficulties that are holding them back. To our surprise (and hers) our eldest daughter was diagnosed as dyslexic at University; likewise someone I interviewed (and appointed) this week. We have a friend whose child's dyspraxia is marked by clumsiness.
If you are able to access help to see if there is any reason behind your problems, knowing why can help you understand and its an easier way of confronting others prejudice. However, even if you cannot go this route or its unproductive, its important you learn to accept yourself, to love yourself as you are.
If you were confined to a wheelchair or only 4 foot in height, there would be restrictions on what you could do. But people wouldn't laugh at you in this derogatory (it sounds) way. And if they did laugh at you, you wouldn't think it funny.
Use your own positive affirmations to remind yourself of what you can do, what you are good at. I'm sure there are many things you can do better than me, and vice versa - we all have strengths. If you must dwell on weaknesses, work out how you can use them to your advantage.
The bottom line is,
does it matter what you look like? As I describe in this post, there are some things we have to learn to live with and accept as part of ourselves. If people laugh or question your "quirkiness", tell them its as much you as your gender, height or hair/skin color - would they question that?