Hmm..
I think you just need more self love. So what if you just told your neighbours you have fear and anxiety? Can't you see it is your low self esteem that is telling you, you need other people to accept you?
(I'm not holier-then-thou, I'm still on this journey myself, but I was in your shoes before, so sorry ifthis comes across a bit harsh.)
The only person you need to accept you is yourself. No matter what you do, no matter what happens, love yourself. Put your hand over your heart and tell yourself "I love you". Or "I love myself" if that works.
Once you accept and love yourself totally, the opinions of others won't mean as much. (Of course you can't turn into a sociopath, but you know what I mean).
It's great that you have already identified the cause of your problems, I lived for years without knowing why I was living in fear.
Check out this article for more:
Personal Development - The teachings of the Urban Monk » Blog Archive » Happiness: What is your relationship to you? Part 1