Could you take a self-defense course? Sign up for martial arts?
Whatever you can do to increase your intelligence at the same time reduces your fear.
Here's an example, hypothetical but you can imagine this.
You're on a plane and you don't know how to fly. You at the controls and you know your life depends upon you landing this plane. But you know nothing about flying! FEAR!!! TERROR!!!!
Now put a seasoned pilot in the same situation and tell him his life depends upon him landing this plane. He will laugh and with incredible finesse, bring the plane to the runway with a smooth landing.
The difference? Experience, awareness, intelligence about the task at hand. If you're worried about your physical safety, then go learn how to protect yourself from professionals. Get hands on training and learn what devices you can carry. Get some experience with these things like pepper spray so that you get an idea how powerful it is, e.g.
Life is too short to live in fear. I was a pro....spent 44 years scared to live. Fear Self Help - Why Am I Afraid of Living - Profound-Self-Help.com