Just as many women leave sensitive men behind in the dust as men do to women. Stuffing every potentially awesome sex partner in your life into the, "He's going to abandon me..." category will only shackle you to this fear.
People love each other and people break up with each other to go love another - it is a natural thing.
Go have sex and stop freaking out about it. Being scared silly of having sex is akin to being scared silly of tasting chocolate mousse because you think there is a possibility (but don't know for sure!) you would be allergic to cocoa beans and therefore die after the first bite (don't you dare say, "I heard about this one person who...").
By the way, I had awesome sex with a 41yr old when I was 19; she was definitely in her prime and I was (still am) a ravenous sex monster that genuinely appreciates and loves women.