I agree with Andrea. Leaving a site (so to speak) isn't the same as puposely doing something to harm others. You can't control how others react, but you do control your thoughts, feelings and intentions.
Erin also makes a good point. Your presence may be helping others. If there's a way to sign up to receive email or notification that someone on the site would like to contact you, that could be useful. You'll naturally evolve to spend less time on some pursuit of you outgrow it. Stop listening to your fear of what may happen. Do what feels right.
