I recommend stabbing the buggers in the eye with a fork. (I'm walking the tightrope I know, but honestly, my intentions are pure Cro, and I hope you're laughing. If you're not, ooops!)
You don't need to be agressive. I know you don't like this but just as an experiment, try imagining you want to help everybody happier. It sounds like a backwards approach, but I guarantee that after a while you'll find The Line. When people cross it you'll feel comfortable letting them know in an assertive but not reactive manner that their behaviour is not cool.
People naturally test other people to find their boundaries... if you don't let them know in a non reactive manner when they've crossed it, they'll keep pushing and pushing, to try and find your boundary. Aiming to make other people happy will remind you of your value. Promise.