I donate to carefully chosen causes. Perhaps you can do the same and just say that your donation dollars are already allocated there. I donate to Acumen Fund. My other cause is women's education in third world countries.
However, there's some interesting issues in there: there's how you'll be perceived by others, there's sympathy for the less well off, and perhaps a desire to help but you're not really sure. I suggest you work through these and figure out what you wanna do about giving. Social judgement may be one of the factors but it's probably best to leave it out.
Edit: perhaps this applies: The Importance of Setting Personal Boundaries » Personal Development - UrbanMonk.Net
If you're looking for a sort of script of how to deal with this sort of person. You could try saying, "I'm sorry, I"m not interested." and leave it at that. If you don't qualify this, and don't add anything to it, they're more likely to leave you alone. If they continue to talk, just repeat that line like a broken record. Eventually, they'll just give up and leave.