Thank you very much for all the replies. I really appreciate them.
I want to clarify my earlier post a little bit. His comment that I didn't have a kind heart didn't bug me so much. Rather, I am looking for a better way to cut off the conversation in case that they pressed me further. Your answers would definitely help.
I agree with ns123 that giving out money is not the indication of kind heart. Many of us pay them to go away. I also don't think the money would help them in any way except encouraging him to go on with that kind of lifestyle