Here's a really simple concept that has been slowing dawning on me over the last few months.
"Thank you" is never the wrong answer.
"Never" is a strong word, but it doesn't sound as good as a rule if I make it "almost never"
.
Practically speaking, there are two possibilities when you say "thank you" in response to a compliment you're not totally sure of the sincerity of. 1. It was meant sincerely, in which case it's a good habit and good manners to say thank you. 2. It was not meant sincerely, in which case you say something that is either funny in response, or disarms the person who's trying to make fun of you.
Oddly enough, I've always been more comfortable saying thank you in the second situation than in the first, but you really can't beat it in either one.