I think if you just smile and nod politely, it's almost like from your child's point of view, you're agreeing with the other adults. Then your child might pick up the idea that something is "wrong" with him.
We are very small, my husband and I This is fine to say. You can add, if you want, "and there's nothing wrong with that." Or say, "he's just fine."
If they continue to keep up with the comments, if it were me, I would start to get irritated and say a few choice words. You don't need their approval.