Just let it go. If it wasn't your height it would be something else. Just focus on your child. So what if he doesn't grow up to be a giant. I'm sure he has other qualities that he can focus on. I personally judge a man or woman by the size of their heart not their physical stature.
I have 4 children myself and I constantly tell them that a person is best defined by his character not by cloths, physical appearance or the things they may have. Because when you are alone by yourself all those other things are not going to matter it is who you are on the inside that will.