Quote:
Originally Posted by Chris Anthony  That'd be my job, actually. Full disclosure: I'm a data analyst working for a doctor at the School of Medicine at Johns Hopkins University. I'm the guy who runs the numbers for the studies you're dismissing, and it's in my personal and professional interest to make sure that those numbers are as accurate as they can possibly be, regardless of what conclusion they support. The funding for our studies comes from the National Institutes of Health, and we are not beholden to them to produce particular results or to study particular subjects.
|
Hey there, data analyst, Chris. I'm a data person as well. I agree with most of what you are saying, but have this to add:
Confirmation Bias: The numbers may be right, but they can be gathered to
support a hypothesis, rather than
test one. And you'd never know the difference.
Optimizing vs. Statisfying: Optimizing your health outcomes requires exposure to thousands of peer reviewed studies, integration and comparison of effect sizes, and shifting through various external and internal validity issues as well as issues surrounding statistical power, examining missing data patterns, construct validity of the independent and dependent variables, etc . . . and then, and only then, letting the cream rise to the top. This requires a level of critical thinking, detail orientation and methodological familiarity that most people either don't have, aren't capable of, don't have time for or aren't motivated to do. So the only option most people have is to statisfy: take what "seems good enough" and leave their particular outcomes to chance. They then build off this little bit of empiricism various inductive principles that define the workings of the world. The whole thing sort of goes like behavioral shaping via operant conditioning. But that's the best most people can do.
I mean, seriously, most people don't even read CR, before buying a car . . . they act on reputation or other such heuristics. Someone here, who's a self-proclaimed tech geek, didn't even do a head to head comparison of MP3 players before dumping money out for an ipod. Statisfying is expected, optimizing is exceptional--hence, the boldness of "Health Studies are Worthless to Those Who Care About Health."
Hawthorne Effect: Any change = a positive outcome. Therefore, everything works.
Black Box Syndrome: Put something in the box, another thing magically comes out of the box. Most people never ask what's inside the box that lets one thing go in and something else come out. Also most people don't understand how to find out what's in the box without opening the box. Some people would even argue that it's pointless thinking that there actually IS and inside of the box as it doesn't matter as long as the magical property keeps coming out.
Framing and schematics: There are 4892358729875... different ways to look at any given problem. Only 500 may actually shed light on relevant mediators and moderators and the rest may yield statistical significance due to their relationships with these mediators and moderators. From a theory POV, you can never really know what it is that you are looking at.
So yes, in the most roundabout way, for those who only care about outcomes "Health studies are worthless . . ."