Maybe we shouldn't publish until results are replicatedThe accompanying link was to a case of a researcher who fabricated data and the published papers have been withdrawn.
I can understand the frustration that causes a lot of non-scientists to complain that too much gets published that turns out to not be true. Rarely is this because of something as blatant as fabricating data, though it is quite often for actions that ought to be considered similar, like researchers intentionally fishing for a model that gets the result they want and hiding that fact. Sometimes, though, it is the completely honest and proper error that comes from random sampling or an unrecognized flaw in the study. Readers want simple right answers, like they learned in high school science classes. But science in action does not work that way.
Probably some of you immediately thought of the point I wanted to make about the above quote: How, exactly, can anyone replicate something that has never been published? They might stumble on the same analysis and do it themselves, but even then they would not know they had replicated something. Why? Because the way researchers let other researchers know that they have seen something that might be worth replicating is to publish it. Scientific publication is primarily designed to be communication among scientific experts.
The problem is, then, when research results that, to non-experts, are barely more understandable than raw data are communicated to people who do not understand what they are: They are just one cut at a question and might be different from the future received wisdom, or even the existing received wisdom. And this even ignores the problem of readers not knowing how to interpret the results in a useful way, even apart from what other studies might show.
Everyone in sight is guilty here. Researchers (and their institutions, and the journals) tout results to the press even when no one other than a few experts can really make sense or use of them. Most health reporters do not know enough to critically assess results or put them in context, except in the way they cover politics (which is to say, treating it like a "he said, she said" game and finding someone to just assert that the result is wrong). Science teachers create the mistaken impression that scientific results are always Truth. And the public demands simpler answers than can really exist (though perhaps all the blame there lies with the others on the list).
As you can surmise from my writings, I think the crux of the problem is science reporters who simply do not understand science, and health researchers who ...well… simply do not understand science. But no one is going to get those groups to refrain from publishing until they know what they are doing. And demanding replication of the errors will not help any -- there is no shortage of that.
