One of the great frustrations I have, trying to exist in the borderlands between science and science-related policy and popular discourse about scientific topics, is the naive view of what it means for a paper having "peer reviewed journal article". The failure to understand how little peer review means is shared by some lay people, many policy makers, and even a lot of people who write scientific papers or have titles in that category (being a scientist is a way of thinking, not a credential). I have a collection of motivating observations, details, and other bits which I will cover in some follow-up posts. Which to facilitate reading I will come back and catalog in the following space with updates:
[
NOTHING HERE YET
]
But for today, for #UnhealthfulNews 200, I wanted to skip the news and explore an extended metaphor that might help clarify what I am trying to explain.
My father and his contemporaries, who were on the road in middle America a lot, recall the appearance of hundreds of McDonalds restaurants along the highways as a great thing. Before McDonalds, there were diners and burger joints along the road, of course. But you always had to worry about quality, including in particular whether the food might make you sick. It was not necessarily quick or easy to find someplace to eat, the menu had to be navigated and second-guessed, and there was always the possibility that "your type" was not welcome in that particular establishment. Readers who have never experienced such a world have probably heard related rules-of-thumb, like eat where the truckers are stopping because they have better data on those issues than you do (though I don't think the word "data" appears in the cliche version of the saying).
Readers without such travel experience probably wonder what is so great about homogenized roadside fast food places, especially those who would never eat at McDonalds, or perhaps even despise the very existence if it and its ilk. Figuring out the local dining is part of the fun of travel, isn't it.? Indeed, it sometimes is, but sometimes you are just trying to make time on the highway. Fast food is as beneficial for that as being able to drive at 60 miles per hour. From that perspective, the invention of McDonalds (and a few of its contemporary early chains) was a technological breakthrough at the level of having a GPS and search engine on your mobile phone.
 |
| Pier Review |
Convenience: McDonalds has roadside billboards directing you to the next location and easily spotted buildings. Journals have indexes. Whatever someone might claim about really liking the fries (believing in the superiority of the content), this is undoubtedly the biggest draw for a non-local (non-expert) venturing into unknown territory and needing to rapidly get nourished (very roughly educated about a scientific topic).
Safety: McDonalds has amazing quality control; the chance of getting at least a little bit sick from a random roadside eatery might be as high as 10%, while at McDonalds (assuming the food is something you can eat) it is about 0% . The journal review process lowers the chance that what you are reading suffers from utterly incompetent analysis or gross bias. Unfortunately, in public health it does so only by something like the same 10 percentage points. In other areas of health science it does rather better, but certainly does not eliminate those problems, though few non-experts probably understand the limits of this benefit.
Familiarity: The same quality control means that McDonalds food tastes the same everywhere in the country, and the fare is the same too so for those who do not want to read a menu no thinking is required. Journal articles almost always follow the same familiar format, and the abstract tells unsophisticated readers what they are supposed to think so for those who do not want to read the study, no thinking is required.
Some of the preceding praise is, of course, not exactly high praise. Still, I would not want to imply there is no value in either institution. (Though, I will note for the record that even though I liked it as a child, I have not eaten at McDonalds in 25 years; exception: I just had to try it in India where the beef is replaced with veggie burgers.) For centuries, what passed for scientific communication in health consisted mostly of the half-assed opinions of physicians, reporting from the basis of their "professional experience". The current journal system is obviously as much as improvement on that as McDonalds is over a rat-infested diner that lacks a working sink in the employee restroom.
But then there is the problem of the downsides that exist, even beyond recognizing that the benefits are more modest than many people think:
Homogenization: Even as standardization and strict recipes create some benefits, they also eliminate the variety of what can be eaten (analyzed) and force everything into the same mold -- perhaps appropriate for fast food, but clearly not good for making truly great food (studies that require something other than the standard recipe). So, for example, a paper that uses atypical methodology that is clearly better than the standard practice is very likely to get rejected. It might seem like this is not really about peer review as practiced, and it is not entirely, but that is part of the problem. Assessing what is better requires a lot more effort and specific skill; but the current review process creates a McDonalds-like assembly line, where it is assumed that any vaguely skilled person can be made into a cook (reviewer) and the incentives are for only speed and adhering to standard practices. Indeed, because so many papers need to be peer reviewed and the qualification for being a reviewer is basically "submitted a paper vaguely related to the topic", the average quality of the people who review papers in health science is necessarily the about the same as the average quality of those who write them. They are no more capable of judging what is better (or even good) than McDonalds cooks are of becoming chefs; with a few exceptions, they are both technicians, with no plan, and perhaps no potential, to become chefs (serious scientific thinkers). Assembly lines fill stomachs and journals with amazing efficiency, but do nothing to create either cuisine or good science.
Frozen in time: Science thrives on creativity, as does dining (as opposed to mere feeding). The tyranny of the health science journal publishing system has stifled advances in thinking and methodology for longer than I have been in the field. If you were to Copy a research report on a topic that is still of interest from moderately decent journal from 1980 and submit it today, and chances are none of the reviews would suggest the methods are archaic. This definitely differs from many sciences, of course, where the editor would immediately reject, perhaps asking "have you been in a coma for the last few decades?" Yes, the McDonalds menu has changed, with much fanfare, by about one item a year since the 1970s, and health science journal articles have seen a similar level of evolution, has but science should be a bit more dynamic.
Unhealthfulness exclusivity: Some would argue that eating any McDonalds at all is actively harmful for your health. That does not fit my metaphor -- there is obviously no harm in consuming journal articles, and indeed there would be terrible harm from ignoring them all (as opposed to never eating at McDonalds, which would be harmless). So we shall acknowledge the departure and set it aside, and instead continue the metaphor with a "Supersize Me" diet of just McDonalds. Eliminating from your diet anything that is not served at McDonalds and its ilk would obviously be a terrible idea. We would probably agree that anyone who insisted on doing that was seriously misguided. We should realize that the same applies for anyone who claims to be expert on a public health topic, but whose scientific knowledge is limited to the information that appears in journals.
Just like a chain fast food restaurant is a sensible place to eat when you are just passing through, a quick review of just what appears in the journals is not a bad strategy if you only have an hour or perhaps half a day to learn something about a topic. But if you are in town two days, such an approach is kind of sad. Claiming to be an expert but only being familiar with the journal literature is kind of like writing a local restaurant guide based on eating at only the restaurants with a drive-through. Finally, believing something to be true just because it is in a journal is kind of like believing that a McDonalds cheeseburger offers perfect nutrition because it contains all four food groups (if you are under 30, you might have to look up that reference and also search "reagan ketchup").
