Unhealthful News 124 - Commentators question epidemiologic result: news in itself

A new study generated news reports yesterday that questioned the accuracy and generalizability of the study's result.  It happened to be about salt consumption and cardiovascular disease risk, and had results that suggested that more salt is better for your health than less.  I am not going to address the substance of the evidence about salt intake, however, because it was interesting that someone actually suggested we should not over-interpret the result. 

You know the cliche, "if you spent less time criticizing others and more time just [doing X] yourself then [something better would happen]"?  Well in epidemiology and health news reporting, the opposite is true.  If epidemiology researchers spend 10% as much time critically analyzing and commenting on what was being churned out in the field rather than just churning more, the field would not be dominated by junk science.  As for health reporters, they generally seemed to have missed the day of class in journalism school when it was explained that reporters should not just believe everything someone tells them.

The New York Times headline read, "Low-Salt Diet Ineffective, Study Finds. Disagreement Abounds."  WebMD headlined "Study Shows Salty Diet Good; Heart Group Disagrees".  Another was more direct about who was leading this supposed parade of disagreement: "Study: Low Salt Diet Not Helpful, CDC Disagrees".

To avoid overstating this rare burst of skepticism in the health press, it should be noted that rather more common were headlines like "Low-salt diet may be risky" (ABC), "Sodium won't kill you? Scientists shake up what we know about salt" (CBS), and "New study: Low-salt diet kills" (Canada Free Press).  Meanwhile some others offered balanced in their phrasing, though it varied in quality with "New Study Suggests Salt May Not Be as Bad as Once Thought" (Fox) not really capturing the "it may be bad for some people" message picked up, albeit perhaps too subtly, by the Boston Globe: "Study questions value of salt reduction in healthy people".

Most of the articles, whatever the headline, reported the background that the conventional wisdom is that salt is bad for CVD risk and the basic results of the new study, and discussed in varying detail the various American initiatives to try to force people to eat less salt.  Many of them then chose to emphasize reasons why we should not over-interpret the new result.

Unfortunately, as you may have guessed by now, this is not the good-news story about skeptical reporting or measured scientific assessment of the study.  Rather, the reporters just dutifully reported that public health authorities – emphasis on the authority part of that, not the health part – launched an attack on the new study.  Employees of the U.S. CDC raced to the press to denounce the study.  They were clearly motivated by their aggressive support for anti-salt policy interventions, as were pretty much all of the other critical commentators that were quoted.  Yes, of course, there were limits to the value of the study, including possible measurement error and choice of a healthy population that might miss the highest risk.  But it was clear that a desire to doubt the result motivated the identification of weaknesses, not the other way around.

So even when there appears to be critical analysis in epidemiology and the health news, it is almost worse than the usual absence of critical thought.  It is acting like a hired gun in defense of a threatened position.

When working in litigation or other adversarial settings, it is typical for one side to not like what some or all of the scientific evidence says and to try to poke holes in the studies by hyping their inevitable limitations.  That is not a proper role of a testifying science expert, who is brought in to assess the overall weight of the evidence and is supposed to provide an honest assessment.  I realize that not all of us adhere to those standards – many are just hired guns who will say whatever their clients want them to.  But an honest assessment is what we are supposed to provide.  However, it is sometimes a central part of a legal strategy when the overall weight of the science does not support the particular side, to try to attack inconvenient studies individually, pointing out flaws and suggesting those render the results completely uninformative.  In case the comparison is not obvious, the anti-salt campaigners are behaving like litigators who do not have the facts on their side. 

Of course, the evidence as a whole might support the anti-salt efforts, but honest public policy is based on analyzing all the evidence and making a balanced assessment.  Any public official who has the urge to attack a new study rather than say "we need to incorporate this information into our decision making, considering its value and limits" should arouse distrust.

Equally sadly, the press reports that covered the controversy seemed to mostly just create confusion about the practical implications of the evidence.  Credit to the Reuters report, that emphasized the sensible words of the new study's author:

It's clear that one should be very careful in advocating generalized reduction in sodium intake in the population at large.  There might be some benefits, but there might also be some adverse effects.

The NYT report, by contrast, descended into a bizarro-world discussion of how we should want randomized trials which (a) would not address the problems discussed about the new study (no one suggested confounding was its major limitation) and (b) would not be possible.  And the (b) is not for the reason that the various quoted commentators suggested: that people would not volunteer to be assigned to a lifetime of a particular diet.  It is actually more fundamental even than that:  If you assigned a cohort of people to many decades of controlled-salt diets then when you got to the end of the study, you would have...,  ...well, mostly you would have to hope that some sort of time travel had been invented by then, so you could send the results back to when they were relevant.  You see, the effect of an exposure usually varies across relevantly-different populations, and the population that will exist fifty years from now will differ a lot from today's population, in terms of dietary choices other than salt and medicines in particular, but also a host of other environmental differences.  So the study would have been about a population that no longer exists.  It would not be useless – just as no study that bears on a question is useless, including the new one – but it would have major limitations – like every study that addresses this question, including the new one.

Any one who understands epidemiology would know that.  It is just such a shame that no one who really understands epidemiology has a phone or email that would allow them to be contacted by reporters.

Unhealthful News 123 - The breast cancer empire strikes back

An accusation of being soft on breast cancer is about as politically problematic as being accused of being soft on smoking or terrorism.  There are interesting observations to write about all three of those topics today, but I will choose the one that is most relevant to Unhealthful News.  Any new claim about breast cancer is pretty much guaranteed to be all over the news, and no exception is to be found for today's claims that we are not sufficiently wasting our medical budget enriching an overpaid special interest group by doing too much mammography protecting women from cancer by screening aggressively.

As background, in 2009 an expert panel recommended that screening mammograms not be started until age 50 for women at average risk (that is, those who do not have some characteristics that create higher risk of cancer and thus suggest more aggressive screening).  The panel was headed by a former colleague of mine when I was at University of Texas Medical School (she is now at Baylor), Virginia Moyer, one of the smartest and clearest thinking scholars of medical policy I have known.  Their recommendation was undoubtedly right – I base that not just on trusting Moyer, but also on my own expertise on this topic.  The costs of screening a low risk population, like those aged 40-49 are enormous.  We pay for the direct costs of the mammograms themselves, and the huge costs of further invasive testing (to confirm that most of the positive mammograms were really false positives) and treatment (which is often unnecessary – see below), and the patients suffer from the lost time and pain from the millions of mammograms, the fear created by false positives, and invasive procedures.

Moreover, it is quite plausible that screening at age 40, when cancer risk is very low and the breast tissue is difficult to see through, is more likely to cause cancer than to help cure it.  That is, the ionizing radiation from the mammogram has some risk of causing cancer, and since screening drops in value as the women gets younger, at some point it must be that the risk is actually greater than the benefit (e.g., mammograms for 20-year-olds are of basically zero value, but still create a risk from the radiation).  I am not expert in the risk from the radiation, but back when I worked in this area, someone who I was inclined to trust estimated that age 45 was the time when this crossed over, and the benefit exceeded the risk (and notice that this considers only cancer risk, not all the other downsides of screening, testing, and treatment).

In short, doing mammography too early or too often (the new guidelines also said every two years, not every year) is expensive in many ways.  It is therefore important to put proper limits on how much is done to avoid creating excessive expense, misery, and actual disease risk.  Not so, according to the story touted by the trade and lobbying association ...oops sorry I keep doing that, I mean professional learned society that represents radiologists, the people who make a fortune from doing mammograms.  That group seems to have declared war on the sensible guidelines made by the experts in evidence-based medicine and cost-effectiveness analysis.  Funny how not a single news article bothered to mention this rather obvious conflict of interest, which makes the press release as much a special-interest commercial act as those that come out of the Macworld expo.  No one reading the news reports would even know there was reason to be skeptical -- unless perhaps they noticed how utterly lame the content was.

The sales pitch included results from two studies (I use that phrase charitably).  The first found that following the publication of the 2009 guidelines, screening of women aged 40-49 went down.  Good news, right?  A sensible guideline is created and people start following the guideline.  Not according to the radiologists, since this is "possibly negatively impacting on the benefits of prompt detection."  Well, we cannot have that, can we?  Saving zillions of dollars and needless trauma is no good if it possibly negatively impacts something to an unknown degree.

Actually, if they had been referring to screening less frequently at older ages, it would have been undoubtedly the case that there was some negative effect due to the reduction in detection.  That loss would be warranted by the reduction in other costs (according to the guidelines), but clearly more screens means more detection, ignoring all else.  However, for the under-50 age group, the radiation effect means that it is actually conceivable that the cancer benefits of not doing the screens is actually positive.  To my knowledge, we do not know for sure.

The second study touted to the press claimed:

Cancer diagnoses among 40-49-year-old women who had undergone screening mammography occurred at a much earlier stage of the disease compared to women of the same age who had not undergone screening.

Well, yeah.  The rare cases of serious cancer in this age range are caught, absent screening, because they advanced to a stage that can be detected due to a lump or other symptoms.  But contrast, most of the cases detected by mammography are very small and of a type that might never become serious.  That is, many of the cancers that are detected never would have turned into a disease, but the medics who detect them will claim credit for intercepting and curing a disease that never would have happened.  (This phenomenon is know as "over-diagnosis".  I explained it in detail in UN84, along with the concept of "lead-time bias".  If you are interested in fully understanding my points today and are not familiar with those, I suggest reading that.) 

Also it is not clear how much worse the outcomes are for cases that do advance because they are not caught by under-50 screening, to be detected and treated later.  Worse, no doubt, but it is difficult to know how much worse because it is difficult to sort the successful treatments that prevent the need for a later-stage treatment from the "successful treatments" of the cases that never would have been bad.  It becomes especially difficult to perform this complicated analysis when you like the answer better when you do not bothering with it.  After all, who is going to call you out for doing an analysis that is known to be incorrect?  The health news reporters?

Further complicating the situation is that women who are doomed to die from breast cancer, but are detected by screening at age 45 rather than 55 will "survive" ten extra years even if they die at the exact same time.  The screening will be credited with providing a benefit even though it merely started the clock sooner (this is lead-time bias).

Yes, it is possible that the radiologist researchers properly adjust for these challenges, but it seems unlikely.  Certainly the quotes that showed up in the news strongly suggested they were completely ignoring them.  Frankly, even though they live in that world, my experience suggests that they may not understand these problems.  Physicians as a group (with obvious exceptions like Ginny Moyer) strike me as less able to understand these points than average college educated and numerate people (UN84 was motivated by an economist instantly understanding and clearly articulating lead-time bias, something that takes a very long time to explain to medics).  Perhaps this has to do with the classic explanation, that their paychecks benefit from them not understanding it.  But blaming simple venal conflict of interest might be letting off too easy those who make claims like:

It's very important that we continue to do all that we can to catch breast cancer in the earliest stages so that we can continue to save lives.

Isn't it interesting how much the "arguments" for screening at age 40 and annually are the exact arguments that could be used in favor of starting at age 25 and screening every month.  Basically they say "if you ignore all of the health, psychological, and resource costs of screening, false positives, and unnecessary treatment then doing more screening is a good idea".  Actually, I think I am letting them off the hook a little too easily with that "25 and every month" thing.  The "we must do everything we can" claims often imply that every woman who is not expecting to nurse a baby in the future should get a double mastectomy.  That will shut this breast cancer problem right down, and stopping breast cancer is worth any cost, right?  Or is that only if the radiologists are the ones scoring in the profits?