Unhealthful News 122 - More examples of why epidemiology and anti-tobacco are seen as junk science

I was going to cover three separate research claims that fit the theme of the title, but the first one kept expanding until it was a post by itself.  I might pick up the others again later in the week.  I went ahead and kept "examples" plural in the title, though, because this one case definitely offers multiple examples of bad science and dishonest reporting thereof.

It was claimed in this press release  that exposure to second hand smoke (aka environmental tobacco smoke, ETS) raises blood pressure in boys.  It would be difficult to make up a better example of why epidemiology is widely considered to be junk science.  It did not generate many news reports (just a handful of newspapers, British and Asian for some reason, and the usual websites that just reprint all health press releases), but it the type of claim might have made the news, and might still, so it worth dissecting based on the press release.  The press release was touted by the American Academy of Pediatrics (yes, that would be the same people who publish the junk science journal, Pediatrics) who appear to be the sponsor of the conference where the results were presented.  Based on that, we might charitably think that the study authors were just honestly reporting simple study results and the publicity seekers tried to turn it into something else.  However, one of the authors actively contributed the the absurd claims in the press release, so the researchers are equally to blame.

[Note:  I had already written most of this earlier today when I noticed that Chris Snowdon posted about it, including a request for my input.  I added a few more points in response to the requests for clarification he emailed me.  Since my goal in this series is to address points of confusion, I always welcome questions from any reader about what people think needs explaining.]

The press release began with several paragraphs about how terrible high blood pressure is in adults and then claimed, "that boys ages 8 to 17 years old who were exposed to secondhand smoke had significantly higher systolic blood pressure than boys not exposed to tobacco smoke."  But a glance at the abstract of the study reveals that the increase was only a bit over 1 mmHg (i.e., about one point on the familiar scale with numbers like "your blood pressure is 116-over-82").  This is hardly "significant"; a better word would be "trivial". 

The press release tries to spin this as important at the population level despite being small for any individual.  The is rather disingenuous when juxtaposed against calling the result significant.  More important, even if the claimed effect followed from their data (it doesn't – keep reading) there is no evidence of a population effect.  The author is mistaking a small increase in risk for a disease with a small increase in a characteristic.  That is, if some ubiquitous environmental toxin increased everyone's risk of liver cancer by 1%, it would not be a huge worry for any one of us, but it would be causing thousands of extra cancers in the population.  But this tiny change in blood pressure may or may not represent any extra risk of anything, so multiplying the extra risk of epsilon or perhaps zero by lots of people still does not necessarily add up to much.  Moreover, it seems almost certain that even if this study had actually shown the claimed effect of ETS exposure in childhood, any residual effect in adulthood would be substantially less.  That is, by the time higher blood pressure created a risk for bad outcomes, the trivial effect of ETS exposure during one's long-past childhood would have faded away.

But, as I said, there is not really evidence of a bad effect.  The numbers actually show a good effect, though what they really say is "this study showed nothing of importance".

The obvious fatal flaw in the attempt to make a big deal of the study that the result for girls was exactly the opposite the touted result for boys.  Actually it was bigger.  The girls had a reduction in blood pressure of well over 2 mmHg, so if this study is to be believed, any bad effect for males is more than offset by a good effect for females.  If we are supposed to believe that one result from the study is right, then we are obliged to believe the corresponding results for other subpopulations.  Moreover, the result for the population as a whole is always more important than the subpopulation results.  Funny that they did not bother to mention that, since it would be a "significant" (by their standards) protective effect of about 1 mmHg (.5*(+1) + .5*(-2) = -1).  So the real first headline from the study is:

Second hand smoke exposure reduces blood pressure in children

It is not just about boys having an effect in one direction and girls in the other.  I noticed that several commentators already made that observation today, but they overlooked the even more important point that the (unreported) overall results show that ETS is beneficial on average.  A better interpretation would, of course, be to consider all three results (all, male, female) to be a bit of technical information that might contribute to some honest analysis someday, but that have no meaningful implications in themselves.  It follows from this that touting one result in a press release or writing the conclusion that this "provides further incentive to expand policies that create smoke-free environments for children and adolescents" is a crime against honest science.

In the press release, the study author, Jill Baumgartner from University of Minnesota (surprise! - though the study apparently was done at Wisconsin), seeks an excuse for why we should believe the result she likes politically, and ignore the one she does not like by claiming, "These findings support several previous studies suggesting that something about female gender may provide protection from harmful vascular changes due to secondhand smoke exposure."  Um, yeah, except the girls were not "protected" from an effect, they actually had a rather larger effect – just in the "wrong" direction.  So, again, if the results were to be believed, the conclusion should be that we have "further incentive to create smoke-free environments for boys, but should encourage girls to suck down some smoke."  Of course, an honest analysis would be that this study provides no support for either of those recommendations.

Also, I did not have the impression that sex has been shown to be a major effect modifier for ETS.  The three choices there are (a) she is cherry-picking one or two studies that tend to support her claim as if they were all the evidence (she certainly has demonstrated skill at cherry picking), (b) she know more than I do (you can be the judge), or (c) she is just making it up (I would not rule it out).

Further review of the abstract (all that is available other than the press release) shows that there is no dose-response trend.  The one-fifth of the population who have approximately zero serum cotinine (a measure of nicotine exposure and thus degree of ETS exposure) have lower blood pressure compared to the other four fifths, but among the latter there is no trend as the cotinine increases.  Among other things, this means that the subjects who actually smoke (who would comprise about half of the one-fifth of the population with the highest cotinine) were the same as those exposed to a bit of ETS.  I suppose this is in keeping with the current propaganda that ETS is so bad for you that if you are exposed to any of it then you might as well smoke, but no one honestly believes that is true.  But propaganda aside, it does create some serious doubt about the results.  Equally damning is that the difference between the lowest and second-lowest fifth is the same as the lowest compared to those with higher exposure.  But even by the standards of ETS exposure, the difference in exposure between the lowest two groups is trivial; they would both be characterized by no one smoking in their home or any place else they spend a lot of time.  This is actually as much of a fatal problem with over-interpreting the results as the boy-girl contrast is, it just requires a bit deeper an analysis to recognize it.

My epistemic point is not that the result should not have been presented.  Even though it is full of oddities and the numbers are trivial, the data is what it is and might usefully inform something someday.  Real scientists do not suggest that research results be censored.  But touting a blatant exaggeration and misinterpretation of the results to the press is lying, pure and simple.

As a final note, some readers might think that this is just something that anti-tobacco or the American Academy of Pediatrics might tout, but real epidemiologists would not be so dishonest or fall for this.  But it reminds me of an example that was my favorite teaching and methodologic example for years, about a decade ago (I won an award for my article about it).  The decongestant medicine phenylpropanolamine (PPA) was banned in the US and then most of the world based on a single sketchy study.  The touted headline from that study was that PPA caused an increase in stroke risk in young women.  What everyone failed to notice or care about was that it had a protective effect for men – less than half of magnitude of the bad effect for women, unlike the greater effect in the present example, but still protective.  Funny how everyone, in their inexplicable frenzy to ban this drug, decided that only part of the evidence from the same study counted rather than simply warning women against taking it.  Junk science indeed.

Unhealthful News 121 - Adamance and conflict of interest (part 2)

Two days ago I started writing about a coincidental collection of essays that had crossed my screen over the last week, all touching on the themes in the title.  When I started, I thought that today I would bring it together with a unifying message, but nothing seemed to emerge.  I think the topic is just too big to pull together all the threads so easily.  So instead I will just add just one additional example its implications.

I had two interesting revelations while reading a really bad, naive analysis about the nature of COI.  An anti-acupuncture advocate was annoyed that an acupuncture practitioner/advocate/researcher criticized her for failing to disclose her COI in a journal commentary.  Her commentary briefly praised the new study it accompanied, which reported a lack of support for acupuncture in the existing literature.  It then went on to present an adamant broad attack on the claims that there is scientific support for acupuncture in general (she reposted the commentary here, with no paywall, if you are interested). 

Her best defense against the criticism of COI would have been:  "My commentary was so obviously anti-acupuncture that no reader could possibly accuse me of having a hidden agenda.  I wrote a general anti-acupuncture editorial that merely used the new study as a hook, so since I did not contribute any analysis about the study itself, there was no COI as there would have been if I purported to be neutrally assessing the quality of that work."  Picking up on the points I made two days ago, being an adamant supporter of a position – which this author clearly is – is not a COI so long as you are working from the clearly-communicated premise that that position is correct.  It is a COI if you are trying to offer an assessment of whether the position is correct or not.

But instead of making that argument, the commentator blogged that because she had no financial benefits or any material stake in anti-acupuncture, she did not have a disclosable COI under any circumstances.  This is a demarcating that is often offered, almost always without any analysis or support (or apparent brain involvement).  But she wrote pages purporting to support the claim but offered no convincing arguments.  She offers no compelling reason why an author who wants all readers to reach a particular conclusion because of passionate personal conviction differs importantly from one who wants them to reach that conclusion due to a financial or other material incentive.  In both cases, the reader deserves to know that the author has that goal, and it matters little why the author has this goal.  I found this to be informative simply because a good test of a position is whether its proponents can make a plausible case for it when the opponent's views are not even considered; this particular proponent certainly could not do so.

Her most substantive defense seemed to be that she was not anti-acupuncture, she was just pro-science, and her years of studying this particular topic had led her to a scientific conclusion that acupuncture was all bunk.  But someone who turns the corner and becomes an advocate rather than just a knowledgeable reader is at great risk of (a) having biases that she herself cannot fully recognize and compensate for and (b) being so focused on a goal that she fails to recognize that her actions have ethical importance and are no longer just some kind of cold pure science.  After all, when someone writes the following in her blog post about the journal commentary, it is pretty clear that she does have an interest:  "It gave me a soapbox in a major medical journal to say all the things I thought needed to be said about acupuncture."  Anyone who thinks there are things that "need to be said" on a topic has an adamance that is a disclosable interest. 

Does it matter that she has not made any money pursuing this work?  Well, imagine someone with her adamant views, and many years of advocacy, who you then learned also got travel expenses and a modest honorarium to give a talk at a medical conference.  Could that possibly change you opinion about the nature of her possible biases?  Obviously not.

But I found some important takeaway lessons and reminders nevertheless (so credit to the author for writing something provocative, which is perhaps even better than something that is correct but insipid).  First, it pointed out the enormous COI that is involved in opinions about the nature of COI.  Most of the time when someone tries to demarcate COI from lack-of-COI, they seem to draw the line so that they are on the latter side.  People who get grants to work on a topic claim that COI is an issue only if you are salaried.  People who have salaried jobs working on an issue with an NGO or advocacy arm of government claim that COI exists only if you have investment assets or involvement with corporations.  Those few people who work on an issue from a purely voluntary basis (like the author of the present example) claim that COI is all about monetary payments, and cannot exist merely because someone feels so strongly about an issue that they will donate their time to it.  But then, of course, even people who get paid to advocate on an issue will counter by figuring out a way to define COI so that it refers to anyone working on the opposite side of the issue, paid or not.

In other words, it is just like the dilemma of people working in highly politicized fields, where all the real experts have strong opinions about the controversial issue and so most end up as part of a political faction.  Most everyone who knows a lot about the travails of COI is expert because they are enmeshed in a world where it is a problem, and thus they have a worldly stake in how it is defined.  That is, they have a COI about defining COI.  There are only a handful of serious historian-of-science types who are experts as a result of their arms-length studies (who should not be confused with the countless armchair pundits who fancy themselves experts on the topic because they read a couple of books on ethics; my experience is that most of them are actually motivated by a personal stake, or are quite clueless about the issue, or both.)  When I have written studies about COI, I and my coauthors have disclosed my interests in how COI is defined; I do not recall ever seeing anyone else do that.  But the more I think about it, the bigger a problem it clearly is.

Even closer to home for me, the anti-acupuncture commentary was an excellent reminder of how an author with an adamant interest in a particular position cannot simply make blanket assertions.  Expert advocates often fall into the habit of writing for people who already know a lot of the background, and the habit of treating the many bad (in the opinion of the author) arguments on the other side as so groundless as to hardly be worth addressing.  The problem is that this does not work as communication for thinking readers who are not expert on the fight already.  As a reader, I read the anti-acupuncture essay as a series of definitive statements that, if true, would present a very strong case, and yet I found the argument to be absolutely unconvincing.  Why?  Because it felt analogous to reading about a single case study of a coincidence (e.g., someone who got acupuncture and had an amazing cancer remission) – you know such a coincidence exists somewhere, and that someone can spin it into an adamant essay.  So the fact that someone presented that essay tells you nothing – there was never any doubt that it could be done, or that someone could have presented something equally adamant (though perhaps with fewer allusions to studies) on the other side.  When the author adamantly presented one side of the argument, I was no more inclined to just take her word for it than I would have been had she disclosed that she was a paid advocate for her side of the issue.

So in the spirit of my Sunday sub-series on who to believe, I did not believe someone who wrote adamant assertions like she did, even though I suspect what she wrote is probably right.  So I, and the rest of you who are also advocates for some science-based cause, need to face that challenge – most claims of financial COI are just ad hominem attacks, but adamance can arouse doubt (even when it is inspired by ferocious support for good science).  Even though those who are very familiar with me know that I am probably the most effective critic of bad arguments made on my side of an issue (example), more so than the opponents of the position, why is what I write more convincing than the anti-acupuncture broadside?  Or is it?