FDA on menthol cigarettes, some suggestions for research methods

The U.S. FDA is probably the most respected and influential medical research organization in the world.  Sure they have their hiccups and there are criticisms (many legitimate, many not) about the science and the choices about what risks to take (historically entirely in the direction of them being too quick to keep a potentially useful drug off of the market, more recently in both directions).  But all in all, it is hard to imagine engineering an institution that does much better.

But tobacco controllers (including a former head of FDA during his tenure) and their pet congressmen got the bright idea of adding a category of non-medical consumer goods to FDA's scientific purview.  To these non-scientists, it probably seemed that there was no contradiction here.  Science is science, right?  Clinical trials of medicines, monitoring food safety, consumer preferences, social forces, climate change research, isolating the Higgs boson -- if they are good at the first two, and experienced in the related ethical questions, then they must be able to do the rest, right?

Um, no.

It is clear that the FDA scientists who have been asked to look at tobacco products are trying, and it is equally clear they are frustrated.  Their latest report on menthol in cigarettes and the predicted effects of banning it [I got it here but that link seems to not work now], offers an opportunity for some unsolicited advice.  After I collect comments and my thoughts, I might include some of it as a public comment, [UPDATE: link fixed] which they are currently soliciting.

Their evaluation concludes that there is no measurable difference in the biological risk between menthol and non-menthol cigarettes (keeping the quantity of exposure constant).  This is based on research that FDA is pretty good at; it is not quite the same as their core competencies, medical trials and safety testing, but in the neighborhood certainly.

But the crux of the decision about whether to ban menthol is presented in terms of economics.  Unfortunately they do not say this.  This is presumably because economics -- the relevant science when you are looking at consumer choices  -- has never been part of what FDA does.  Drugs and medical devices are evaluated based on people who "need" them and are generally assigned by gatekeeper rather than chosen, so there is limited need to consider economics.  To the extent that economics is considered, it is the rump economics of "cost-effectiveness" and "quality-of-life" measures, which are definitely useful and nontrivial, but only a bit of the picture.  The food side is rather closer to normal consumer goods, but the focus is still on what everyone "needs" (are not willing to give up at any plausible cost), like non-infectious food.  When legitimate economic questions do come up (e.g., some people prefer to be able to consume raw dairy products, even though they are judged not safe enough by the simple bright-line standards), the system has no mechanism for balancing competing preferences, a hint of the challenge in regulating tobacco products.

It is natural that there is no historical capacity to do much economics within FDA.  I would argue that this is the biggest problem the normative side (i.e., ethics -- assessing what is the right thing to do to make people better off), though selling that message is a tough fight.  But it is also a problem on the positive side (assessing how the world works when individual free choices are involved), which ought not to be too tough to sell.  Understanding consumer choices, and being open and explicit about the science, really should be part of the Center for Tobacco Products.

The specific economic question at hand is the effect of menthol on the number of people who choose to smoke, the quantity they choose to smoke, and whether they choose to stop smoking.  These are all economic questions, and when someone tries to address them ad hoc, using epidemiology rather than welfare economics, it does not work out so well.  Indeed, even the description of the question at hand, with the key word "choose" in it, defies the standard narrow medicalized method of looking at products (and does not seem to appear at all in the FDA report, despite how crucial the concept is).

The questions being asked are in the form of "whether" -- e.g., is there likely to be more smoking if menthol is available -- rather than "how much".  But a tiny bit of economics reveals that the "whether" question is like asking "does this object have a weight" rather than "how much does it weigh".  Of course the availability of a flavor that some people like leads to more product use, and therefore its elimination would reduce how much people like the product.  Somewhere out there is someone who is barely on the positive side of indifferent between smoking and not, and very much likes menthol.  Remove the availability of menthol and he would not smoke.

(Notice that I am avoiding the question of implementation here, and simply positing the "removal" outcome.  Merely banning menthol cigarettes rather than magically removing them from the world creates all kinds of interesting complications about black markets and do-it-yourself mentholation, which is quite easy.  I will come back to that in a later post.)

The "how much" question is quite a bit more difficult to answer.  To have any hope of making a useful prediction, it is critical to understand what is going on:  people are using a product because they like it better than its close substitute (nonmenthol cigarettes), most of them probably prefer the substitute to abstinence, some of them like their product enough that they would defy the law, etc.  Without these economic points, it is difficult to imagine making a useful prediction.  Indeed, if you look at the models that have been used for prediction, they are clearly based on premises that are indefensible but probably the only premises that someone can come up with if they ignore economics.

Some consist of assuming that any additional use or initiation of cigarettes that is associated with choosing menthol (i.e., smoking rates are higher in subpopulations that use menthol more often) is causal, and thus without menthol the rate would drop to the average.  The economics shows that while this outcome is in the plausible range, it would be mere luck if it really turned out to be right because the basis for the claim does not actually support the claim.  Which is to say, the prediction has no validity because the premise of it is clearly wrong.  It actually gets worse than that, with some of the modeling going so far as to predict that all consumers of menthol cigarettes would be abstinent if menthol were not available, which is not even in the plausible range of values.  (No, I am not kidding -- one of the most cited predictions about the effect of banning menthol is based on this premise.)

Some of the most dramatic errors in the history of science, to say nothing of incorrect claims you see in the news today, result from confusing statistics with mechanism.  That is, researchers who do not know why something is happening (e.g., atomic theory has not yet been discovered so they have no idea why samples of pure elements weigh what they do, to take a classic historical example; or they seem to not realize that people make choices based on preference, to take the example of tobacco control industry researchers) sometimes go to great lengths to make measurements.  But when they try to interpret the observations without understanding the underlying phenomenon, and basically just assume that the measurements are the phenomenon (an example of which is assuming that all observed association is causal), then whether they are right becomes just a matter of luck.

As I mentioned, the other problem with not understanding the underlying mechanism when dealing with worldly questions is that wrong (in the sense of accuracy) can also be wrong (in the sense of unethical).  When tobacco control activists hide the phenomenon of people's preferences, choices, and happiness behind naive statistics, they avoid having to admit that they are a special interest group trying to impose a narrow "moral" view.  Our nation's government is not a special interest group and generally does a pretty good job of resisting imposing narrow moral views on the citizens (thank you, James Madison et al.!).  But if FDA research ignores the economics, it tends to prevent decision-makers from realizing they are making ethical, not technical, decisions.  (And it allows those who know they are imposing narrow "moral" views to pretend they are not doing so.)

Banning menthol would serve only one purpose: intentionally lowering the welfare of people who currently choose a particular product.  As soon as you express the economic situation in economic terms, this becomes immediately apparent.  So, is that justified by the (legitimately predicted) benefits it would produce?  Is such an action by a for-the-people government ever justified?  Do smokers deserve to have their welfare lowered?  None of these questions are answered by economics or any other science, but economics has the advantage of forcing a recognition that those are the questions that need to be addressed.  Anyone who suggests that the question "should we ban menthol cigarettes" can be answered scientifically, rather than ethically, is doing the wrong science.

Finally, as a comparatively minor aside about how to do social science, I note that the first paragraph of the FDA report makes a claim about the portion of the US cigarette market that is menthol, citing it to a 2004 paper.  2004??!  Folks, social science does not work that way.  People who are used to dealing with biology and other sciences that study phenomena that do not change much over time get into the habit of ignoring when a measurement was made.  This is a mistake even then, but it is a fatal error when dealing with social science -- just think about how much has changed in the tobacco product markets in the last decade.  An economist wanting to make such a summary claim would either find a more recent estimate, make a rough general claim without citation (making clear that the exact number is not known to the author but that it does not matter much), or look at the most recent statistics themselves and do the calculation.  Part of the problem is that people who are used to dealing with only one area of science, medical research, get the mistaken impression that all or most useful information is contained in journal articles.  That is clearly not the case in social science, where constantly updated statistics, working papers, and the blogosphere generally contain much better current information and thinking.  Indeed, as with many serious sciences, when an article appears in a good journal it is more like an archiving and awarding of a trophy, and is not really the publication, since anything worth reading has already been circulating long before the final version is etched in stone.  These are just a few of the things that FDA researchers need to understand now that they have entered the world of studying people as people, and not just as biological agents.

[More on this theme in the next post.]