Models v. Mechanisms at FDA Center for Tobacco Products

Observed coincidences occur far more often than chance would suggest because we look for them and define our list of what would constitute an interesting coincidence based on what actually occurred (we have no intuition for just how huge the denominator is).  I know that.  Still, I find it pretty remarkable that for the last couple of days I have been trying to nail down exactly how to explain what is missing from the behavioral modeling by FDA CTP, and then discovered in my morning econoblogosphere reading, the answer I needed seems to be the topic of the hour.

Yesterday I posted some advice to FDA CTP about the need to understand social science (mainly economics) in their modeling of behavior. There is relatively little economics that needs to be considered in traditional FDA missions, and that which is needed is relatively simple.  But regulations that are intended to affect preferences about a freely-chosen consumer product where preferences vary across the population (i.e., like tobacco products, in contrast with medicines or food safety) are all about economics.  The failure to include explicit economic analysis in the recent report on the possibility of banning menthol cigarettes illustrated the problems, both scientific and ethical.

But if you were to suggest to the people working in the FDA orbit that they do not really have a model of people's choices about tobacco products (as I have argued), they would probably reply that they do have models.  Several of them.  (For those who are familiar with this field I am, of course, talking about Levy, Mendez, Environ.  For those not familiar, that should present no obstacle to understanding this post -- just know that I am talking about a handful of well-known specifics.)

Ok, there are models.  But there is something fundamentally wrong with them.  They do not offer us any reason to believe in what they say will occur at the micro level (that is, why each individual person whose actions, collectively, result in the outcome, will do what they suggest they will do).  What I mean is that they tell us things like "if X% fewer people start smoking each year", say, due to a menthol ban, "then this graph shows the number of smokers in the future, which is lower than current trends by Y" (if they fill in other information and make a bunch of other assumptions about what is happening, of course).  But as for why X% fewer people would start smoking, there is nothing at all.  There is just the number.

I have criticized these as being more like calculators than models.  If a population starts at 1 and doubles, every period then the number after n periods is 2^n.  But it is hard to call the equation "2^n" a model.  Similarly, I argued, the preferred "models" used by the tobacco policy inner circle are just more complicated equations.

I learned, however, that this point was not widely convincing based on sociological empiricism -- i.e., I tried to make the point to people and did not have much success.  I realized perhaps why this was based on my blog reading from this morning:  I was using the wrong words to make my point.  It is not that these are not models; any calculation, no matter how simple, can be called a model if it is representing a worldly phenomenon in some useful way (even that lowly 2^n).  The problem with these models, and the reason they failed as legitimate models, is the lack of mechanism.

A model uses numbers and equations to show how one variable/construct/point-in-time/etc. affects others.  But the model may not capture why a particular effect occurs (the mechanism), as with that X% reduction in smoking initiation.  In such cases, it is really just answering a hypothetical question ("if X were true, then Y") rather than making real predictions ("X appears to be true, therefore Y").  But the models that would be useful for FDA purposes are ones that tell us "therefore" not just "if...then".  Of course, every model is going to have some simplified or hypothetical elements (if there are no simplifications it is not a model, it is reality) and, once again, there are no bright lines since the mechanisms generally are abstractions (i.e., models) in themselves.  But for a model to offer predictions that do not just result from hypothetical inputs, there has to be some "why" built into it.

It seems that the problem is that these models have been developed in a world where the only familiar social science is epidemiology.  Epidemiology usually fails as a social science, and as a science more generally, because there is very little attention paid to mechanisms.  That it fails as a social science is fairly easy to explain: most people doing it do not realize they are engaging in social science, and most people teaching it have no background in social science.  They think they are just doing medical trials.  Sometimes this is literally true, of course, and sometimes the observational epidemiology is legitimately an attempt to substitute for medical trials.  But as soon as what is being studied is not purely biological, and involves people as people, not just as organisms, it is social science.  Medical trials are easy because either the mechanism is obvious or it does not matter -- e.g., this drug makes cancer go away, and we probably have a guess about why, but that guess does not matter because the mechanism does not matter to the epidemiology (though obviously it does for the drug development process).

The failure to be good science at all is less easy to explain or defend.  For almost 15 years, mechanism-oriented methods have been developed and taught (in the few good epidemiology departments).  These tend to be pretty simple, just boxes and arrows that show what is causing what, but that is most of what you need.  Unfortunately these are (a) seldom used at all and (b) almost exclusively used just for identifying confounders.  The latter is useful, of course, and doing it is far better than not doing it.  But what is missing is use of these mechanistic models to address questions like "if X is really causing Y by affecting Z, then I should be able to observe not just an association between X and Y, but also..."  Such scientific hypothesis testing is close to completely absent from epidemiology.  Instead, mechanisms in epidemiology exist entirely in the untested conclusion statements.  You have seen it: An association is observed and there is a discussion of how X must be to be causing Y as a result of Z, or whatever, but whether that really seems to be true is never addressed scientifically.  It is worth reiterating:  In epidemiology, mechanisms live almost entirely in the conclusions and not in the science.

So circling back to the question of tobacco behavior modeling, when the models are developed in the tradition of epidemiology, it is little wonder that there is no mechanism.  The "why" of what happens when a variable changes is not part of epidemiology, and so not part of the models.  It is just assumed that if the effects of a particular variable changing were observed in the past -- or more likely, merely if there was just some association observed in the past, with no effects of changes observed -- then that same association will still occur if an intervention is imposed (e.g., menthol is banned).  But there is usually no reason to believe that, and indeed, often a lot of reason to not believe it.  To take an extreme case, one of the popular models assumes that without menthol, the rate of smoking initiation would drop by the rate at which smoking is initiated with menthol cigarettes.  Put a little more simply, this basically is the assumption is that everyone who would have initiated with menthol will therefore never smoke (it is even a bit worse than that because it is based on past associations which might themselves change).  I suspect I do not need to explain why the implicit mechanism about people choosing to initiate smoking menthol cigarettes is rather absurd.  The absurdity of that seems unfathomable unless you recognize the mechanism-free mindset: "all we know [the mindset goes] is the association we observed before, so we just have to assume that association will always exist".

FDA on menthol cigarettes, some suggestions for research methods

The U.S. FDA is probably the most respected and influential medical research organization in the world.  Sure they have their hiccups and there are criticisms (many legitimate, many not) about the science and the choices about what risks to take (historically entirely in the direction of them being too quick to keep a potentially useful drug off of the market, more recently in both directions).  But all in all, it is hard to imagine engineering an institution that does much better.

But tobacco controllers (including a former head of FDA during his tenure) and their pet congressmen got the bright idea of adding a category of non-medical consumer goods to FDA's scientific purview.  To these non-scientists, it probably seemed that there was no contradiction here.  Science is science, right?  Clinical trials of medicines, monitoring food safety, consumer preferences, social forces, climate change research, isolating the Higgs boson -- if they are good at the first two, and experienced in the related ethical questions, then they must be able to do the rest, right?

Um, no.

It is clear that the FDA scientists who have been asked to look at tobacco products are trying, and it is equally clear they are frustrated.  Their latest report on menthol in cigarettes and the predicted effects of banning it [I got it here but that link seems to not work now], offers an opportunity for some unsolicited advice.  After I collect comments and my thoughts, I might include some of it as a public comment, [UPDATE: link fixed] which they are currently soliciting.

Their evaluation concludes that there is no measurable difference in the biological risk between menthol and non-menthol cigarettes (keeping the quantity of exposure constant).  This is based on research that FDA is pretty good at; it is not quite the same as their core competencies, medical trials and safety testing, but in the neighborhood certainly.

But the crux of the decision about whether to ban menthol is presented in terms of economics.  Unfortunately they do not say this.  This is presumably because economics -- the relevant science when you are looking at consumer choices  -- has never been part of what FDA does.  Drugs and medical devices are evaluated based on people who "need" them and are generally assigned by gatekeeper rather than chosen, so there is limited need to consider economics.  To the extent that economics is considered, it is the rump economics of "cost-effectiveness" and "quality-of-life" measures, which are definitely useful and nontrivial, but only a bit of the picture.  The food side is rather closer to normal consumer goods, but the focus is still on what everyone "needs" (are not willing to give up at any plausible cost), like non-infectious food.  When legitimate economic questions do come up (e.g., some people prefer to be able to consume raw dairy products, even though they are judged not safe enough by the simple bright-line standards), the system has no mechanism for balancing competing preferences, a hint of the challenge in regulating tobacco products.

It is natural that there is no historical capacity to do much economics within FDA.  I would argue that this is the biggest problem the normative side (i.e., ethics -- assessing what is the right thing to do to make people better off), though selling that message is a tough fight.  But it is also a problem on the positive side (assessing how the world works when individual free choices are involved), which ought not to be too tough to sell.  Understanding consumer choices, and being open and explicit about the science, really should be part of the Center for Tobacco Products.

The specific economic question at hand is the effect of menthol on the number of people who choose to smoke, the quantity they choose to smoke, and whether they choose to stop smoking.  These are all economic questions, and when someone tries to address them ad hoc, using epidemiology rather than welfare economics, it does not work out so well.  Indeed, even the description of the question at hand, with the key word "choose" in it, defies the standard narrow medicalized method of looking at products (and does not seem to appear at all in the FDA report, despite how crucial the concept is).

The questions being asked are in the form of "whether" -- e.g., is there likely to be more smoking if menthol is available -- rather than "how much".  But a tiny bit of economics reveals that the "whether" question is like asking "does this object have a weight" rather than "how much does it weigh".  Of course the availability of a flavor that some people like leads to more product use, and therefore its elimination would reduce how much people like the product.  Somewhere out there is someone who is barely on the positive side of indifferent between smoking and not, and very much likes menthol.  Remove the availability of menthol and he would not smoke.

(Notice that I am avoiding the question of implementation here, and simply positing the "removal" outcome.  Merely banning menthol cigarettes rather than magically removing them from the world creates all kinds of interesting complications about black markets and do-it-yourself mentholation, which is quite easy.  I will come back to that in a later post.)

The "how much" question is quite a bit more difficult to answer.  To have any hope of making a useful prediction, it is critical to understand what is going on:  people are using a product because they like it better than its close substitute (nonmenthol cigarettes), most of them probably prefer the substitute to abstinence, some of them like their product enough that they would defy the law, etc.  Without these economic points, it is difficult to imagine making a useful prediction.  Indeed, if you look at the models that have been used for prediction, they are clearly based on premises that are indefensible but probably the only premises that someone can come up with if they ignore economics.

Some consist of assuming that any additional use or initiation of cigarettes that is associated with choosing menthol (i.e., smoking rates are higher in subpopulations that use menthol more often) is causal, and thus without menthol the rate would drop to the average.  The economics shows that while this outcome is in the plausible range, it would be mere luck if it really turned out to be right because the basis for the claim does not actually support the claim.  Which is to say, the prediction has no validity because the premise of it is clearly wrong.  It actually gets worse than that, with some of the modeling going so far as to predict that all consumers of menthol cigarettes would be abstinent if menthol were not available, which is not even in the plausible range of values.  (No, I am not kidding -- one of the most cited predictions about the effect of banning menthol is based on this premise.)

Some of the most dramatic errors in the history of science, to say nothing of incorrect claims you see in the news today, result from confusing statistics with mechanism.  That is, researchers who do not know why something is happening (e.g., atomic theory has not yet been discovered so they have no idea why samples of pure elements weigh what they do, to take a classic historical example; or they seem to not realize that people make choices based on preference, to take the example of tobacco control industry researchers) sometimes go to great lengths to make measurements.  But when they try to interpret the observations without understanding the underlying phenomenon, and basically just assume that the measurements are the phenomenon (an example of which is assuming that all observed association is causal), then whether they are right becomes just a matter of luck.

As I mentioned, the other problem with not understanding the underlying mechanism when dealing with worldly questions is that wrong (in the sense of accuracy) can also be wrong (in the sense of unethical).  When tobacco control activists hide the phenomenon of people's preferences, choices, and happiness behind naive statistics, they avoid having to admit that they are a special interest group trying to impose a narrow "moral" view.  Our nation's government is not a special interest group and generally does a pretty good job of resisting imposing narrow moral views on the citizens (thank you, James Madison et al.!).  But if FDA research ignores the economics, it tends to prevent decision-makers from realizing they are making ethical, not technical, decisions.  (And it allows those who know they are imposing narrow "moral" views to pretend they are not doing so.)

Banning menthol would serve only one purpose: intentionally lowering the welfare of people who currently choose a particular product.  As soon as you express the economic situation in economic terms, this becomes immediately apparent.  So, is that justified by the (legitimately predicted) benefits it would produce?  Is such an action by a for-the-people government ever justified?  Do smokers deserve to have their welfare lowered?  None of these questions are answered by economics or any other science, but economics has the advantage of forcing a recognition that those are the questions that need to be addressed.  Anyone who suggests that the question "should we ban menthol cigarettes" can be answered scientifically, rather than ethically, is doing the wrong science.

Finally, as a comparatively minor aside about how to do social science, I note that the first paragraph of the FDA report makes a claim about the portion of the US cigarette market that is menthol, citing it to a 2004 paper.  2004??!  Folks, social science does not work that way.  People who are used to dealing with biology and other sciences that study phenomena that do not change much over time get into the habit of ignoring when a measurement was made.  This is a mistake even then, but it is a fatal error when dealing with social science -- just think about how much has changed in the tobacco product markets in the last decade.  An economist wanting to make such a summary claim would either find a more recent estimate, make a rough general claim without citation (making clear that the exact number is not known to the author but that it does not matter much), or look at the most recent statistics themselves and do the calculation.  Part of the problem is that people who are used to dealing with only one area of science, medical research, get the mistaken impression that all or most useful information is contained in journal articles.  That is clearly not the case in social science, where constantly updated statistics, working papers, and the blogosphere generally contain much better current information and thinking.  Indeed, as with many serious sciences, when an article appears in a good journal it is more like an archiving and awarding of a trophy, and is not really the publication, since anything worth reading has already been circulating long before the final version is etched in stone.  These are just a few of the things that FDA researchers need to understand now that they have entered the world of studying people as people, and not just as biological agents.

[More on this theme in the next post.]