“How Much Does the Cardinal Treatment of Ordinal Variables Matter?”—Forthcoming

I am very excited to share that my paper, “How Much Does the Cardinal Treatment of Ordinal Variables Matter? An Empirical Investigation” is now (finally) forthcoming in the journal Political Analysis. I wrote the first draft of this paper in my 2nd-year paper class at the University of Minnesota. So, publishing this paper in the official methods journal of the American Political Science Association is particularly rewarding.

Here is the abstract:

Many researchers use an ordinal scale to quantitatively measure and analyze concepts. Theoretically valid empirical estimates are robust in sign to any monotonic increasing transformation of the ordinal scale. This presents challenges for the point-identification of important parameters of interest. I develop a partial identification method for testing the robustness of empirical estimates to a range of plausible monotonic increasing transformations of the ordinal scale. This method allows for the calculation of plausible bounds around effect estimates. I illustrate this method by revisiting analysis by Nunn and Wantchekon (2011, American Economic Review, 101, 3221–3252) on the slave trade and trust in sub-Saharan Africa. Supplemental illustrations examine results from (i) Aghion et al. (2016, American Economic Review, 106, 3869–3897) on creative destruction and subjective well-being and (ii) Bond and Lang (2013, The Review of Economics and Statistics, 95, 1468–1479) on the fragility of the black–white test score gap.

Now, you may be thinking, “Why do I need such a method?” “Is not this exactly what an ordered probit or ordered logit regression specification is designed to address?”

If you are thinking this, you are correct. However, in practice, applied researchers tend to prefer linear regressions to an ordered logit or an ordered probit despite the known concern with the cardinal treatment of an ordinal dependent variable. This is because causal inference is typically easier to interpret and defend when using linear regression, and most applied researchers care first-and-foremost about causal inference.

Moreover, recent work on the “Sad Truth about Happiness Scales” highlights that ordered logit or ordered probit regression specifications implicitly assume either a normal or logistic distribution on the error term, when alternative distributions of the error term are also theoretically permissible.

So, what are applied researchers to do? Well, as I’ve previously blogged about, one approach is to pursue partial identification and this paper provides a methodology to do just that.

The method involves three steps: (1) Limit the universe of monotonic increasing transformations defined by a parameterized function representing a (relatively extreme) range of transformations. (2) Based on this range of transformations, the researcher estimates a set of effect estimates. (3) If effect estimates are not robust to this initial range of transformations the researcher graphically assesses the plausibility of the transformation associated with specific effect estimates. This method, therefore, tests the sensitivity of the parameter of interest to a range of plausible monotonic increasing transformations and generates a set of plausible effect estimates.

I apply this method to several papers that use an ordinal dependent variable and yet use linear regression methods. Some results are more robust than others. For example, in the figure above, I show results from Nunn and Wantchekon (2011) on the effect of the trans-Atlantic slave trade on trust in sub-Saharan Africa. In each panel, the vertical axis measures the estimated effect and the horizontal axis represents a parameter value characterizing a different monotonic increasing transformation of the ordinal scale. As the figure shows, the estimates are quite robust to these transformations.

If this sounds useful to your research, check the paper out and let me know if you have any questions. A pre-print of the paper is available here. Also, keep an eye on this space for a follow-up paper on this topic co-authored with Andrew Oswald.