Have an ordinal dependent variable? Use this robustness test.

Ordinal variables are everywhere. Data providing information about happiness, levels of customer satisfaction, employees’ satisfaction, mental stress, psychological well-being, societal trust, and other important variables are now regularly collected and analyzed by national governments, large multinational companies, and researchers. However, because these data are not directly observable or quantitatively measurable, they are thus not measured on objective cardinal units. This presents a key challenge when performing standard quantitative analysis.

Even more concerning, this challenge casts doubt on many existing results that use an ordinal dependent variable in regression analysis.

In a new paper, co-authored with Andrew Oswald, we develop a simple robustness test that we think can help assuage concerns about the potentially invalid cardinal treatment of an ordinal dependent variable.

Here is the abstract:

“Governments, multinational companies, and researchers today collect unprecedented amounts of data on human feelings. These data provide information on citizens’ happiness, levels of customer satisfaction, employees’ satisfaction, mental stress, societal trust, and other important variables. Yet a key scientific difficulty tends to be downplayed, or even ignored, by many users of such information. Human feelings are not measured in objective cardinal units. This article aims to address some of the ensuing empirical challenges. It suggests an analytical way to approach the scientific complications of ordinal data. The article describes a dichotomous-around-the-median (DAM) test, which, crucially, uses information only on direction within an ordering and deliberately discards the potentially unreliable statistical information in ordered data. Applying the proposed DAM approach, this article shows that it is possible to check and replicate some of the key conclusions of previous research—including earlier work on the effects upon human well-being of higher income.”

We apply our DAM test to numerous applications in the paper, but I will highlight just one in this blog post. Haushofer and Shapiro (2016) estimates the effect of experimentally provided cash transfers on psychological well-being (e.g., happiness and life satisfaction) in Kenya. They use survey questions from the World Values Survey, which measure happiness and life satisfaction on an ordinal scale. The authors use a standardized version of this ordinal scale as their dependent variable. We find that the original findings of Haushofer and Shapiro (2016) are robust to our DAM test.

Check out the rest of the paper if this robustness test sounds useful and applicable to your work. We hope that this simple robustness check will help researchers achieve the duel goal of (i) taking seriously the analytical challenge of using ordinal variables and (ii) using the important information embedded in ordinal variables in serious and policy-relevant analysis.