How reliable is survey data on personality traits from low- and middle-income countries?

Last week in the Weekly Links, David McKenzie shared a new paper recently published in Science. The paper, by Laajaj et al., examines the validity of quantitative measurements of the “Big 5” personality traits (e.g., openness, conscientiousness, extraversion, agreeableness, and emotional stability) in developing countries. Here is the punchline:

This paper compiles and analyzes Big Five data collected through a very diverse set of surveys from low- and middle-income countries in different parts of the world and covering all types of education levels, including large nationally representative surveys and surveys collected on targeted samples for impact evaluation purposes. It shows that commonly used personality questions fail to measure the intended PTs in these settings and do not pass standard validity tests. The analysis of the psychometric properties of Big Five measures raises serious concerns about their validity and hence about their use and subsequent interpretation in many studies in low- and middle-income countries when collected through surveys. In contrast, data collected through the internet from the same set of countries reflect the Big Five factor structure, suggesting that low validity is not primarily driven by cultural or contextual differences.

Yikes!

The figure below shows correlations between each of the Big 5 and income in a pooled data set including data from 14 low- and middle-income countries. The estimated effects are all over the place. Strikingly, conscientiousness—a characteristic often found in data from the US as being a relatively high predictor of earnings—is statistically insignificant in 10 out of the 14 countries.

It gets a little worse! The figure below (from the article’s supplemental material) shows a simple correlation matrix between the 15 sub-items measuring the Big 5 personality traits in the World Bank’s Skills Towards Employment and Productivity (STEP) data. The good news is that the correlation of individual items within each of the Big 5 categories is strictly positive for all items. However, items under the conscientiousness category all correlate more with items under the agreeableness and emotional stability categories than compared to other items within the conscientious category. This is contrary to the robust findings of psychometricians that the Big 5 personality traits each broadly emerge from repeated factor analysis of a set of questions describing individual personality in the United States.

So, should we give up analyzing data on personality traits in developing countries? I’d emphatically say, absolutely not! And I think the authors of this study agree.

The key takeaway of this study, for me, is to be more careful when analyzing data on personality traits collected via a survey in low- and middle-income countries. Even when using data collected by experienced organizations, we (i.e., empirical researchers) should take care to validate the quality of the data. (Indeed, this is the topic of my JDS paper where we tried to quantitatively measure “hope” in rural Myanmar.) The authors list several other suggestions for ensuring the validity of data on personality traits, such as (1) balancing reversed and nonreversed items in aggregate scales, (2) using self-administered surveys, rather than surveys administered by an enumerator that can induce measurement error, (3) if using enumerators take time to train the enumerators to minimize measurement error, and (4) randomly assign enumerators or at least ensure that bias does not directly follow the variation of treatment in the data.

The paper itself includes much more detail than I note in this short blog post. If you use data on personality traits (or other psychological characteristics), I’d recommend digging into the full article.