TOMM Cut Scores May Not Generalize to Spanish-Speaking Adults - CONCEPT Professional Training

Alicia Nijdam-Jones, Fordham University
Diego Rivera, BioCruces Health Research Institute, Cruces University Hospital
Barry Rosenfeld, Fordham University
Juan Carlos Arango-Lasprilla, BioCruces Health Research Institute, Cruces University Hospital, and IKERBASQUE – Basque Foundation for Science, Bilabo, Spain

Cognitive efforts tests, such as the Test of Memory Malingering (TOMM; Tombaugh, 1997), are widely used internationally, yet there is a dearth of research that has assessed the utility of these measures in different cultures, countries, and languages. This study evaluated the specificity of the TOMM Trial 2 among a sample of 3,590 Spanish-speaking adults residing in 8 Latin American countries (Argentina, Bolivia, Chile, Colombia, Mexico, Paraguay, Peru, and Puerto Rico). Trial 2 TOMM scores were negatively associated with participants’ age and positively associated with level of education. Country development, as measured by the United Nations Human Development Index, was also positively associated with TOMM scores. With the widespread use of cognitive efforts tests, this study offers useful insights into the utility of the TOMM in the assessment of Spanish speakers and highlights potential cultural biases that may impact test performance. Furthermore, this study raises concerns about the cross-cultural applicability of the TOMM, particularly when using cut scores established and evaluated primarily on North American, English-speaking samples. Forensic psychologists should be cautious when interpreting TOMM performance with Spanish-speaking adults from Latin American countries because it appears that there are cross-cultural differences that influence test performance.

memory malingering, Latin America, feigning, cross-cultural

“Although the assessment of feigning is a universal challenge for forensic psychologists, very few measures have been adequately examined with regard to their cross-cultural validity” (p. 422).

“One of the most widely used cognitive effort tests is the Test of Memory Malingering (TOMM; Tombaugh, 1997). The TOMM is a visual recognition performance validity test that assesses for poor cognitive effort and/or exaggerated memory problems using a forced-choice approach. Research with the TOMM has demonstrated adequate sensitivity and high specificity (i.e., very few false-positive results) in English-speaking, North American samples” (p. 423).

“[D]espite the widespread use of cognitive efforts tests such as the TOMM, there is a dearth of research that has assessed the utility of these measures in different cultures, countries, and languages (Nijdam-Jones & Rosenfeld, 2017). The potential for biases in a measure’s linguistic, functional, cultural, and metric equivalence across languages and cultures highlights the importance of evaluating these tests in a wide range of settings (Peña, 2007). Of particular concern is the risk of false-positive results, incorrectly identifying a genuinely impaired (or cognitively intact) individual as exhibiting insufficient cognitive effort because of biases in the measure. Even measures such as the TOMM, which rely on visual rather than verbal stimuli, may be vulnerable to systematic biases, perhaps because of differences in the familiarity of the test stimuli or in the relative strength of visual versus verbal memory” (p. 423).

“A handful of studies have examined the TOMM with Spanish-speaking samples, revealing a range of specificity (79–100%) using the established cut score” (p. 423).

“It was hypothesized that education levels and overall socioeconomic development of the country would be associated with performance on the TOMM. Moreover, we anticipated differences across the eight countries in the strength of the association between education and TOMM scores (i.e., A Country × Education interaction effect)” (p. 424).

“This study sample was comprised of 3,590 healthy adults residing in Argentina, Bolivia, Chile, Colombia, Mexico, Paraguay, Peru, and Puerto Rico who were involved in a larger study generating normative data for a series of neuropsychological instruments. […] To be eligible for study participation, individuals must have met the following requirements: (a) between 18 and 84 years old; (b) born and currently living in the country where the study was conducted; (c) native Spanish speaker; (d) completed at least 1 year of formal education beginning in the first grade; (e) grossly intact on cognitive testing […]; (f) no evidence of significant depression […]; and (g) evidence of sufficient independent daily functioning” (p. 424).

“All participants were administered the Spanish version of the TOMM (Tombaugh, Vilar-López, García, & Puente, 2011) […] with Trial 2 (T2) scores serving as the basis for all analyses described below” (p. 424).

“Unlike studies conducted in the United States, we found widely diverging levels of specificity across the eight samples, with excellent specificity rates in some countries (e.g., Argentina, 99.7%; Puerto Rico, 98.7%) and surprisingly poor specificity in others (e.g., Paraguay, 75.9%; Bolivia, 90.0%). We also identified significant mediators of TOMM T2 performance, including age (negatively associated) and education (positively associated). Moreover, the strength of these associations varied significantly across countries, indicating that the confounding influence of these variables is greater in some countries and cultures than others” (p. 426).

“Age was negatively correlated with TOMM scores within the total sample as well as within each Latin American country […] indicating that younger participants performed better on the TOMM than older participants” (p. 426).

“Although no relationship between education and TOMM T2 scores has been previously observed among English-speaking North American samples, it has been identified in other research studies with Latino and Hispanic adults, with less educated individuals failing the TOMM at greater rates than those with higher levels of education” (p. 426).

“[P]ast findings demonstrating an association between socioeconomic status and test performance was also supported, albeit using a proxy variable based on the United Nations Human Development Index” (p. 426).

“TOMM specificity rates among Spanish-speaking adults in Latin America varied markedly between countries, and there were significant associations between participants’ age and education with TOMM scores. Furthermore, a country’s development was also correlated with TOMM scores” (p. 427).

“The limited cross-cultural validity stands in contrast to ethical guidelines and principles, which mandate psychologists to use culturally appropriate assessment instruments that account for the demographic factors relevant to test performance” (p. 422).

“A false-positive test result can have serious repercussions for the individual being assessed. It could prevent the provision of appropriate medical treatment or legitimate financial compensation for an injury, or in a criminal justice context, it could lead to the unjust prosecution and/or stigmatize the individual, perhaps leading to even more punitive sanctions” (p. 423).

“Education, literacy, acculturation, racial socialization, and socioeconomic status have all been recognized as influencing neuropsychological test performance in culturally diverse samples” (p. 423).

“The importance of identifying assessment tools that have cross-cultural validity has become increasingly recognized across a range of settings. Particularly in the context of forensic evaluations in which the potential for symptom or deficit exaggeration exists, the need for reliable and valid measures of cognitive effort is clear. One such instrument that has widespread support is the TOMM, a nonverbal performance-based measure that uses visual recognition memory to identify suboptimal cognitive effort” (p. 426).

“[T]he main effects of age and the interaction of age by country were significant, even after controlling for the influence of education, which may (suggest that the influence of age is not simply due to better access to educational opportunities among younger participants) [sic]. Because age has not been associated with performance on the TOMM in prior research with North American, English-speaking samples, the reasons for this association are not clear. One possible explanation is a cohort effect, whereby this type of testing (or testing more generally) is less familiar to older than younger individuals” (p. 426).

“[A] false-positive rate of nearly 25% among respondents from Paraguay, all of whom had an incentive to put forth their best effort, raises serious questions about the cross-cultural validity of even a nonverbal measure such as the TOMM. On the other hand, the impressive specificity rates in Argentina, Peru, and Puerto Rico should provide reassurance that the measure is not likely to yield inflated false-positive classifications in those countries” (p. 426).

“Although most research collapses Hispanic/Latino participants into a single group, conceptualizing culture on a more micro level reveals important differences that would have otherwise been obscured. […] Clearly, cross-cultural research needs to consider regional differences rather than simply summarizing diverse cultures into a single sample on the basis of a common language or shared border” (p. 427).

“Forensic psychologists should be cautious when interpreting TOMM performance with Spanish-speaking adults from Latin American countries because it appears that there are cross-cultural differences that influence test performance” (p. 427).

“Sensitivity, also known as the true-positive rate, refers to the proportion of feigning individuals that a test accurately identifies as feigning. Specificity, on the other hand, refers to the proportion of genuine honest responders who are accurately identified by the test as not feigning (i.e., true negatives)” (p. 423).

“Research conducted in Spain has also revealed lower accuracy rates using the TOMM than studies conducted in the United States” (p. 423).

“Participants who were illiterate were not included in the study, and therefore, the findings may not represent the diverse body of adults living in the countries examined. Indeed, inclusion of illiterate individuals may have decreased the specificity rate even further, given the association with education. Another important consideration is that the assessments occurred with Spanish-speaking residents within several Latin American countries. Thus, the generalizability of these results to migrants living in the United States may be limited” (p. 427).

“Future research should examine the cultural biases associated with individual items in the TOMM as well as how bilingualism might affect test performance more generally” (p. 427).

As always, please join the discussion below if you have thoughts or comments to add!