AI Brief: Is the Intellectual Functioning Component of AAIDD’s 12th Manual Satisficing?

AI Brief:  Is the Intellectual Functioning Component of AAIDD's 12th Manual Satisficing?

 (McGrew, 2021)

 

Dr. Kevin McGrew with assist from Google NotebookLM

 

In a commentary published in Intellectual and Developmental Disabilities, Kevin S. McGrew  evaluated the intellectual functioning section (prong 1) of the AAIDD’s 12th edition manual (2021) for diagnosing intellectual disabilities (ID). He commends the organization for finally adopting the Cattell-Horn-Carroll (CHC) theory, which aligns the manual with modern scientific consensus on cognitive abilities. However, the author expresses significant concern about the manual’s contradictory guidance on part scores, arguing that its ambiguous stance could lead to legal and diagnostic confusion. McGrew also highlights various technical measurement issues and numerous copyediting errors that he believes undermine the manual's status as an authoritative resource. He suggests that while the manual is satisfactory in its theoretical shift, it does not provide the precise clarity needed for high-stakes clinical and judicial settings.

 

_______________________

 

In his review of the 12th edition of the American Association on Intellectual and Developmental Disabilities (AAIDD) manual, McGrew (2021; link for downloading article) evaluates whether the "Intellectual Functioning Component" (aka.,prong 1 of a three-prong definition of intellectual disability—ID) provides a "satisficing"—or satisfactory and sufficient—solution for practitioners and scholars.[1] McGrew draws on over 45 years of experience in school psychology and intelligence research, theory, and test development. In addition, he draws on his expert work and consultation (since 2009) on Atkins intellectual disability (ID) death penalty cases in legal settings. McGrew provides an evaluation of the AAIDD’s manual's prong 1 (intellectual functioning) theoretical grounding, technical guidance, and professional polish. He does not evaluate the other two ID prongs (adaptive behavior and age of onset).

Advancement in Intelligence Theory

McGrew awards the manual a Grade B+ for its formal adoption of the Cattell-Horn-Carroll (CHC) theory of intelligence. This shift aligns the AAIDD manual with the contemporary consensus taxonomy of cognitive abilities, moving away from outdated models. However, McGrew notes that the manual "muddies the CHC waters" by giving preferential treatment to fluid (Gf) and crystallized (Gc) intelligence while neglecting other broad CHC abilities like learning efficiency (Gl), working memory (Gwm), retrieval fluency (Gr), auditory processing (Ga), visual-spatial processing (Gv), and processing speed (Gs). He suggests that a visual-graphic model of the CHC hierarchical model would have been a beneficial addition for users.

Measurement and Organizational Challenges

The manual receives a Grade B- for its treatment of major measurement issues. While it provides adequate coverage of such measurement issues as the standard error of measurement (SEM), confidence intervals, and the Flynn effect (aka., norm obsolescence), McGrew criticizes the lack of a topic index, which makes finding specific guidance very frustrating. For instance, practice effects are obscurely placed under "progressive error" in the glossary, and the Flynn effect is curiously categorized under "Making a Retrospective Diagnosis," despite being relevant to historical and current intellectual assessments.

The Part-Score Controversy

The most critical evaluation—a Grade C—is reserved for the manual's handling of part scores. McGrew identifies three primary failures in this area:

 

●      Inconsistency: The manual contradicts itself by advising against the use of part scores as proxies for general intelligence (psychometric g) while simultaneously suggesting that their valid use requires 3–6 subtests of Gf and Gc.

●      Variance with Other Authorities: This "just say no to part scores" stance conflicts with other major authoritative sources, such as the DSM-5, which acknowledges that highly discrepant subtest scores may invalidate an overall IQ score.

●      Scientific and Legal Tensions: McGrew argues that the manual fails to address the "General-2-individual" (G2i) legal principle which acknowledges that group-based scientific research (e.g., suggesting full-scale scores are always superior) may not apply to every unique individual case—the G2i principle conundrum is that scientists generalize; but courts must particularize to an individual. He warns that without clearer guidance; legal entities may fill the void with "remedies of dubious quality.”

Editorial Quality and Professionalism

McGrew gives the manual a Grade D for style and substance, citing at least 20 copyedit errors in the sections relevant to the intellectual functioning prong alone. These include misspellings of prominent researchers, incorrect terminology like "test e-norms," and frequent "misplaced italics.” He contends that such preventable errors tarnish the manual’s status as an "authoritative" and "definitive" source for diagnosing intellectual disabilities.

Conclusion

McGrew concludes that while the endorsement of CHC theory is a significant positive revision, the manual’s obfuscation regarding part scores and its numerous editorial flaws represent major missed opportunities. He emphasizes that practitioners cannot wait another decade for the next edition to offer more robust guidance, particularly in high-stakes legal and diagnostic settings. He concludes that while he may be a "tough grader," his critiques are intended to push AAIDD toward more robust and clearer guidance in future editions or supplements

[1] Nobel laureate Herb Simon advanced the behavioral economics concept of satisficing (Simon, 1956)—the idea that, although we may aspire to optimal solutions, real-world constraints often require us to settle on what is both satisfactory and sufficient (hence, the portmanteau term satisficing).

IQ score and the “seduction of quantification”: Concise overview of the historical #eugenics use of #IQ and emerging new conceptualizations—#g #WJV #CHC #POT #processoverlap #schoolpsychology #schoolpsychologists #emergentproperty

Click on image to enlarge for easy reading

I stumbled on this relatively concise article that provides a nice (and brief) overview of the historical “bad days” of IQ test and score misuse.  Don’t let the title’s focus on epilepsy deter you from reading—the content is relevant to thinking about intelligence and IQ scores in general.  After the succinct overview of the horrible historical uses of IQ tests and scores, the article touches on contemporary theories and thinking (e.g., process overlap theory or POT; CHC cognitive abilties theory) that view the IQ score as nothing more than a statistical emergent property index—and the need to focus on broad CHC abilities from cognitive ability tests. 

Recommended reading--available as open access here. 

Click here for prior relevant post about IQ scores being emergent property scores.  Click here for WJ V authors view’s on relevance of global IQ scores.  See recent McGrew et al. (2023) article for more information and discussion.

Journal of Intelligence “Best Paper Award” for McGrew, Schneider, Decker & Bulut (2023) Psychometric network analysis of CHC measures - #psychometric #networkanalysis #intelligence #CHC #WJIV #bestpaper #schoolpsychology #schoolpsychologist

Today I (Kevin McGrew), and colleagues Joel Schneider, Scott Decker, and Okan Bulut, were pleased to learn that our recent 2023 Journal of Intelligence article listed above (open access—click link to read or download) was selected as 1 of 2 “Best Paper Awards” for 2023.  

As stated at the journal award page, “The Journal of Intelligence Best Paper Award is granted annually to highlight publications of high quality, scientific significance, and extensive influence. The evaluation committee members choose two articles of exceptional quality that were published in the journal the previous year and announce them online by the end of June.”

Below is the abstract and two figures that may pique your interest. We thank the members of the JOI evaluation committee.

Abstract

For over a century, the structure of intelligence has been dominated by factor analytic methods that presume tests are indicators of latent entities (e.g., general intelligence or g). Recently, psychometric network methods and theories (e.g., process overlap theory; dynamic mutualism) have provided alternatives to g-centric factor models. However, few studies have investigated contemporary cognitive measures using network methods. We apply a Gaussian graphical network model to the age 9–19 standardization sample of the Woodcock–Johnson Tests of Cognitive Ability—Fourth Edition. Results support the primary broad abilities from the Cattell–Horn–Carroll (CHC) theory and suggest that the working memory–attentional control complex may be central to understanding a CHC network model of intelligence. Supplementary multidimensional scaling analyses indicate the existence of possible higher-order dimensions (PPIK; triadic theory; System I-II cognitive processing) as well as separate learning and retrieval aspects of long-term memory. Overall, the network approach offers a viable alternative to factor models with a g-centric bias (i.e., bifactor models) that have led to erroneous conclusions regarding the utility of broad CHC scores in test interpretation beyond the full-scale IQ, g.

Keywords: 

CHC; Cattell–Horn–Cattell theory; intelligence; cognitive abilities; psychometric network analysis; process overlap theory; dynamic mutualism; factor analysis; general intelligence

Click on images to enlarge for easier viewing/reading

Stay tunned!!!! #WJV g and non-g multiple #CHC theoretical models to be presented in the forthcoming (2025) technical manual: Senior author’s (McGrew) position re the #pscyhometric #g factor and #bifactorg models.

(c) Copyright, Dr. Kevin S. McGrew, Institute for Applied Psychometrics (11-14-24)

Warning, may be TLDR for many. :).  Also, I will be rereading again multiple times and may tweak minor (not substantive) errors and post updates….hey….blogging has an earthy quality to it:)

        In a recent publication, Scott Decker, Joel Schneider, Okan Bulut and I (McGrew, 2023; click here to download and read) presented structural analysis of the WJ IV norm data using contemporary psychometric network analysis (PNA) methods.  As noted in a clip from the article below, we recommended that intelligence test researchers, and particularly authors and publishers of the respective technical manuals for cognitive test batteries, needed to broaden the psychometric structural analysis of a test battery beyond the traditional (and almost exclusive) relieance on “common cause” factor analysis (EFA and CFA) methods to include PNA analysis…to compliment, not supplant factor based analyses.

(Click on image to enlarge for easier reading)

         Our (McGrew et al., 2023) recommendation is consistent with some critics of intelligence test structural research (e.g., see Dombrowski et al., 2018, 2019; Farmer et al., 2020) who have cogently argued that most intelligence test technical manuals typically present only one of the major classes of possible structural models of cognitive ability test batteries.  Interestingly, many school psychology scholars who conduct and report independent structural analysis of a test battery also do something similar…they often only present one form of structural analysis—-namely, bifactor g analyses.  

        In McGrew et al. (2023) we recommended future cognitive ability test technical manuals embrace a more ecumenical multiple method approach and include, when possible, most all major classes of factor analysis models, as well as PNA. A multiple-methods research approach in test manuals (and journal publications by independent researchers) can better inform users of the strengths and limitations of IQ test interpretations based on whatever conceptualization of psychometric general intelligence (including models with no such construct) underlies each type of dimensional analysis. Leaving PNA methods aside for now, the figure below presents the four major families of traditional CHC theoretical structural models.  These figures are conceptual and are not intended to represent all nuances of factor models. 

(Click on image for a larger image to view)

         Briefly, the four major families of traditional “common cause” CHC CFA structural models (Carroll, 2003; McGrew et al., 2023) vary primarily in the specification (or lack thereof) of a psychometric g factor. The different families of CHC models are conceptually represented in the figure above. In these conceptual representations the rectangles represent individual (sub)tests, the circles latent ability factors at different levels of breadth or generality (stratum levels as per Carroll, 1993), the path arrows the direction of influence (the effect) of the latent CHC ability factors on the tests or lower-order factors, and the single double headed arrow all possible correlations between all CHC broad CHC factors (in the Horn no-g model in panel D).  
        The classic hierarchical g model “places a psychometric g stratum III ability at the apex over multiple broad stratum II CHC abilities” (McGrew et al., 2023, p. 2). This model is most often associated with Carroll (1993; 2003) and is called (in panel A in the above figure) the Carroll hierarchical g broad CHC model. In this model the shared variance of subsets of moderately to highly correlated tests are first specified as 10 CHC broad ability factors (i.e., the measurement model; Gf, Gc, Gv, etc.). Next the covariances (latent factor correlations) among the broad CHC factors are specified as being the direct result of a higher-order psychometric g factor (i.e., the structural model). 

        A sub-model under the Carroll hierarchical g broad CHC model includes three levels of factors—several first-order narrow (stratum I) factors, 10 second-order broad (stratum II) CHC factors, and the psychometric g factor (stratum III). This is called the Carroll hierarchical g broad+narrow CHC model in panel B in the figure above. In the above example, two first-order narrow CHC factors (auditory short-term storage-Wa; and auditory working memory capacity-Wc, which, in simple terms, is a factor defining auditory short-term memory tasks that also include heavy attentional control-based (AC as per Schneider & McGrew, 2018) active manipulation of stimuli—the essence of Gwm or working memory).  For illustrative purposes, a narrow naming facility (NA) first-order factor, which has higher-order effects or influences from broad Gs and Gr is specified for evaluation.  Wouldn’t you like to see the results of this hierarchical broad+narrow CHC model?  Well……..stay tunned for the forthcoming WJ V technical manual (Q1 2025; LaForte, Dailey, & McGrew, 2025, in preparation) and your dream will come true.

        The third model is the Horn no-g model (McGrew, et al., 2023).  John Horn long argued that psychometric g was nothing more than a statistical abstraction or artifact (Horn, 1998; Horn & Noll, 1997; McArdle, 2007; McArdle & Hofner, 2014; Ortiz, 2015) and did not represent a brain or biologically based real cognitive ability. This is represented by the Horn no-g broad CHC model in panel D. The Horn no-g broad CHC model is like the Carroll hierarchical g broad CHC model, but the 10 broad CHC factor intercorrelations are retained instead of specifying a higher- or second-order psychometric g factor. In other words, the measurement models are the same but the structural models are different. In some respects the Horn no-g broad CHC model is like contemporary no-g psychometric network analysis models (see McGrew, 2023) that eschew the notion of a higher-order latent psychometric g factor to explain the positive definite correlation variance between individual tests (or first-order latent factors in the case of the Horn no-g model) in an intelligence battery (Burgoyne et al. 2022; Conway &Kovacs, 2015; Euler et al., 2023; Fried, 2020; Kan et al. 2019; Kievit et al. 2016; Kovacs & Conway, 2016, 2019; McGrew, 2023; McGrew et al., 2023; Protzko & Colom 2021a, 2021b, van der Maas et al. 2006, 2014, 2019).  Over the past decade I’ve become more aligned with no-g psychometric network CHC models (e.g, process overlap theory or POT) or Horn’s no-g CHC model, and have, tongue-in-check, referred to the elusive psychometric g ability (not the psychometric g factor)  as the “Loch Ness Monster of Psychology” (McGrew, 2021, 2022).

        Three of these common cause CHC structural models (viz., Carroll hierarchical g broad CHC model, Carroll hierarchical g broad+narrow CHC, and Horn no-g broad CHC), as well as Dr. Hudson Golino and colleagues hierarchical exploratory graph analysis psychometric network analysis models (that topic is saved for another day), are to be presented in the structural analysis section of the forthcoming WJ V technical manual validity chapter.  Stay tunned for some interesting analysis and interpretations in the “must read” WJ V technical manual. Yes….assessment professionals, a well written and thourough technical manual can be your BFF!

        Finally, the fourth family of models, which McGrew et al. (2023) called g-centric models, are commonly known as bifactor g models. In the bifactor g broad CHC model (panel C in figure) the variance associated with a dominant psychometric g factor is first extracted from all individual tests. The residual (remaining) variance is modeled as 10 uncorrelated (orthogonal) CHC broad factors. The bifactor g model was excluded from the WJ V structural analysis. Why…..after I (McGrew et al., 2023) recommended that all four classes of traditional CHC structural analysis models should be presented in a test batteries technical manual????
        Because…the complexity involved in specifying and evaluating bi-factor g models with 60 cognitive and achievement tests was found to be extremely complex and fraught with statistical convergence issues.  Trust me…I tried hard and long to run bifactor g models for the WJ V norm data.  It was possible to run bifactor g models separately on the cognitive and achievement sets of WJ V tests, but that does not allow for the direct comparison to the other three structural models that utilized all 60 cognitive and achievement tests in single CFA models.  Instead, at of the time the WJ V technical manual analyses were being completed and are now being summarized, the Riverside Insights (RI) internal psychometric research team was tackling the complex issues involved in completing WJ V bifactor g models, first in the separate sets of cognitive and achievement tests.  Stay tunned for future professional conference paper presentations, white papers, or journal article submissions by the RI research team.

        Furthermore, the decision to not include bifactor g models does not suggest that the evaluation of WJ V bifactor g-centric CHC models is not important. As noted by Reynolds and Keith (2017), “bifactor models may serve as a useful mathematical convenience for partitioning variance in test scores” (p. 45; emphasis added). The bifactor g model pre-ordains “that the statistically significant lion’s share of IQ battery test variance must be of the form of a dominant psychometric g factor (Decker et al., 2021)” (McGrew, et al., 2023, p. 3). Of the four families of CHC structural models, the bifactor g model is the conceptual and statistical model that supports the importance of general intelligence (psychometric g) and the preeminence of the full-scale or global IQ score over broad CHC test scores (e.g., see Dobrowski et al., 2021; Farmer et al., 2021a, 2021b; McGrew et al., 2023)—a theoretical position inconsistent with the position of the WJ V senior author (yours truly) and with Dr. Richard Woodcock’s legacy (see additional footnote comments at the end). It is important to note that there is a growing body of research that has questioned the preference for bifactor g cognitive models based only on statistical fit indices, as structural model fit statistics frequently are biased in favor of bifactor solutions. Per Bonifay et al. (2017),“the superior performance of the bifactor model may be a symptom of ‘overfitting’—that is, modeling not only the important trends in data but also capturing unwanted noise” p. 184–185). For more on this, see Decker (2021), Dueber and Toland (2021), Eid et al., (2018), Greene et al. (2022), and Murray and Johnson(2013). See Dombroski et al. (2020) for a defense of some of the bifactor g criticisms.

        Recognizing the wisdom of Box’s (1976) well known axiom that “all models are wrong, but some are useful” the WJ V technical manual authors (LaForte, Dailey, McGrew, 2025, in preparation) encourage independent researchers to use the WJ V norm data to evaluate and compare bifactor g CHC models with the models presented in forthcoming WJ V technical, as well as  alternative models (e.g., PASS, process overlap theory, Cattell’s triadic Gf-Gc theory, etc.) suggested in the technical manual.

Footnote:  Woodcock’s original (and enduring) position (Woodcock, 1978, 1997, 2002) regarding the validity and purpose of a composite IQ-type g score is at odds with the bifactor g CHC model. With the publication of the original WJ battery, Woodcock (1978) acknowledged the pragmatic predictive value of statistically partitioning cognitive ability test score variance into a single psychometric g factor, with the manifest total IQ score serving as a proxy for psychometric g. Woodcock stated “it is frequently convenient to use some single index of cognitive ability that will predict the quality of cognitive behavior, on the average, across a wide variety of real-life situations. This is the [pragmatic] rationale for using a single score from a broad-based test of intelligence” (p.126). However, Woodcock further stated that “one of the most common misconceptions about the nature of cognitive ability (particularly in discussions characterized by such labels as ‘IQ’ and ‘intelligence’) is that it is a single quality or trait held in varying degrees by individuals, something like [mental] height” (p. 126). In several publications Woodcock’s position regarding the importance of an overall general intelligence or IQ score was clear—“The primary purpose for cognitive testing should be to find out more about the problem, not to obtain an IQ” (Woodcock, 2002, p.6; also see Woodcock, 1997, p. 235). Two of the primary WJ III, WJ IV, and WJ V authors have conducted research or published articles (see Mather & Schneider, 2023; McGrew, 2023; McGrew et al., 2023) consistent with Woodcock’s position and have advocated for a Horn no-g or emergent property no-g CHC network model. Additionally, based on the failure to identify a brain-based biological g (i.e., neuro-g; Haier et al., 2024) in well over a century of research since Spearman first proposed g in the early 1900’s, McGrew (2020, 2021) has suggested that g may be the “Loch Ness Monster of psychology.” This does not imply that psychometric g is unrelated to combinations of different neurocognitive mechanisms, such as brain-wide neural efficiency and the ability of the whole-brain network, which is comprised of various brain subnetworks and connections via white matter tracts, to efficiently adaptively reconfigure the global network in response to changing cognitive demands (see Ng et al., 2024 for recent compelling research linking psychometric g to multiple brain network mechanisms and various contemporary neurocognitive theories of intelligence; NOTE…click link to download PDF of article and read sufficiently to impress your psychologist friends!!!!).

McGrew on #IQ scores: In what ways are a car engine, a starling bird #murmuration, and #g (general #intelligence) alike..how are they the same?

Kevin McGrew on IQ scores, borrowing from Detterman (2016) and McGrew et al., (2023)

“General intelligence (represented by a composite IQ score or the factor-analysis derived psychometirc g factor) is a fallible summary statistical (numerical) index of the efficiency of a complex system of dynamically interacting multiple brain networks.  Like the emergent statistical index of horsepower of a car engine, which does not represent a “thing” (a mechanism) in the engine, it reflects the current estimated efficiency of the processing of multiple interacting cognitive abilities and brain networks. It should not be interpreted as being the result of a single brain-based entity or mystical mental energy, as fixed, or reflecting biological/genetic destiny.  The manifest expression of this statistical emergent property index is also influenced by other non-cognitive (conative) (click for relevant article) traits and temporary states of the individual and current environmental variables” (K. McGrew, 11-07-24)

Question.  In what ways are a car engine, a starling bird murmuration, and general intelligence alike..how are they the same?  See slides and comments below for answer

 

(A starling bird murmuration)

Double click on larger more readable images 

More on the conflation of #psychometric #g (general #intelligence): Is g the Loch Ness Monster of psycholgy?

From McGrew et al. (2023) article (click here for prior post and access to the article in Journal of Intelligence.)”  Click here for a series of slides regarding the theoretical and psychometric conflation of g.

The Problem of Conflating Theoretical and Psychometric g

“Contributing to the conflicting g-centric and mixed-g positions (regarding the interpretive value of broad CHC scores) is the largely unrecognized common practice of conflating theoretical and psychometric g. Psychometric g is the statistical extraction of a latent factor (via factor analysis) that accounts for the largest single source of common variance in a collection of cognitive abilities tests. It is an emergent property statistical index. Theoretical g refers to the underlying biological brain-based mechanism(s) that produce psychometric g. The global composite score from IQ test batteries is considered the best manifest proxy for psychometric g. The conflation of psychometric and theoretical g in IQ battery structural research ignores a simple fact—“general intelligence is not the primary fact of mainstream intelligence research; the primary fact is the positive manifold….general intelligence is but one interpretation of that primary fact” (Protzko and Colom 2021a, p. 2; italic emphasis added). As described later, contemporary intelligence and cognitive psychology research has provided reasonable and respected theories (e.g., dynamic mutualism; process overlap theory; wired cognition; attentional control), robust methods (psychometric network analysis), and supporting research (Burgoyne et al. 2022; Conway and Kovacs 2015; Kan et al. 2019; Kievit et al. 2016; Kovacs and Conway 2016, 2019; van der Maas et al. 2006, 2014, 2019) that accounts for the positive manifold of IQ test correlations in the absence of an underlying latent causal theoretical or psychometric gconstruct.” (p.3; bold font emphasis added).