IQ (ISIR) Scholar Spotlight: David Lohman--CogAT, NNAT, Ravens research

David Lohman presented at the 2008 ISIR conference. I've been a big fan of Lohman's work as much of it has direct application to the work of practicing school and educational psychologists. Lohman was a student of the late Richard Snow, whose work has had a significant work on my thoughts regarding non-cognitive factors important for school learning (see prior post today). Lohman is an author of the group CogAt (click here to see prior post re: study with WJ III). Aside from being an excellent applied psychometrician, Lohman has written papers on a wide variety of topics in educational psychology and intelligence. He is also very generous in making his various publications available for download at his web page.

At this conference he presented a paper comparing scores and norm characteristics from the CogAT, NNAT, and Ravens. The name of the paper and abstract (italics added by me) is below. The focus was on the use of nonverbal measures of intelligence in the identifcation of gifted students. The results presented were a bit disconcerting regarding possible technical issues with the norms of two of the tests---I've featured Lowman's conclusion in the abstract below. Lohman's research raises significant issues re: the accuracy of gifted identification via the NNAT and Ravens. Of course, and appropriately so, Lohman made it clear that his findings and research needed to recognize his potential conflict of interest as author of the CogAT, a direct competitor to the other tests, esp. the NNAT. It is refreshing to see such scholarly integrity in person.

• Ethnic Differences on Fluid Reasoning Tests: Is the NNAT the Panacea? David F. Lohman, University of Iowa

• Abstract: Nonverbal, figural reasoning tests such as the Raven Progressive Matrices are often used as markers for Gf in research on intelligence. These tests are also widely used in schools to help identify academically gifted students. The Naglieri Nonverbal Ability Test (NNAT) has been particularly popular following a recent report that in a large, representative sample of U.S. school children, it was found to identify equal proportions of high-scoring White, Black, and Hispanic students. Although questions have been raised about the integrity of the data analyses used in this study, the author of the study continues to defend it as the most important research yet conducted with the NNAT. The goal of this investigation was to compare the NNAT with two other nonverbal assessments: the Raven Progressive Matrices and the Nonverbal Battery of the Cognitive Abilities Test (CogAT). All three tests were administered by trained examiners in counterbalanced order to 1,200 children in grades K to 6 in an ethnically diverse school district. Results showed provided no support for the assertion that the NNAT reduced ethnic differences – either at the mean or at the tails of the distribution. Rather, ethnic differences were actually somewhat larger on the NNAT than on the other two tests. Furthermore, it was discovered that the variance of basic normative score on the NNAT (M = 100, SD = 15) substantially exceeded the reported value of 15 at all but one test level. Re-analyses of the standardization data for the NNAT confirmed this finding. A similar normative score on the Raven (computed from the most recent U.S. national norms) was 10 points too lenient. Consequences of invalid or outdated normative scores for research and practice are discussed.

If you are interested in learning more about this issue, you should check out his full length publication in Gifted Child Quarterly (which Lohman makes available from his web page). You can view a copy by clicking here. The reference citation is: Lohman, D. F., Korb, K., & Lakin, J. (2008). Identifying academically gifted English language learners using nonverbal tests: A comparison of the Raven, NNAT, and CogAT. Gifted Child Quarterly, 52, 275-296. Apparently this paper received the "Research Paper of the Year Award" from the National Association of Gifted Children.

The bottom line take-away: Buyer beware. Educators need to do their due diligence when evaluating and comparing psychometric instruments (group or individual) that impact important educational decisions regarding children.

[Conflict of interest disclosure - I'm a coauthor of the WJ III mentioned above, which is an instrument that competes with a different individually administered cognitive battery of the author of the NNAT, which was a focus of Lohman's presentation and paper].

IQ Research bytes #2: WJ III, CoGAT, RIAS

Check out the following research articles re: commonly used group (CoGAT) and individually administered (WJ III: RIAS) intelligence tests.

1. Although not that recent (2003), I just discovered a manuscript by David Lohman, coauthor of the CoGAT (group IQ test), where he presents the results of an investigation between the relations of the group-based CoGAT scores and scores from the individually administered Woodcock-Johnson Tests of Cognitive Ability--Third Edition (WJ III ; conflict of interest disclosure - I'm a co-author of the WJ III, and both the CoGAT and WJ III are published by Riverside Publishing). The title, abstract, and a link to a pdf copy of manuscript is below.

• Lohman, D. (2003, March). The Woodcock-Johnson III and the Cognitive Abilities Test (Form 6): A Concurrent Validity Study (click here to view)

• Abstract - This study investigated the concurrent validity of the Woodcock-Johnson III (WJ-III; Woodcock, McGrew, & Mather, 2001) and Form 6 of the Cognitive Abilities Test (CogAT; Lohman & Hagen, 2001). A total of 178 students in grades 2, 5, and 9 were administered 13 tests from the WJ-III and the appropriate level of the CogAT. Interbattery confirmatory factor analyses showed that the general factors on the two batteries correlated r = .82. Correlations between broad-group clusters on the WJ-III and battery-level scores on the CogAT generally supported the construct interpretations of each, but also suggested important differences in the abilities measured by both batteries.

2. In another WJ III related study (see conflict of interest disclosure statement above), Craig Firsby and Steven Osterlind present a follow-up to an earlier descriptive analysis of the WJ III Test Session Observation Checklist norm data. Their most recent analysis focused on potential differences in examiner ratings for self-identified Hispanic's in the WJ III standardization sample. The abstract and article speak for themselves. See information below.

• Frisby, C. & Osterlind, C. (2007). Hispanic Test-Session Behavior on the Woodcock Johnson Psychoeducational Battery– Third Edition. Journal of Psyhoeducational Assessment, 25 (3), 257-270. (click here to view)
  

• Abstract--This study examined potential differential examiner ratings for a large sample of self-identified Hispanics on the Woodcock Johnson Psychoeducational Battery–Third Edition (WJ-III) Test Session Observation Checklist (TSOC). Both between-group (Hispanics vs. non-Hispanics) and within-group analyses (Hispanics disaggregated by first spoken language, language spoken in the home, and mother’s highest educational level) were conducted. Four research hypotheses were tested through 44 analyses. Most comparisons were not statistically significant, and across- and within-group differences had minimal influence in analyses that were statistically significant. The authors conclude that there is no compelling evidence of substantial systematic differences in examiner ratings of Hispanics’ test-session behaviors on the WJ-III.

3. Finally, Nelson et al. (2007) report an independent investigation of the internal (structural) validity of the Reynolds Inellectual Assessment Scales (RIAS). The article information is below. The long story short - Nelson et al. conclude that their factor extraction methods were more rigorous and appropriate than the exploratory (EFA) and confirmatory factor analysis (CFA) reported by the RIAS authors. More importantly, Nelson et al. conclude that the RIAS should only be interpreted as a single-factor measure of g (general intelligence). Their analysis did not support a three-factor structure presented by the RIAS authors. As we all know, the "my factor analysis methods are better than your factor analysis methods" has been a common battle cry in the factor analysis literature for decades. Regardless of this caveat, I do have a couple of comments based on my quick skim of the article.

First, I concur with Nelson et al. that the EFA methods should include oblique correlated factors...and not orthogonal (uncorrelated) factors as reported by the test authors. Cognitive abilities are correlated and this should be modeled in the analyses. Second, in defense of the RIAS authors, the Nelson et al. sample consists of referred subjects, and not a nationally representative sample. Although referral samples are much better for investigating the factor structure of instruments than clinical samples (e.g., LD), it is possible that the Nelson et al. results are influenced by characteristics of the non-normal sample. Third, I believe that the continued criticism of the over-use of CFA methods in test development is misdirected. If one reads the writings of one of the fathers of SEM-based methods (Joreskog), SEM methods (in this case CFA) can be used not only for model confirmation, but also may play a critical role in "model development and generation." Those of us who develop measures of cognitive abilities often use CFA as a model generation/building tool as per Joreskog . And..this is OK.

Fourth, since all test manuals must report test intercorrelation matrices as per the Joint Test Standards, it is possible for independent researchers to import the published correlation matrices into standard stat software and factor analyze the orignal norm-based correlations. Given this possibility, it would have been nice if Nelson et al. would have included, as a companion to their factor analysis in the referral sample, a similar analysis (using the same logic and methods) of the same correlation matrices the RIAS authors used. If similar results were found vis-a-vis the application of their methods in both the norm and referred samples, it would make their arguement stronger.

Cecil.....I know you catch my blog every now and then. If you would like to prepare a response to the Nelson et al. article, I'd be happy to post it as a guest blog post.

• Nelson, J., Canivez, G., Lindstrom, W. & Hatt, C. (2007) Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439-456 (click here to view)

• Abstract--The factor structure of the Reynolds Intellectual Assessment Scales (RIAS; [Reynolds, C.R., & Kamphaus, R.W. (2003). Reynolds Intellectual Assessment Scales. Lutz, FL: Psychological Assessment Resources, Inc.]) was investigated with a large (N=1163) independent sample of referred students (ages 6–18). More rigorous factor extraction criteria (viz., Horn's parallel analysis (HPA); [Horn, J.L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185.], and Minimum Average Partial (MAP) analysis; [Velicer, W.F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321–327.]), in addition to those used in RIAS development, were investigated. Exploratory factor analyses using both orthogonal and oblique rotations and higher-order exploratory factor analyses using the Schmid and Leiman [Schmid, J., and Leiman, J.M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61.] procedure were conducted. All factor extraction criteria indicated extraction of only one factor. Oblique rotations resulted in different results than orthogonal rotations, and higher-order factor analysis indicated the largest amount of variance was accounted for by the general intelligence factor. The proposed three-factor solution was not supported. Implications for the use of the RIAS with similarly referred students are discussed.

Technorati Tags: psychology, educational psychology, school psychology, neuropsychology, RIA, WJ III, CoGAT, intelligence, IQ tests, IQ scores, factor analysis, statistics, hispanic, bias, g, general intelligence

Powered by ScribeFire.