#ITC webinar: #Validity and #Fairness Considerations in Innovative #Digital #Assessments."

When and how can you access this webinar?

The webinar is live online on Thursday, April 3rd, at 10 a.m. (New York, USA time, UTC-4). 

Register for free here!

 

Research Brief 7-4-10: Investigation of prediction bias in WISC-IV

Excellent article that shows how one form of empirically-defined test bias (differential prediction) should be pursued.

Konold, T. R., & Canivez, G. L. (2010). Differential Relationships Between WISC-IV and WIAT-II Scales: An Evaluation of Potentially Moderating Child Demographics. Educational and Psychological Measurement, 70(4), 613-627.

Considerable debate exists regarding the accuracy of intelligence tests with members of different groups. This study investigated differential predictive validity of the Wechsler Intelligence Scale for Children—Fourth Edition. Participants from the WISC-IV—WIAT-II standardization linking sample (N = 550) ranged in age from 6 through 16 years (M = 11.6, SD  = 3.2) and varied by the demographic variables of gender, race/ethnicity (Caucasian, African American, and Hispanic), and parent education level (8-11, 12, 13-15, and 16 years). Full Scale IQ and General Ability Index scores from the WISC-IV were used to predict scores on Mathematics, Oral Language, Reading, Written Language, and the total composite on the Wechsler Individual Achievement Test—Second Edition. Differences in prediction were evaluated between demographic subgroups via Potthoff’s technique. Of the 30 simultaneous tests, 25 revealed no statistically significant between group differences. The remaining statistically significant differences were found to have little practical or clinical influence when effect size estimates were considered. Results are discussed in the context of other ability measures that were previously investigated for differential validity as well as educational implications for clinicians.

Technorati Tags: psychology, forensic psychology, forensic psychiatry, neuropsychology, intelligence, school psychology, psychometrics, educational psychology, IQ, IQ tests, IQ scores, adaptive behavior, adaptive functioning, intellectual disability, mental retardation, MR, ID, criminal psychology, criminal defense, criminal justice, ABA, American Bar Association, Atkins cases, death penalty, capital punishment, AAIDD, Atkins MR/ID listserv, ICDP blog, bias, test bias, differential prediction bias, race, WISC-IV

Research brief 7-26-10: CHC theory and measures (WJ III) found invariant (no psychometric bias) across blacks and whites

Kane, H. D., & Oakland, T. D. (2010). Group Differences in Cognitive Ability: A CHC Theory Framework. Mankind Quarterly, 50(4), 318-331.

Abstract

The Cattell-Horn-Carroll (CHC) model of cognitive ability as represented in the Woodcock-Johnson Tests of Cognitive Ability-III (WJ-III) was examined for Black and White adults matched on various demographic variables. Although Whites performed higher than Blacks (i.e., race differences were found in test scores and accompanying factor means), the results of multisample confirmatory factor analyses found that the same constructs are measured in different groups. Therefore results are directly comparable, and in this sense measured differences can be interpreted as “real” differences on the dimensions that the test is meant to measure.

Part of authors conclusions:

With respect to the primary purpose of this study, although White-Black differences in cognitive ability are affirmed in favor of Whites, these analyses reveal no source of psychometric bias (i.e., differences in loadings, test intercepts, and error variance). Constructs are represented adequately and without undo influence of error. The structural fidelity of the WJ-III factor model is psychometrically sound, making it a suitable instrument for psychologists when estimating general and broad cognitive abilities for individuals and groups. The reported indices of fit (e.g., TLI, GFI, and RMSEA) suggest that the threestratum CHC model fits the WJ-III data fairly well and provides evidence of construct validity. This finding substantiates a growing body of research literature that upholds the WJ-III as a strong representation of CHC theory (e.g., Edwards & Oakland, 2006; McGrew & Woodcock, 2001). Further, the data support Carroll’s (1993) belief that the CHC theory is essentially invariant across racial-ethnic groups. Notably, the group differences in test performance are smaller than in most other studies (e.g., Osborne & McGurk, 1982). This particular result is likely due to the samples being matched by parental education and occupational status. In the US, this kind of control is expected to remove approximately one third of the Black-White difference that may be expected in demographically representative samples (Jensen, 1998;
Lynn, 1998).

Technorati Tags: Psychology, school psychology, developmental psychology, educational psychology, forensic psychology, neuropsychology, special education, intelligence, cognitive abilities, cognition, intelligence theories, CHC theory, CHC, Cattell-Horn-Carroll, intelligence, cognition, IQ, IQ tests, Gf, Gc, Gv, Ga, Gsm, Glr, Gs, Gq, Grw, WJ-R, WJ III, Woodcock-Johnson, WJ III NU, race and IQ, racial differences, psychometric bias, CFA, structural invariance

Australian Psychologist special issue on culture, language, and cognitive assessment

The March 2009 issue of the Australian Psychologist (44-1) is a special issued dealing with cultural and language issues in cognitive assessment in Australia. The opening editorial by Stolk (2009) is "Approaches to the influence of culture and language on cognitive assessment instruments: The Australian context"

Technorati Tags: psychology, school psychology, educational psychology, neuropsychology, Australia, Australian Psychologist, IQ, IQ test, IQ score, intelligence testing, culture, language

Black-white IQ gap: Flynn and Murray present

I just learned of a special conference presentation (next week in DC) sponsored by the American Enterprise Institute (AEI) that will feature James "Flynn Effect" Flynn and Charles "Bell Curve" Murray presenting and discussing new research on the "Black-White IQ Gap: Is it Closing? Will it Ever Go Away?" Follow the links for additional information. The program abstract is presented below:

• For decades, the difference in the test scores of blacks and whites on the SAT, National Assessment of Educational Progress test, Armed Forces Qualification Test, and traditional IQ tests has been a vexed issue for American educational policy. Two of the leading scholars of this controversial topic, James R. Flynn of the University of Otago (New Zealand) and Charles Murray of AEI, will debate the causes of the difference, its implications, and recent trends. New studies of the subject by Professor Flynn and by Mr. Murray will be available for distribution at the session.

Technorati Tags: psychology, educational psychology, school psychology, education, neuropsychology, intelligence, IQ, IQ scores, IQ tests, cognition, cognitive, race differences, race, black-white, IQ gap, AEI, Flynn Effect, Bell Curve, James Flynn, Charles Murray

powered by performancing firefox

New black white IQ (g) comparison study

Bias in intelligence testing, particularly black-white IQ differences, has been a contentious area of study in the field of intelligence. A new study by Edwards and Oakland (reference, abstract, select findings, and article link provided below) adds new information to this area of study.

Briefly, using the K-12 (school-age) standardization data from the Woodcock-Johnson--Third Edition (WJ III; conflict of interest disclosure: I'm a co-auther of the WJ III), Edwards and Oakland examined, across blacks and whites, (a) the similarity of the structure of g (general intelligence), (b) mean differences in general intelligence, and (c) the similarity in predictive validity (do the WJ III g-scores predict achievement similarily across both groups). Given my potential conflict of interest, I'm only going to report the abstract and select direct quotes from the article. Readers are encouraged to read and digest the complete article.

The one comment I will make is not specifically WJ III related, but is theory related. Consistent with Carroll's (1993) conclusion that the structure of cognitive abilities is largely the same (invariant) as a function of gender and race, Edwards and Oakland's findings indicate that the structure of g (general intelligence), when operationalized by seven different CHC ability indicators (Gf, Gc, Glr, Gsm, Gv, Ga, Gs), is similar across whites and blacks.

Edwards. O. & Oakland, T. (2006). Factorial Invariance of Woodcock-Johnson III Scores for African Americans and Caucasian. Journal of Psychoeducational Assessment, 24 (4), 358-366. (click here to view)

Abstract

• Bias in testing has been of interest to psychologists and other test users since the origin of testing. New or revised tests often are subject to analyses that help examine the degree of bias in reference to group membership based on gender, language use, and race/ethnicity. The pervasive use of intelligence test data when making critical and, at times, life-changing decisions warrants the need by test developers and test users to examine possible test bias on new and recently revised intelligence tests. This study investigates factorial invariance and criterionrelated validity of the Woodcock-Johnson III for African American and Caucasian American students. Data from this study suggest that although their mean scores differ, Woodcock-Johnson III scores have comparable meaning for both groups.

Select author conclusions from the article

• Results from factor analysis, SEM, congruence coefficients, correlations coefficients, and Fisher’s Z statistic are uniform in indicating the factor structure of the WJ III is consistent for African Americans and Caucasian Americans.

• The high congruence coefficient of .99 suggests the g factor structure is essentially identical for African Americans and Caucasian Americans. In addition, all fit indices are > .95, indicative of excellent fit and suggests covariant structural equivalence between the two groups. Although the mean IQs for the groups differ, the WJ III scores from the Cognitive Battery have comparable meaning for African American and Caucasian American students. Additionally, correlations between GIA and three achievement clusters and nine achievement subtests are similarly high and statistically significant for both groups.

• The collective findings from this and other studies using the WJ III provide some support for Carroll’s (1993) assertion that CHC theory, one that forms the theoretical basis for the WJ III, is essentially invariant across racial/ethnic groups.

• Thus, when using the WJ III Cognitive with African American and Caucasian American students, practitioners can be somewhat assured that possible score differences reflect differences in the underlying latent constructs rather than variations in the measurement operation itself (Watkins & Canivez, 2001).

Technorati Tags: psychology, educational psychology, school psychology, neuropsychology, WJ III, Woodcock Johnson, race, IQ scores, IQ score differences, g, general intelligence, bias, test bias, psychometrics, cognition, cognitive, Jack Carroll, genetics, race differences

Test fairness - simple definition

I continue to enjoy my current reading of "Knowing what students know"

The book provides many succinct and conrete explanations of various congitive and measurement concepts and, more importantly, is providing me a framework for integrating various developments I've read over the past decade in both cognitive science and measurement.

Today I ran across one nice little example....the books description of "test fairness" (p. 214)

• "Fairness in testing is defined many ways (see AERA et al., 1999; NRC, 199b), but at its core is the idea of comparable validity: a fair test is one that yields comparably valid inferences from person to person and group to group. An assessment task is considered biased if construct-irrelevant characteristics of the task result in different meanings for different subgroups."
  

Technorati Tags: psychology, psychometrics, measurement, statistics, fairness, test fairness.validity, educational psychology, neuropsychology, cognitive, cognition, assessment, education, test standards, validity

powered by performancing firefox