Monday, June 17, 2013

Errror in Dr. James Flynn's (2009) WAIS-IV norming data: Quest blog post by Dr. Dale Watson

This is a guest blog post by Dr. Dale Watson.  The opinions expressed do not necessarily reflect the official position of the ICDP blog or the blogmaster.  However, it is of interest to note that the error Dr. James Flynn (2009) made in reporting the WAIS-IV norming date (here reported by Dr. Dale Watson) is true, and was also in a published review that I received a few days after I received Dr. Watson's guest post.  This second verification source (Kaufman, Dillon, & Kirsch, 2013) will be the subject of my next post.

Dr. Dale Watson's guest post 

In an article entitled, The WAIS-III and WAIS-IV: Daubert motions favor the certainly false over the approximately true, Dr. James Flynn analyzed data from a number of IQ tests, including the WAIS-R, WAIS-III, and WAIS-IV to estimate the rate of the “Flynn Effect” on the Wechsler scales in the U.S. over time.[i] He concluded, as have others, that in order to account for the obsolescence of aging IQ test norms, a “Flynn Effect” adjustment of 0.30 points per year from the date of a tests norming should be applied to the obtained IQ test scores (Flynn, 2009; Fletcher et al., 2010). For example, if the WAIS-III (normed in 1995) was administered to an individual in 2005, the obtained IQ should be downwardly adjusted by 0.30 x 10 or 3.0 points. Thus, an obtained IQ score of 72 would result in a Flynn-adjusted score of 69. Such adjustments have been recommended for use in Atkins evaluations (Flynn, 2009; Gresham & Reschly, 2011; cf Hagen et al., 2010).[ii]

Flynn compared the IQ scores obtained on the WAIS-III and the WAIS-IV in a sample of 240 examinees reported in the Technical and Interpretive Manual for the WAIS-IV (2008).[iii] The Technical Manual reported that the mean IQs differed by 2.9 points with the sample mean for the WAIS-IV being 100 and for the WAIS-III 102.9 (Wechsler, 2008, p. 75). However, because these IQ scores were calculated using different combinations of subtests, Flynn re-calculated the IQ scores utilizing the same combination of 11 subtest scores used on the WAIS-III to calculate the IQs. Flynn (2009) noted, “The list of subtests used to compute Full Scale IQ had not only changed, but had dropped from 11 to 10. But, once again, they gave the comparison group all 11 of the old WAIS-III subtests, and once again that was fortunate because it meant that the true obsolescence of the WAIS-III could be measured. I calculated the total standard score the group got on the same 11 WAIS-III and WAIS-IV subtests. Using the totals and the WAIS-III conversion table, I calculated Full Scale IQs for the two tests” (p. 102). 

In examining Flynn’s Table 2, it appears that these calculations included scores for the Picture Arrangement subtest for both the WAIS-III and WAIS-IV. However, the Picture Arrangement subtest is not included in the WAIS-IV so it is quite unclear how this calculation was performed. Moreover, there is a footnote to this table indicating that the “WAIS-IV estimate is eccentric in carrying over WISC-III subtests (and scoring vs. the WAIS-III tables)…” but the meaning of this statement is also uncertain. In addition, substitution of the Symbol Search subtest for Picture Arrangement appears to yield very similar results.

In any case, the point of this note is not to recalculate Flynn’s estimates but rather to point out what appears to be a discrepancy between WAIS-IV norming date provided by Flynn and that found in the Technical and Interpretive Manual for the WAIS-IV. Flynn indicated that the WAIS-IV was normed in 2006 (Table 1) whereas the Manual reported, “The WAIS-IV normative data was established using a sample collected from March 2007 to April 2008.” [iv] If we use 2007 as the mid-point norming date, the time between the norming of the WAIS-III and WAIS-IV is 12 years and not 11 as provided by Flynn. Using the Flynn 2006 date resulted in a calculated Flynn Effect between the WAIS-III and WAIS-IV of 0.306 points per year (+3.37 / 11 years). Using the norming date provided in the manual resulted in a calculated score of 0.281 points per year (+3.37 / 12 years). It is understood that this discrepancy of just 0.025 points is of little practical significance but it should be noted nonetheless. Moreover, the metaphorical splitting of hairs is not uncommon when discussing the Flynn Effect. Hagan et al. (2010) asserted, “Decades of FE research and testimony… depict the amount of this shift as a moving target. For example, Flynn (1998) once identified the annual shift as 0.25 rather than 0.30, but later testified in Ex Parte Eric Dewayne Cathey (2010) that 0.29 would be appropriate. Schalock et al. (2010) have called for an annual adjustment of 0.33” pp. 1-2.[v] Flynn has acknowledged that the results reported in his report are estimates for the Wechsler scales, writing, “It is quite possible that the rate of gain on Wechsler tests is 0.275 or 0.325 points per year” (Flynn, 2009, p. 104). The recalculation noted here is consistent with this judgment. Further, the weight of the available evidence, including that of a recent meta-analysis, continues to support the Flynn Effect adjustment of 0.3 points per year.[vi]

[i] Flynn, J. R. (2009). The WAIS-III and WAIS-IV: Daubert motions favor the certainly false over the approximately true. Applied Neuropsychology, 16(2), 98-104. doi: 10.1080/09084280902864360
[ii] Gresham, F. M., & Reschly, D. J. (2011). Standard of practice and Flynn Effect testimony in death penalty cases. Intellectual and Developmental Disabilities, 49(3), 131-140. doi: 10.1352/1934-9556-49.3.131
[iii] Wechsler, D. (2008). Wechsler Adult Intelligence Scale: Technical and interpretive manual (4th ed.). San Antonio, TX: Pearson.
[iv] Id., p. 22.
[v] Hagan, L. D., Drogin, E. Y., & Guilmette, T. J. (2010). IQ scores should not be adjusted for the Flynn Effect in capital punishment cases. Journal of Psychoeducational Assessment, 28(5), 474-476. doi: 10.1177/0734282910373343
[vi] Fletcher, J. M., Stuebing, K. K., & Hughes, L. C. (2010). IQ scores should be corrected for the Flynn Effect in high-stakes decisions. Journal of Psychoeducational Assessment, 28(5), 469-473. doi: 10.1177/0734282910373341

No comments: