Applied #psychometrics 101: Strong programs of #constructvalidity—the #theory - #measurement framework with emphasis on #substantive & #structural validity - #WJIV #WJV #shoolpsychology #psychology

The validity of psychological tests “is an overall judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment” (Messick, 1995, p. 741).

The ability to draw valid inferences regarding theoretical constructs from observable or manifest measures (e.g., test or composite scores) is a function of the extent to which the underlying program of validity research attends to both the theoretical and measurement domains of the focal constructs (Bensen, 1998; Bensen & Hagtvet, 1996; Cronbach, 1971; Cronbach & Meehl, 1955; Loevinger, 1957; Messick, 1995; Nunnally, 1978). 

The theoretical—measurement domain framework that has driven the revisions of the WJ test batteries, particularly from the WJ-R to the forthcoming WJ V cognitive and achievement test batteries (Q1, 2025; COI disclosure: I am a coauthor of the current WJ IV and forthcoming WJ V), is represented in the figures below.  

The goal of this post is to provide visual-graphic (Gv) images that hopefully, if properly studied by the reader (and if I did a decent job), provide the basic concepts of what constitutes the substantive component (and to a more limited extent the structural component) of a strong program of construct validity—in particular, the theoretical-measurement domain mapping framework used in the WJ-R to the forthcoming WJ V. The external stage of construct validity is not highlighted in this current post.  The goal is for conceptual understanding…thus the absence of empirical data, etc.

For those who want written background information, the most succinct conceptual overview of a “strong program of construct validation” is Bensen (1998; click to download and read).  

Otherwise…sit back and enjoy the Gv presenation…where five images equal at least one or more chapter in a technical manual :). 

Be sure to click on each image to enlarge (and make readable)

This figure below was first published in a book on CHC theoretical (then known as Gf-Gc) interpretation of the Wechsler intelligence test batteries (Flanagan, McGrew, & Ortiz, 2000).

Yea…I know.  The following figure uses Gv as the sample cognitive ability construct domain and not Gf as in the prior figure.  I first crafted the figure below in 2005 and don’t have the time (nor attentional control focus bandwidth) to make a new version.  Consider the switch from the Gf domain (above) to Gv (below) as a test of your understanding of the material…and ability to generalize what you have learned.  And yes, I do see there is a spelling error (“on” for “one”)…but it is an old image file and I don’t have time to “clean it up” as noted above.  The primary new feature is the addition of the concept  of developing developmentally (difficulty) ordered sets of test items for the underling ability trait scales for manifest indicator tests C and D under the CHC theoretical narrow ability domain of spatial scanning, under the broad ability domain of Gv. This is where IRT (Rasch model) item scaling is involved.

The following figure is drawn from the WJ IV technical manual (McGrew, LaForte, Schrank, 2014) and illustrates the three-stage structural validity process used in the WJ IV.  The same process, with slightly different age groups and the addition of exploratory hierarchical psychometric network analysis (see exciting and ground-breaking work of Dr. Hudson Golino and colleagues) during stage 2A, will be presented in the WJ V technical manual (LaForte, Dailey & McGrew, Q1-2025).

CAI January 2025 WJ V McGrew Webinar: An overview of the WJ V Tests of Cognitive Abilities (A “look behind the curtain” of the WJ IV to WJ V ‘cognitive’ (COG+VTL batteries) revision.

Click to start brief (5 minute) overview of agenda for Dr. Kevin McGrew’s “An overview of the WJ V Tests of Cognitive Abilities”.  January 22, 2025 CAI webinar.

Registration for CAI webinars (mine included) - Click here.

Link to my repost the the CAI’s LinkedIn description and webinar info can be found here.

Gv-IM: Has the time finally come for Gv visual imagery tests?

In our 2018 CHC chapter  Joel Schneider and I made our long standing affection for visual imagery (Gv-IM per CHC taxonomy) very clear.  See text excerpt below. (double click on images to enlarge)

Today I ran across a potential free PDF book (via Research Gate) that may make IM tests viable.  There is hope.

The Model of Achievement Competence Motivation (MACM) Part B: An overview of the MACM model

The Model of Achievement Competence Motivation (MACM) is a series of slide modules.  By clicking on the link you can view the slides at SlideShare.  This is the second (Part B) in the series--An overview of the model.  There will be a total of five modules.  The modules will serve as supplemental materials to "The Model of Achievement Competence Motivation (MACM)--Standing on the shoulders of giants" (McGrew, in press, 2021 - in a forthcoming special issue on motivation in the Canadian Journal of School Psychology)

Click here for first of the series (Part A:  Introduction and Background)

Click here for prior "beyond IQ" labeled posts at this blog.

CHC taxonomy of human cognitive abilities v 2.5: A silent movie animationed presentation

 An oldie but goodie prior post.

 

Spatial ability (Gv) and math (Gq; Gf-RQ): A meta-analysis

Is Spatial Ability Related to Mathematical Ability: a Meta-analysis

Fang Xie & Li Zhang  & Xu Chen & Ziqiang Xin

Abstract

The relationship between spatial and mathematical ability is controversial. Thus, the current study conducted a meta-analysis of 73 studies, with 263 effect sizes to explore the relationship between spatial and mathematical ability. Furthermore, we explored potential factors that moderate this relationship. Results showed that the relationship between mathematical and spatial ability was not simply linear. Specifically, logical reasoning had a stronger association with spatial ability than numerical or arithmetic ability with spatial ability. Intrinsic-dynamic, intrinsic-static, extrinsic-dynamic, extrinsic-static spatial ability, and visual–spatial memory showed comparable associations with mathematical ability. The association between spatial and mathematical ability showed no differences between children, adolescents, and adults and no differences between typically developing individuals and individuals with developmental disabilities. The implications of these findings for theory and practice are discussed.

Keywords Spatial ability . Mathematical ability . Meta-analysis . robumeta package . Spatial training.

Implications for Theory and Practice

“Our study can shed light on our understanding of the relationship between spatial and mathematical abilities. The relationship between spatial and mathematical abilities is not simply linear. Our moderation analyses suggested that logical reasoning was more strongly associated with spatial ability than numerical and arithmetical ability. As such, when examin-ing the mechanism of the association between spatial and mathematical ability, each domain of mathematical ability should be separately examined. The current study has important educational implications. Although we did not prove the causal relationship between spatial and mathematical ability, our findings might provide some pedagogical suggestions about how to train spatial ability to improve children's mathematical abilities. Notably, a recent intervention study by Sorby et al. (2018) demonstrated the positive effect of spatial interventions on STEM-related skills, and several studies have shown that spatial training can improve mathematical achievement (Cheng and Mix 2014; Clements et al. 2011; Sorby and Baartmans 2000). Firstly, our findings shed light on what kind of spatial ability training should be chosen. The current study indicated that different domains of spatial ability are associated with mathemat-ical ability to a similar degree. Therefore, training in other domains of spatial ability, not just intrinsic-dynamic spatial abilities (Cheng and Mix 2014; Clements et al. 2011; Taylor and Hutton 2013), should be encouraged in educational practice. Further, our findings shed light on when to begin spatial ability training. This study showed that the close association between spatial and mathematical abilities exists in childhood and adolescence. Therefore, spatial training can be beneficial for both children and adolescents. For children, spatial training can be rooted in the real world to develop direct experience by using regular activities such as paper folding, paper cutting (Burte et al. 2017), and Lego construction (Nath and Szücs 2014). For adoles-cents, it is better to carry out spatial training through comprehensive courses involving theory and practice in a series of spatial skills (Miller and Halpern 2013; Patkin and Dayan 2013; Sorby et al. 2013).”

Educational Psychology Review

Beyond individual intelligence tests: Application of Cattell-Horn-Carroll Theory

A most excellent CHC theoretical and methodological article.

Click on image to enlarge

CHC taxonomy update: Large-scale Gv spatial navigation abilities proposed entry into taxonomy

Joel Schneider and I, in our new CHC chapter, recommend that large-spatial Gv navigation abilities be included in the CHC taxonomy. Evidence is strong. Measures are far and few between—a potential place for good dissertation research and instrument development, especially with computer-administered test technology.

Click images to enlarge.

CHC theory update: Emotional intelligence (Gei) proposed as newest CHC family member

Part of my FASP presentation next week.  See Joel Schneider and my new CHC chapter for info.

https://www.guilford.com/books/Contemporary-Intellectual-Assessment/Flanagan-McDonough/9781462535781

The new Cattell-Horn-Carroll (CHC) periodic table of cognitive elements: Build your own

The latest and greatest CHC periodic table of cognitive elements is now available!

I've been meaning to make this post for a long time.  Below are the front and back images of my official CHC periodic table of cognitive elements.  This revision reflects Dr. Joel Schneider and my CHC theory update chapter to be published this fall (I believe next month) in the latest CIA (contemporary intellectual assessment) book.

Some of you may be familiar with the fact that, for the prior version, I had the table professionally printed (one sheet...both sides used) and laminated.  I distributed these during some of my presentations and at conferences.  The response was always positive.  My original plan was to have a new batch professionally printed and laminated and to make them available for a nominal fee via PayPal.  However, I was stalled by the details of having to deal with different state sales taxes, accepting payments, packaging, and mailing.  Instead, I am now providing interested people with the actual JPEG images (click here to download).  You can print your own copy and laminate if you want.  Build your own CHC periodic table of cognitive elements!

A PDF copy of the latest definitions, abstracted from our chapter, can be found here.

Enjoy.

Click on images to enlarge.

Beyond IQ: Mining the “no-mans-land” between Intelligence and IQ: Journal of Intelligence special issue

I am pleased to see the Journal of Intelligence addressing the integration of non-cognitive variables (personality; self-beliefs; motivational constructs; often called the “no-mans land” between intelligence and personality— I believe this catchy phrase was first used by Stankov) with intellectual constructs to better understanding human performance. I have had a long-standing interest in such comprehensive models as reflected by my articulation of the Model of Academic Competence and Motivation (MACM) and repeated posting of “beyond IQ” information at my blogs.

Joel Schneider and I briefly touched in this topic in our soon to be published CHC intelligence theory update chapter. Below is the select text and some awesome figures crafted by Joel.

Our simplified conceptual structure of knowledge abilities is presented in Figure 3.10. At the center of overlapping knowledge domains is general knowledge—knowledge and skills considered important for any member of the population to know (e.g., literacy, numeracy, self-care, budgeting, civics, etiquette, and much more). The bulk of each knowledge domain is the province of specialists, but some portion is considered important for all members of society to know. Drawing inspiration from F. L. Schmidt (2011, 2014), we posit that interests and experience drive acquisition of domain-specific knowledge.

In Schmidt's model, individual differences in general knowledge are driven largely by individual differences in fluid intelligence and general interest in learning, also known as typical intellectual engagement (Goff & Ackerman, 1992). In contrast, individual differences in domain-specific knowledge are more driven by domain-specific in-terests, and also by the “tilt” of one's specific abilities (Coyle, Purcell, Snyder, & Richmond, 2014; Pässler, Beinicke, & Hell, 2015). In Figure 3.11, we present a simplified hypothetical synthesis of several ability models in which abilities, interests, and personality traits predict general and specific knowledge (Ackerman, 1996a, 1996b, 2000; Ackerman, Bowen, Beier, & Kanfer, 2001; Ackerman & Heggestad, 1997; Ackerman & Rolfhus, 1999; Fry & Hale, 1996; Goff & Ackerman, 1992; Kail, 2007; Kane et al., 2004; Rolfhus & Ackerman, 1999; Schmidt, 2011, 2014; Schneider et al., 2016; Schneider & Newman, 2015; Woodcock, 1993; Ziegler, Danay, Heene, Asendorpf, & Bühner, 2012).

Click on images to enlarge.







- Posted using BlogPress from my iPad

Using Gt distribution parameters to predict executive functions in AHDH: Study consistent with Schneider & McGrew 2018 CHC update chapter

Interesting article consistent with what Joel Schneider and I discussed in our latest CHC Intelligence theory update chapter. Click here for info.

Using inspection time and ex-Gaussian parameters of reaction time to predict executive functions in children with ADHD. Intelligence, 69 (2018) 186–194.

Hilary Galloway-Long, Cynthia Huang-Pollock


A B S T R A C T

Slower and more variable performance in speeded reaction time tasks is a prominent cognitive signature among children with Attention Deficit Hyperactivity Disorder (ADHD), and is often also negatively associated with executive functioning ability. In the current study, we utilize a visual inspection time task and an ex-Gaussian decomposition of the reaction time data from the same task to better understand which of several cognitive subprocesses (i.e., perceptual encoding, decision-making, or fine-motor output) may be responsible for these important relationships. Consistent with previous research, children with ADHD (n = 190; 68 girls) had longer/ slower SD and tau than non-ADHD peers (n = 76; 42 girls), but there were no group differences in inspection time, mu, or sigma. Smaller mu, greater sigma, longer tau, and slower inspection time together predicted worse performance on a latent executive function factor, but only tau partially mediated the relationship between ADHD symptomology and EF. These results suggest that the speed of information accumulation during the decision-making process may be an important mechanism that explains ADHD-related deficits in executive control.

Click image to enlarge.



Assessment Recommendations for Gt (from Schneider & McGrew, 2018)

To be published shortly in:




Tasks measuring Gt are not typically used in clinical settings (except perhaps in CPTs). With the increasing use of low-cost mobile computing devices (i.e., smartphones and iPads/other slate notebook computers), we predict that practical measures of Gt will soon be available for clinical use. Some potential clinical applications are already apparent. We present three examples.

Gregory, Nettelbeck, and Wilson (2009) demonstrated that initial level of and rate of changes in inspection time might serve as an important biomarker of aging. Briefly, a biomarker for the aging process “is a biological parameter, like blood pressure or visual acuity that measures a basic biological process of ageing and predicts later functional capabilities more effectively than can chronological age . . . a valid biomarker should predict a range of important age-related outcomes including cognitive functioning, everyday independence and mortality, in that order of salience” (p. 999). In a small sample of elderly individuals, initial inspection time level and rate of slowing (over repeated testing) was related to cognitive functioning and everyday competence. Repeated, relatively low-cost assessment of adults' inspection times might serve a useful function in cognitive aging research and serve as a routine measure (much like blood pressure) to detect possible early signs of cognitive decline.

Researchers have demonstrated how to harness the typical non-normal distributions of RT as a potential aid in diagnosis of certain clinical disorders. Most RT response distributions are not normally distributed in the classic sense. They are virtually always positively skewed, with most RTs falling at the faster end of the distribution. These distributions are called ex-Gaussian, which is a mathematical combination of Gaussian and exponential distributions. It can be characterized by the mean (m), the standard deviation (s),and an exponential function (t) that reflects the mean and standard deviation exponential component (Balota & Yap, 2011). (Don't worry; one does not need to under-stand this statistics-as-a-second-language brief description to appreciate the potential application.) The important finding is that “individuals carry with them their own characteristic RT distributions that are relatively stable over time” (p. 162). Thus, given the ease an efficiency with which RT tests could be repeatedly administered to individu-als (via smart devices and portable computers), it would be possible to readily obtain each person's RT distribution signature. Of most importance is the finding that all three RT distribution parameters are relatively stable, and t is very stable (e.g., test–retest correlations in the high .80s to low .90s). Furthermore, there is a robust relation between t and working memory performance that is consistent with the worst-performance rule (WPR) discovered in the intelligence literature. The WPR states that on repeated trial testing on cognitive tasks, the trials where a person does poorest (worst) are better predictors of intelligence than the best-performance trials (Coyle, 2003). It has been demonstrated, in keeping with the WPR, that the portion of each person's RT distribution representing the slowest RTs is strongly related to fluid intelligence and working memory.

In the not-too-distant future, assessment personal armed with portable smart devices or computers could test an individual repeatedly over time with RT paradigms. Then, via magical software or app algorithms, a person's RT distribution signature could be obtained (and compared against the normative distribution) to gain insights into the person's general intelligence, Gf, or working memory over time. This could have im-portant applications in monitoring of age-related cognitive changes, responses to medication for attention-deficit/hyperactivity disorder (ADHD) or other disorders, the effectiveness of brain fitness programs, and so forth. Finally, using the same general RT paradigms and metrics, research has indicated that it may be possible to differentiate children with ADHD from typically developing children (Kofler et al., 2013) and children with ADHD from those with dyslexia (Gooch, Snowling, & Hulme, 2012), based on the RT variability—not the mean level of performance. It is also possible that RT variability might simply be a general marker for a number of underlying neurocognitive disorders.

We have the technology. We have the capability to build portable, low-cost assessment technology based on Gt assessment paradigms. With more efficient and better assessments than before, build it . . . and they (assessment professionals) will come.


- Posted using BlogPress from my iPad

Visualization, inductive reasoning, and memory span as components of fluid intelligence: Implications for technology education

File under CHC domains of Gf, Gwm, Gc and STEM

Visualization, inductive reasoning, and memory span as components of fluid intelligence: Implications for technology education. Article link.

Jeffrey Buckleya, Niall Seerya, Donal Cantyc, Lena Gumaelius

International Journal of Educational Research, 90 (2018) 64–77

ABSTRACT

The philosophy and epistemology of technology education are relatively unique as the subject largely focusses on acquiring task specific relevant knowledge rather than having an explicit epistemological discipline boundary. Additionally, there is a paucity of intelligence research in technology education. To support research on learning in technology education, this paper describes two studies which aimed to identify cognitive factors which are components of fluid intelligence. The results identify that a synthesis of visualization, short-term memory span and inductive reasoning can account for approximately 28% to 43% of the variance in fluid intelligence. A theoretical rationale for the importance of these factors in technology education is provided with a discussion for their future consideration in cognitive interventions.


- Posted using BlogPress from my iPad

Can emotional intelligence (Gei) be trained: A meta-analysis

Can emotional intelligence be trained? A meta-analysis

Please cite this article as: Mattingly, V., Human Resource Management Review (2018), https://doi.org/10.1016/j.hrmr.2018.03.002

Victoria Mattingly, Kurt Kraiger

Keywords: Emotional intelligence, Training Meta-analysis

A B S T R A C T

Human resource practitioners place value on selecting and training a more emotionally in-telligent workforce. Despite this, research has yet to systematically investigate whether emo-tional intelligence can in fact be trained. This study addresses this question by conducting a meta-analysis to assess the effect of training on emotional intelligence, and whether effects are mod-erated by substantive and methodological moderators. We identified a total of 58 published and unpublished studies that included an emotional intelligence training program using either a pre-post or treatment-control design. We calculated Cohen's d to estimate the effect of formal training on emotional intelligence scores. The results showed a moderate positive effect for training, regardless of design. Effect sizes were larger for published studies than dissertations. Effect sizes were relatively robust over gender of participants, and type of EI measure (ability v. mix-edmodel). Further, our effect sizes are in line with other meta-analytic studies of competency-based training programs. Implications for practice and future research on EI training are discussed.

See prior Gei posts here and here.


- Posted using BlogPress from my iPad

MindHub Pub #3: WJ IV Norm-Based and Supplemental Clinical Test Groupings for “Intelligent” Intelligence Testing with the WJ IV

I am pleased to announce the availability of MindHub Pub #3 (WJ IV Norm-Based and Supplemental Clinical Test Groupings for "Intelligent" Intelligence Testing with the WJ IV).  Click the link to view or download.

The material in this document is based on my work during the development of the WJ IV as well as significant post-WJ IV publication analyses.  I have been completing considerable post-WJ IV data analysis in response to questions on listservs and to develop advanced and clinical interpretation information for convention presentations and workshops.  In the past I had the luxury of time to write professional books re: clinical "intelligent" intelligence testing with the WJ (1986) and WJ-R (1984).  I was unable to find time for the WJ III nor the WJ IV.  So much to do....so little time.

I have presented early versions of this material at conventions and workshops.  However, I never felt comfortable with the final product.  The most important reason for not distributing widely was my knowledge that the CHC model was in the process of responding to new research and insights--to be published this fall 2018 in a chapter by Joel Schneider and myself.  I only wanted this"supplemental grouping strategy" worksheet material (ala, Dr. Alan Kaufman's shared ability approach to test interpretation) to be made available once the revised CHC model had been described.  This event will occur this August with the publication of our chapter.  An early visual-graphic overview of the chapter, presented in a nifty animated YouTube video was released at this blog approximately a week ago.

So...enjoy the material.  This is not a book or article--more of a detailed PPT presentation.  It should be understandable to clinicians familiar with the WJ IV, CHC theory, and Kaufman's "intelligent" intelligence test interpretation approach.

Below is a sample worksheet--for Gc related tests.  Click on images to enlarge.

Cattell-Horn-Carroll (CHC) intelligence theory update (2.5): Animated summary of Schneider & McGrew (2018; fall) chapter theory update

A static PDF copy of this animated video PPT can be downloaded here.

AJT CHC Intelligence Test launch in Jakarta - a measure of 9 broad CHC abilities

Yesterday’s AJT CHC cognitive test launch yesterday in Jakarta was a big success. I was taken aback by the special “event” flavor. Extremely professional. As I’ve stated before, the AJT is based on an Indonesia norm sample of 4,800 and will be one if the most comprehensive intelligence tests in the world (on par with the WJ IV COG). It measures 9 broad CHC domains (Gf, Gc, Gwm, Ga, Gv, Gs, Gl, Gr, and some of Gp-separate from cognitive). This has been the most personally rewarding and important project I have worked on in my 40+ years in psychology and education. It is bringing the core concept of individual differences to the education system of the fourth largest country in the world.

George and Laurel Tahija (see picture below), and their YDB foundation, are the visionaries behind this project and other projects focused on helping unique learners in their country. In my five years on this project I can say that I’ve never worked with so many nice people. It was a grand effort by many. I am very impressed how together we built such a comprehensive and technically sound battery of tests from scratch. I have developed a fondness for Indonesia and the people of this wonderful country. The genuine warmth and enthusiasm of the participants was personally moving.

For more information check out these two links (one; two)

Conflict of interest disclosure.  I have been a paid consultant for this project but do not receive any royalties from sales of the test.

Click images to enlarge.












Posted using BlogPress from my iPad

Indonesia AJT CHC Intelligence test first official launch at UGM in Jogyakarta.

Next stop Jakarta.

One of the most comprehensive intelligence tests in the world (on par with WJ IV). When I have time I will post more. The most important project I have worked on in my 40+ years in psychology and education.

The importance of differential psychology for school learning: 90% of school achievement variance is due to student characteristics

This is why the study of individual differences/differential psychology is so important. If you don’t want to read the article you can watch a video of Dr. Detterman where he summarizes his thinking and this paper.

Education and Intelligence: Pity the Poor Teacher because Student Characteristics are more Significant than Teachers or Schools. Article link.

Douglas K. Detterman

Case Western Reserve University (USA)

Abstract

Education has not changed from the beginning of recorded history. The problem is that focus has been on schools and teachers and not students. Here is a simple thought experiment with two conditions: 1) 50 teachers are assigned by their teaching quality to randomly composed classes of 20 students, 2) 50 classes of 20 each are composed by selecting the most able students to fill each class in order and teachers are assigned randomly to classes. In condition 1, teaching ability of each teacher and in condition 2, mean ability level of students in each class is correlated with average gain over the course of instruction. Educational gain will be best predicted by student abilities (up to r = 0.95) and much less by teachers' skill (up to r = 0.32). I argue that seemingly immutable education will not change until we fully understand students and particularly human intelligence. Over the last 50 years in developed countries, evidence has accumulated that only about 10% of school achievement can be attributed to schools and teachers while the remaining 90% is due to characteristics associated with students. Teachers account for from 1% to 7% of total variance at every level of education. For students, intelligence accounts for much of the 90% of variance associated with learning gains. This evidence is reviewed


- Posted using BlogPress from my iPad

The AJT Cattell-Horn-Carroll (CHC) test of intelligence for Indonesia: Launch in April 2018 as one of the most comprehenvise intelligence tests in the world

For the past 5 years I've served as a consultant to the Yayasan Dharma Bermakna foundation for the development of the AJT test of intelligence in Indonesia.  The AJT is a Cattell-Horn-Carroll (CHC) based measure of intelligence.  The AJT is the first individually-administered, comprehensive, test of intelligence developed and normed in Indonesia.

The AJT will be one of the most comprehensive intelligence tests in the world--18 individual tests; 8 broad CHC ability domains).  The formal launch of the AJT will occur next month (April, 2018) in Indonesia.  During one of my recent visits I was filmed discussing various aspects of the AJT. This material is now available in a series of 9 videos [which I don't like to view as I don't like hearing my voice on tape, let alone seeing myself on video :) ..and now I wish I would have worn a suit--but it gets awful hot and humid in Indonesia, especially for someone who lives in Minnesota].  The first is included in this post.  You can find the complete list at this link.

Enjoy.  Take it easy on me...I am not a professional actor.