Monday, July 16, 2018

What is an applied psychometrician?

I wear a number of hats within the broad filed of educational psychology.  One is that of an applied psychometrician.  Whenever anyone asks what I do, I receive strange looks when that title rolls out of my mouth.  I then always need to provide a general explanation.

I've decided to take a little time and generate a brief explanation.  I hope this helps.

The online American Psychological Association (APA) Dictionary of Psychology defines psychometrics as: n. the branch of psychology concerned with the quantification and measurement of mental attributes, behavior, performance, and the like, as well as with the design, analysis, and improvement of the tests, questionnaires, and other instruments used in such measurement. Also called psychometric psychology; psychometry.

The definition can be understood from the two components of the word. Psycho refers to “psyche” or the human mind. Metrics refers to “measurement.” Thus, in simple terms, psychometrics means psychological measurement--it is the math and science behind psychological testing.  Applied psychometrics is concerned with the application of psychological theory, techniques, statistical methods, and psychological measurement to applied psychological test development, evaluation, and test interpretation. This compares to more pure or theoretical psychometrics which focuses on developing new measurement theories, methods, statistical procedures, etc. An applied psychometrician uses the various theories, tools and techniques developed by more theoretical psychometricians in the actual development, evaluation, and interpretation of psychological tests. By way of analogy, applied psychometrics is to theoretical psychometrics, as applied research is to pure research.

The principles of psychometric testing are very broad in their potential application., and have been applied to such areas as intelligence, personality, interest, attitudes, neuropsychological functioning, and diagnostic measures (Irwing & Hughes, 2018). As noted recently by Irwing and Hughes (2018), psychometrics is broad as “It applies to many more fields than psychology, indeed biomedical science, education, economics, communications theory, marketing, sociology, politics, business, and epidemiology amongst other disciplines, not only employ psychometric testing, but have also made important contributions to the subject” (p. 3).

Although there are many publications of relevance to the topic of test development and psychometrics, the most useful and important single source is “the Standards for Educational and Psychological Testing” (aka., the Joint Test Standards; American Educational Research Association [AERA], American Psychological Association [APA], National Council on Measurement in Education [NCME], 2014). The Joint Test Standards outline standards and guidelines for test developers, publishers, and users (psychologists) of tests. Given that the principles and theories of psychometrics are generic (they cut across all subdisciplines of psychology that use psychological tests), and there is a standard professionally accepted set of standards (the Joint Test Standards), an expert in applied psychometrics has the skills and expertise to evaluate the fundamental, universal or core measurement integrity (i.e., quality of norms, reliability, validity, etc.) of various psychological tests and measures (e.g., surveys, IQ tests, neuropsychological tests, personality tests), although sub-disciplinary expertise and training would be required to engage in expert interpretation by sub-disciplines. For example, expertise in brain development, functioning and brain-behavior relations would be necessary to use neuropsychological tests to make clinical judgements regarding brain dysfunction, type of brain disorders, etc. However, the basic psychometric characteristics of most all psychological and educational tests (e.g., neuropsychological, IQ, achievement, personality, interest, etc.) assessment can be evaluated by professionals with expertise in applied psychometrics.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, DC: Author.

 Irwing, P. & Hughes, D. J. (2018). Test development. In P. Irwing, T. Booth, & D. J. Hughes (Eds.), The Wiley Handbook of Psychometric Testing: A Multidisciplinary Reference on Survey, Scale and Test Development (pp. 3-49. Hoboken, NJ: John Wiley & Sons

Sunday, July 15, 2018

Excellent conceptual suggestion for organizing mind wandering research

Trends in Cognitive Sciences, June 2018, Vol. 22, No. 6


As empirical research on mind-wandering accelerates, we draw attention to an emerging trend in how mind-wandering is conceptualized. Previously articulated definitions of mind-wandering differ from each other in important ways, yet they also maintain overlapping characteristics. This conceptual structure suggests that mind-wandering is best considered from a family-resemblances perspective, which entails treating it as a graded, heterogeneous construct and clearly measuring and describing the specific aspect(s) of mind-wandering that researchers are investigating. We believe that adopting this family-resemblances approach will increase conceptual and methodological connections among related phenomena in the mind-wandering family and encourage a more nuanced and precise understanding of the many varieties of mind-wandering.

Click on image to enlarge

- Posted using BlogPress from my iPad

Saturday, July 14, 2018

Using Gt distribution parameters to predict executive functions in AHDH: Study consistent with Schneider & McGrew 2018 CHC update chapter

Interesting article consistent with what Joel Schneider and I discussed in our latest CHC Intelligence theory update chapter. Click here for info.

Using inspection time and ex-Gaussian parameters of reaction time to predict executive functions in children with ADHD. Intelligence, 69 (2018) 186–194.

Hilary Galloway-Long, Cynthia Huang-Pollock


Slower and more variable performance in speeded reaction time tasks is a prominent cognitive signature among children with Attention Deficit Hyperactivity Disorder (ADHD), and is often also negatively associated with executive functioning ability. In the current study, we utilize a visual inspection time task and an ex-Gaussian decomposition of the reaction time data from the same task to better understand which of several cognitive subprocesses (i.e., perceptual encoding, decision-making, or fine-motor output) may be responsible for these important relationships. Consistent with previous research, children with ADHD (n = 190; 68 girls) had longer/ slower SD and tau than non-ADHD peers (n = 76; 42 girls), but there were no group differences in inspection time, mu, or sigma. Smaller mu, greater sigma, longer tau, and slower inspection time together predicted worse performance on a latent executive function factor, but only tau partially mediated the relationship between ADHD symptomology and EF. These results suggest that the speed of information accumulation during the decision-making process may be an important mechanism that explains ADHD-related deficits in executive control.

Click image to enlarge.

Assessment Recommendations for Gt (from Schneider & McGrew, 2018)

To be published shortly in:

Tasks measuring Gt are not typically used in clinical settings (except perhaps in CPTs). With the increasing use of low-cost mobile computing devices (i.e., smartphones and iPads/other slate notebook computers), we predict that practical measures of Gt will soon be available for clinical use. Some potential clinical applications are already apparent. We present three examples.

Gregory, Nettelbeck, and Wilson (2009) demonstrated that initial level of and rate of changes in inspection time might serve as an important biomarker of aging. Briefly, a biomarker for the aging process “is a biological parameter, like blood pressure or visual acuity that measures a basic biological process of ageing and predicts later functional capabilities more effectively than can chronological age . . . a valid biomarker should predict a range of important age-related outcomes including cognitive functioning, everyday independence and mortality, in that order of salience” (p. 999). In a small sample of elderly individuals, initial inspection time level and rate of slowing (over repeated testing) was related to cognitive functioning and everyday competence. Repeated, relatively low-cost assessment of adults' inspection times might serve a useful function in cognitive aging research and serve as a routine measure (much like blood pressure) to detect possible early signs of cognitive decline.

Researchers have demonstrated how to harness the typical non-normal distributions of RT as a potential aid in diagnosis of certain clinical disorders. Most RT response distributions are not normally distributed in the classic sense. They are virtually always positively skewed, with most RTs falling at the faster end of the distribution. These distributions are called ex-Gaussian, which is a mathematical combination of Gaussian and exponential distributions. It can be characterized by the mean (m), the standard deviation (s),and an exponential function (t) that reflects the mean and standard deviation exponential component (Balota & Yap, 2011). (Don't worry; one does not need to under-stand this statistics-as-a-second-language brief description to appreciate the potential application.) The important finding is that “individuals carry with them their own characteristic RT distributions that are relatively stable over time” (p. 162). Thus, given the ease an efficiency with which RT tests could be repeatedly administered to individu-als (via smart devices and portable computers), it would be possible to readily obtain each person's RT distribution signature. Of most importance is the finding that all three RT distribution parameters are relatively stable, and t is very stable (e.g., test–retest correlations in the high .80s to low .90s). Furthermore, there is a robust relation between t and working memory performance that is consistent with the worst-performance rule (WPR) discovered in the intelligence literature. The WPR states that on repeated trial testing on cognitive tasks, the trials where a person does poorest (worst) are better predictors of intelligence than the best-performance trials (Coyle, 2003). It has been demonstrated, in keeping with the WPR, that the portion of each person's RT distribution representing the slowest RTs is strongly related to fluid intelligence and working memory.

In the not-too-distant future, assessment personal armed with portable smart devices or computers could test an individual repeatedly over time with RT paradigms. Then, via magical software or app algorithms, a person's RT distribution signature could be obtained (and compared against the normative distribution) to gain insights into the person's general intelligence, Gf, or working memory over time. This could have im-portant applications in monitoring of age-related cognitive changes, responses to medication for attention-deficit/hyperactivity disorder (ADHD) or other disorders, the effectiveness of brain fitness programs, and so forth. Finally, using the same general RT paradigms and metrics, research has indicated that it may be possible to differentiate children with ADHD from typically developing children (Kofler et al., 2013) and children with ADHD from those with dyslexia (Gooch, Snowling, & Hulme, 2012), based on the RT variability—not the mean level of performance. It is also possible that RT variability might simply be a general marker for a number of underlying neurocognitive disorders.

We have the technology. We have the capability to build portable, low-cost assessment technology based on Gt assessment paradigms. With more efficient and better assessments than before, build it . . . and they (assessment professionals) will come.

- Posted using BlogPress from my iPad

Thursday, July 12, 2018

Great psychometric resource: The Wiley Handbook of Psychometric Testing.

I just received my two volume set of this excellent resource on psychometric testing.  There are not many good books that cover such a broad array of psychometric measurement issues.  This is not what I would call "easy reading."  This is more like a "must have" resource book to have "at the ready" when seeking to understand contemporary psychometric test development issues.

Sunday, July 08, 2018

Practice or retest effects in measures of working memory capacity (Gwm): A meta-analysis

Retest effects in working memory capacity tests: A meta-analysis
Jana Scharfen, Katrin Jansen, Heinz Holling. Article link

© Psychonomic Society, Inc. 2018


The repeated administration of working memory capacity tests is common in clinical and research settings. For cognitive ability tests and different neuropsychological tests, meta-analyses have shown that they are prone to retest effects, which have to be accounted for when interpreting retest scores. Using a multilevel approach, this meta-analysis aims at showing the reproducibility of retest effects in working memory capacity tests for up to seven test administrations, and examines the impact of the length of the test-retest interval, test modality, equivalence of test forms and participant age on the size of retest effects. Furthermore, it is assessed whether the size of retest effects depends on the test paradigm. An extensive literature search revealed 234 effect sizes from 95 samples and 68 studies, in which healthy participants between 12 and 70 years repeatedly performed a working memory capacity test. Results yield a weighted average of g = 0.28 for retest effects from the first to the second test administration, and a significant increase in effect sizes was observed up to the fourth test administration. The length of the test-retest interval and publication year were found to moderate the size of retest effects. Retest effects differed between the paradigms of working memory capacity tests. These findings call for the development and use of appropriate experimental or statistical methods to address retest effects in working memory capacity tests.

Keywords Meta-analysis · Retest effect · Practice effect · Working memory

- Posted using BlogPress from my iPad

Nice visual graphic of cognitive load theory

Waves move across the human brain to support memory

Waves move across the human brain to support memory

Sent from Flipboard

Wednesday, July 04, 2018

How to keep your memory as you age: Annual Review of Psychology

Annual Review of Psychology: Successful Memory Aging. Article link.

Lars Nyberg and Sara Pudas


For more than 50 years, psychologists, gerontologists, and, more recently, neuroscientists have considered the possibility of successful aging. How to define successful aging remains debated, but well-preserved age-sensitive cognitive functions, like episodic memory, is an often-suggested criterion. Evidence for successful memory aging comes from cross-sectional and lon-gitudinal studies showing that some older individuals display high and sta-ble levels of performance. Successful memory aging may be accomplished via multiple paths. One path is through brain maintenance, or relative lack of age-related brain pathology. Through another path, successful memory aging can be accomplished despite brain pathology by means of efficient compensatory and strategic processes. Genetic, epigenetic, and lifestyle fac-tors influence memory aging via both paths. Some of these factors can be promoted throughout the life course, which, at the individual as well as the societal level, can positively impact successful memory aging.

Click on image to enlarge.

- Posted using BlogPress from my iPad