ITEMS: Ed measurement/statistics web-based instructional modules

I just read about the ITEM project in the latest issue of Educational Measurement: Issues and Practice. All school-based assessment professionals might want to take a look and see....the materials may be useful in educating others about what scores mean, what they do and don't tell us, and what score differences mean.

Project Description from ITEMS web page (emphasis and links added by IQ's Corner blogmaster)

• In the current No Child Left Behind era, K-12 teachers and administrators are expected to have a sophisticated understanding of standardized test results, use them to improve instruction, and communicate them to others. Many educators, however, have never had the opportunity to acquire the "assessment literacy" required for these roles. The goal of the ITEMS project, directed by Rebecca Zwick of the University of California Santa Barbara, was to develop and evaluate three Web-based instructional modules in educational measurement and statistics to address this training gap. We created three 25-minute modules: "What's the Score?" (2005), "What Test Scores Do and Don't Tell Us" (2006), and "What's the Difference?" (2007). Overall, 250 K-12 teachers and administrators participated in our research, which demonstrated the effectiveness of the modules in communicating educational measurement and statistics concepts, especially for teacher education students. Our modules are now freely available on our website, http://items.education.ucsb.edu, in low- and high-bandwidth versions, with optional closed captioning. Also posted are supplementary materials, including glossaries, formulas, reference lists, and quizzes corresponding to each module. The provision of this training in a convenient and economical way is intended to assist schools with the successful implementation and interpretation of assessments. Several school districts have let us know they are using the materials, and at least one teacher education program has incorporated them into its curriculum.

Technorati Tags: psychology, educational psychology, school psychology, education, educational testing, educational measurement, statistics, testing, NCLB, test scores, psychometrics

Latest "nations report card (NAEP)" results available

The latest results form the United States "Nations Report Card" (NAEP) are now available for viewing.

Technorati Tags: psychology, education, educational psychology, NAEP, NCLB, achievement

Powered by ScribeFire.

More NCLB backlash

More NCLB backlash...this time from the GOP. Click here for prior posts (at this blog) re: NCLB.

Technorati Tags: psychology, education, educational psychology, school psychology, neuropsychology, NCLB

powered by performancing firefox

Beyond NCLB

The high standards set by the No Child Left Behind legislation is in the news again...this time a Washingtonpost.com article posted at MSNBC.

For more detailed information regarding the "pulse" of NCLB, as reflected by the report (Beyond NCLB) of the non-partisan Commission on No Child Left Behind, check out the official report of the commission. The National Center on Learning Disabilities (NCLD) also has a nice summary posted.


Technorati Tags: psychology, education, educational psychology, school psychology, education policy, NCLB, No Child Left Behind, Commisson onf No Child Left Behind


powered by performancing firefox

NCLB Reading First controversies

I'm not tracking the politics of NCLB and Reading First with any diligence and count on readers to alert me to some of the political controversies that have arisen (click here for prior FYI post about possible problems revealed in an audit of Reading First).

Today someone sent me a copy of an article (Reading for Profit) in the Chronicle of Higher Education. A copy of the article can be read by clicking here. I have no comment as I've not kept up with the issues, studied the charges and counter-charges, etc. All I can say is that most academic-scholars who are involved in commercial products (like myself---coauthor of the WJ III) need to stay extra-vigilant with regard to potential conflicts of interest in the current high dollar stakes of NCLB-driven local/state educational policy decisions.

It is within this ethical spirit that readers need to be aware that some folks see a fundamental tension between norm-referenced testing (like the WJ III battery I coauther) and CBM measures (which are mentioned in the article). Thus, folks may think I'm posting this FYI as I want to see the CBM camp have less influence. This is not my motive as, individuals who have heard me present, know that I find norm-referenced and CBM approaches as complimentary...not mutually exclusive. They perform different functions and are both needed to better educate high risk students and students with disabilities.

Technorati Tags: psychology, education, educational psychology, reading, Reading First, NCLB, DIBELS, WJ III, testing, achievement, learning

powered by performancing firefox

IQ and school learning - Charles Murray speaks out

If you haven't heard the buzz yet, yesterday Charles "Bell Curve" Murray wrote the first of a three part series on general intelligence (g) and education in the Wall Street Journal. To say that Dr. Murray is no stranger to the controversial research on general intelligence (g) and race, gender, and education, is a understatement. He has published extensively, and not without controversy, on black/white IQ differences, male/female IQ differences, the Flynn Effect, etc.

Dr. Murray's first installment in the WSJ was called "Intelligence in the Classroom." A free copy (available from the AEI think tank where he is a scholar) can be found by clicking here.

Dr. Murray, in his series, is attempting to lay out "three truths about the mediating role in intelligence that should bear on the way we think about education and the nation's future." His first truth is - "Half of all children are below average in intelligence. We do not live in Lake Wobegon."

Blog readers can digest the article themselves. But, I do want to offer a few comments.

Briefly, the article is largely the articulation of the standard nativist's position that there are inherent (and largely genetically determined) individual differences in general intelligence (g) and educational reformers simply need to wake up and small the coffee...not all students, and esp. those below the 50th percentile, should be expected to achieve the same common set of high standards as per recent educational reforms as NCLB.

Dr. Murray is correct that general IQ is the single best predictor of school achievement. However, at best, general intelligence has been repeatedly shown to account for no more than 40% to 50% of the variance of achievement. This suggests, IMHO, that higher achievement can indeed be obtained by many individuals who may "suffer from the soft bigotry of low expectations"...this time grounded in the concept of fixed general intelligence.

As I've written before (click here and here for starters), non-cognitive factors (self-efficacy; self-regulated learning strategies; motivational orientation; etc.), human characteristics that are more amenable to intervention and change, exert significant effects on school learning and can be targeted to improve achievement for most all students. There is more to school learning than IQ. There is intelligence (IQ)...but then there is "intelligent performance and behavior" (IQ + conative or non-cognitive variables).

I could go on and on. Instead, I refer readers to a report I previously wrote re: this issue. It is often referred to as the "Forrest Gump" report. The report, IMHO, points out a serious flaw in Dr. Murray's logic...namely, that a specific IQ (esp. low IQ dooms a person to a lower set of achievement expectations). The bottom line...there is a normal distribution of achievement around any IQ score, whether it be below the 50th percentile or above. My point in the Gump report is that for any individual person, their general IQ score is not accurate enough to deny any individual to reasonably appropriate and high academic expectations.

However, Dr. Murray's position does fit with the extant body of group (vs individual) based intelligence research that suggests that, on the average, interventions to raise IQ and achievement have meet with limited success. This is one reason why the mastery learning experiment (which I have always maintained has conceptually been reborn in the form of NCLB) ran up against a major dilemma in individual differences/learning research....the "time-achievement equality dilemma." [That is another who series of potential posts...readers can check out one of the best literature reviews on this dilemma--Time, Equality,and Mastery Learning in 1984, by Arlin, in Review of Educational Research, which is at the heart of Dr. Murray's thesis, by clicking here.]

In simple terms....educational psychology research has repeatedly shown us that at the group level (not to be confused with my individual Forrest Gump expectations report), education policy can either hold instructional time constant, and then achievement will vary as per the normal curve, or can attempt to hold achievement constant (standard expectations for all), and time will then vary as per the normal curve. This is the rub against which NCLB is now bumping up against. We have been here before. The educational research suggests, IMHO, that we can likely move the mean level of educational achievement (I'm not sure how far) via attention to variables "beyond IQ", such as instructional time, quality of instruction, high standards and expectations, interventions focused on non-cogntive/conative characteristics of students, etc., but that there still will be a normal distribution around the new mean. This is still a laudable goal.

Enough said by me.

Dr. Murray's second WSJ article, "What's wrong with vocational school?" can be found by clicking here. I've not yet read it.

Technorati Tags: pscyhology, education, educational psychology, school psychology, neuropsychology, NCLB, intelligence, IQ, IQ test, IQ score, Charles Murray, Bell Curve, general intelligence, g, conative, aptitude, learning
powered by performancing firefox

Comments on the achievement gap

Interesting thoughts and analysis over at the Quick and the ED regarding national trends in narrowing the "achievement gap"...as reflected in changes in the most recent NAEP test scores.

Technorati Tags: psychology, education, educational psychology, NAEP, achievement gap, NCLB, learning, achievement

powered by performancing firefox

NCLB negatively impacting kids with disabilities?

Some interesting food for thought regarding a potential unintended negative side effect of the standards and test-driven impact of NCLB on kids at the extremes. Check out the Quick and the ED blog.

For a prior post (at IQs Corner) related to NCLB and teacher expections, check out the Forrest Gump post and report.


Technorati Tags: psychology, educational psychology, educatoin, school psychology, NCLB, testing, educational standards, disabilities, special education, teacher expectations

powered by performancing firefox

Audit suggests ethical/legal problems with President Bush's Reading First education program

Ouch! NBC news, as other media outlets, reported today that President Bushs's Reading First program has been plauged by all kinds of ethical problems. Check out the link and read and decide for yourself. I'm sure more information will be coming via education blogs and sites in the next week or so.

Technorati Tags: psychology, educational psychology, education, reading, Reading First, Bush, President Bush, NBC, NBC News, audit

powered by performancing firefox

Clarification of value of IQ tests in education: Response to the Reid Lyon DC gang

It has come to my attention that a report I previously directed blog readers to (Expectations for Students with Cognitive Disabilities: Is the Cup Half Empty or Half Full? Can the Cup Flow Over?) (IQ Scores, Forest Gump & NCLB: Run Forrest..Run), has made it to high places....which is not always good....especially when these places are awash in a sea of politics.

Briefly, in the recent announcement from the U.S. Department of Education that there should be an increase in the percent of students with disabilities who can be excluded from state NCLB-related assessments, (Raising Achievement: Alternative Assessment for Students with Disabilities), yours truly (and, indirectly, my associate Jeff Evans), is cited in support of, what can only be called, and anti-intelligence testing position in special education. As a coauthor of the Woodcock-Johnson III battery, which includes a comprehensive CHC-based intelligence battery, the attribution of an anti-IQ position to me may strike readers as bizarre, and, if believed, possibly suggestive of myself experiencing a break from reality.

Below is the select quote (emphasis added by me) from the ED.gov web URL cited above:

• "Research also supports the idea that IQ does not dictate achievement and, thus, cannot be used as a predictor. Kevin McGrew of the Institute for Applied Psychometrics notes that for most children with below average IQ scores, it is not possible to predict expected achievement with much accuracy. Lower-than-average IQ does not automatically translate into lower achievement or less ability to learn reading, language arts, mathematics, or other subjects. Other important variables affecting achievement appear to be interpersonal skills, motivation, engagement, and study skills, all of which can be positively influenced by high standards and expectations. Unfortunately, students are too often given a curriculum that is driven by educators' expectations of their students (based in part on a misunderstanding of IQ).

First, the later half of this statement is 100% true. As I've written elsewhere, non-cognitive factors (e.g., conative variables - motivation, self-regulated learning strategies, social and interpersonal abilities, etc.) are often ignored when making statements about a student's "aptitude" for school learning. I'm a firm believer in the Richard Snow approach to defining school aptitude as a combination of both cognitive and conative abilities. Nowhere have I ever written that valid measures of theory-based intelligence tests are not useful or predictive. In fact, I've authored/coauthored four different books on the interpretation of intelligence batteries. In the report drawn upon, I simply describe the normal curve of achievement scores that surround any specific IQ score, and this variability makes it difficult to predict, with precise accuracy, the exact level of expected current or future achievement for a specific student.

What I did write, and which is misconstrued in the ED.gov announcement, is the honest truth about our best measures of cognitive abilities. Namely, they are fallible predictors. Intelligence tests do not, nor have they ever, nor will they likely ever, account for school achievement beyond a threshold of approximately 40-60% of academic achievement variance....which is a hell of a lot in the field of psychology!

Cognitive measures, as per all respected models of school learning (e.g., Carroll's Model of School Learning; Walberg's Model of Educational Productivity), all include student characteristics, and cognitive characteristics in particular, as important contributing/predictive variables in explaining school learning. The point made in the Forrest Gump report is that even today's best available intelligence batteries are fallible predictors that do not allow educators to predict, beyond a reasonable doubt, specific expected levels of current or future achievement without a known degree of error. But, this degree of uncertainty (or error of prediction) is known and can be quantified. When combined with other variables (e.g., conative characteristics; home environment variables; quantity and quality of instruction), intelligence tests can provide valuable explanatory and predictive information.

The anti-IQ rhetoric coming out of many politically correct academic school psychologist channels, as well as the powers-that-be driving the special education reforms at the federal level, is accurate in the statement that global/full-scale IQ scores, and their use in ability-achievement formula's for LD determination, is not an empirically-based defensible procedure. I couldn't agree more.

However, cognitive measures, especially those that are based on contemporary psychometric theory (e.g., CHC theory), and those that provide reliable and construct valid measures of most of the major broad CHC ability domains, can, and do provide useful information in the hands of skilled clinicians. In fact, those advocating against IQ tests offer, as alternatives, measures or "markers" of phonemic awareness, rapid automatized naming, working memory, and vocabulary to identify students at risk for reading disabilities. Anyone with any knowledge of CHC theory recognizes that these marker/screening abilities can be directly mapped to the CHC abilities of phonetic coding (Ga-PC), naming facility (Glr-NA), working memory (Gsm-MW), and lexical knowledge (Gc-VL). AND, all of these abilities are measured on the most comprehensive CHC-based intelligence batteries. Isn't the addage "don't throw the baby out with the bathwater" applicable?

The real irony (in the misrepresentation of the information in the Forrest Gump report) in the U. S. Dept. Of Education statement, is that the primary premise of the report was that students with disabilities should not be denied high expectations, inclusion in state assessments, or access to the general education curriculum based only on a single point IQ score. The primary theme of the report was that, for too long, some educators, psychologists, and policy makers have believed in the supreme power of IQ tests, to the point that inappropriately low academic achievement expectations may be formed. Furthermore, the primary thesis of that report is that kids with disabilities (that are so classified based on intelligence test scores) should be NOT be automatically excluded from high standards and state accountability systems--the exact opposite message that the ED.gov statement uses our report to support (viz., increasing the percent of students with disabilities in NCLB accountability activities).

Whatever happened to President Bush's statement, when unveiling NCLB, that for too long, many children have been victims of "the soft bigotry of low expectations." Isn't the exclusion of more students with disabilities from high-stakes state accountability systems, systems that require schools to raise their expectations for students, promoting the same "soft bigotry of low expectations?"

As I've often said, and I don't know who to attribute the original quote to (I want to say Ralph Reiten of the Halstead-Reitan Neurospcyhological Battery fame), "if you give a monkey a Stradivarius violin and you get bad music, you don't blame the violin." By extension, if you give a "politically motivated federal bureaucrat honest scholarly information....... [never mind, you get the point].

IQ scores, NCLB & Forrest Gump: Run Forrest..Run

As a professional who has written about intelligence theories and tests and who is a coauthor of a frequently used individual measure of intelligence (WJ III), I often find other professionals and educators shocked when I highlight the less than perfect predictive capability of IQ tests. Although test manuals and research reports often report high correlations between IQ and achievement tests (e.g., .60-.70's), I occassionaly hear statements that suggest that some educators and assessment professionals fail to recognize that such high correlations (although some of the highest in all of psychology) are evidence of the fallibility of intelligence tests--they are less than perfect predictors.

Correlations of this magnitude tell us that IQ tests, on their best days, predict 40-50% of school achievement (Applied Psychometrics 101 – square the correlations and multiply by 100 to get the percent of variance explained). This is very good. Yet….50-60% of a person’s school achievement is still related to factors “beyond IQ!”

An unfortunate unintended negative side effect of the success of IQ tests can be the implicit or explict use of global IQ scores to form substandard or low expectations for individuals. How often have we all heard someone state, after hearing a child's general IQ score that, after using some norms or forumula to generate an "expected" achievement score, that teachers and parents should expect the child to achieve "at or below" these already below average expectations? It is often not recognized that for any given level of IQ score, half of the individuals with that score will achieve at or above predicted levels of expected achievement (based on that score). In the context of NCLB (No Child Left Behind), there is a real fear that IQ test scores may seduce educators and other education-related professionals into the “soft bigotry of low expectations” (it was either G. W. Bush or his then Education Secretary who coined this phrase).

The National Center on Educational Outcomes (NCEO) has recently published a report dealing specifically with this issue in the context of NCLB. Expectations for Students with Cognitive Disabilities: Is the Cup Half Empty or Half Full? Can the Cup Flow Over? (McGrew & Evans) is a report that addresses this issue. This report is pubished on the NCEO web page (click here). [If you go to the following page(click here) and right click on PDF (after the report title) you can download a pdf copy to your hard drive.] --- isn't technology wonderful?

As stated in the report introduction,

This report…includes an analysis of nationally representative cognitive and achievement data to illustrate the dangers in making blanket assumptions about appropriate achievement expectations for individuals based on their cognitive ability or diagnostic label. In addition, a review of research on the achievement patterns of students with cognitive disabilities and literature on the effects of teacher expectations is included. The literature raises numerous issues that are directly relevant to today’s educational context for students with disabilities in which both the Individuals with Disabilities Education Act (IDEA) and the No Child Left Behind (NCLB) Act of 2001 are requiring improved performance. Particularly for those students with cognitive disabilities, the information on expectancy effects should cause us much concern. Is it possible that expectancy effects have been holding students back in the past? Are we under the influence of silently shifting standards, especially for students with cognitive disabilities? It is anticipated that the information in this report will help guide decisions about appropriately high and realistic academic expectations for students with cognitive disabilities.

The fictitious story of Forrest Gumpis used in the report to illustrate the potential danger IQ score based generalized low expectations for students with disabilities---food for thought for educators, parents, and professionals involved in the education of students with disabilities during the current wave of NCLB-driven education reform.

After reading the report readers should feel compelled to yell “run Forrest run….from the potential negative impact of the soft bigotry of low expectations.”

3-14-05 note. Please note, in the spirit of full disclosure and potential conflicts of interest, that I was a coauthor of the NCEO report. I received $$ for writing the report but receive nothing for the number of copies that will be printed or distributed. Sorry for not pointing this out in the first post. I'm new at this....baby steps.