全球智库信息集成服务系统

On May 16, I got into a Twitter argument with Campbell Brown of The 74, an education website.  She released a video on Slate giving advice to the next president.  The video begins: “Without question, to me, the issue is education. Two out of three eighth graders in this country cannot read or do math at grade level.”  I study student achievement and was curious.  I know of no valid evidence to make the claim that two out of three eighth graders are below grade level in reading and math.  No evidence was cited in the video.  I asked Brown for the evidentiary basis of the assertion.  She cited the National Assessment of Educational Progress (NAEP).

NAEP does not report the percentage of students performing at grade level.  NAEP reports the percentage of students reaching a “proficient” level of performance.  Here’s the problem. That’s not grade level. 

In this post, I hope to convince readers of two things:

1.  Proficient on NAEP does not mean grade level performance.  It’s significantly above that.
2.  Using NAEP’s proficient level as a basis for education policy is a bad idea.

Before going any further, let’s look at some history.

NAEP history 

NAEP was launched nearly five decades ago.  The first NAEP test was given in science in 1969, followed by a reading test in 1971 and math in 1973.  For the first time, Americans were able to track the academic progress of the nation’s students.  That set of assessments, which periodically tests students 9, 13, and 17 years old and was last given in 2012, is now known as the Long Term Trend (LTT) NAEP. 

It was joined by another set of NAEP tests in the 1990s.  The Main NAEP assesses students by grade level (fourth, eighth, and twelfth) and, unlike the LTT, produces not only national but also state scores.  The two tests, LTT and main, continue on parallel tracks today, and they are often confounded by casual NAEP observers.  The main NAEP, which was last administered in 2015, is the test relevant to this post and will be the only one discussed hereafter.  The NAEP governing board was concerned that the conventional metric for reporting results (scale scores) was meaningless to the public, so achievement standards (also known as performance standards) were introduced.  The percentage of students scoring at advanced, proficient, basic, and below basic levels are reported each time the main NAEP is given.

Does NAEP proficient mean grade level? 

The National Center for Education Statistics (NCES) states emphatically, “Proficient is not synonymous with grade level performance.” The National Assessment Governing Board has a brochure with information on NAEP, including a section devoted to myths and facts.  There, you will find this:

Myth: The NAEP Proficient level is like being on grade level.

Fact: Proficient on NAEP means competency over challenging subject matter.  This is not the same thing as being “on grade level,” which refers to performance on local curriculum and standards. NAEP is a general assessment of knowledge and skills in a particular subject.

Equating NAEP proficiency with grade level is bogus.  Indeed, the validity of the achievement levels themselves is questionable.  They immediately came under fire in reviews by the U.S. Government Accountability Office, the National Academy of Sciences, and the National Academy of Education.[1]  The National Academy of Sciences report was particularly scathing, labeling NAEP’s achievement levels as “fundamentally flawed.”

Despite warnings of NAEP authorities and critical reviews from scholars, some commentators, typically from advocacy groups, continue to confound NAEP proficient with grade level.  Organizations that support school reform, such as Achieve Inc. and Students First, prominently misuse the term on their websites.  Achieve presses states to adopt cut points aligned with NAEP proficient as part of new Common Core-based accountability systems.  Achieve argues that this will inform parents whether children “can do grade level work.” No, it will not.  That claim is misleading.

How unrealistic is NAEP proficient? 

Shortly after NCLB was signed into law, Robert Linn, one of the most prominent psychometricians of the past several decades, called ”the target of 100% proficient or above according to the NAEP standards more like wishful thinking than a realistic possibility.”  History is on the side of that argument.  When the first main NAEP in mathematics was given in 1990, only 13 % of eighth graders scored proficient and 2 % scored advanced.  Imagine using “proficient” as synonymous with grade level—85 % scored below grade level! 

The 1990 national average in eighth grade scale scores was 263 (see Table 1).  In 2015, the average was 282, a gain of 19 scale score points.

Table 1.  Main NAEP Eighth Grade Math Score, by achievement levels, 1990-2015

That’s an impressive gain.  Analysts who study NAEP often use 10 points on the NAEP scale as a back of the envelope estimate of one year’s worth of learning.  Eighth graders have gained almost two years.  The percentage of students scoring below basic has dropped from 48%  in 1990 to 29% in 2015.  The percentage of students scoring proficient or above has more than doubled, from 15% to 33%.  That’s not bad news; it’s good news.

But the cut point for NAEP proficient is 299.  By that standard, two-thirds of eighth graders are still falling short.  Even students in private schools, despite hailing from more socioeconomically advantaged homes and in some cases being selectively admitted by schools, fail miserably at attaining NAEP proficiency.  More than half (53 percent) are below proficient. 

Today’s eighth graders have made it about half-way to NAEP proficient in 25 years, but they still need to gain almost two more years of math learning (17 points) to reach that level.  And, don’t forget, that’s just the national average, so even when that lofty goal is achieved, half of the nation’s students will still fall short of proficient.  Advocates of the NAEP proficient standard want it to be for all students.  That is ridiculous.  Another way to think about it: proficient for today’s eighth graders reflects approximately what the average twelfth grader knew in mathematics in 1990.   Someday the average eighth grader may be able to do that level of mathematics.  But it won’t be soon, and it won’t be every student.

In the 2007 Brown Center Report on American Education, I questioned whether NAEP proficient is a reasonable achievement standard.[2]  That year, a study by Gary Phillips of American Institutes for Research was published that projected the 2007 TIMSS scores on the NAEP scale.  Phillips posed the question: based on TIMSS, how many students in other countries would score proficient or better on NAEP?  The study’s methodology only produces approximations, but they are eye-popping.

Here are just a few countries:

Table 2.  Projected Percent NAEP Proficient, Eighth Grade Math