« Test Scores and Reinforcing the Wrong Connections | Main | Why Not Strengthen the NAEP as a Tool for Seeing Trends? »

What NAEP Long-Term Trend Scores Tell Us About NCLB

| 32 Comments

Dear Deborah,

I watched with some amusement as the media tried to figure out how to report the latest results from the National Assessment of Educational Progress (NAEP). Margaret Spellings said that the results vindicated the success of No Child Left Behind. The story by Sam Dillon of The New York Times reported that the achievement gaps—which the law was designed to eliminate—remained unchanged, and the headline of the story was “’No Child’ Law Is Not Closing a Racial Gap.”

So which is it? Were the results heartening or not? I’ll try to parse them here for the benefit of our readers and perhaps to kick off a renewed consideration of NCLB. Readers can make their own judgments by reading the report here.

Most people (and reporters) do not realize that there are actually two different versions of NAEP. There is Main NAEP, the tests that are given every other year to measure national and state achievement in reading and math, along with occasional tests in other subjects, such as science, history, civics, economics, writing, etc. Main NAEP’s tests of reading and mathematics are based on frameworks that periodically are revised, to reflect changes in the field.

And then there is Long-Term Trend NAEP, which is given less frequently and which tests more or less the same reading and math questions and concepts that have been tested since the early 1970s, with only minor revisions to remove obsolete references (such as outmoded technology).

Another difference is that Main NAEP tests grades 4 and 8, while Long-Term Trends tests students at ages 9, 13, and 17.

The Long-Term Trend results were released last week; the previous administration was in 1999. LTT reading scores for 9-year-olds, 13-year-olds, and 17-year-olds were up significantly, which is why Spellings felt vindicated.

But the scores for 9-year-olds were up by less than in the previous five years, so the rate of progress seems to have slowed. As for 13-year-olds, their scores have risen back to where they were in 1992; that’s progress, but only in the sense of recovering lost ground. And while reading scores went up for the 17-year-olds, they are still not as high as they were in the late 1980s and early 1990s. Again, better to see the scores going up rather than down, but we don’t seem to have made any real breakthroughs.

In mathematics, the story was similar. A significant gain for 9-years-olds, but not as large as the gain posted pre-NCLB. A significant gain for 13-year-olds, but not as large as the one posted pre-NCLB. No change for 17-year-olds, whose scores have hardly changed since 1973 (even though many low-performing students have dropped out by this age).

As for the racial gaps, they narrowed more pre-NCLB than post-NCLB in every age group.

So Spellings is right; the scores are moving in the right direction. But since the passage of NCLB and its implementation, the rate of improvement on the federal tests has slowed. Perhaps there are other strategies that would improve academic achievement with greater consistency.

By the way, you might be interested in reading my debate with John Chubb in Education Next about the future of NCLB. It was just posted.

Diane

The following was added on 5-6-09:
P.S. Thank you to a reader for pointing out that I did not clarify the dates of the tests that I was comparing. The results that were released last week were for the tests given in 2008. The previous tests were given in 2004. Before that, they were offered in 1999. Most of the pre- and post-NCLB comparisons that I make are related to comparing the results for these two periods: 1999-2004 and 2004-2008.

Thanks, and sorry for the error,
Diane


32 Comments

Thank you so much for this very thoughtful and helpful analysis of NAEP results. Is it too much of a stretch to say that perhaps what we were doing with students (or trying to do) before NCLB test-induced teaching may have been better for student learning? It seems what we really needed (and still do) are better methods of assessment.

Thank you for this helpful analysis, but it's hard to see how Spellings is "right" especially after reading Aaron Pallas at Gotham Schools (see http://gothamschools.org/2009/05/04/wishful-thinking/). The Main NAEP gains for 4th and 8th graders were prior to 2003 and NCLB was passed in 2002. Pallas has also spoken publicly about ways in which the accountability systems are gamed by pouring all resources into nudging scores just below AYP to just above it, essentially ignoring students who perform either way above or way below that line.

Yes, Renee, it is too much of a stretch. First off, what we were doing was all over the map, and to a great degree remains so today, despite statewide standardization of academic content. If, in fact, what existed pre-NCLB was thoughtful, responsive practice, that took into account the individual needs of children, there would not have been the rush that we have seen in totally non-sensical directions (cancelling recess, dedicating the last month before testing to "review" of concepts that students most likely didn't understand on the first go around, identifying kids as having/not having disabilities based on which would be most beneficial to the school's AYP rating, formula writing--not to mention outright cheating and things that come perilously close to it). In essence--if we were all doing the right things before, we might have been able to enhance what we were doing instead of shooting ourselves (and our students) in the foot. Certainly none of this has been helped by a lot of "bad attitude" choices on the part of individuals and groups. Some pretty big and influential groups hung their hats on telling the world that NCLB wouldn't "work." They have issued statements about how bad NCLB would be for those poor misunderstood fatherless hungry and distitute children that only teachers see in a true light. I wouldn't say that anyone intentionally sabotaged their work--but then on the other hand, it's pretty hard to be successful on the one hand while your are telling all the world on the other hand that you are being asked to do the impossible.

Recall that some groups were not even being tested prior to NCLB. So, we don't know a whole lot about how they were doing before--only that in the initial go arounds they looked pretty pathetic--and in some cases have improved.

I would say that what we were doing pre-NCLB was pretty disorganized and aimless. This does allow for a certain amount of individualized experimentation and the emergence of isolated excellent practice. But at the same time it allows for practitioners at the other end of the spectrum to either drop off, or to muddle along for years on nothing more than "survival skills." To the extent that there is a "system" of education, it has worked in the direction of ensuring that those excellent teachers tend to move in one direction--based on preferred salary or working conditions, or particular students, while the survivors tend to drift in another. This is true of administrators as well as teachers. For the most part there has been little response to these kinds of conditions.

In fact, there has been very little in the way of systemic change, even within buildings. Far too many have focused on minimal efforts (after all, why put a lot of energy into something that you believe is impossible) and waiting this one out. Many, many have placed great hopes on Obama to lead them back to where they were, where they were comfortable. No pesky standards, no assessments with published scores, and overall, no expectations of improvement. Just let everyone (that is, all teachers) set their own expectations, individually, and close the door.

Let me suggest an alternate explanation for earlier gap closing. Going back to the 1970s there were a number of factors that might very likely account for some narrowing. First, there was the money. Title I initially meant an increase in funds for low-income schools. Despite supplement not supplant requirements--this has tended to erode over time. Next, there were many accompanying changes in the lives of both low-income and minority families. Jobs became more open to African Americans, as did higher education. Support for people in poverty (welfare, food stamps, public housing, Medicaid) was on the increase. Finally, there was a fair amount of idealism. We believed that change was possible. This really brought about something of a "perfect storm" with regard to the narrowing of gaps (not to mention the desegregation of schools). Many of these things have been actively derided since that time. Others have just eroded. Our schools are more segregated now than they were then. Financial aid for low income students going to college is minimal--even those who are highly qualified to succeed. We have been through multiple drug epidemics that decimated the inner cities, and their residents. Jobs have left the rust belt.

And I caution against taking any of these as excuses for not improving schools. In fact, I would suggest that the intensify the need to improve education. The factory jobs that provided union wages to minimally educated workers are not going to come back. What was "good enough" then, is not "good enough" now. We have to get over our hurt feelings that someone is pointing out the need to improve. We just have to.

Diane,

Very interesting debate.

My hope has been to get everyone on the same page as to what to be aiming for, as in national standards and assessments with a common definition of "proficient." I also hoped we could reduce the amount of testing.

You state, "Perhaps there are other strategies that would improve academic achievement with greater consistency." Would you care to elaborate on what strategies you think would be more helpful. While you're at it, give us your thoughts on how to intelligently address the achievement gap.

Alexandra,
Spellings is right that the scores are trending upward, but as I pointed out, the gains were larger before NCLB than since it was implemented. So the rate of progress has slowed since NCLB was imposed.
And Pallas is right that she made a phony claim about the scores for 1999-2008, since this span of years includes years of large gains that were prior to the passage of NCLB. One cannot claim credit for gains registered before the reforms were introduced! This is the same thing that Joel Klein does in NYC, where he takes credit for big score gains in the year prior to the introduction of his reforms.
We have to watch our public officials closely when they start manipulating data for their political benefit.
Diane

Margo,

If, in fact, what existed pre-NCLB was thoughtful, responsive practice, that took into account the individual needs of children, there would not have been the rush that we have seen in totally non-sensical directions...

And you know this how?

Maybe some greedy bastards saw an opening and took it?

Or, maybe you are right that education has been getting worse, despite the NAEP scores we are commenting on in this post !

If NAEP were the only evidence available, the "new" report might merit the attention it is receiving. Actually, the report per se isn't receiving that much attention. It's merely a take-off point for venting ideological-based opinion. (Bridging Differences company excluded. Certainly, Diane gets it right. But I think you did omit 2004 in the Long Term cycle, didn't you, Diane?).

But the "no change to speak of" is confirmed by the IES Impact Study of Reading First and by other solid data bases such as ECLS-K, studies of "tutoring" and "school restructuring.

One doesn't need a "test" or a "study" to draw general conclusions.

"State content standards" were formulated by committee before NCLB. They haven't changed. And the states that have changed haven't based the changes on any performance feedback. New "standards" are just a different rehash of "old" standards.

The fact that texts and tests are used interchangeably across states indicates a de facto National Curriculum. It's instruction, not curriculum, that varies within-and-among school sites across the nation. But prevailing testing practice provides no feedback on instructional consequences that could be used to create "change we can believe in."

One thing it would seem we could learn from NCLB is: "Ed Slogans Suck." But there isn't any indication that this has yet happened. To date, Secretary Duncan seems interested only in "rebranding" NCLB and in promoting an even emptier slogan, "Race to the Top."--What "top"? appears to be guided by an implicit "Don't ask. Don't tell" policy.

We might also learn that prevailing standardized achievement test construction and reporting relies on psychological "trait and factor" theory long since generally abandoned. The fact that Long Term NAEP remains essentially unchanged since the 1970's should itself tell us something.

"Going back to pre-NCLB" is a fanciful bugaboo. We can only address the future. The "big deal" in that future is likely to be an economic "train wreck" for public schools, with California being the first to crash. (Watch for the results of the CA bond initiatives May 19. It isn't going to be pretty.)

Short-falls at state and local levels will more than eat up the "ed stim" moneys. But the pending "crisis" is no cause to hunker down. Improving productivity in el-hi as in every other sector means doing more with less. "Less" may very well provide the political and economic stimulus for responsible and transparent change in the el-hi enterprise.

Dick Schultz,
No, I did not skip the 2004 results. When I referred to the last administration of LTT, I meant 1999 was the "previous" or "last" before the current set of test results. Forgive me for not being clear. I can see why you misread what I wrote.
It is useful to be reminded that test results for a state or nation change slowly. But as you can see, the LTT results do not show any big victory for NCLB.
I personally have come to believe that NCLB has become a drag on education, that it is having many unintended consequences, and that we should scrap it and start over.
That is not going back to pre-NCLB.
That's what you do with a failed program.
Diane

Diane, much food for thought...if used right. I'm all for 'something better'. Yet, 'tis it really a fair claim your headline makes, saying you are telling us 'about NCLB'?

Isn't it also telling us something about the good faith efforts put forth--or not put forth--by districts, schools, and teachers toward executing the NCLB strategy?

You were, I believe, with me at Fixing Failing Schools - Is the NCLB Toolkit Working? There we learned much about how the remedial gifts included in the law were received and forwarded in the local districts.

The supplemental services, for example. These after-school aids, well funded by the law, were then among the bright spots of the NCLB remedy story. Here, funds for tutors and other services were specifically made available to students of schools identified as falling short.

The data were, in 2005-6, 180,000 students taking advantage of these services. Makes sense, if you need to catch up, you may need a little extra help.

Yet how did the schools deal with these external services? The reporting was, not well. Schools were charged with 1) making such services available directly if possible, or 2) making parents aware of the availability of external service providers.

In 36 urban districts surveyed, only 6 had developed any sort of supplemental services (SES). In the 6 that did, participation was thrice the rate of districts without their own SES (29% vs 11%).

So, good news for the lucky kids in those six districts! Yet what about the rest? If just a few more districts had had developed SES services, how many more of the 1.1 million eligible children would have participated? And maybe advanced?

The second duty of schools, notifying parents of options, did not seem to fare well, either. Audits at the time had commended states and districts to try again.

Since 94% of the supplemetal services were being provided by entities other than the schools, notifying parents of their options was a critical part of the process. These outside providers could sit there all day, but if parents didn't know they could send their kids, help was not on the way.

Also critical for schools was answering parents' questions. In another survey, Jay Greene emailed 2500 schools (from parent-appearing addresses) asking for information on SES or choice options. 95% of those schools did not provide any return information.

84% never even answered the email.

The second major remedy provided for was school choice. If your kid's school is not performing, moving them is an option you might consider. Yet, due to caps and limitations demanded by the education associations, the opportunities for choice were more limited than they might have been.

Even so, again, parents have to know about their options; there has to be some notification of Mom or Dad. Audits suggested that the school choice notification was also not adequately performed.


OK, so your concluding thesis was, "Perhaps there are other strategies that would improve academic achievement with greater consistency." Is strategy really the problem here? Or is it more in organization and execution in the field?

The strategy was to identify the specific failing schools, make services and options available specifically to the students of those schools, and if that didn't work, provide for more aggressive actions.

In other words, in the SES case, we were directly giving schools and teachers additional resources to help kids. You may have a bit more current data. Did more come to take the offer?

Hi All.... a point of clarification from new jersey.

"The supplemental services, for example. These after-school aids, well funded by the law, were then among the bright spots of the NCLB remedy story. Here, funds for tutors and other services were specifically made available to students of schools identified as falling short." Ed Jones

Not how it works here Ed.

The SES money is a set aside that comes directly out of the districts allocation.... not a well funded mandate.

What it required was that districts in need of improvement had to "set aside" money out of the allocation for the SES services...which took money away from the services the district was providing.

It then required the districts to notify parents about the availability of the SES services.

( A side note... NCLB talks at length about having teachers meet "highly qualified requirements in the schools....however...this is not the case concerning those outside the school providing the SES services... a bit odd don't you think? )

In effect Ed, the SES provisions require districts to set aside needed money for supplemental programs that comes directly from the money they need to provide the services.

"In other words, in the SES case, we were directly giving schools and teachers additional resources to help kids." Ed Jones

Not really...Again... no additional money for SES... at least here in NJ.

be well... mike

With the tutoring I couldn't help but think of Edward Thorndike's Law of Effect; (paraphrase) people tend to repeat those things they've had success with and avoid those things in which they've been unsuccessful. If a youngster has had problems academically in school what is there to motivate him/her to attend tutoring on any kind of a regular basis? (S)He can't appreciate school. Beyond that, how are parents going to motivate these kids to attend the tutoring? In summer? When everyone else is working, playing, sports, enjoying leisure time?

The tutoring sounds like a bit of a stretch to me, a very BIG stretch. The kid is going to resist it with every instinct in their body.

Is the "gold standard" LTT NAEP really as good a test as we need?

Look at the data on the achievement gap over time and compare that to the differences in the highest and lowest scores reported for all students. Since the true highest (ceiling) and lowest (floor) observed scaled score on each form are not reported, the best guesstimate we have is the scaled score for the 90th percentile and 10th percentile. Those are reported for every test, Reading and Math, ages 9, 13, and 17, for every administration of the test. By taking the 90th percentile and subtracting the 10th percentile, we get a measure of the spread of scores on the test.

If one test had a spread of 100 scaled score points and another had a spread of 80 points, we would know that either the test or the students were different. Either the two tests included test items with a different range of difficulty or the students themselves varied more in what they knew. The ceiling could be higher or students smarter or floor lower and students less well-educated or all of the above.

The LTT NAEP is an interesting case because the sampling is so carefully done, controlling for many background factors and weighting the data. Klein in NY claims that you need to test all of the students, but if NCLB has taught us anything it is that the definition of "all" is even less certain than what the definition of "is" is, something with which Klein may also be familiar. The likelihood of wild swings in inclusion of the highest or lowest achieving students is far less likely than the imprecision of item data for the easiest and most difficult items, which are usually not very reliable.

Looking at the data for the six tests, the spread of observed scores changes considerably from one administration to the next. For whatever reason, that spread correlates very well with the achievement gap between whites and blacks. For all of the tests, aggregating over subject, age, and year (N=66), the correlation between the achievement gap and test spread is .485. That means that the greater the spread of scores (ceiling to floor), the greater the achievement gap. The correlation holds for every subject/age combination except one.

Math, Age 9: 0.808
Math, Age 13: 0.882
Math, Age 17: 0.748
Reading, Age 9: 0.518
Reading, Age 13: 0.110
Reading, Age 17: 0.454

Is there some error in this analysis (data at bottom of post)? Is this just a coincidence?

If this is true, why is it? It looks like variations from one administration to the next are having a sometimes enormous effect on the estimate of the achievement gap. This is not a simple ceiling effect in which scores of one group are suppressed, narrowing the gap. The spread did not always narrow from one administration to the next. Furthermore, these are scaled scores, not proficiency data where a cut score can be set so low everyone passes and there is no gap between any group.

Diane, can one of your friends at NAEP explain this? Does this effect disappear when you use the actual observed highest and lowest scaled scores for each administration, or is the correlation even stronger?

My guess is that these correlations reflect the reality that data-driven test advocates want to ignore. Differences in form construction or item sampling have a big impact on estimates of achievement. The achievement gap reported on NAEP is real, but the estimates are highly imprecise -- too imprecise to measure changes in the gap over time, even with thousands of students in the data, even with a "gold standard" test. Now, imagine how many misleading conclusions are drawn about schools and teachers and students because those decisions were driven by state tests that are only as good as the "silver" or "bronze."


Here are all of the data used. Bottom10 is the scaled score at the 10th percentile and Top10 is the scaled score at the 90th percentile. You will need to calculate the spread (Top10-Bottom10). Ach_Gap is the mean scaled score difference between white and black students on that administration. (Anyone checking will see that the 2004 new form data are used for that year.)

Test Year Age Bottom10 Top10 Ach_Gap
M 1978 9 171 264 32
M 1982 9 173 263 29
M 1986 9 177 264 25
M 1990 9 186 271 27
M 1992 9 185 271 27
M 1994 9 187 272 25
M 1996 9 187 274 25
M 1999 9 187 275 28
M 2004 9 193 280 24
M 2008 9 198 284 26
M 1978 13 213 313 42
M 1982 13 225 311 34
M 1986 13 230 309 24
M 1990 13 230 310 27
M 1992 13 233 312 29
M 1994 13 233 315 29
M 1996 13 233 314 29
M 1999 13 234 317 32
M 2004 13 234 322 30
M 2008 13 237 323 28
M 1978 17 254 345 38
M 1982 17 256 341 32
M 1986 17 263 343 29
M 1990 17 264 345 21
M 1992 17 267 345 26
M 1994 17 267 345 27
M 1996 17 267 346 27
M 1999 17 268 347 31
M 2004 17 266 343 27
M 2008 17 267 343 26
R 1971 9 152 260 44
R 1975 9 159 258 35
R 1980 9 165 262 32
R 1984 9 157 263 32
R 1988 9 157 263 29
R 1990 9 150 266 35
R 1992 9 156 260 33
R 1994 9 156 260 33
R 1996 9 160 260 29
R 1999 9 158 259 35
R 2004 9 162 263 27
R 2008 9 171 265 24
R 1971 13 208 300 39
R 1975 13 209 300 36
R 1980 13 213 302 32
R 1984 13 210 302 26
R 1988 13 213 302 18
R 1990 13 210 302 21
R 1992 13 208 309 29
R 1994 13 205 307 31
R 1996 13 206 306 32
R 1999 13 209 308 29
R 2004 13 205 304 25
R 2008 13 211 306 21
R 1971 17 225 342 53
R 1975 17 228 340 52
R 1980 17 231 337 50
R 1984 17 236 340 32
R 1988 17 241 337 20
R 1990 17 237 343 29
R 1992 17 233 343 37
R 1994 17 230 343 30
R 1996 17 232 341 29
R 1999 17 233 341 31
R 2004 17 224 337 27
R 2008 17 227 341 29

Mike, Good morn. Much as I feel for you, good buddy, dealing with NJ's educational woes and all; smoking and passing the NEA koolaid won't help us help these kids.

Title I funding to the LEA's was increased by $5,400,912,000 (61%) between FY2000 and 2005, in good measure to cover the Supplemental Services.

Hi Ed... hope this finds you well.

Point of clarification... somehow you seem to connect me with the teachers union Ed and i have not been a part of that organization since i left the classroom over 20 years ago... many teachers are in unions Ed.... but that doe not prevent independent thinking and judgement!

On the ground....at the actual school level is very different state to state Ed from the statistic you have provided.

The actual schools that are most in need that i work with did not receive anywhere near a 60% increase....
there actual on the ground in the school allocations are down.

My poorest schools actual title i allocations:
2006- 212,205
2008- 196,770
Out of that real schools allocation--- 20% must be set aside for SES services.

None a roughly 40,000 dollar set aside may not seem like a lot of money but in poor schools.... it is significant.

You may also wish to comment on the SES providers that do not need to be "highly qualifed".

Ed... been working on helping kids for over 25 years.... what i am trying to point out is that this is no real help to anyone.

Have read or seen no data on the effectivenss of SES services.... even when they are provided as the law requires ( which the above school does)

For some reason Ed...you see my information as "defending the system"... no...the system has many issues that need to be addressed.....

what i am trying to point out.... this is not the direction we need to go to change the system.

be well... mike


Ed and mike:

As regards the SES provision. This has never been, to my mind, the strongest feature of NCLB. But I understand the logic of it. If we look at the intent of Title I dollars over a couple of decades--which was to "level the playing field" for otherwise disadvantaged (both in terms of life and education) kids--and the outcome of lots and lots of dollars spent, I think most will agree that the overall impact has been nearly negligible. I would suggest that a good bit of this relates to a lack of evaluation regarding outcomes. Most criteria had far more to do with how dollars were spent and various input measures than on whether anything was really "working." Enter a largely Republican administration and the natural question arises--why spend all the money. And the corollary--if the money going to the schools isn't making a difference, let it go to the parents so they can go to their local vendor of tutoring services and BUY something.

Now--we know that nothing is ever that simple. First off--most of those vendors (who may or may not have been able to provide evidence of success) were catering to a different market--people with disposable income and a concern for their kid keeping up with the Vanderbilts. Second--the whiff of money always awakens a lot of folks, opening the door to both scoundrels and people who were already doing similar things for poor kids on a shoestring in church basements and the like. Third, any loss of those dollars to districts meant loss of jobs for teachers.

Well that mess (like everything in NCLB) got handed off to the states to deal with in fifty different ways. The states weren't in any better position to evaluate SES than they were to evaluate any of the ongoing uses of Title I dollars. So they put together some rules and handed it on to the districts. As interested players (not eager to give up the bucks or the jobs), they half-heartedly did the minimum that was required (to the extent that anyone was checking).

Personally, as a parent of a kid in one of the schools that was required to provide SES (because of low and unimproved achievement), it took two years of advocacy to get less than a year of after-school tutoring. It happens that the tutor was quite good. We ended up with one-on-one, not because that was intended, but because the district was very good at ensuring that parents, programs and students did not connect. They refused building space (even though this would have benefitted their students). They set up a convoluted registration process (requiring face-to-face enrollment, at the convenience of the school). By year two they had set up their own program, which they actively "sold" in the process of enrollment. Enrollment was supposed to include goal setting based on school progress--but the district or principal withheld the records (from district staff), so this was purely arbitrary--making evaluation of progress an impossibility. When the district was banned from providing their own tutoring program (due to that odd little supposition that if they couldn't succeed during school hours they shouldn't get paid extra to do the same thing after school), they laid low for a year before the union turned up as a provider. But even the union gave it up after a year, discovering that the environment that they had created made it impossible to cover expenses with the number of kids that made it through enrollment.

All of which, I suppose, would be to point out that in evaluating the impact of NCLB, as any intervention, a responsible researcher would want to take into account "faithfulness of implementation." Is there in fact some magical cut-off of 2003 that signals a groundswell of implementation? Recall that NCLB is widely regarded as having "put teeth" into requirements of ESEA--so that some states were already marching forward pre-NCLB.

I don't know that I see clarity one way or the other in the recent NAEP scores with regard to NCLB. What I do see, and I find this disturbing, is a lot of cheering from the side that says NCLB has failed. I don't see much to cheer about in letting our kids down, if that is what we have done. And I have seen way too many adults eager to gum up the intended works. Congratulations. We've failed our kids.

At the Math Panel meeting in Palo Alto a couple of years ago. Jim Milgram who is on the IES oversight board presented his analysis of the NAEP math examination. He demonstrated very clearly that 20% of the NAEP math questions had either incorrect answers or were not math questions. As far as I know NAEP has made no effort to correct this. If your blood pressure meter was off by 20% the data is not useful.

Mike, that seems terrible.

Of course, I don't know how many students your poorest school serves, but... why should your share decrease when the actual Title I funds to LEA's were budgeted to increase ('06 to '08) by 1.2 Billion!?

In fact, this school appears to receive much less than the average Title I school would seem to get. Is there a reason for this? How do your schools do on average?

The next question is, why do you receive so little if these are the very schools we want to fix? Well, its probably because you share the $14 Billion in Title I funds with 50,000 other schools! One (or more) of them is in Jackson Local SD, a rather wealthy suburban district (with which I've had a small association). It will see $250K of new Title I funds from the ARRA. Our small rural district will see $409,000.

Should we be getting this money? If you looked at family income and the gaping hole in the floor where the HS boiler pipes are annually repaired, you'd say yes. If you looked at our high graduation and 50+% college attendance rate, you might just say no. It doesn't really matter, though, as this is the price of politics in funding local programs via a national congress.

So back to the context of NCLB: It wasn't all about testing; it was about providing catch-up services to those who need it. Finding those who need it was step one; focusing the aid to them was step two.

You seem to be saying we (or at least NJ) did a poor job focusing that aid. This is a productive line of inquiry! I'm glad to see my old town of Little Silver got no funding, but what about Freehold?

Is this the fault of the NCLB strategy? Its hard to imagine how; the Title I setup long precedes the 2000 election.

Does the next version of the Act need to revamp who gets these moneys? If so, lets say that. Are supplemental services not effective when used? Then lets say that.

Yet if the fed offers supplemental help, sends funds to do it, and provides a list of the most academically needy, and the states react poorly, is the national strategy all at fault? Are we to send in the National Guard to the states to make wise allocation decisions?

Detroit is getting $175,000,000 in ARRA. How will they spend it? Will the kids be there to take advantage?

I don't know the answers to these questions. What I do know is that I am trained to look for solutions, not just hold up alleged failures.


BTW, not accusing anyone of belonging to any org; just not naive about how the spin from the State Ed Assoc. makes its way to headlines and talk on the street. We need to turn this around; to make citizens in districts like mine see, for example, that it may be in their interest to pass a levy more than once every 40 years; that they cannot forever rely on the buildings their grandparents built from '13 to '59.

--
The preview shows Margo's response with her description of 2 years of begging needed to get 1 year of tutoring! That's an implementation problem! Yet not unexpected. What is the solution?

Ed and Margo....

I think these conversations are worth having and reading and listening to others is of value.

The stimulus money will help... however districts will need to be careful as to there use.... because they will go away.

As i look at my small section of the world... i continue to be drawn to the idea of reorganizing districts to begin to address some of the challenges that occur with-in districts that have very high rates of poverty.

Many southern states have county wide school districts and are able to balance the schools, racially and ecconomically, much better than the schools in the north.

Wondering... as many states move toward consolidation of school districts if this should not at least be on the table?

Mike,

Funny you should mention school (and other agency) consolidation attempts. I've suggested it as a possible solution to budgetary woes here in Massachusetts through several papers while I was a student at Harvard and several pieces in the Boston Globe.

I believe it is a direction that must be considered for our financial survival but the history in this state of "local" control is omnipotent. The 361 cities and towns in this state still think they’re back in the eighteenth century and have to have a say in every minute detail of government. It’s costing taxpayers outrageously with ubiquitous redundancies between state and local agencies. Officials are looking past FY 20102 before any relief materializes.

To date, my pleas have fallen on deaf ears. As a result of our state officials' reluctance to even consider amending the state Constitution it looks as though Massachusetts is in for some very difficult days ahead.

Hi All... hope this finds you well.

Paul thanks for the update. Here in New Jeresy we have a similar situation with over 600 school districts.

I am certainly not advocating bigger is better.... and i really do like the idea of local control....

however.... i think this is something that can be explored. As we continue to see the enormous differences in what schools can do.... money is a major player.

we are moving faster and faster toward districts with significantly more opportunities vs districts that barely can hold on.

NJ's pooerest districts have been in court for decades concerning funding...however....the probelm is larger than even our 31 Abbott districts.

be well.... mike

Mike, Paul, I like exploring solutions!!

Consolidation: No. And Yes.

Consolidation is part of what got us in so many of these messes. New Yorkers, for example, thought that gathering all the tax revenues and spreading it across the city would even out the field for kids. Hah! 1.1 Million students, none served well. High schools with 5000 students. Arg. Plus overhead. DC schools on Rhee's arrival had 50,000 student, 5000 teachers, and 500 central office workers! What a 1950's dinosaur that is!

On the flip side, Conotton Valley Schools here serves 578 students pK-12. That's after the 1950's consolidations were. Of course they can't employ many of the services they need.

Thereto we add Educational Service Centers which support these small districts and schools with services for speech, assessment, art, drivers ed, PD, etc. Different schools have different needs; they sign up only for their needs.

The ESC's then are an effective sort of consolidation. Moreover, they allow the small district relationship to citizens for which Deb so often calls.

Yet when I think of 'consolidation' opportunities for the future, I don't so much think physical or command consolidation! I think of virtual collaboration.

I think we need a restructuring much like that which swept through here in the '50's, not consolidating students into bigger buildings, but bundling services into manageable (and improvable) packages.

(Paul, not sure whether your post was tongue in cheek (I hope!) but local control is central to the idea of America. I know my sheriff and Judge and my police chief and I like it that way. The inverse, in LA county where the sheriff took police responsibility for most of the 8 million residents, was absurd and predictably disastrous.)

Fortunately, this marverlous new web gives us many new options. Let us look to the commercial sector for ideas and possibilities.

Lets just think of Amazon.com and UPS supply chain management. Do we really think all those among the 500 DC bureacurats are now necessary? If I need TP and light bulbs for my building, the general schedule purchasing just seems an anachronism. Any minute savings from the bulk purchases will be frittered away ten-fold by overhead. So give each building/department a debit card and be done with it.

On the other hand, what if we consolidated the math departments of failing schools across central Ohio (I71 corridor)? What if we agreed to hire a support organization that would work with teachers in urban Cleveland, Columbus, Dayton, and Cincinnati with bringing up their math proficiency?

Said organization might have what types of duties/powers?
- They might deliver tutoring.
- They might deliver a set of aides to classrooms to help during common work periods.
- They might scour the net for emerging ideas, resources, and web-based tools for improving math understanding in this population.
- If the organization were larger, with contracts in other states and corridors, they might further be able to support a research staff or even a software development staff.
- They might be empowered to reward teachers who went the extra mile.
- They might be an independent voice which says, Mr. Jones needs training; or Mr. Jones just isn't cutting it.
- They would have a larger voice in the assessment mechanisms. Better, if there were 4-5 national organizations, they would work together as professionals to solve the assessment dilemma.
- They might have a development arm which garnered support from industry, perhaps partnering with Wright Patterson to send students to explore math in the Air Force, or arranging a series of interaction with NASA Ames.

This is just a brainstorming example. I think both of you are correct in that we need to bring more support to the teacher on the front lines. How we do this, we need to be really, really, flexible and creative!

What do you think, all?

Ed:

Drucker offers up some discussion of something similar with regard to large corporations "outsourcing" various components of HR. The example he gave was of hospitals (knowledge workers), where you have a whole lot of highly specialized workers. While the volume of workers has in the past dictated an ability of sorts to meet all HR needs (hiring, firing, tracking benefits, overseeing licensing and professional development) in-house, he observed trends in the direction of contracting with numbers of much more highly specialized groups. Such a group might provide physician services, and due to their specialization be able to deliver up very well in terms of recruiting and evaluation, as well as following the intricacies of licensure, meeting professional development needs etc.

Imagine in education if such a group (perhaps taken on by professional organizations, or entrepreneurial ESCs, or even districts who wanted to specialize) were to take on the provision of language teachers, for instance. They might be able, through the virtual world, to ensure that three sparsely populated, but geographically separated, areas of the state are able to offer a class in, say, Farsi.

A mathematics group, with the ability to focus adequately on vertical integration from K-12, might be able to provide, through a contract, a constant supply of both teachers and professional development to (gasp) increase student performance at an agreed upon rate. The union would contract with them, but they would then have the responsibility to ensure that the right teachers were in the right places.

Like mike, I have often envied some of the states with county-wide districts. They have managed, in some cases, to pull off some redistributions that the rest of us can only dream about--as large urbans surrounded by a high performing ring of "leech" districts--where parents depend on the city for employment, but take the fruits of the labor out to the 'burbs where they are shared only with those similarly situated. It's a bad situation and ought to have been halted ages ago. But, I think its likely too late now. Them that have ain't likely to give it up. I think we need to look to the future in terms of new ways of organizing.

Margo, your hospital image puts me in mind of an image from Canton, a small city: When I were a child, the two schools and two hospitals were much the same. A couple hundred people in a building serving maybe 2000 people.

The hospital, of course, was a resource for your family doctor. He had an office, as did a few specialists. Yet any type of care beyond the basic brought you to the hospital.

Fast forward to today, where a hospital is merely the center of huge rings of providers. My brother never saw a hospital for his broken leg and screw insertion; the surgical center did it. Diagnostics? Another center. Eye surgery? Another. The offices and centers go on and on for blocks. Far more resources outside the main hospital than in; most privately owned; though the hospital itself adds many external services, even to include fitness centers.

The school? Pretty much the same. (Although recent funding has replaced some 50-80 year old buildings).

Bring on more such ideas! I like it!

Mike, Ed, Margo,

Here is part of a proposal I suggested for Massachusetts:

If local districts decided to regionalize, even as large as (14) county districts, they would be able to realize significant savings of tax dollars on their schools.

EXAMPLE: If County X had fifty local school districts and decided instead to make itself one district, the savings could be enormous. In central office positions alone; Superintendent, Assistant Superintendent, Business Director, Special Education Director, etc., salaries plus benefits for these central office positions can easily exceed half a million dollars (times the 50 districts=$25 million). If the towns in that county consolidated into one school district it would save taxpayers well in excess of $20 million on that aspect of the school budget alone - even if the salaries of these central office positions doubled or tripled for their increased responsibility.

Now, factor that out to all other aspects of a school system and imagine the additional savings. Then do the same consolidation for police, fire, and DPW. Then multiply that again times the fourteen counties.

Even in small New England towns where local control has been the hallmark of government run agencies: DRASTIC TIMES CALL FOR DRASTIC MEASURES. Continue to pay exorbitant tax dollars for local services or regionalize the services (by county) and reduce the redundancies and the tax dollars significantly.

Paul - I think that is a grass is greener issue. In my state by constitution school districts are set by county ... we all wish we could break them up. Huge districts have a plethora of their own issues, they save money in some areas but cost more money in others - and things and people and students get lost in the shuffle of huge districts.

I've also taught in a small district that worked with all the districts around to purchase many things like a big district, so that is another way you can achieve savings. Also taught in a Catholic school that went in on things with the local public schools when they could legally.

The "let's make huge districts to save money" idea is much more of a wash than it often seems I'm afraid, and the "hugeness" has too many pitfalls when it comes to the "people" side to be worth it. Not saying it shouldn't be considered at times to realize a savings, but sure isn't the saviour you might think it is.

Paul meant to mention the cost and inefficiency of the HUGE beauracracy that becomes entrenched in a large district and is hard to manage and loses touch with the "customer" (students, teachers and parents) all by itself eats up most of the "savings."

On this sizing thing, I wrote earlier a bit:

A high school should be able to field a varsity, reserve, and freshman football team (if 9th graders are there). There should not be so many students that many who want to play get left off.

The school should be large enough for a decent speech and debate organization, drama activities, a band with all the instruments filled, swimming activities, dabbling with robotics and Arabic or Mandarin or Farsi at least via tele-learning.

I don't think they should be much larger. These 5000 student high schools seem absurd to me.

How many high schools in a district? Around here it is one. One school board, one high school, 1-8 elementary buildings. It works. In the near small city, two hs.

As a general rule, how many principles should a superintendent oversee? A large handful? Twenty? Not 100.

The Army recently scrapped its Division structure in favor of Brigade Combat Teams guided by Corps commanders, a CINC, and MNF leadership. The division structure made sense in the days of lesser communications ability. Now that a CINC can see graphics of every unit, that layer is less necessary.

Stripping out middle management and improving the tooth-to-tail ratio is always a good thing.

Education is not a top-down thing, though, or shouldn't be. It has become so in the 1000-1500 schools serving as dropout factories; the national interest is at stake in reversing this.

In our community you can pretty much count on the police' support; if a teen is standing on the street 4 days in a row at noon, someone is gonna ask. Getting such cooperation was often harder in large cities; one of the reasons mayoral control made sense. Yet we should be ever striving to drive the power back downward, use the net, create new relationships.

Another example from the Army: as Iraq spun upward, young (~28) Captains discovered the Training and Doctrine was not preparing them to win an insurgency of this type. No one could give them a manual or daily instruction.

They they formed a network, Company Command, where, as on classroom 2.0, they exchanged ideas for surviving and winning at their level. It became one of the more powerful tools we had.

Hi All.... thanks for the thoughts.

As i continue to look at the idea of consolidation.... certainly do not what to create what Nev above mentioned.

As both Ed and Nev say... too big is crazy and i really like the concepts of
Deborah Meier's work on small schools....See in Schools We Trust.... a very good read ( Thanks Deb )

As i look at my county here in southern new jersey i think there are some possiblities to create more diverse...smaller distircts with-in the formation of consolidation of parts of schools distirct... for instance transportation... Business office duties..cross curricular work between and among the schools and even some variety of programing that is a bit different from highschool to highschool.

Is there a way to combine the correct things without interfering with the idea of diversity and some autonomy with-in the structure?

be well... mike

Two questions were raised in the comments, each by a different person. I didn't feel that I have the technical expertise to answer them, so I turned to Andrew Porter, who is dean of the graduate school of education at the University of Pennsylvania and currently a member of the National Assessment Governing Board.
One question (or comment) from John Stallcup asserted that Jim Milgram said that 20% of NAEP math questions are not really math. Porter says that he has no idea how anyone could reach this conclusion without reviewing all the NAEP math items, most of which are secure. In any event, people do disagree about what really is math, which may be the source of the issue here. My own experience as a NAGB member was that the math questions were rigorous, and were defnitely math.
The other question or comment came from "gooenuf," who implied or stated that the LTT NAEP really didn't tell us much about the achievement gap. Again, I am not a psychometrician, so I asked Porter (and sent him the entire comment). He said that LTT NAEP is probably the best national measure of the achievement gap.
Not the place to dig deeper, but there you are.
Diane

Nev

Thank you for your feedback.

I have run the numbers for Massachusetts and on paper county government could be a blessing. However, as you intimated the grass can be greener if the other fellows yard...until you get there. Much would have to happen to allow for the perfect storm of government to actually make this model more effective.

I certainly appreciate Dr. Porter's clever answer. I agree that the LTT NAEP may be the "best" measure of the achievement gap, but is it good enough? Is the LTT NAEP valid for the purpose of tracking the achievement gap over time, especially given that the size of that estimate would appear to be dependent on the test items selected from one administration to the next?

Dr. Porter is keenly aware of the fact that two test forms can be "equated" and measure very different content. His "Alignment Index" measures the degree to which one form of a test matches the published curricular standards and the curriculum being taught. Using raters reading actual test items and standards and observing actual teaching, you can measure the degree to which these are in sync. Dr. Porter's easily understood, easy-to-calculate index tells you the degree to which the test is measuring what is being taught. See his Chapter 8, "Curriculum Assessment" (Handbook of Complementary Methods in Education Research, 2006, Edited by Green, Camilli, & Elmore. Lawrence Erlbaum Associates, Inc., pages 141-160").

Dr. Porter understands what many blind-faith test advocates ignore. "...At the same time, alignment of a test with content standards for only one form of the test should not be perfect. One form of a test is a sample of the content in the standards. If the state uses a different form of the test each year, as it should, then content analyses could be conducted across multiple forms of the test. As the number of forms increases, the total number of items increases. At a point when sufficient numbers of items are present so that the sample becomes close to being the population, the test should be perfectly aligned to the content standards..." (Porter, Page 151)

Unfortunately, students, teachers, and schools don't get the chance to take every form over multiple years until we can be sure that their scores are valid. The fact that they take another test next year and the year after that and so on doesn't help. Driving a car on four misaligned tires is not better than driving on only one misaligned tire.

In the case of NAEP, changes in the test items selected for the administration can lead to misinterpretation of trends over time. In the case of NY's own tests, any improvement may be due solely to the increased alignment of taught curriculum with the tests over time. This doesn't reflect better instruction or better learning, just a different decision about what to instruct. It is called being "data-driven" if you like that process; "teaching to the test," if you don't.

For some state tests, the content of a test may actually narrow over time, even while the overall difficulty remains the same statistically. More difficult to teach content can also be more difficult to test and early efforts to include those items on tests are abandoned in subsequent forms. Then, everyone starts to figure out that the really difficult algebra questions are always on the multiplicative property. What can't be learned from reviewing sample test items can be gleamed from idle chatter in the teachers' lounge.

At the end of chapter 8, Dr. Porter suggests that others might find a good use for the Alignment Index. The Alignment Index is a very straightforward way to show that differences over time are due to differences in student achievement and not differences in test item selection. Without compromising any test items, the Alignment Index provides a metric by which we can defend the validity of an assessment as a consistent measure of the standards over time. It should be run for every test.

Dr. Porter could tell you how many states are currently using it. I will save him the task of devising a careful reply. The choices are: A) Damn few, if any; B) Not enough; C) Who would even dare?; and D) All of the above.

GoodEnuf,
You asked me lots of technical questions that I am not qualified to answer. So I went directly to Andrew Porter--I ran into him recently at a meeting at Education Week headquarters--and he was kind enough to respond to your questions and to give me permission to reproduce his comments. I hope you find them satisfactory.

Diane,
I guess we don't know who the person is who commented on your blog, but she certainly does know some things about some of my work (or he).

I like the person's idea that the content of the test may actually narrow over time. I hadn't thought about that, but it's a possibility. Everybody talks about the curriculum narrowing over time to become better aligned with what is tested, but I hadn't heard of a test actually narrowing. Interesting.

Supposedly, each form of a test is a random sample of items from the domain to be tested. In that sense, then, the extent to which content tested varies from one form to the next should be random as well. This should not affect estimates of the achievement gap. Of course, it is possible that the items for one form differ in some systematic way from items in another form and at least in theory, that could affect estimates of the achievement gap.

A great deal of time and money is spent on equating one test form to another as different forms are used over time. At least in the states that I'm working as chair of their technical advisory committee, all of them are using a different form each year, though there are typically some overlapping items. But, even so, there are difficult issues in getting one form equated to another and in virtually every state where I have been working, there have been times when there was a worry that the forms were not perfectly equated. I see this as a big issue in tracking state-wide achievement levels over time. I see less of a risk that it should affect the size of the estimate of the achievement gap if the gap is defined in the same way over time (e.g. white versus black student achievement). The problem with equating is most severe for the entire sample. I'm guessing that the metric of one form is relatively comparable to the metric of the other form other than some overall mean. I don't know that this is true, but my experience suggests that it is.

Andy

Comments are now closed for this post.

The opinions expressed in Bridging Differences are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Follow This Blog

Advertisement

Most Viewed on Education Week

Categories

Archives

Recent Comments

  • hertfordshire security installers: Greetings. Great content. Have you got an rss I could read more
  • http://blog.outsystems.com/aboutagility/2009/04/challenges-of-scoping-and-sizing-agile-projects.html: I would like to thank you for the efforts you've read more
  • http://acousticwood.net/mash/2008/03/yeah_off_to_the_uk.html: Between me and my husband we've owned more MP3 players read more
  • buy cheap metin2 yang: When you play the game, you really think you equipment read more
  • Nev: Anne Clark - If a Dr. instructs a patient that read more