« The Vision Vacuum | Main | Thinktank Thursday: Fordham's Boy Band Breaks Up! (Inside the Music) »

Nobody Beats the Biz

| 5 Comments
bee-thumb.gif
When you're looking for measured, careful, and thoughtful analysis, there is no better blogger to turn to than Dean Millot at edbizbuzz. He's consistently able to take complex debates and lay out the issues raised in an incisive and even-handed way. Millot follows up on the thinktank/peer review debate here, and promises another post tomorrow. Stay tuned.

Update: Dean Millot posts his second installment.
5 Comments

Dean Millot is being fundamentally dishonest in that he misquotes me. He says that I argue: "In short, I see no problem with research becoming public with little or no review.”

In fact I wrote: "In short, I see no problem with research initially becoming public with little or no review." (See http://jaypgreene.com/2008/07/08/eduwonkette-apologizes/ )

The absence of the word "initially" makes quite a difference and sets up the straw man that he wishes to knock down. The issue is not whether research can benefit from peer review, but whether it is inappropriate to make it publicly available INITIALLY, before it has received peer review.

So much for Eduwonkette's praise of Millot's "measured, careful, and thoughtful analysis."

Hi Jay,

Three things:

1) With all due respect - though in this case Millot's post was specifically about your original post - this debate is not about you, and it puzzles me that you continue to see this as a referendum on Jay Greene. It's not. You're not the first and will not be the last to release non peer-reviewed research with a large PR office behind you. If you read back on Millot's blog since it started, or on Alexander Russo's blog, or on my blog, this is an ongoing debate, one that far predated your entry into the blogosphere.

2) I seriously doubt that the word "initially" was left out intentionally, and I don't think the issues at hand are different even if research is ultimately peer reviewed. What paper have any of us ever put through the peer review process that doesn't result in our conducting new sensitivity analyses, answering new questions, and as a result, presenting a slightly more nuanced storyline? Did your follow-up peer reviewed articles land themselves back in the press?

3) I suggest that you take some time and read back on Millot's blog, because he has a tremendous body of careful and systematic work up that make your claim that he is not "measured, careful, and thoughtful" a canard. That reputation extends well beyond the omission of one word - a word that doesn't even change the meaning of what you wrote - so you are really grasping at straws here to personally discredit those writing about a topic you'd rather see left untouched.

I suggest that readers check out this excerpt from Millot's post, and then read the whole thing:

"Greene argues that he was victimized by a strawman I set up last week. In his words, I asserted that Greene believes “research doesn’t benefit from peer review.” Readers will have to decide that for themselves by re-reading my post. I suggest that in his initial effort to swat down eduwonkette, Greene got cocky and overstated his own position. It is hard for him to say he does see some problems with his research for Manhattan becoming public with little or no review. Claiming to be the aggrieved party is his best chance of walking back the cat. But in making this claim, Greene is the one setting up the straw man.

I doubt any other edbizbuzz reader inferred that last week’s Letter From was intended to address the vast spectrum of education “research.” I think a reasonable person would understand that my comments were prompted by, and addressed to, a specific situation – the release of a study offered to the media, and by implication to policymakers, with all the fanfare of research subject to peer review, but without the peer review.

In that context, I addressed Greene’s statement of having “no problem with research becoming public with little or no review.” I did not say that Greene does or does not believe in the value of peer review as a general proposition. I suggested he ought to have a problem with this specific fact-pattern because 1) post-hoc review by the market is not a reliable means of quality control, and 2) there are no compelling reasons to forego the review process.

I’m arriving a bit late to this conversation, and I want to be careful not to simply repeat what’s already been said by Dean Millot, Sherman Dorn, Eduwonkette or the other posters here.

In Eduwonkette’s original post ( http://blogs.edweek.org/edweek/eduwonkette/2008/07/the_trouble_with_the_education.html ), she linked to a published review of an earlier report authored by Greene and Winters: “even when researchers working in the policy advocacy industry make sloppy, indefensible errors - for example, when Greene and Winters used data that the Bureau of Labor Statistics warned against using ( http://epicpolicy.org/thinktank/review-how-much-are-public-school-teachers-paid ) to show that teachers are overpaid - they're not approached with caution by the press when the next report rolls around.”

That review, written by Professor Sean Corcoran (NYU), was part of the Think Tank Review Project ( http://thinktankreview.org ), which I co-direct. Our project has reviewed, over the past three years, four different reports from Greene and Winters, offering some praise but also documenting errors: overstating effects, omission of key information, weaknesses in data and analyses as well as research design, unsubstantiated assumptions, poor use of existing literature, and (in the instance noted by Eduwonkette) inappropriate use of a database. Comparable mistakes have been found in most other think tank reports.

I’ll briefly note here that I see our Project, as well a comparable project started recently by the What Works Clearinghouse ( http://ies.ed.gov/ncee/wwc/quickreviews/index.asp ), as being part of a dialogue rather than as some sort of objective final word. In fact, that’s how I also see the blind peer reviews that I receive from journals, regarding my own work.

Here are the urls for the four “Think Tank Reviews” of Greene and Winters reports:
http://epicpolicy.org/thinktank/review-effect-of-special
http://epicpolicy.org/thinktank/review-getting-ahead-staying-behind-an-evaluation-floridas-program-end-social-promotion
http://epicpolicy.org/thinktank/review-how-much-are-public-school-teachers-paid
http://epicpolicy.org/thinktank/review-getting-farther-ahead-staying-behind-a-second-year-evaluation-floridas-policy-end-s

These reviews should not necessarily be taken as a criticism of the authors’ scholarship. If anything, it’s a criticism of the publication process used by think tanks. Most of us who publish our research have been humbled when our mistakes are pointed out in the peer review process, but we’ve also been relieved that those mistakes were identified before actual publication.

I’m also sympathetic to Jay Greene’s timeliness argument; the peer review and pre-publication process for many academic journals can take literally years. But think tanks could set up their own, streamlined peer review process, as I believe has been done at the Hoover Institution’s Education Next ( http://www.hoover.org/publications/ednext/about ). I believe that non-advocacy think tanks like Rand and AIR have also set up rigorous (although usually internal) review processes. My own policy center created, about three years ago, a streamlined peer-review process for the policy briefs that we release ( http://epicpolicy.org/publications ), and I’ve never regretted the decision. Even our think tank reviews go through an abbreviated peer-review process.

The other part of this conversation -- concerning the role of the press in reporting on different studies -- is also of great interest to me. I hope to add to that discussion here soon.

"In short, I see no problem with research initially becoming public with little or no review."

Wanting to invoke "initially" as the key word, but still sending out a press-release about the work seems to me to be wanting to have your cake and eat it too.

If it's "a preliminary result" it's a working paper tossed out to the world to have holes poked in it. If it has a press-release with it, most people would take it to be in its final form.

I'll also toss into this discussion a quote from physicist Juan Collar that impressed me when I ran into it on at Cosmic Variance (http://cosmicvariance.com):

I try to teach my students that a good experimentalist does not need any critics: he or she is his/her own worst enemy. If you don’t feel a sincere drive to debunk, test and revise your own conclusions, you should be doing something else for a living.

Comments are now closed for this post.

Advertisement

Recent Comments

  • Rachel: I'll also toss into this discussion a quote from physicist read more
  • Rachel: "In short, I see no problem with research initially becoming read more
  • Kevin Welner: I’m arriving a bit late to this conversation, and I read more
  • eduwonkette: Hi Jay, Three things: 1) With all due respect - read more
  • Jay P. Greene: Dean Millot is being fundamentally dishonest in that he misquotes read more

Archives

Categories

Technorati

Technorati search

» Blogs that link here

Tags

8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability
accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
admissions
AERA
AERA annual meetings
AERA conference
AERJ
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
ATR
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
D3M
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
DC
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
desegregation
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
dropouts
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
ETS
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
IES
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
McGraw-Hill
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
NCLB
neuroscience
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
Pearson
picking a school
press office
principal bonuses
proficiency scores
push outs
pushouts
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
skoolboy
small schools
small schools in New York City
social justice teaching
Sol Stern
SREE
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing
testing and accountability
Texas accountability
TFA
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
Tweed
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-addded
value-added
value-added assessment
Washington
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School