Are You Making Good Data-Driven Decisions?
Today's guest post is written by Steve Ventura, an educational consultant and former school superintendent.
If you are a teacher or leader, you know that "using data" has become one of the most emotional issues in all of education. Many organizations profess that they are "data driven" but what does that mean? Ask any teacher from a "data driven" district about the numbers and they'll tell you straight up-"teachers do not want more data...they want better data!"
How do we know that we are collecting and analyzing the right data? What is good data and why is it important to consider this? Well, when the results of an assessment are reliable, we can be confident that repeated or equivalent assessments will provide consistent results. This puts us in a better position to make generalized statements about a student's level of achievement which is especially important when we are using the results of an assessment to make decisions about teaching and learning.
Factors Affecting Reliability and Validity
It's important for schools to calculate effect sizes to determine not only achievement, but progress. An effect size measures the impact of a particular influence or strategy on learning achievement. It is a calculation that is relatively easy to undertake and helps compare different influences on the same scale (Visible Learning, 2009).
In terms of reliability, there are several things to consider:
- Length of the test
- Consistency in administration
- High quality questions stems with probable distractors (answer choices)
- Alignment of the assessment to the task
Can you think of a situation in your school where a test was not reliable? What are the consequences of assessment results if the administration of that assessment is not consistent? If an assessment has very low reliability, it will also have low validity.
If high stakes testing and assessment are to be used as feedback to teachers and leaders, then we have to ensure that tests are reliable and valid. If not, we can never make accurate inferences about student achievement. One thing is for certain - if the data we collect is not being used to inform learning, then why are we gathering it?
Pre-Assessment 101: Why and How
As educators continue to become better consumers of assessments, there is a need to identify and define the purpose of pre-assessment:
- a true pre-assessment measures the same knowledge and skills that will be measured on the post-assessment, is in the same form, of the same length, given under the same conditions.
- in order to be able to truly compare a pre and post-tests, they must be comparable in terms of difficulty, length, and manner in which they were administered.
- if the assessment is not based on knowledge or skills that progress in a linear manner, such as reading fluency, pre-tests may not be appropriate.
- pre-and post tests are based on what is going to be taught
The most common concerns I hear about pre-assessment are as follows:
- If aware that a pre-test "doesn't count" students may consciously or unconsciously underperform, thus deflating pre-test scores.
This concern can be diverted by explaining to students the purpose of pre-assessment: to gauge the teachers's ability to prepare and adjust instruction. Teachers should explain to students that this information is critical to both teacher and student success.
- If the post-test too closely resembles the pre-test, scores may be inflated due simply to familiarity and exposure.
This may be a valid concern. James Popham describes this as pretest reactivity (Popham, 2003, p. 152). I might suggest, however, that the reason a student performs better on a post assessment than a pre-assessment is not simply because they have seen the assessment. I recently turned 50 years old (well, more recently, 57) and I still encounter problems that I have seen dozens of times over the course of my life. I still can't solve them.
Students may perform better on post assessments because they received purposeful and aligned instruction in between the two assessments.
With an understanding of the assessment criteria listed above, it is possible to calculate an effect size for teacher made assessments, as long as they are valid and reliable.
The Visible Learning Research and the Role of Effect Size
Many schools and districts are familiar with the work of Professor John Hattie, especially when it's time to discuss what matters most. His understanding of the most powerful influences on student learning is based on evidence from 800+ meta-analysis of 50,000 research articles, 200 influences and over 250 million students.
It is possible to calculate effect sizes for not only the teacher, but for individual students. In this manner, we are able to discuss not only achievement, but progress. After all, if we can't measure our magnitude as teachers, how can we react to the impact we are having?
Effect sizes involve two sets of data from the same students, comparing one set with the other. We call these two sets Time 1 and Time 2. Effect sizes can be negative or positive and the typical effect size is 0.4. This is the progress that we should see in a year for our students. Data that shows progress (effect size) can reveal a different story from data that shows achievement.
For example, a student who scores an 85 on a pre-assessment and a 90 on a post assessment has high achievement, but low progress. An important caveat to consider-when using teacher made assessments, the hinge point of 0.4 may be less relevant. This is because students have little or no prior knowledge on the pre-assessment and considerably more knowledge on the post-assessment. Other causes might be an easier assessment at the start and then a comparatively difficult assessment later on.
The challenge is-what story does the data tell? Student achievement data is often reported for whole populations, like cohorts, whole classes and grade levels. This is commonly referred to as aggregated data. It is not until the data is disaggregated that trends, patterns and other important information are uncovered.
Reflecting on Your Own School Data
The ultimate requirement is for teachers and leaders to develop the skill of evaluating the impact they have on their students. It's easy to slip back into our default mode and keep doing what we have always done. The challenge is to develop plans that will build new learning!
For more information on effect size calculations, please visit this link to increase your understanding about effect sizes and how to calculate and interpret them.
Connect with Steve Ventura on Twitter.