Many school districts are starting to choose common assessments from third party vendors. Some of this is due to new federal or state regulations, and some of this is due to allowing for an assessment that might be more reliable than one created internally. The purpose of this article is to provide practical advice on how to choose an assessment; it is NOT an endorsement of one company over another.
The first aspect to consider when researching assessments is the psychometrics behind the assessment. In other words... How is this test put together? For example, the assessment should be one that contains field tested and leveled questions--the most common being Webb Leveling. Some other questions you should ask include:
1. How do you scale your assessment?
2. Can we get the Pvalue of each question?
3. Does the assessment correlate to the state assessment?
4. Is a metric being used in the background so that the assessment will be tailored to the student dependent on their answers? If so, what goes into the metric? The time it took students to answer to each question?? Right answers?
Another area to review when selecting a third party assessment is its alignment to the Common Core or your state's standards in a given subject area. Such an exam will help students and teachers identify areas of strengths and areas in need of improvement. Some guiding questions are:
1. Is there an item map that shows the alignment between the test questions and Common Core or state standards?
2. Who did the alignment of the questions and what is their background?
3. What resources are available to support teachers as they identify students' needs and develop interventions?
The assessment is only the first piece of an effective assessment plan. A second piece to the puzzle that is just as important is related to data analysis. Teachers and administrators will need easy to read and understandable data analysis; therefore, asking the right questions to third party vendors is essential. Some questions to ask include:
Is there a way to get the results out of the system via Excel or other flat format?
Does the system come equipped with CBM (Curriculum Based Measurement) reports?
How is RTI supported through the use of the assessment?
Is there a model report accessible to customers to review prior to purchasing the system? Are the reprorts available in color? If so, are the colors standardized throughout the tool? For example, does red consistently indicate an area in need of improvement?
Is there a parent portal or a parent component for school to home communication about student progress?
How soon are reports generated after testing?
Is a benchmarking to other clients or to national norms provided? How many records is the benchmarking based?
Nuts and Bolts
Just as important as all of the above is the backend of the system. You need to make sure that it will fit into your current technology. You will also want to inquire about the company and their history in the industry.
How do we get the students' information into the system? Is there any automated way? SIF, Web folder, auto extract, etc.
Is your solution Web based or Web enabled (Enabled means it is mostly Web delivered but you may need to install certain files on each computer)?
Can students take the test on both Mac and PC?
If it is Web based, then what browsers work the best and what plugins are necessary?
Will the assessment and management of the reports be accessible and editable on a tablet and/or IPad or Iphone?
Can you provide references? Can we do a site visit to any of them?
How long have you been in this specific marketplace?
The last part that needs to be addressed is the actual implementation of the assessment. Teachers who administer the test will have more confidence doing so if they can experience the tests firsthand. Finally, knowing what the timeline is between signing a contract and obtaining the product should be discussed and confirmed.
1. How is training provided? Onsite training or Web ex? How much professional development is included?
2. What are the credentials of trainers?
3. How quick after the contract is signed should we expect to be up and running?
4. What kind of contract do I need to sign? Make sure the contract does not allow the company to use your district in any promotional material.
There are several aspects to consider when researching third party assessments. The questions noted above provide a glimpse into the types of inquiries educators may make when faced with the decision to purchase of an assessment. Companies in the assessment field are competitive, and educators do not have to settle for anything less than what is best for their districts. Asking challenging questions will help narrow the selection process; thus, you will be well on your way to picking an assessment that will fit the culture and needs of your district.
James Yap and Dr. Teresa Ivey