Since our spring launch of reviews that evaluate the alignment and usability of instructional materials to the Common Core State Standards, we have heard from educators, publishers, and members of the mathematics community about the strengths in our reviews and their suggestions for how to improve the reports to best serve the needs of the field.
We spoke with many educators who are using our reviews to help inform their purchasing decisions and inform how they utilize their current textbooks. They have told us that they want even more information – such as more evidence on how materials were scored against specific indicators, more details about strengths and gaps for key focus and coherence categories, and more information about how publishers developed these materials. We also heard that the additional information provided cannot compromise our high bar for alignment and usability.
EdReports.org staff and the educators who designed our tool took all of these suggestions into consideration and identified enhancements to the website and the review process. On June 5, our Board of Directors approved these refinements to our review tool, methodology and reporting protocol:
Review more materials for rigor and mathematical practices. In our first set of reviews, instructional materials were analyzed for mathematical practice and rigor only if they met our expectations for focus and coherence alignment. Beginning this summer, and including materials that are already on the site, we will review materials that partially met expectations for focus and coherence (gateway 1) against our criteria for mathematical practice and rigor (gateway 2). Materials that do not meet expectations for focus and coherence will still not be reviewed for rigor and mathematical practices. This will provide more information and honor what we heard from the field: namely that we need to keep a gateway system to ensure that the right content, in the right balance, is present in materials for which we will review for rigor and the mathematical practices.
Our teams will review all materials that were found to partially meet expectations for focus and coherence (8 points or higher in gateway 1) for rigor and mathematical practices, including the 13 texts that are currently on the site.
Revise evidence collection and scoring for indicator 1a. As a critical indicator for focus, this aspect of EdReports.org reviews looks at whether students are assessed and held accountable to future grade-level standards. In order to better identify and make determinations about the significance of the inclusion of future grade-level work, we are creating even higher standards for the amount of evidence for this indicator and providing further details on the rationale behind the final scores. Our methodology and tool will be updated to show additional steps our reviewers take, including how they will:
Bolstering the evidence we provide will ensure our analysis reflects the deep conversations reviewers have to calibrate scores fairly within and across teams. We will look again at the 46 texts that did not score full points on 1a in the first set of reviews and also apply this standard moving forward.
Revise website with more detailed visuals and published evidence guide. We are enhancing our rating visuals and the website to include more detail and illustrate the relative range of possible scores within our reviews. This visual refinement will provide a better sense of the strengths and gaps in each series at each grade level and will be updated in July 2015. We are also publishing our evidence guides. These guides will help educators to better use our tool and methodology to conduct their own reviews and to understand how EdReports.org reviewers conduct reviews, including how they collect and consider evidence of all of the content standards to make determinations about focus and coherence.
Provide more series information from publishers. We will invite publishers to share more background information on the mathematical foundations upon which their materials were developed. We also will offer additional space on our site for publishers to share some of the supplementary services and any evidence of effectiveness they may provide. We know these additions will help educators make the best of the instructional materials already in use in their school/district.
Our ratings system will continue to send a clear signal to the field about the primacy and importance of focus and coherence in instructional materials while staying adaptive to educators’ needs.
EdReports.org’s goal is to provide the field with more information about instructional materials and alignment in order to supplement the professional judgment of educators across the country. These revisions will help provide even more, updated information about our reviews and the rationale behind our ratings so that districts and educators can complement their own professional judgment and make informed decisions on which instructional materials will best serve their teachers and students.