“The failure of educational systems to integrate research evidence productively into practice impedes progress towards making schools and colleges more effective ….” So state the authors of Practical Measurement, a Carnegie Foundation paper describing tenets of measurement and research that support the improvement of practice.
For the past three years, the participants of Building a Teaching Effectiveness Network (BTEN) have sought to build the type of integrated system of measurement described in Practical Measurement that is so often lacking in our educational systems—one that hews closely to the processes occurring at the ground level of schools, and that supports the day-to-day efforts of practitioners to improve their work.
The BTEN measurement system, designed to support the improvement of processes by which new teachers receive feedback on their instructional practice from school leaders, consists of varying grain sizes of measures to support and spur improvement efforts. The measurement system spans the conceptual space from the micro-level process being improved to outcomes of this process to the ultimate aim of the effort, teacher development and retention.
As a result of the experiences gained through using measures in BTEN, we can take next steps with the ideas originally discussed in Practical Measurement. One of these ideas concerns the purposes of measurement in the improvement context, which we have revisited, augmented, and reorganized, and now propose anew.
Measures in the improvement context are used for:
- Learning About Your System
- Priority Setting
- Testing the Practical Theory of Improvement
- Tailoring Interventions to Individual Participants’ Needs
- Developing Social and Psychological Stances Necessary for Improvement
Enacting widespread and sustained improvements in a system requires knowledge of that system.
Leaning About Your System
Enacting widespread and sustained improvements in a system requires knowledge of that system. Often, practitioners are steeped in their context – their department, their classroom, the role that they play – but they do not see the larger system within which their work is embedded. Knowledge of a system precedes improving that system, and practical measurement can support this type of learning. Furthermore, improvement teams can gain important knowledge about enhancing performance from successful members.
BTEN EXAMPLE
BTEN improvement teams collected information about the frequency with which new teachers receive feedback. In one of the partnering districts, improvement teams discovered that 42 percent of new teachers responding to the survey indicated that they had not received any feedback from their principal, assistant principal, or mentor from September to October. This baseline knowledge provided a reference point from which the improvement team could gauge their success in increasing the frequency and regularity of feedback on instruction.
Priority Setting
Tackling any complex problem requires improvement teams to narrow their focus, to have a place to commit their improvement efforts and resources. Measures in the improvement context can enable practitioners to choose a priority area. These areas may be those of greatest weakness, the “low hanging fruit” for which improvements are likely to be quick and easy, or the area of greatest collective interest among the improvement team members.
BTEN EXAMPLE
When the improvement team at one BTEN school surveyed their new teachers about their feedback experience, they discovered that some teachers in the school felt that the feedback they received from different feedback providers was not consistent. The team decided to focus their improvement efforts on “coordination meetings” where feedback providers met to discuss the new teachers to align their support efforts. In the next survey administration, they saw an improvement in teachers’ responses about the degree of feedback consistency.
A measurement system supporting improvement work provides data against which improvement teams can test their theory of practice improvement.
Testing the Practical Theory of Improvement
A measurement system supporting improvement work provides data against which improvement teams can test their theory of practice improvement. As practitioners make changes in their work processes to spur improved outcomes, they can use data from the measurement system to see if the changes are actually happening and if they are resulting in the outcomes they hoped to see.
BTEN EXAMPLE
The BTEN improvement teams improved their feedback processes by making feedback more frequent, giving feedback providers a conversation protocol to guide the feedback conversation with the new teacher, and establishing coordination meetings among the feedback providers. They tracked these changes with tools that allowed them to see whether these changes were happening. They were able to consider this information in conjunction with teachers’ survey responses to see if these changes led to improved teacher perceptions of their feedback experiences.
Tailoring Interventions to Individual Participants’ Needs
As improvement teams work on improving a process, they are likely to find that the intended beneficiaries of the improvement differ in their starting points, their needs, and how they respond to the changes being tested. Certain individuals or subgroups may be at higher risk of failure than others, or may otherwise need a specific intervention to achieve the outcome sought. Improvement teams can use measures to identify these individuals or subgroups, ascertain their areas of struggle, and craft targeted interventions.
BTEN EXAMPLE
At one of the BTEN schools districts, data from a survey of new teachers provided evidence that teachers entering the system through alternative certification routes were more likely to show signs of disengagement and burnout than teachers who entered through traditional certification routes. This prompted school leaders to consider how they might partner with the alternative certification providers to better support these teachers.
Developing Social and Psychological Stances Necessary for Improvement
When improvement teams collectively engage with data relevant to their improvement work, they can develop a psychological approach and a social dynamic that supports their efforts. These cultural and mindset shifts include the development of a shared language and understanding about the problem being addressed and the theory of improvement; increased will and engagement with the improvement work; and a sense of internal accountability, both personally and within a team, to enact changes.
BTEN EXAMPLE
On this last point about a culture of internal accountability, one BTEN principal explained:
“[The BTEN data] gave us a reason to not make excuses about why we weren’t doing it or how other things got in the way. I guess it sort of prioritized the work for us so that we made a commitment to a certain timeline, we made a commitment to each other that we would stay on that timeline …. I think in that way it kept us accountable.”
As Carnegie moves forward in developing a robust methodology of measurement for improvement, we will no doubt be revisiting and revising this list.