Introduction

One feature of practical measurement that makes it distinct from measures that are used for other purposes is that it is “up close and personal” – measures are tied to the actual processes that are the focus of change efforts. This means that if you are trying to improve instruction, you’ll likely want to identify measures that will help you understand the effectiveness of specific classroom-level practices. However, in accountability-focused environments, practitioners can be apprehensive about collecting and sharing data that is so closely tied to their work. To counteract these sentiments, improvement leaders need to be intentional about creating an organizational culture and the associated social processes to foster an environment that encourages honest reflection and learning from data.

The Need for Measurement

The Teaching Matters Network for School Improvement (NSI) is a network of middle schools focused on integrating culturally responsive practices. Utilizing continuous improvement methods, the network’s aim is to increase the number of students who end 8th grade on track for high school and beyond. To meet this goal, the network is focusing on practices that build community, affirm identity, and deepen learning in order to improve literacy (see Figure 1).

Figure 1. Teaching Matters NSI Driver Diagram

The Teaching Matters NSI wanted to understand the extent to which such practices were enacted in classrooms from the students’ perspective. Incorporating student voice would be critical for helping the network as sess the impact of their changes in this area and make progress toward their overall aim.

The Measure: Inclusive Classrooms Survey

After considering a few different options, the network decided to utilize questions from a survey developed by Panorama Education – Panorama Student Survey. The Panorama is a nationally normed survey instrument focused on school climate, teaching and learning, relationships, and belonging. The Teaching Matters NSI selected Panorama because the vendor was already approved by the New York City Department of Education, reducing hurdles to implementing it in network schools. To ensure that this measure met their goals, network leaders selected survey items across six domains related to inclusive classroom practices. Network leaders deliberately chose questions focused on classrooms, not schools (e.g., rigorous teacher expectations instead of schoolwide expectations), and students are asked to report their perceptions of their language arts (ELA) teacher and classroom. 

Two key features of measurement for improvement are that the measure is 1) tied to a working theory of improvement and 2) specific to the processes that are the focus of change efforts. Utilizing the Panorama allows the Teaching Matters NSI to track the impact their efforts are having in fostering inclusive classroom practices. Additionally, Panorama provides for the delivery of the survey and displays an analysis of the results in a dashboard tailored to specific roles (e.g., teacher, school leader, and network leader). This reduces the burden on network and school staff to administer the measure and process data, which allows for results to be reported in a timely manner and frees up the hub team to invest in additional analyses, such as identifying bright spots. Finally, and perhaps most importantly, to ensure that data from the Panorama actually transforms practices within schools and classrooms, the network has developed social learning routines and resources that help network members learn from each other.

Insights from Using the Measure: Creating a culture that fosters learning from data 

The network intentionally creates a social environment that encourages honest reflection and learning from the Panorama data among teachers. Network leaders noted that because Panorama data is reported based on individual teachers and their classroom practices instead of an entire school, it can be difficult for teachers not to take this feedback personally regarding their instruction and how students perceive classroom culture. “It’s not saying that anyone is a bad teacher,” Network Director Jacobē Bell reflected on the challenge of using teacher-level data to drive improvement efforts. “How do you take the ‘but I work so hard’ out of it? Yes, you work hard, and you’re lovely, and this is what your kids are saying.”

Network leaders needed to shift the culture and lay the groundwork for teachers to collectively engage in data analysis by creating a supportive environment focused on two key questions: What are our students telling us? and What can we learn from each other? At the school level, teachers and school leaders work with coaches to analyze Panorama data using a protocol that helps teams determine actionable next steps and change ideas to support areas of need. To support learning across the network, the hub uses a P-chart model to analyze variation across each domain in order to discover bright spots and unearth best practices (see Figure 2). 

Figure 2. Sample P-chart for “Classroom Belonging” Domain

The hub team reviews these charts with coaches, focusing their conversations on specific schools or teachers that are getting positive results. The hub team visits identified schools and classrooms and interviews teachers to unearth what is working well. They capture these strategies in videos and blog posts that are shared on the network’s online platform and during convenings. (Check out this video and summary of a bright spot visit in one of the schools in the Teaching Matters NSI.) By integrating knowledge management practices focused on elevating effective strategies, the hub team hopes to accelerate learning across the network around best practices for building inclusive classrooms.

Impact of Measurement for Improvement

Using Panorama as a measure to evaluate the Teaching Matters NSI’s efforts to advance inclusive classroom practices encourages network members to keep students at the center of their work. Students’ perceptions of their classroom environment guide the creation of change ideas and prompt deep reflection from teachers on their practice. Use of this measure has also shifted mindsets that teachers in the network had around using data. As conversations related to Panorama data focus on learning and not on accountability or evaluation, The measure has fostered a culture that is supportive of improvement work. Teachers have moved beyond the initial apprehension they might have felt about sharing data about their teaching practices and classroom culture and are building mental muscles to meaningfully engage with the data and allow it to inform their practice.