Measuring teacher quality is complicated. In recent years, value-added measures that are based on an analysis of student growth (typically determined by test scores) have received the most public media attention. But an important component of teacher evaluation systems—and the one often given the most weight in calculating teachers’ evaluation scores—is also the oldest and most common: observations of teacher performance.
Through observing a teacher’s practice, evaluators are able to see instruction in context and in real time. And unlike value-added measures or other test-based assessments of student growth and achievement, classroom observations can be used to assess all teachers, irrespective of grade level or subject. Perhaps most importantly, if done well, observations not only give an indication of the quality of teaching, but also can provide precise and constructive feedback that helps teachers understand and improve their practice, no matter their skill level.
For these reasons, observations of teacher practice are critically important to ensuring quality instruction. Yet not much is known about how observers themselves are trained, supported, and evaluated. What do observers need to learn, and how effective are their training opportunities? The Carnegie Foundation’s latest policy brief, Improving Observer Training: The Trends and Challenges, explores these questions. The third brief in a series examining trends in teacher evaluation, the report details findings both from recent research on observer training and from conversations with experts from district officials in five districts across the country: Boston, MA; the District of Columbia; New Haven, CT; Santa Fe, NM; and Maricopa County, AZ.
What training do observers need?
In order to do their jobs well—to accurately assess teacher quality and to provide helpful feedback to teachers—observers need to know how to identify evidence of teacher practice according to their district’s evaluation rubric, how to interpret that evidence to arrive at reliable and accurate scores, and how to frame and deliver productive feedback for teacher improvement.
Districts are training observers to be more evidence-based when watching and recording what happens in the classroom. This means that, instead of trying to take in everything they see and synthesize a final judgment based on those impressions, observers are increasingly instructed to identify specific teacher and student behaviors that may correspond with strands of the rubric. As one of our interviewees put it, observers now must “ground their observations of teachers in evidence rather than just making normative statements like, ‘The teacher has good classroom management.’” Instead, using examples, observers must define what “good classroom management” actually means in terms specifically aligned to the district’s rubric.
A significant challenge to training observers is reducing variability in scoring.
A significant challenge to training observers, research shows, is reducing variability in scoring. Districts are devoting significant time to calibration—the process of improving scoring accuracy and consistency by checking and adjusting ratings through comparison with standard scores. Districts vary in their approaches to calibration. Some have observers individually score videos of teacher practice and compare their scores to a “master” or “anchor” standard score; others have groups of observers watch a lesson in real time and debrief afterwards before comparing to a standard score. But all districts view ongoing calibration as essential training, as do some states, meaning that some observers even get a double dose of this support.
Training on how to give actionable feedback to teachers is less comprehensive, but the districts that are not currently offering significant opportunities in this area plan to do so in the near future. Research shows that conversations about teacher performance can be emotional or even threatening to teachers, if observers are not adequately prepared to conduct these meetings in a collegial and productive manner. But the feedback process is critical to the success of any observation system; it is where observers translate what they have seen of a teacher’s practice into recommendations for actionable improvements.
What challenges are districts facing?
Districts must consider not only what observers need to learn, but also how training sessions should be conducted. Many districts have moved some components of training online, for example, both to reduce costs and to make training more convenient for observers, given their often packed schedules. In-person training is now often collaborative; observers in many districts calibrate scores together and share with one another what they have learned from their own practice.
A major challenge for districts is the capacity to fund and execute quality training.
A major challenge for districts is the capacity to fund and execute quality training opportunities for observers. All five districts highlighted in the report have used or currently rely on outside support of some kind—grants to build online training platforms, ready-made training materials, or training facilitators from external providers—a practice that may not be sustainable over the long term. Even with external support, most districts struggle to find the time and staff they need. Some are creating new positions at the district level to help conduct trainings, while others are shifting existing roles so that people share training responsibilities.
And though training of observers is crucial to ensuring high-quality observation, which in turn is key to improving teacher quality, not much is known about the effectiveness of observer training. Districts can and do collect qualitative feedback, such as observers’ satisfaction with training opportunities or teachers’ perception of the usefulness of observations. But the field has yet to connect observer training to student outcomes. Until this information is measured and analyzed, districts will likely struggle to make true improvements to observer training.
Among the five districts profiled in the brief, the trend is to add more and more training opportunities for observers. This makes sense, given how much observers must know and be able to do in order to do their jobs well. But, in addition to the challenges described above, districts must keep in mind the purpose of observation—to accurately gauge teacher quality and to provide feedback for improvement—to make strategic training decisions.