Editor’s note: The Carnegie Foundation’s fourth annual Summit on Improvement in Education is March 27-29, 2017. Over the course of three days, teachers, school and district leaders, researchers, foundations, and innovators will discuss their successes and challenges, and offer tools and tips related to effective school change. This is the fourth post in a series discussing key insights and reflections from Summit presentations.

The debate over the usefulness of once-a-year standardized test scores to measure student, teacher, and school quality has been winding down in recent years, as the annual, high-stakes exams are losing favor as the sole arbiter of achievement. That leaves a question with a complicated answer; what other factors should be included to provide a comprehensive assessment that takes into account differences in school resources, community health, student backgrounds, and year-to-year growth?

After grappling with this issue, California’s Oakland Unified School District developed a school performance framework (SPF) to give schools a detailed look at what’s working, what’s not working, and for which students. The SPF assesses schools on up to 17 indicators (depending on grade level) in both academics and culture/climate. Schools receive an annual report that shows the school’s current status and its growth from the previous year for each indicator. Eliminating inequity in the district is a critical aim of the SFP, so results are further disaggregated for specific groups, including English language learners, students with disabilities, low-income students, and the lowest performing racial or ethnic group in that measure.

The district had a soft rollout in the 2015-2016 academic year, followed by a formal launch in the fall of the current term. In between, district officials fanned out giving presentations at school professional learning sessions, staff meetings, parent meetings, and school site council meetings (a decision-making group at each school representing parents, teachers, and other stakeholders).

“A lot of what we’ve been doing this fall is trying to communicate [about SPF], to give people a baseline understanding,” said Corey Donahue, the school performance coordinator in Oakland Unified’s Office of Continuous School Improvement.

Donahue will also be providing information about the program during a session on district-led improvement methods at the upcoming 2017 Carnegie Foundation Summit on Improvement in Education. He said the SPF evolved from the district’s school balanced scorecard (PDF) and a school quality improvement index developed by the California Office to Reform Education, a coalition of districts working on school improvement, but the SPF offers a more comprehensive and holistic view of how the school is performing. Growth over time, for example, is important because it not only shows how a school is doing now, but where it’s coming from.

“It’s really valuable for schools that otherwise would be seen as lower performing, but have shown growth,” explained Donahue. “In showing the culture/climate measures of the school, you get a better sense of how the school is treating its students beyond just the academic standardized test scores.

Academic indicators include scores on both the Scholastic Reading Inventory and the Smarter Balanced assessment in math and English language arts and high school readiness (for middle schools) based on student grade point averages, attendance, suspensions, and failure rates in core academic courses. High schools are measured by college, career, and community readiness indicators, including graduation rates, percentage of students on track to graduate, enrollment in college-level courses (such as advanced placement) and career pathways programs, and the number of students who have completed the courses required for admission to the University of California and California State University systems.

To get a picture of the culture and climate of a school, the SPF tracks chronic absence rates; suspension rates; surveys of students, parents, and staff; reclassification rates for English language learners; and surveys asking students about their social-emotional learning and attendance rates.

The district is developing strategies for using the results to make decisions about budget priorities and establishing guidance on when the district will intervene. “We want to be able to reward schools that are performing well with the kind of autonomies that they might not always get from the district level, while at the same time, better support schools that have not seen growth,” said Donahue. The goal is that schools will also use the reports when setting their priorities for the following year and to better communicate with staff and families. The district has developed a guide (PDF) for school site councils that describes six steps for using the results to pursue continuous improvement.

1. Review and analyze student outcome data;
2. Identify critical questions related to the efficacy of the school’s program strategies;
3. Research effective-practices;
4. Establish student outcome goals and benchmarks;
5. Establish a data-informed theory-of-action that leads to the achievement of goals; and
6. Establish a resource allocation plan that is aligned with the theory-of-action.

Some schools are already digging into the data. “The SPF has been a powerful visual tool to use with my staff and my parent community,” said Chelsea Toller, principal of Glenview Elementary School, which has a very diverse student population spanning from high income families to those that are homeless, as well as newly arrived immigrants. “The SPF pulls apart the various forms of data in a visual way so that all of us can understand the access and achievement gaps in its many forms.” As a result of having this deeper level of information, Glenview’s PTA shifted its priorities this year to fund a reading intervention specialist.

Because the SPF is an annual measure, it’s not sufficient by itself to use for continuous improvement, but does fit the improvement science principle of seeing the system and helping districts identify their problems, which are necessary steps, said Donahue. He does want to expand its use by having schools conduct root cause analyses after setting their priorities to dig into why they’re getting poor results, if that’s the case. In order to accelerate the process of implementing and evaluating changes, the district has also added key performance indicators to the mix this year. These are weekly engagement reports on each school’s rate of chronic absences, suspensions, and students sent to the office. Less frequently, but still several times a year, the district sends out school assessment results.

Ideally, Donahue wants schools to quickly identify, implement, and evaluate interventions based on these reports. For example, if a school’s weekly reports show ongoing problems with chronic absenteeism, the attendance team would identify the most chronically absent students and decide to spend the next month visiting their homes to talk to the students’ families. At the end of the month, the team would review the latest key indicators report to see if the activity worked and should be expanded or didn’t work and should be replaced with a different intervention.

“In terms of ongoing evolution, we are definitely looking for ways to make SPF better and to make it more aligned with the priorities of our schools,” said Donahue. “What we’re hoping to be doing is developing a system in which people have measures at their fingertips that could help them continuously improve.”