Introduction

Key characteristics of practical measures are that they should be minimally burdensome and relatively easy and quick to use. However, whether a measure is seen as “practical” depends somewhat on who you are asking. In particular, front-line practitioners are often very constrained for time and reluctant to take on “one more thing.” To collect essential data from practitioners to inform improvement efforts, it can help to get a little creative. The Society for Improving and Measuring Procedural Learning’s (SIMPL) performance assessment app (SIMPL OR) is an example of how technology can be used to extend the possibility of what is considered feasible when collecting data from busy front-line practitioners like attending, or supervising, surgeons.

The Need for Measurement

As a quality improvement collaborative focused on investigating and developing tools, curriculum, and policy to improve the training of physicians, SIMPL first addressed gathering data on surgical trainee performance over time. Most assessment in surgical training is too infrequent and does not capture the full range of procedures and learning experiences a trainee engages in. This gap highlighted a key area to address in order to improve overall trainee performance: increasing the data and feedback trainees received on their developing skills. Given the time constraints that both training and attending surgeons faced, the SIMPL Collaborative faced a critical and common challenge for those involved in change efforts focused on improving practice: How can we assess and provide feedback to trainees in a rapid, easy way?

The Measure: SIMPL OR (Smartphone-Based Performance Assessment App)

The team developed a smartphone-based formative assessment tool that consists of three questions. Using three separate rating scales, attendings rate the overall complexity of the case, amount of autonomy given to the trainee, and how well they performed the focal part of a procedure. If the supervisor would like to provide information beyond the three questions, he or she can also dictate feedback that the trainee can later listen to. Based on the data collected from evaluations, the measure provides simple data displays so that trainees can quickly see how many procedures they have completed and their performance, and it also allows trainees to review feedback from supervising physicians.

Figure 1. Screenshots of SIMPL OR app

Whether utilized in a hospital with training surgeons or in a school with beginning teachers, measures for improvement must be minimally burdensome and timely. SIMPL OR utilizes what most people have readily available — their phone — and provides a quick and simple way for trainees to request and review feedback. Supervisors are only required to complete three questions, which they can do on a short elevator ride or over lunch. This system allows trainees to receive feedback quickly and signals changes that they might make to improve their performance in future tasks. Another key feature of measures for improvement is that they should be perceived as valuable to the target users. Both training physicians and their supervisors are invested in seeing trainees develop key skills and improve performance over time. The SIMPL OR app creates opportunities for more frequent, regular feedback to be shared with trainees that can then be applied to their practice.

Unlike many performance evaluation and feedback systems, the SIMPL OR app provides timely information through a low-stakes approach. Many teacher evaluation systems, for example, rely on classroom observations that take place once or twice a year. Given that these evaluations are often tied to compensation or certification decisions, the lack of frequency creates a high-pressure environment for teachers and also reduces the usefulness of the feedback provided. Illustrating key characteristics of practical measures, the SIMPL OR app promotes the use of many brief, frequent evaluations to better understand a trainee’s performance and provide more actionable feedback.

Insights from Using the Measure: Embedding measures into the system and daily practice

Even with a relatively simple tool, a common hurdle to measurement in improvement efforts is actually getting people to use the measure. The SIMPL OR app is trainee driven — after completing a procedure, the trainee requests feedback from their supervisor in the application. As might be expected, the team found that while some trainees frequently requested feedback, others requested very little. This reluctance to seek feedback was often influenced by social and cultural norms in a particular setting.

The team noticed that hospitals that had higher engagement or utilization of the SIMPL OR app had found ways to embed the use of the tool into their daily practice. For example, training surgeons are required to log their procedures for certification purposes, but there is not typically a record of how well they performed the procedure. Some hospitals mandated using the SIMPL OR app for logging cases. This created a uniform system for tracking the procedures each trainee performed. Additionally, because logging a procedure in the SIMPL OR app triggers a request for feedback, requiring its use supports a cultural shift in the institution regarding the value of feedback and makes it part of its normal way of operating. The team identified this type of integration within institutional structures and processes as critical in their efforts to scale the use of the SIMPL OR app.

Impact of Measurement for Improvement

The work of improvers is never done, and collecting key data can point a team toward other areas to focus on. Beyond just increasing the feedback that trainees are receiving, the use of the SIMPL OR app has launched other initiatives aimed at improving the training of physicians and their resulting practice. For example, given the variability of ratings across supervising physicians utilizing the SIMPL OR app, the team began to wonder what was driving this variation. This has led to new improvement opportunities focused on the following questions: As raters, do we agree on what “good” performance is? What are we actually measuring? A clear understanding of the expectations for performance can help shape training experiences and the type of feedback that trainees receive. Additionally, a teaching and learning committee is selecting and formalizing strategies that supervising surgeons will do with trainees; they are using the SIMPL OR app as an outcome measure for their interventions.