TECHNICAL PAPER

SIMPL OR

Society for Improving and Measuring Procedural Learning in the Operating Room (SIMPL OR) app as a Practical Measure

Characteristics of Practical Measures	Descriptions of the SIMPL OR App
Is closely tied to a theory of improvement	The SIMPL OR app is used to support multiple improvement efforts with a common aim to increase practice-readiness among surgery trainees. When it is used as a measure for the aim of “every trainee achieves practice-readiness on one specialty-defined index procedure within each postgraduate year,” observations within individuals are transformed into readiness scores using a prediction model. The readiness score is also used as an outcome measure for testing teaching and learning strategies. Another metric of interest is the number of times trainees request feedback. This metric is used as a PDSA measure to test change ideas designed to improve trainee engagement in feedback-seeking behaviors. (See network improvement project and article on readiness)
Provides actionable information to drive positive changes in practice	The SIMPL Collaborative learned from the users that the SIMPL OR app has face validity. In other words, it is not difficult for users to understand what gets measured. On the one hand, trainees can use the SIMPL OR app to reflect on their performance and to receive formative feedback from faculty, which can guide their efforts in improving their practices. On the other hand, faculty can use aggregated SIMPL OR app data to personalize their teaching and supervision in the operating room. Moreover, program directors can use the data to identify procedures that are particularly challenging and design change ideas to support faculty and trainees. Recently, to maximize the actionability of the data, the Collaborative has invested in optimizing and differentiating data visualizations.
Captures variability in performance	The SIMPL OR app captures variability in trainee performance across operating procedures over time. It sounds straightforward, but when implemented in practice, various implementation challenges may emerge and can add noise to the data — this can potentially make the results less interpretable and hence less actionable. For example, in programs with a smaller faculty size, trainees may not be able to gather feedback from multiple faculty members.
Demonstrates predictive validity	A statistical prediction model has been developed by the SIMPL Collaborative to make sense of observations. Once a data point is entered into the system, a readiness score is generated to estimate the likelihood of success in the next operation. The Collaborative is in the process of understanding how trainees and faculty make sense of readiness scores. In addition, the SIMPL OR app results are expected to predict end-of-rotation performance evaluation results. However, because the latter lacks variability, the Collaborative is planning on shifting focus to studying the predictive relationships between SIMPL OR app results and post-graduation/career outcomes.
Is minimally burdensome to users	Being a phone application makes the SIMPL OR app highly accessible and minimally burdensome. On top of that, the SIMPL OR app embraces user autonomy. Trainees can choose whether to request feedback after a case closes, while faculty can choose whether to accept a feedback request and if they are to dictate a qualitative response. This design helps both trainees and faculty perceive the use of the SIMPL OR app as something they want to do instead of something they have to do.
Functions within social processes that support improvement culture	The SIMPL OR app is used as an assessment for improvement, not an assessment for accountability. Trainees are allowed to request feedback as little as one time or as many times as needed to inform their learning. Given the voluntary nature of asking for feedback, the SIMPL Collaborative designed change ideas to encourage trainees to request more feedback.
Is reported on in a timely manner	Trainees are only required to be formally evaluated through direct observation twice a year. The SIMPL OR app makes the evaluation process simpler and thus encourages more frequent feedback. Once a case closes, the trainee can request feedback from the supervising surgeon. After the surgeon provides feedback by answering three questions on the app, the system will instantly notify the trainee about the survey responses.

Question on Practical Measures Inspired by the SIMPL OR App

Are practical measures synonymous with quick-and-dirty measures?
Practical measures like the SIMPL OR app can often be implemented swiftly and easily. Yet, the process of developing and refining practical measures is usually far from easy. “In diving into the technical assessment properties of [SIMPL OR], we have opened up interesting improvement opportunities,” said Dr. Andrew Krumm, a member of the SIMPL Collaborative. What contributes to a surgeon’s rating of trainee performance? The Collaborative found that an attending surgeon’s rating reflects not only trainee competence, but also a number of other factors, such as the rarity of the procedure and, most of all, the attending surgeon’s own perception of good performance — a factor that contributes a majority of the variance in attending surgeons’ ratings. To make sure that the SIMPL OR app provides accurate assessment of trainee performance to inform improvement efforts, the Collaborative tested change ideas (e.g., specifying key procedures to be assessed, retraining raters) to minimize unwanted sources of variation. Examining the psychometric properties of a measure, identifying problems, and testing potential solutions take expertise, time, and resources; however, these efforts allow improvers to be confident about the efficacy of change being measured using the app’s data.