Entrust Exam Platform
Surgical Trainees have to pass a competency-based exam (called EPAs) to be entrustable surgeons. This process is organized and maintained by surgical professionals in different EPAs levels through an assessment system.
The existing system has multiple key design weaknesses which makes it extremely time-consuming to maintain and hard to scale.
We built an ecosystem that enables the surgical professionals in different EPAs levels to create exam contents, take exams and assess trainee's performance in terms of surgical decision-making.
Feb 2020 - Present
Edward M. | Principal Investigator
Oleksandra K. | PM
Fatyma C. | Character Artist
Jason T. & Ruonan C.| Engineer
How might we provide surgical trainees a trustworthy, realistic, and user-friendly digital environment to complete the surgical decision-making competency-based exam?
To solve the HMW problem, we designed a game-based virtual patient platform that offers an immersive and realistic environment in which trainees demonstrate their management of a given clinical scenario.
The platform including preoperative evaluations and treatments, intraoperative decision-making, and postoperative management.
Preoperative Evaluations & Treatments
The patient scenario begins in a preoperative clinical setting, either inpatient or outpatient, where users initiate workup of the patient. A life-like patient is depicted on the left side. The patient's vital signs appear in the central upper area and reflect the clinical status of the patient. The patient chart is located on the right side of the screen.
Order tests and treatments
We started conducting user interviews in order to gain insights on trainee needs and pain points when taking the exam. We asked as many clients' inputs as possible to make sure the business goal is satisfied. A usability test of the old platform is conducted followed by an interview to find potential design opportunities.
Below is the interface for the old platform. It's divided into three parts, left, right, and bottom. The left part is patient avatars and a timeline. The right part has medical records and treatment information. The bottom side is the place where users could press the corresponding button to conduct tests, evaluations, and dispositions.
The interview and usability test results are sorted into 3 categories based on the three user experience elements: flow, information architecture, and interface.
By using an affinity map, I have a better understanding of the relationship of each user pain point and synthesized it into the following seven points.
Pain Points - Flow
Multiple patients show up at the panel simultaneously distracts trainees, which is unrealistic.
Error prevention is missed in the irreparable go-to-surgery decision.
Pain Points - Information Architecture
Action Button contains dense and important options, which, however, are stacked on the bottom and away from user visual concentration.
Treatment progress is not clearly presented to users and makes the information update easily get neglected.
Pain Points - Interface
Patient image is oversimplified, which is unrealistic and fails to provide sufficient information.
Vital signs are important but lack realistic simulation to be easily perceived.
Physical exam is an important process in preoperative evaluation, but examination process is oversimplified and far from real scenarios.
Below are three versions of the user flow we came up with and iterated in different stages. Working closely with engineers help me to understand the structure thoroughly.
Only one patient appears at a time, only when finished, a new patient case is available.
The confirmation dialog showed in a pop-up window to prevent user errors when deciding to go to surgery.
In the meantime, we brainstormed different layouts, including order panel options, clinic room layout, and operating room layout. We onboard the clients at the very early stage and gained insightful feedbacks from them.
Order Panel Options
Gamification panel on the left side
Computer monitor panel on the middle
Preoperative Evaluation & Treatment Interface
The patient lay down horizontally, the chart on the right side
The patient standing on the right side, the chart on the left side
Intraoperative Decision-making Interface
The decision-making questions overlay the patient
The patient lay on the right side, questions on the right side
Based on the feedback and suggestions from clients, users, and engineers, we came up with and iterated for several rounds of high-fidelity prototypes. Below are some highlights of the prototype and accommodations to the above-mentioned pain points.
The treatment progress is displayed in a structured view.
The order panel, which is displayed on a computer monitor, is in the center of the panel, to retain the user’s attention.
The layout of the order panel mimics the real-world workspace.
The patient avatar is lifelike and fully customizable via the authoring platform.
A monitor is displayed to precisely show dynamic vital signs.
The physical exam process is simulated based on the real-world environment. The doctor's hand replaces the regular cursor to onboard users to conduct a virtual physical exam.
The correlation between the order panel and medical records
Lab tests & results
Imaging scans & results
Interventions & records
I built a design system that integrates the guidance and patterns for this ecosystem from scratch to make it easy to scale and manage. For detailed information, please refer to the Design System case study
To take care of a broader audience, we’ve considered the accessibility features. We’ve carefully chosen the colors to make sure the color contrasts satisfy the WCAG 2.1 standard. We also run the platform from a color blindness simulator called Funkify, to ensure it is colorblind-friendly.
To align with the UX design and make the scenario look professional, I referred to the images that clients provided, designed medicine equipment for the corresponding scenario. I invited valuable inputs from doctors and polished them as follow. The carefully illustrated visual assets make the overall exam process vivid.
Before the pilot test, we ran a mini-pilot test with 38 medical students at Stanford University School of Medicine. We logged their exam activities data and let them fill a survey after the section. Here're the survey feedbacks:
In addition to the survey, we also conducted a think-aloud test to gather qualitative feedback. This is an ongoing test. We'll have more detailed information when it's finished.
Compared with the traditional in-person verbal interrogation via the oral boards and the old platform, Entrust has improvements in the following aspects:
I'm glad this project has been going very well and received beyond expected results:
Improved trainees' performance in the pilot test
Won Excellence in Research Award in ACSCC 2021
Won two grants
Collaborated with institutions from 15 counties
After graduation, I‘m hired by the clients as a contractor to keep working on it and further expand the platform in order to reach out to broader audiences.
Working with the team and clients who are passionate about making the idea come true is so lucky. Everyone is eager to learn to make things perfect. Through this process, I learned: