Entrust Exam Platform

Problem

Surgical Trainees have to pass a competency-based exam (called EPAs) to be entrustable surgeons. This process is organized and maintained by surgical professionals in different EPAs levels through an assessment system.

The existing system has multiple key design weaknesses which makes it extremely time-consuming to maintain and hard to scale.

Outcomes

We built an ecosystem that enables the surgical professionals in different EPAs levels to create exam contents, take exams and assess trainee's performance in terms of surgical decision-making.

This case study mainly focuses on the redesign process of the exam platform. Other case studies of this ecosystem: the content creation platform and the design system.

Time
Feb 2020 - Present

Client
Stanford Medicine

Role
UX/UI Design
Visual Design

Team
Edward M. | Principal Investigator
Oleksandra K. | PM
Fatyma C. | Character Artist
Jason T. & Ruonan C.| Engineer

    How might we provide surgical trainees a trustworthy, realistic, and user-friendly digital environment to complete the surgical decision-making competency-based exam?    

Solutions

To solve the HMW problem, we designed a game-based virtual patient platform that offers an immersive and realistic environment in which trainees demonstrate their management of a given clinical scenario.

The platform including preoperative evaluations and treatments, intraoperative decision-making, and postoperative management.

Preoperative Evaluations & Treatments

The patient scenario begins in a preoperative clinical setting, either inpatient or outpatient, where users initiate workup of the patient. A life-like patient is depicted on the left side. The patient's vital signs appear in the central upper area and reflect the clinical status of the patient. The patient chart is located on the right side of the screen.

1

Physical exam

Users can hover over and click on regions of the body to elicit physical exam findings. Then, the findings will appear under the physical exam section of the patient's chart.

2

Order tests and treatments

The action button to look up or treat the patient are located on this computer monitor in the middle of the screen. Users can order labs or imaging studies. The status of the order can be tracked on the in-progress screen. Results can be accessed via the corresponding tabs in the patient chart.

3

Make Disposition

Once users are satisfied with a workup and confident of the diagnosis, they can make a disposition for the patient.
Disposition options are different for inpatient and outpatient, include: non-operative management, discharge home, consent to the operating room, admit to ward room, ICU, etc.

Intraoperative Decision-making

If users make a disposition to Operation, the system will transport them to a highly simulated environment of the operating room.
Users will answer a series of multiple-choice surgical decision-making questions.
The preoperative evaluation is the helpful information in this part.

Postoperative Management

Some patient cases require postoperative management, which is presented on a separate page with a corresponding environment to help trainees better access the patient’s condition. Users will answer the corresponding complications management questions in this part.

Design Reserch

Primary Research

We started conducting user interviews in order to gain insights on trainee needs and pain points when taking the exam. We asked as many clients' inputs as possible to make sure the business goal is satisfied. A usability test of the old platform is conducted followed by an interview to find potential design opportunities.

Usability Test

Below is the interface for the old platform. It's divided into three parts, left, right, and bottom. The left part is patient avatars and a timeline. The right part has medical records and treatment information. The bottom side is the place where users could press the corresponding button to conduct tests, evaluations, and dispositions.

Old Platform

Synthesize

The interview and usability test results are sorted into 3 categories based on the three user experience elements: flow, information architecture, and interface.

Affinity Map

By using an affinity map, I have a better understanding of the relationship of each user pain point and synthesized it into the following seven points.

Pain Points - Flow

F1

Multiple patients show up at the panel simultaneously distracts trainees, which is unrealistic.

F2

Error prevention is missed in the irreparable go-to-surgery decision.

Pain Points - Information Architecture

IA1

Action Button contains dense and important options, which, however, are stacked on the bottom and away from user visual concentration.

IA2

Treatment progress is not clearly presented to users and makes the information update easily get neglected.

Pain Points - Interface

I1

Patient image is oversimplified, which is unrealistic and fails to provide sufficient information.

I2

Vital signs are important but lack realistic simulation to be easily perceived.

I3

Physical exam is an important process in preoperative evaluation, but examination process is oversimplified and far from real scenarios.

Ideate & Iterate

User Flow

Below are three versions of the user flow we came up with and iterated in different stages. Working closely with engineers help me to understand the structure thoroughly.

1

Version 1

In this version, we figured out the overall flow of preoperative evaluation.
However, the relationship between exam, treatment, and the medical record is not very clear in this phase.

2

Version 2

In this version, we reorganize the flow of preoperative evaluation.
A new issue came out. Users felt distracted about evaluating multiple patients at the same time, which helped us come up with the final user flow.

3

Final Version

In this version, we changed the mechanism of the patient’s show-up and addressed F1-F2 pain points.

F1

Only one patient appears at a time, only when finished, a new patient case is available.

F2

The confirmation dialog showed in a pop-up window to prevent user errors when deciding to go to surgery.

Wireframe

In the meantime, we brainstormed different layouts, including order panel options, clinic room layout, and operating room layout. We onboard the clients at the very early stage and gained insightful feedbacks from them.

1

Order Panel Options

Gamification panel on the left side

Computer monitor panel on the middle

2

Preoperative Evaluation & Treatment Interface

The patient lay down horizontally, the chart on the right side

The patient standing on the right side, the chart on the left side

3

Intraoperative Decision-making Interface

The decision-making questions overlay the patient

The patient lay on the right side, questions on the right side

Hi-fi Prototype

Based on the feedback and suggestions from clients, users, and engineers, we came up with and iterated for several rounds of high-fidelity prototypes. Below are some highlights of the prototype and accommodations to the above-mentioned pain points.

1

Information Architecture

IA1

The treatment progress is displayed in a structured view.

IA2/a  

The order panel, which is displayed on a computer monitor, is in the center of the panel, to retain the user’s attention.

IA2/b

The layout of the order panel mimics the real-world workspace.

2

Interface

I1

The patient avatar is lifelike and fully customizable via the authoring platform.

 I2 

A monitor is displayed to precisely show dynamic vital signs.

I3

The physical exam process is simulated based on the real-world environment. The doctor's hand replaces the regular cursor to onboard users to conduct a virtual physical exam.

3

The correlation between the order panel and medical records

Lab tests & results

Imaging scans & results

Interventions & records

Integration

Design System

I built a design system that integrates the guidance and patterns for this ecosystem from scratch to make it easy to scale and manage. For detailed information, please refer to the Design System case study

Accessibility Design

To take care of a broader audience, we’ve considered the accessibility features. We’ve carefully chosen the colors to make sure the color contrasts satisfy the WCAG 2.1 standard.  We also run the platform from a color blindness simulator called Funkify, to ensure it is colorblind-friendly.

Visual Design

To align with the UX design and make the scenario look professional, I referred to the images that clients provided, designed medicine equipment for the corresponding scenario. I invited valuable inputs from doctors and polished them as follow. The carefully illustrated visual assets make the overall exam process vivid.

Validation

Quantitative Research

Before the pilot test, we ran a mini-pilot test with 38 medical students at Stanford University School of Medicine. We logged their exam activities data and let them fill a survey after the section. Here're the survey feedbacks:

  • 95% of users agreed that the exam platform is user-friendly.
  • 90% of users strongly agreed that the exam is useful and professional.
  • 98% of exam platform users can totally understand the process of the exam.

Qualitative Research

In addition to the survey, we also conducted a think-aloud test to gather qualitative feedback. This is an ongoing test. We'll have more detailed information when it's finished.

Achievement

Overall Improvement

Compared with the traditional in-person verbal interrogation via the oral boards and the old platform, Entrust has improvements in the following aspects:

Factors
Oral Board
(Traditional Way)
Sicko
(Old Platform)
Entrust
(New Platform)
1. Features standardized scenarios
2. Less time-and resource-intensive
3. Minimizes bias and subjectivity of faculty
4. Reduces the burden upon faculty to complete evaluations
5. Reproducible
6. Use as clinical judgement for rarely encountered yet critical scenarios

Project Impact

I'm glad this project has been going very well and received beyond expected results:

Improved trainees' performance in the pilot test

Won Excellence in Research Award in ACSCC 2021

Won two grants

Collaborated with institutions from 15 counties

After graduation, I‘m hired by the clients as a contractor to keep working on it and further expand the platform in order to reach out to broader audiences.

Reflection

Working with the team and clients who are passionate about making the idea come true is so lucky. Everyone is eager to learn to make things perfect. Through this process, I learned:

  • Clients have great insights into the process and have a lot of valuable feedback. It's helpful to create an environment that both designers and clients are confident to discuss.
  • It's professional to conduct the research in each stage, whether or not it's just a quick interview or playtest.
  • Defining the opportunity and success metric for each stage is important.
  • It's important to sync the progress regularly in order to get instant feedback.