De-Risking Software Investment with Usability Testing

By now, most of us have interacted with a doctor or nurse via video. COVID’s increased demands on medical professionals, combined with the need to prevent community transmission wherever possible, have accelerated an already-developing practice in the delivery of medical care: remote patient monitoring, or RPM.

Clinicians are presented with many technical service options designed to help bridge this “white space” between office visits. Grand Studio recently helped a large hospital network evaluate RPM vendor candidates for their RFI process, and usability testing offered an important set of criteria for this evaluation.

An emerging technology with profound usability implications

Care delivery modalities are often dependent on comprehension and adoption from both clinicians and patients, and our client understood this before we even had to suggest it. 

Clinicians can find themselves short on time and presented with a long list of biometric data to quickly assess and address. They may be accessing this data during busy clinic hours or between surgical procedures. Missing or misunderstanding something important can have serious implications for the patient’s wellbeing. Good interfaces will amplify clinician focus and mitigate their fatigue.

Patients may be familiar with digital technologies in general. Still, their ability to learn new interaction modes might be challenging when focusing their attention on coping with a health condition, especially a condition that may be new to them. Also, because many of these applications reside on a smartphone, it’s fair to assume that patients will be distracted when interacting with these interfaces.

We knew that context of use for all users would be critically important for our RPM solution. We decided that ranking the offerings against foundational and accepted usability rubrics would allow us to objectively assess how patients and clinicians might interact with this technology. These rankings would provide the decision-makers with a non-biased set of acceptance criteria to consider when choosing a technology partner.

Step 1: Measure for fundamental heuristics

The market for RPM solutions is large and varied. As with any technology industry sector, some products are more mature than others. Additionally, some solutions were removed from consideration due to no fault of their own – they may be too narrow in their utility, too difficult to integrate with sibling platforms, or too general in their functionality.

We were presented with a collection of eleven vendor demo videos to evaluate. Grading on a curve, we scored their clinician-facing dashboards on commonly-accepted foundational rubrics.

• Orientation, context, & wayfinding: How easy is it for users to find what they’re looking for?
• Visual hierarchy & module differentiation: Is it clear that some things are more important than others?
• System feedback/confirmation of action: Does the software validate the user’s actions?
• Constructive failure & mistake recovery: What happens when a mistake is made? Is it easy to correct?
• Affordances & interaction cues: Are interactive elements intuitive?
• Language & terminology: Does the system present commonly accepted terms?
• Three of the eleven scored very well, four were well below average, three were unacceptable, and one of the vendors was dismissed for other reasons.

Step 2: Test the finalists with real users

Testing software with real users is an essential part of any usability evaluation. No matter how much research you do and how deep your professional expertise is, you’ll never be able to consider real people’s comprehension patterns and work habits.

To simulate a real clinical scenario, we collaborated with our clinical partners to create a “dummy” data set with 100 fictional patient records. Each patient was given hypothetical biometric readings for blood pressure, blood glucose level, heart rate, respiration rate, weight, fall detection, and SPO2. We also mapped these patients to a small selection of conditions such as congestive heart failure, diabetes, and hypertension. Finally, each patient was assigned to one of eight doctors and one of five monitoring nurses.

The vendors were given these datasets, and the three finalists were asked to stand up a “sandbox” environment to support our task observation exercises.

With the help of four monitoring nurses and one care coordinator, we asked these testers to execute a series of tasks within each finalist platform. Watching these users closely and interviewing them for feedback after completing the tasks yielded interesting and clear results.

The dashboards were evaluated on both general and feature-specific sets of heuristic criteria, allowing for some overlap with the first round of scoring. We measured the systems for the following feature-specific heuristics:

• Dashboard clarity: Are the contents of the clinician dashboard presented in a scannable way?
• Dashboard filter sets: Can the user reduce and refine the contents in the dashboard in an intuitive and content-relevant way?
• Patient details: Are these details clear and contextual, presenting the clinician with both a single measurement and trending data presentation?
• Alert clearing & documentation: How easy is it to clear and document the clinical details when a patient’s biometrics are out of range?
• Patient contact: Does the application provide functionality that enables text, phone, or video contact with the patient?
• Clinician contact & collaboration: Does the platform support secure patient information sharing when a patient case requires escalation to a physician or collaboration with a nursing peer?
• We also evaluated each of these offerings from a patient point of view, emphasizing program onboarding and user comprehension. 

• Onboarding & guidance: Is the patient provided a clear and easy-to-use introduction to the software?
• Next-best action clarity: As with clinicians, many of the patient’s tasks need to happen in sequence. Is that sequence clear?
• Troubleshooting & support: How does the platform support users encountering technical challenges?
• Biometric reading context: Patients are often confused by their biometric readings. Does the interface provide helpful context for understanding this information?
• Wayfinding & signage: Is functionality clearly marked?
• System feedback, reminder cues, & task validation: Patients also need confirmation of their actions in the application. They may also need scheduled reminders to take biometric readings. Is this functionality clear and flexible?
• Clinician interaction functionality: Does the system provide a means of interaction between the patient and the clinician?
• Physical space needed for devices: How much physical space does the kit take up? 
• Portability of devices: Are the kit devices easy to carry, or is moving them challenging?
• Device consistency & ease of connectivity: Do the kit devices and interfaces feel like they’re part of a suite of products? How easy is it to connect them with each other, personal smartphones, and the web?

Findings: One vendor scored much higher than the others

Remote Patient Monitoring offers a great example of the value of human-centered design. Each of the platforms we evaluated is built around technologies unavailable only a few years ago, technologies that will fundamentally change how healthcare is delivered in the near future. Many of the systems we evaluated failed to deliver that technology effectively because the interface and product design did not adequately support the real people, clinicians, and patients, who would use these new tools.

Given these stakes, the role of usability testing was elevated in prominence. The testing results were clear – one of the eleven vendors stood out as a clear favorite.

While the usability testing was only one important piece of the vendor evaluation process, it ensured that user needs were considered, helping to facilitate onboarding and adoption as a result.