Published on in Vol 6, No 2 (2021): Apr-Jun

Preprints (earlier versions) of this paper are available at, first published .
Using Medical Device Standards for Design and Risk Management of Immersive Virtual Reality for At-Home Therapy and Remote Patient Monitoring

Using Medical Device Standards for Design and Risk Management of Immersive Virtual Reality for At-Home Therapy and Remote Patient Monitoring

Using Medical Device Standards for Design and Risk Management of Immersive Virtual Reality for At-Home Therapy and Remote Patient Monitoring

Authors of this article:

Joseph Peter Salisbury1 Author Orcid Image


Playhab R&D, Cognivive, Inc., Davis, CA, United States

Corresponding Author:

Joseph Peter Salisbury, PhD

Playhab R&D

Cognivive, Inc.

2003 Baywood Ln

Davis, CA, 95618

United States

Phone: 1 530 361 6006


Numerous virtual reality (VR) systems have received regulatory clearance as therapeutic medical devices for in-clinic and at-home use. These systems enable remote patient monitoring of clinician-prescribed rehabilitation exercises, although most of these systems are nonimmersive. With the expanding availability of affordable and easy-to-use head-mounted display (HMD)-based VR, there is growing interest in immersive VR therapies. However, HMD-based VR presents unique risks. Following standards for medical device development, the objective of this paper is to demonstrate a risk management process for a generic immersive VR system for remote patient monitoring of at-home therapy. Regulations, standards, and guidance documents applicable to therapeutic VR design are reviewed to provide necessary background. Generic requirements for an immersive VR system for home use and remote patient monitoring are identified using predicate analysis and specified for both patients and clinicians using user stories. To analyze risk, failure modes and effects analysis, adapted for medical device risk management, is performed on the generic user stories and a set of risk control measures is proposed. Many therapeutic applications of VR would be regulated as a medical device if they were to be commercially marketed. Understanding relevant standards for design and risk management early in the development process can help expedite the availability of innovative VR therapies that are safe and effective.

JMIR Biomed Eng 2021;6(2):e26942



Virtual Reality as a Medical Device

Therapeutic virtual reality (VR) offers tremendous potential to provide innovative treatments in a broad range of clinical areas, including mental health disorders [1] (eg, traumatic stress [2,3], anxiety disorders [4], depression [5], schizophrenia [6], eating disorders [7]), pain management [8,9], motor and cognitive rehabilitation of neurodegenerative disorders [10,11], traumatic brain injury [12], stroke [13,14], and cognitive disorders [15,16].

While a wide variety of approaches have been referred to as VR in the literature, VR is popularly understood to include the use of a wearable head-mounted display (HMD) that creates a sense of being immersed in a virtual environment. HMD-based immersive VR has only recently begun to approach the same level of affordability as nonimmersive VR. The sense of presence that immersive VR offers has considerable potential to differentiate the impact of VR in clinical contexts, including telerehabilitation, moving forward [17-19]. Critically, immersive VR offers the potential for greater ecological validity in therapy, allowing the brain to respond to stimuli similar to how it does in the real world [20-24].

As VR interventions are developed and evaluated by clinicians and patients—particularly in an at-home environment—it is essential to evaluate the regulatory requirements that may restrict the translation of such technologies to routine clinical practice. For VR interventions that will be classified as a medical device, it is strongly recommended that requirements be identified early in the design and development phase to prevent costly reworkings of the system, software, and associated documentation [25-27].

While VR interventions include both a hardware and software component, many proposed VR interventions (particularly those for at-home use) leverage off-the-shelf (OTS) VR technology. In the most recent wave of technology, this includes the standalone VR headsets with 6-degrees of freedom (6-DOF) tracking, including the Vive Focus Plus, Oculus Quest, and Pico Neo. While designed primarily for nonmedical purposes, this does not restrict their use as a component of a medical device. A consumer VR headset is transformed into a medical device by virtue of the intended use of the software it is running. While additional, built-for-purpose hardware components may be introduced into a therapeutic VR system (eg, custom sensors, adaptive controllers), the software component is necessary and essential for transforming OTS VR devices into a medical device, and thus, can be considered as part of the larger category of software as a medical device (SaMD) [28].

While the design, development, testing, and postmarket surveillance of therapeutic VR include many of the same considerations, HMD-based VR presents unique challenges in comparison to the broader category of SaMD. In addition to the potentially hazardous situations introduced by wearing an occlusive headset that can induce side effects ranging from simulator sickness [29,30] to seizure, fully immersive VR introduces novel challenges for interface design among a population that will typically have little-to-no experience with the technology [31-33]. Thus, it is becoming useful to discuss the requirements of VR as a medical device (VRaMD) in their own right.

Medical Device Quality Requirements

To understand what regulatory requirements may be for a given VR intervention, it is important to first consider whether the intended use is indeed classified as a “medical device” in a particular jurisdiction. The Global Harmonization Task Force published a guidance document toward an internationally recognized definition of a medical device [34]. In the United States, a medical device is defined in section 201(h) of the Food, Drug, and Cosmetic Act [35].

The US Food and Drug Administration (FDA) has published guidance [36] that outlines certain software functions that may meet the definition of a medical device, but as they pose a lower risk to the public, the FDA intends to exercise enforcement discretion. That is, the FDA will not enforce medical device regulatory requirements on this software. Included in the type of software is one that “use video and video games to motivate patients to do their physical therapy exercises at home.” With that said, this guidance document also states that software becomes a regulated medical device by performing patient-specific analysis and providing patient-specific diagnosis or treatment recommendations. Furthermore, there are specific regulatory classifications in the United States that classify “interactive rehabilitation exercise devices” as Class II medical devices, providing a clear regulatory path for a VRaMD intended to provide rehabilitation. Ultimately, manufacturers interested in commercialization in the United States are encouraged to contact the FDA to determine what, if any, regulatory requirements may apply.

Assuming the intended use of a VRaMD is determined to be a regulated medical device in a particular jurisdiction, it is important to understand regulatory requirements early in the device and development process. When developing a novel medical device, those without a background in medical device engineering may assume the burden to demonstrate the safety and effectiveness of a medical device is the domain of clinical investigators. However, it is important to note that the universal expectation of regulatory bodies is that safety and effectiveness be built into the system in early design and development stages. In the United States, Title 21 of the Code of Federal Regulations (CFR) [37] provides specific regulations that define the minimum current good manufacturing practice (cGMP) requirements for drugs, biologics, and medical devices. The cGMP regulations, also known as the quality system regulation (QSR), are based on the “quality-by-design” principle, which calls for quality to be built into the product, as testing alone cannot be relied on to ensure product quality [38].

cGMP regulations require establishment of a quality management system (QMS). The QMS impacts an organization’s daily activities at every level, including product planning, design, development, testing, and change management. Software professionals coming from a nonregulated software development industry may find it difficult to adapt to the planning and documentation requirements imposed by quality requirements [39,40]. Quality requirements for medical device software development may seem to conflict with agile software development methodologies and impose a large amount of overhead when developing medical device software [41]. Still, it is critical that software professionals confront the challenge of medical device quality requirements head on not only to be compliant with regulations, but also to ensure medical device software is safe and effective for its intended purpose. For medical device software, there are clear expectations for how to document the entire software development life cycle, from establishing user needs through to verification, validation, postmarket surveillance, and change management.

Quality requirements for medical devices include the integration of risk management across the product life cycle. As a component of risk management, a systematic risk assessment for a device must be performed with risk controls implemented and verified to mitigate unacceptable device hazards. Implementing risk management as part of the requirements analysis and design process of an SaMD can aid in improving designs early in the development process. This can prevent the need for reworking solutions and changing project scope late in the development process when changes can be more costly. In the case of home-use VRaMD, risk analysis can reveal new system requirements that can help improve system usability and adoption while mitigating risks to patients.

Objectives of This Paper

This paper reviews regulations, standards, guidance documents, and technical reports that can be relevant for the design and development of a VRaMD. To demonstrate the application of these standards in the design and development process, the requirements for a generic home-use VRaMD system for at-home therapy are specified. A risk assessment is performed on the requirements to derive a set of risk control measures. Methods for verifying these risk control measures are discussed. The objectives of this paper are to:

  1. Provide an overview of medical device standards that are applicable to the development of VRaMD intended for home use and remote patient monitoring.
  2. Analyze the requirements of a generic home-use VRaMD and demonstrate how risk management can be used to identify and evaluate hazards, determine appropriate risk control measures, and limit potentially hazardous situations.

General Quality Management System Requirements

Medical device regulations are the legally defined requirements within a jurisdiction for how medical device manufacturers must operate. Requirements for a particular medical device can be determined by classifying the device within the risk-based classification system of a particular jurisdiction. One of the most fundamental requirements of a medical device organization is implementation of a QMS [42]. A QMS is a formal system that documents policies, procedures, and responsibility to manage product or process quality. QMS requirements are specified by regulatory bodies to ensure medical devices will be safe and perform as intended. It is important to note that while QMS regulations and standards outline a range of specific requirements, they are typically broadly defined to allow a variety of ways an organization can achieve their goals. Thus, the scope and complexity of an organization’s QMS can vary widely depending on the device type, organization size and structure, and the nature of specific regulatory requirements.

While the requirement that a QMS be certified varies depending on regulatory jurisdiction and device type, to achieve broad recognition many manufacturers follow ISO 13485:2016 [43]. ISO 13485 specifies requirements for a QMS that can be used by an organization involved in one or more stages of the life cycle of a medical device. These stages can include design, development, and production of a medical device, as well as storage, distribution, installation, technical support, servicing, decommission, and disposal. In the United States, adherence to ISO 13485 is not required, although the US QSR is generally aligned with this standard.

An important aspect of the QMS relevant for VRaMD development is the concept of design controls. Design controls are a set of policies and practices intended to ensure consistent translation of input requirements into a product that meets those requirements. Both ISO 13485 and FDA QSR set out a series of requirements for design controls. Design control is an iterative process following a structured methodology to ensure the device under development will be safe, effective, and meet end-user needs. The design control process is often illustrated with the V-model [44]. Design control requirements specify a general framework where various deliverables are generated and approved at each stage of the design and development process through to device verification and validation activities. These deliverables are necessary for auditing the QMS and meeting regulatory needs, requiring a robust system of procedures for maintaining documentation and approvals.

While the expectations of the QMS design controls are well-defined, there remains considerable room for how an organization decides to carry out these objectives. As part of QMS requirements, it is expected that an organization establishes detailed design and development plans for each product. These plans should specify how the development process is carried out, including assignment of responsibilities to adequately trained personnel and how these procedures are aligned with regulatory requirements and appropriate standards.

Risk Management for Medical Devices

As part of fulfilling regulatory requirements, organizations must perform risk management activities. For example, under the 2017 European Union Medical Device Regulation (EU MDR) [45], manufacturers must have a documented risk management plan, identify and analyze the known and foreseeable hazards for each device, estimate and evaluate the associated risks, and eliminate or control those risks. Risk analysis is required as part of the US FDA’s design control requirements (21 CFR 820.30) [46] and is a component of FDA premarket submissions. ISO 14971:2019 [47], recognized worldwide by regulatory bodies, is widely acknowledged as the principal standard for this purpose. As part of ISO 14971, an organization develops a risk management plan, which includes how device risk assessments should be conducted.

ISO 14971 describes the requirements of a risk management process for medical device development, including 6 key stages: risk analysis, risk evaluation, risk control, evaluation of overall residual risk acceptability, risk management report, and production and postproduction information. Like quality management requirements, the details of how these processes are carried out in practice are left to the manufacturer. To implement ISO 14971, a company must first establish and document how they will conduct a risk management process that includes the required components in the standard. To accomplish risk analysis, Annex G of ISO 14971 provides guidance on some techniques, including preliminary hazard analysis, fault tree analysis, and failure modes and effects analysis (FMEA).

FMEA enables any effect or consequence of individual components to be systematically identified and is more appropriate as the design matures [48]. FMEA can be applied during the design process to understand the impact of potential defects and incorporate changes relatively early when they are less expensive to make. Thus, safety is improved and performance is enhanced by minimizing the probability and severity of hazardous situations.

It is important to note that, although FMEA is a recognized risk assessment tool specified in ISO 14971, completing FMEA according to the FMEA standard IEC 60812:2018 [49] does not fulfill all the requirements of ISO 14971. For example, FMEA focuses on defects, whereas the focus of ISO 14971 is on harm. In ISO 14971, both normal and abnormal circumstances must be considered, as opposed to a focus on failure situations in FMEA. That is, even when the device is functioning as intended, hazardous situations may still occur, which must be identified. For example, the device may function as intended, but a specific subset of patients may experience side effects. Patients may also misinterpret instructions or feedback provided by the system. Of course, hazardous situations that arise from system malfunction, such as damage or misuse of the system leading to degraded system performance, must also be considered.

Furthermore, FMEA can allow for low-priority defects to persist, whereas risks in a medical device should be reduced or eliminated as far as reasonably possible before a medical device can be marketed. Both ISO 14971 and FMEA require the risk parameters of occurrence and severity to be addressed, where occurrence is the probability of occurrence of harm and severity is the extent of its impact or consequences. However, FMEA also considers the probability of detecting the harm before it occurs, which is not part of ISO 14971. Harm may still happen even if it is detected, and harms not easily detectable may unnecessarily raise risk levels. Thus, this parameter is excluded. Once the differences between FMEA and ISO 14971 are understood, it is possible to adapt FMEA to meet the requirements of ISO 14971 (Figure 1).

To conduct the risk management process, the first step is to identify the hazards, hazardous situations, and associated harms of a device. Hazard identification can be performed by reviewing the medical device characteristics, such as intended use, technologies used in the device, how the device is intended to function in clinical procedures, what could occur if the device is misused, and what could occur if information from the device is misinterpreted.

Once hazards are identified, for each hazardous situation, risk estimation is performed whereby the probability of occurrence and severity of that harm is estimated. It is the responsibility of the manufacturer to establish an appropriate quantitative or qualitative method for categorizing probability of occurrence of harm and severity of harm. Tables 1 and 2 provide example ways of categorizing severity of harm and probability of harm occurrence. Note that these tables are intentionally kept simple for illustration purposes and could include greater (or fewer) categories, as appropriate.

Figure 1. Overview of ISO 14971 risk management process requirements and how FMEA can be adapted. Redrawn and adapted from resources developed by Gantus and Semoegy (unpublished data). FMEA: failure mode and effects analysis; ISO: International Organization for Standardization; RPN: risk priority number.
View this figure
Table 1. Example severity table.
5CriticalLoss of limb or life-threatening injury.
4MajorSevere, long-term injury; potential disability.
3SeriousShort-term injury or impairment requiring additional medical intervention to correct.
2MinorSlight inconvenience with little to no effect on product performance; minor injury not requiring medical intervention.
1NegligibleNo significant risk of injury to patient.
Table 2. Example probability of occurrence table.
5Frequent1 in 100
4Probable1 in 1000
3Occasional1 in 10,000
2Remote1 in 100,000
1Improbable1 in 1,000,000

Acceptable methods for estimating risks are provided in ISO 14971 and include published standards, scientific technical data, field data from similar devices, usability tests, clinical evidence, and expert opinion. It is often not practical to assign numerical estimations for the likelihood of an occurrence of a particular harm. Thus, following qualitative descriptors can provide a reasonable method for estimating the probability of occurrence in the absence of precise data.

The “Rank” column is included in Tables 1 and 2 so that, following the FMEA approach, a risk priority number (RPN) can be generated for risk evaluation. In Figure 2, a risk evaluation matrix is generated by multiplying the probability of occurrence ranks with the severity of harm ranks. The risk evaluation matrix is divided into 3 risk regions to define acceptable risks (green), borderline risks (yellow), and unacceptable risks (red). Again, the illustrated risk regions are provided merely as an example, and a manufacturer can establish their own way of delineating acceptable and unacceptable risks, as may be appropriate for their device.

Hazards that are evaluated to have an unacceptable risk level require risk control measures. Borderline risks may also require risk control measures upon further investigation. While not required, risk control measures may also be desirable for acceptable risks as these may still improve the safety and performance of the device and lead to better end-user satisfaction. RPNs provide a way to prioritize the allocation of limited resources within a particular risk region.

Risk control measures defined in ISO 14971 include inherent safety by design, protective measures in the medical device itself or in the manufacturing process, and information for safety. Implementation and effectiveness of risk control measures must be verified and validated by the manufacturer. To evaluate the effectiveness of risk control measures, it is often necessary to conduct usability tests. For example, if information for safety is utilized, it is important that information is perceivable, understandable, and supports correct use of the device by the intended user group in the context of its intended use environment. IEC 62366-1 [50] is an international standard that can be used with ISO 14971 to conduct these evaluations. The US FDA has also developed their own guidance document [51].

After risk control measures are applied, any residual risk is required to be evaluated. Residual risk that is judged not to be acceptable requires further risk control measures. In the event residual risk is not acceptable and further risk control is not practicable, the manufacturer may conduct a risk–benefit analysis by gathering and reviewing data and literature to determine if the medical benefits of the intended use outweigh the residual risk. Information for safety may be used by the manufacturer to disclose risks that may outweigh the benefits of the device.

Figure 2. Risk evaluation matrix with risk priority numbers (RPNs) generated when multiplying the severity of harm rank (Table 1) with the corresponding probability of occurrence rank (Table 2). The risk evaluation matrix is divided into 3 risk regions, with acceptable risks in green, unacceptable risks in red, and borderline risks in yellow.
View this figure

Software Life Cycle Processes

The majority of software problems are traceable to design and development errors, making software design control critical [52]. In both the United States and EU, all software components must be under design control or purchasing control, including design validation that includes software validation and risk analysis [53]. The EU adopted a new essential requirement regarding software in 2007, Essential Requirement 12.1.a [54], addressing the software life cycle. In EU MDR, safety and performance requirements (SPRs) replace the essential requirements and SPR 17 places greater emphasis on the entire product life cycle, as well as introducing specific requirements for mobile computing platforms and information security. Likewise, the US FDA has guidance on general validation principles applicable to medical device software [55].

IEC 62304 [56] provides specific guidance on the processes to be performed for the development of medical device software, including risk management activities. IEC 62304 is an EU harmonized standard and is recognized by the FDA as an approved consensus standard and thus can be used as a benchmark to comply with both markets’ regulatory requirements. This standard provides a life cycle process framework, with activities and tasks necessary for safe medical development. A central theme of IEC 62304 is the need to establish and maintain traceability between system requirements, software requirements, software testing, and risk control measures implemented in software [57]. Established user needs provide the foundation from which all software requirements are derived and must be maintained throughout the product life cycle. It is important to note that while the V-model is often used to illustrate these requirements, a “waterfall” approach is not necessary. IEC 62304 clarifies incremental strategies (eg, Agile), which acknowledge user needs may not be fully defined or may evolve throughout the product life cycle, can still meet the requirements specified in the standard.

Ultimately, it is the responsibility of the manufacturer to define how user needs and system requirements are captured, refined, and tested during product development. Numerous resources are available to help with this process and provide a way for manufacturers to demonstrate traceability and generate reports necessary for regulatory compliance while benefiting from the value of Agile practices. For example, the Association for the Advancement of Medical Instrumentation (AAMI) has published a technical information report (TIR)—AAMI TIR45:2012 [58]—to help provide guidance on how best to align Agile and medical device regulatory perspectives to develop safe and effective medical device software. AAMI TIR45 covers key topics such as documentation, evolutionary design and architecture, traceability, verification and validation, management changes, and “done” criteria. While adopting the practices recommended in AAMI TIR45 is not strictly required, and there are certainly other sensible ways to adapt Agile methodologies for medical device development, the US FDA recognizes AAMI TIR45 as a consensus standard. This recognition provides assurance that Agile practices can be successfully adapted to meet regulatory compliance requirements.

An important feature of AAMI TIR45 is that it provides a framework for reconciling the user story approach [59] for incrementally specifying product requirements with the design input/design output framework used in medical device development. A user story is a short, simple description of a feature told from the perspective of the person who desires the new capability (ie, an end user of the system or other stakeholder). User stories are typically written following the template: As a <type of user>, I want <some goal> so that <some reason>. To elaborate on a story, an accompanying set of acceptance criteria can be specified. User stories can also be broken down into more specific user stories when necessary and appropriate. As AAMI TIR45 explains, the goal of a user story is to be persistent and lightweight, capturing just enough essence of a requirement to allow for future discussions to uncover or elaborate more when needed.

Summary of VRaMD Design and Development Considerations

To summarize:

  1. Depending on the regulatory jurisdiction and classification, VRaMD design and development may need to be captured in a QMS following design control requirements. This may include use of ISO 13485 or local regulations (eg, US FDA QSR).
  2. For software development specifically, IEC 62304 provides a standard framework. AAMI TIR45 can used as guidance for adopting Agile practices, if desired.
  3. Risk management according to ISO 14971 should be performed during the design process. FMEA is one risk analysis tool that can be adapted to ISO 14971. Effectiveness of risk control measures can be evaluated in usability tests following IEC 62366.

To demonstrate these concepts, a VRaMD for home use and remote patient monitoring is specified using a set of generic user stories. Applying FMEA, hazards associated with each user story are identified and risk is evaluated based on estimations of probability of occurrence and severity of harm. Risk control measures are proposed, and the residual risk is determined to demonstrate how a safe and effective VRaMD may be designed.

Requirements Analysis

To specify the requirements for an at-home VR rehabilitation system, it is helpful to review similar devices that are already legally marketed. In the United States, this is a common regulatory strategy. Demonstrating substantial equivalence to a legally marketed device, referred to as a predicate device, enables many devices to be cleared under the premarket notification [510(k)] submission process. In the US FDA medical device classification scheme, devices are classified as Class I, II, or III based on risk level, with Class I devices presenting the lowest risk, and Class III devices presenting the highest risk. Within each FDA class, device types are classified within regulations, which include special control requirements for Class II devices. The intended use and technological characteristics of a system obtaining 510(k) clearance are often made publicly available as a “510(k) Summary” (21 CFR 807.92).

Over the past decade, numerous devices that included at-home physical rehabilitation using video game technology received 510(k) clearance. This began with Jintronix (Jintronix Inc.) [60-62] and went on to include the Recovr Rehabilitation System (Recovr, Inc.) [63], Vera (Reflexion Health, Inc.) [64], the Yugo System (BioGaming Ltd.), the Virtual Occupational Therapy Application (Barron Associates, Inc., marketed as SaeboVR by Saebo, Inc.) [65-67], Uincare Home (UINCARE Corp.) [68], and MindMotion Go (MindMaze SA) [69]. These devices are regulated as Class II devices in the United States. None of these systems utilize HMD-based VR, but rather the Microsoft Kinect motion tracking system. Still, each system includes the intended use of supporting physical rehabilitation of adults at-home by providing therapy guidance for patients and remotely accessible performance metrics for medical professionals.

By examining 510(k) Summaries for these devices, it can be seen they generally include 3 separate applications: a patient-facing application, a clinician-facing application, and a cloud-based server for providing data storage and managing communication between the 2 applications. The patient-facing application (henceforth, patient application) prompts and monitors patients in the performance of a therapy prescribed by their clinicians, reports performance data to the clinician for analysis, and provides an interface for patients to communicate with their clinician. The clinician facing-application (henceforth, clinician application) allows a clinician to define and update a patient’s personal data, a patient’s therapy prescription, monitor a patient’s performance of that therapy, and permit a clinician to communicate with a patient. Thus, the common components of a VR telerehabilitation system have been established. Taking these descriptions into account, the core functionality for an immersive VR therapy system can be specified by replacing the Kinect with a standalone HMD-based VR system with 6-DOF tracking (Figure 3). A generic set of user stories for the patient application can be constructed from the descriptions of these systems (Table 3). Likewise, a generic set of user stories for the clinician application can also be constructed (Table 4). These generic user stories have provided a basis for designing additional therapy systems, including one using HMD-based VR [70,71].

Figure 3. Overview of generic VR as a medical device system for home-use and telerehabilitation. HMD: head-mounted display; VR: virtual reality.
View this figure
Table 3. Overview of virtual reality system requirements—patient stories.
IDSummaryAs a patient I that...
P1Trusted exerciseto trust that exercises I perform have clinical utilityI know the time I spend using the system is worthwhile and will benefit my health.
P2Ease of useto feel confident I will be able to use the system at-home with minimal assistanceI can effectively benefit from the system and recover independently.
P3Portabilityto be able to perform therapy at homeI do not have to rely on transportation and scheduling and can recover on my own.
P4Clinician supervisionmy clinical health care provider to have access to my therapy datacommunication regarding my at-home therapy is streamlined.
P5Instructionsto have clear instructions on how to perform exercisesI am reminded how to properly complete these actions.
P6Feedbackto have feedback as I am performing exercisesI know if I am performing exercises effectively.
P7Motivationexercises to feel more like a video game than homeworkI am motivated to adhere to prescribed exercises.
P8Performance summaryto see the results of my exercise performanceI know if I am performing exercises effectively based on established targets.
P9Track progresssee my progress over timeI am intrinsically motivated to advance my recovery.
P10Remindersto be reminded to do exercisesI do not forget and take longer to recover.
Table 4. A generic set of clinician requirements for a virtual reality as a medical device system designed for home use and remote patient monitoring.
IDSummaryAs a clinician I that...
C1Clinical validityto know how system routines correspond to clinically valid therapiesI know how to apply my current understanding of therapy to the system’s functionality.
C2Instructionsto be able to learn the system quickly and completelyI am confident I understand how to utilize the system and integrate it into my practice without too much difficulty.
C3Ease of useto be able to quickly train patients and caregivers on the systemI feel confident they will be able to utilize the system at home.
C4Manage patientsan online dashboard to manage my patientsI can add patients to the system.
C5Personalizationto be able to give each patient their own profileI can customize individual patients and track their progress over time.
C6Customizationto be able to specify exercises for patientsI can target areas where patients need improvement.
C7Assessmentto be able to assess patient abilityI can determine a baseline challenge level that can be used to monitor progress.
C8Remindersto remind patients to complete therapypatients can be re-engaged if they are not active.
C9Adherence trackingto know when patients complete therapyI know if patients are adhering to recommended frequency.
C10Exercise trackingto know how much patients complete therapyI know if patients how much patients accomplish and whether they are being adequately challenged.
C11Movement trackingto know how patients perform treatmentI know how patient’s exercise and if they are using clinically valid movements.
C12Progress trackingto know how well patients perform exercisesI know a patient’s ability at a given time.
C13Symptom trackingto know how patient symptoms change (eg, improve)I know if the patient can progress in a therapy.
C14Remote feedbacka way to provide feedback to patientsI can communicate with patients, if necessary.
C15Exportabilitya way to export patient performance records (eg, through a printable report)I can share clinically interpretable data with other members of the patient’s clinical care team and with payers.
C16Cybersecuritymy patients’ records to be secure and protected I can manage protected health information responsibly.

Risk Analysis and Evaluation

For each user story, the adapted FMEA process can be used to identify hazards (ie, potential sources of harm) of that feature (Multimedia Appendix 1). Hazards can lead to hazardous situations which may then cause harm. As part of the risk analysis, the probability of occurrence of harm is estimated based on a combination of the likelihood of both the hazard and the hazardous situation. Then, the severity of the harm produced by each hazardous situation is considered. For example, the display screens used in HMD-based VR can be considered a consistent hazard, which may lead to a variety of different hazardous situations. When identifying hazardous situations, it is important to consider both normal and abnormal use of the device. Likewise, hazards can still cause harm without a device failure. Even when using HMD-based VR as intended, there is potential for side effects including eye strain, claustrophobia, overstimulation, anxiety, and seizures. On one extreme, eye strain may be considered a common side effect of using HMD-based VR. However, the severity of resulting short-lasting headaches may be considered low. Alternatively, patients with photosensitivity may experience seizures. While the probability of this occurrence is much lower, if the patient is using the system independently at home, this seizure could fatal, and thus, critically severe.

In addition to the hazardous situations related to HMD-VR specifically, Multimedia Appendix 1 lists many hazards that would be common to non-HMD-VR at-home therapies. For example, data failing to properly synchronize between the clinician and patient applications can lead to the patient not receiving proper treatment. Hazardous situations also arise whenever either user group is unable to properly interpret instructions or data provided to them. This is most severe when it leads to the patient not being able to benefit from treatment. Thus, it can be seen how critical usability engineering can be to ensure proper medical device function. Figure 4 summarizes the identified risks associated with the system.

Figure 4. Risk analysis summary for a head-mounted display-based virtual reality (HMD-VR) medical device for home use and telerehabilitation.
View this figure

Risk Control Measures and Residual Risk Evaluation

ISO 14971 defines “safety” as freedom from unacceptable risk. Based on the risk assessment and acceptability criteria defined earlier, several unacceptable risks were identified that require risk control measures. A variety of borderline risks were also identified, which would require further investigation to determine whether risk control measures are necessary. While the remaining hazards were deemed to have an acceptable risk level, these potential failure points highlight opportunities to improve the system design and create greater patient and clinician satisfaction with the system. Thus, in Multimedia Appendix 2, risk control measures have been proposed for all hazards.

After completing the residual risk evaluation, almost all risks have been brought within the acceptable region. The only remaining borderline risk is associated with the potential for seizures in patients with photosensitivity. While the likelihood of a patient with these issues can be reduced by prescreening patients for a history of seizures, the harm associated with this situation is still considered severe. Overall, an HMD-based VRaMD can be designed to be safe for at-home use and remote patient monitoring when risk control measures are applied. Figure 5 summarizes the risk control measures that are recommended.

When examining the potential sources of unacceptable risks, important areas to consider early in the design and development of a VRaMD can be identified. For example, one source of unacceptable risk is related to insufficient clinician training resources, which can lead to a clinician not understanding the proper intended use of the system and prescribing it to inappropriate patients; clinicians not understanding how to properly configure the system to meet patient needs; and clinicians not understanding how to train patients on the system. To control for this risk, usability testing with clinicians should verify the system’s intended use is understandable and meets clinician needs. For clinicians to be able to utilize the system, they must both understand how to develop individualized patient treatment plans and interpret patient data generated by the system. Furthermore, there should be adequate resources available for clinicians to be able to train patients on the system, if necessary. Likewise, usability testing with patients should verify patient application exercise instructions are adequate to elicit target therapeutic actions. Resources specific to VRaMD usability evaluations should be considered [72]. System feedback provided to patients should also be interpretable by patients and, ideally, provide motivation so patients do not become discouraged by their results for whatever reason.

Hazardous situations can also occur when data between the patient and clinician applications can not properly synchronize due to internet connectivity issues. Depending on the severity of harm, it may be necessary for an internet connection to be provided as part of the system to ensure data can synchronize. This could also alleviate risk caused by patients not being able to connect the device to their home network. However, if a stable internet connection is available, it may be sufficient to provide adequate instruction and have a remote provider verify the patient has completed the necessary set up.

Finally, cybersecurity and patient privacy issues must be addressed. Indeed, protections for these concerns are critical for adoption of a connected health technology [73,74].

Figure 5. Summary of risk control measures to improve safety and end-user satisfaction.
View this figure

Cybersecurity and Patient Privacy Protections

Regulatory bodies generally require cybersecurity considerations as an aspect of the device design and risk management process. MDCG 2019-16 [75] provides guidance on how to fulfill essential requirements regarding cybersecurity specified in Annex I of EU MDR. Likewise, the US FDA provides guidance regarding cybersecurity information to be included in FDA premarket submissions [76], as well as guidance for postmarket management of cybersecurity vulnerabilities [77]. While it is generally up to the manufacturer to determine what cybersecurity controls are necessary for their device, applying recognized standards can help demonstrate implemented capabilities are appropriate and effective.

A variety of standards for addressing medical device cybersecurity are available to help manufacturers ensure they are following industry best practices. Selecting which standards to adopt can depend on the specific technologies and interfaces used by a product (see [78] for further discussion). One starting point is ANSI/AAMI/IEC TIR80001-2-2:2012 [79], which presents an informative set of high-level security capabilities that are intended to facilitate more effective communication of security requirements with stakeholders. The Manufacturer Disclosure Statement for Medical Device Security (MDS2) form [80] is aligned with the security capabilities described in TIR80001-2-2 and provides manufacturers a format for reporting the data assets handled by a medical device, as well as the approach taken to secure it. The MDS2 form thus provides a way for manufacturers to disclose to health care organizations (eg, hospitals) information necessary for them to conduct their own cybersecurity risk analyses [81].

Once necessary security capabilities are identified through risk management and understanding stakeholder needs, AAMI/IEC TIR80001-2-8:2016 can be used to determine specific design requirements from a set of common security standards. This allows a design team to select appropriate standards, as well as provide evidence that each of the applicable security capabilities have been met [78].

When evaluating necessary cybersecurity capabilities (eg, data access controls) and associated procedures (eg, notification of patient’s data rights, notifications of detected data breaches to appropriate stakeholders), medical device manufacturers must understand and comply with the local legal requirements (eg, General Data Protection Regulation [GDPR], Health Insurance Portability and Accountability Act [HIPAA]). Regarding data rights and governance, manufacturers may employ end user license agreements, terms of service, and privacy policies to establish and convey company and user data rights for monitoring, evaluation, and distribution of collected data [73]. Additional precautions may be necessary for certain patient populations, such as children (eg, the Children’s Online Privacy Protection Act).

Summary of Recommendations

HMD-based VRaMDs, depending upon their intended use, will likely be subject to the same regulatory requirements as other medical devices. Quality requirements such as design controls may be unfamiliar to product designers and software professionals coming from unregulated fields. While design control requirements may appear to suggest a “waterfall” approach is necessary, it is not incompatible with Agile practices, which can be used once properly adapted. Incorporating regulatory requirements early in the design process is not only necessary but also helps eliminate costly reworkings later in development. Incorporating a risk management process will help systematically expose ways to make the product safer and improve end-user satisfaction. A comprehensive usability engineering plan is necessary to verify risk control measures are effective.

Using non-HMD-based VR systems already legally marketed in the United States for at-home therapy, a generic set of user stories for both patients and clinicians was specified here. While HMD-based VR introduces unique hazards to at-home therapy, the associated risks can be mitigated with appropriate control measures, demonstrating that HMD-based VR can be designed to be safe for home use and remote patient monitoring.

For clinicians, it is important they understand the proper intended use of the system. This will enable them to prescribe the system to appropriate patients, understand how to configure the system to meet a particular patient’s needs, and be able to interpret system performance metrics as intended to progress a patient through treatment. This can be accomplished through robust user interface design and providing clinicians with the necessary training resources. The effectiveness of these measures should be verified in usability testing. Finally, the accuracy of OTS movement tracking sensors should be verified to be within clinically relevant ranges if 3D motion data are used in assessing patient performance. Studies have already evaluated the accuracy of the Oculus Touch controllers [82] and HTC VIVE motion tracking sensors [83] for clinical use in motor rehabilitation.

Patients must understand risks associated with HMD-VR so that they may avoid hazardous situations at home. General risks may be avoided by properly clearing the environment of obstacles, avoiding standing with the headset on, taking breaks to rest, and stopping use of the device if they experience negative effects. When instructing patients to perform therapeutic actions, it is important they have the necessary guidance and feedback to determine if they are performing the therapy as intended. When providing patients with progress data, it is important these data are easily interpretable. Ideally, performance feedback data should not discourage the patient from continuing treatment. To determine if system safety information is effective, usability testing in patients’ homes is necessary. An iterative human-centered design approach with clinicians and patients can help guide design concepts toward success early in development [84,85]. Assuming transmission of data between clinicians and patients is necessary for effective treatment, measures should be taken to verify the device is connected to the internet upon arrival at the patient’s home.

Finally, appropriate patient privacy and cybersecurity protections are essential. Standards can be utilized to determine necessary security measures and how to implement them effectively. Stakeholder needs, including relevant data privacy regulations, will contribute to the assessment of necessary cybersecurity capabilities. The MDS2 form provides one method for communicating with health care providers data handled by the system and how they are protected. End users should be provided with a privacy notice that describes how data are collected, used, and retained, the types of data that the product obtains, the length of data retention, and how and by whom information is used.


This paper describes a generic VRaMD system, using devices with a similar intended use as a basis. System functionality was specified at only the highest level to provide a reasonable scope for examination and discussion in the paper. Given a more specific intended use, more detailed requirements will be specified that may introduce new hazards to the system. The probability of occurrence and severity of device harms were roughly estimated for practical purposes.

A standalone HMD-based VR system with 6-DOF tracking was used as the core technology. Use of other VR systems may introduce different hazards. For example, a non-standalone system with external sensors (eg, Oculus Rift) may require additional set up and monitoring of sensor placement. More expensive head-mounted augmented and mixed reality systems (eg, Microsoft HoloLens, Magic Leap) were also not considered, although augmented and mixed reality–based medical devices are in development and may resolve certain hazards and limitations associated with occlusive VRaMD [86,87].

Overall, the intention of this paper was to provide an overview of an ISO 14971-compliant risk management process. To accomplish this, it was necessary to review related medical device regulations, standards, and guidance documents. While these requirements and recommendations are applicable to a variety of SaMD, specific devices and regulatory jurisdictions may require additional considerations. This paper was also not intended to be an exhaustive review of applicable standards. For example, the IEC 60601-1 [88] series of standards for electrical medical devices was not discussed. This was done to keep the focus of the paper on software and implementation of IEC 62304. However, IEC 60601-1 may be necessary for demonstrating the safety and electromagnetic compatibility of system hardware. More general (ie, nonmedical device specific) standards may also be useful for the design process, such as ISO 9241-210:2019 [89]. The IEEE Virtual Reality and Augmented Reality Working Group is developing standards for VR design that could be useful to apply to improve the safety, usability, and standardization of VRaMD [90]. Ultimately, a VRaMD manufacturer should communicate with the appropriate regulatory bodies when developing a new product intended for commercialization. The review provided here is intended to help orient those new to medical device development and provide a broad overview of regulatory requirements applicable to a variety of jurisdictions.

Comparison With Prior Work

Recent advances in the widespread availability of VR and its potential in therapy have led to growing interest in the development of industry best practices for translating this potential to a reality. For example, the Virtual Reality Clinical Outcomes Research Experts (VR-CORE) committee has published a framework for iterative clinical trial design for validating VR therapies [91]. Numerous papers have also shared the design process for various HMD-based VR interventions [92-99]. The focus of this paper was less about the design of a particular system, and more about demonstrating a risk management process to develop a VRaMD safe for home use and remote patient monitoring.

Numerous reports have examined challenges and best practices for introducing medical device regulatory requirements, such as design controls and risk management, into contemporary software development practices such as Agile [39,40,100-110]. Here, Agile techniques were introduced primarily as a method for specifying the requirements of the system through user stories. Additional concepts were introduced as necessary to demonstrate the risk management process. It is expected that VR developers working on VRaMD will be coming from nonregulated fields (eg, video games, entertainment), and thus it is important to provide some necessary background. While the transition to medical device development can be challenging, this paper describes how Agile practices can be utilized to develop a safe and effective VRaMD. More recently, the HMD-based REAL Immersive system obtained 510(k) clearance for in-clinic use, providing further evidence that immersive VRaMD can successfully meet regulatory requirements.


HMD-based VR offers tremendous potential for novel at-home treatments. However, for these treatments to be successfully translated into clinical practice, VRaMD will need to be designed following the necessary regulatory requirements. While regulatory requirements can appear challenging, VRaMD designers should find it beneficial to gain an understanding of what is required so they may adapt their design process early in development. While medical device design controls present a need for comprehensive documentation of device design, incorporating risk management early in this process should help further refine system requirements. Following these recommendations will help make VRaMDs safe and effective, as well as improve patient and clinician satisfaction with these novel digital therapeutics.


This work was supported in part by the US National Institutes of Health (R43AG05287).

Conflicts of Interest

JPS is an employee of Cognivive, Inc., which markets CogniviveVR, an HMD-based immersive VR rehabilitation system for home use.

Multimedia Appendix 1

Risk analysis and risk evaluation of patient and clinician user stories.

XLSX File (Microsoft Excel File), 46 KB

Multimedia Appendix 2

Failure modes and effects analysis including risk control measures and residual risk analysis.

XLSX File (Microsoft Excel File), 60 KB

  1. Jerdan SW, Grindle M, van Woerden HC, Kamel Boulos MN. Head-Mounted Virtual Reality and Mental Health: Critical Review of Current Research. JMIR Serious Games 2018 Jul 06;6(3):e14 [FREE Full text] [CrossRef] [Medline]
  2. Knaust T, Felnhofer A, Kothgassner OD, Höllmer H, Gorzka R, Schulz H. Virtual Trauma Interventions for the Treatment of Post-traumatic Stress Disorders: A Scoping Review. Front Psychol 2020;11:562506 [FREE Full text] [CrossRef] [Medline]
  3. Singh S, Nathan-Roberts D. Virtual Reality Exposure Therapy and Military Personnel with Post-Traumatic Stress Disorder: A Systematic Review. 2019 Nov 20 Presented at: Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2019; Los Angeles, CA p. 1378-1383. [CrossRef]
  4. Oing T, Prescott J. Implementations of Virtual Reality for Anxiety-Related Disorders: Systematic Review. JMIR Serious Games 2018 Nov 07;6(4):e10965. [CrossRef] [Medline]
  5. Lindner P, Hamilton W, Miloff A, Carlbring P. How to Treat Depression With Low-Intensity Virtual Reality Interventions: Perspectives on Translating Cognitive Behavioral Techniques Into the Virtual Reality Modality and How to Make Anti-Depressive Use of Virtual Reality-Unique Experiences. Front Psychiatry 2019;10:792 [FREE Full text] [CrossRef] [Medline]
  6. Bisso E, Signorelli MS, Milazzo M, Maglia M, Polosa R, Aguglia E, et al. Immersive Virtual Reality Applications in Schizophrenia Spectrum Therapy: A Systematic Review. Int J Environ Res Public Health 2020 Aug 22;17(17):6111 [FREE Full text] [CrossRef] [Medline]
  7. Clus D, Larsen ME, Lemey C, Berrouiguet S. The Use of Virtual Reality in Patients with Eating Disorders: Systematic Review. J Med Internet Res 2018 Dec 27;20(4):e157 [FREE Full text] [CrossRef] [Medline]
  8. Pourmand A, Davis S, Marchak A, Whiteside T, Sikka N. Virtual Reality as a Clinical Tool for Pain Management. Curr Pain Headache Rep 2018 Jun 15;22(8):53. [CrossRef] [Medline]
  9. Smith V, Warty RR, Sursas JA, Payne O, Nair A, Krishnan S, et al. The Effectiveness of Virtual Reality in Managing Acute Pain and Anxiety for Medical Inpatients: Systematic Review. J Med Internet Res 2020 Nov 02;22(11):e17980 [FREE Full text] [CrossRef] [Medline]
  10. Maggio MG, Russo M, Cuzzola MF, Destro M, La Rosa G, Molonia F, et al. Virtual reality in multiple sclerosis rehabilitation: A review on cognitive and motor outcomes. J Clin Neurosci 2019 Jul;65:106-111. [CrossRef] [Medline]
  11. Triegaardt J, Han TS, Sada C, Sharma S, Sharma P. The role of virtual reality on outcomes in rehabilitation of Parkinson's disease: meta-analysis and systematic review in 1031 participants. Neurol Sci 2020 Mar;41(3):529-536 [FREE Full text] [CrossRef] [Medline]
  12. Maggio MG, De Luca R, Molonia F, Porcari B, Destro M, Casella C, et al. Cognitive rehabilitation in patients with traumatic brain injury: A narrative review on the emerging use of virtual reality. J Clin Neurosci 2019 Mar;61:1-4. [CrossRef] [Medline]
  13. Wiley E, Khattab S, Tang A. Examining the effect of virtual reality therapy on cognition post-stroke: a systematic review and meta-analysis. Disabil Rehabil Assist Technol 2020 May 02:1-11. [CrossRef] [Medline]
  14. Mekbib DB, Han J, Zhang L, Fang S, Jiang H, Zhu J, et al. Virtual reality therapy for upper limb rehabilitation in patients with stroke: a meta-analysis of randomized clinical trials. Brain Inj 2020 Mar 20;34(4):456-465. [CrossRef] [Medline]
  15. Clay F, Howett D, FitzGerald J, Fletcher P, Chan D, Price A. Use of Immersive Virtual Reality in the Assessment and Treatment of Alzheimer's Disease: A Systematic Review. J Alzheimers Dis 2020;75(1):23-43 [FREE Full text] [CrossRef] [Medline]
  16. D'Cunha NM, Nguyen D, Naumovski N, McKune AJ, Kellett J, Georgousopoulou EN, et al. A Mini-Review of Virtual Reality-Based Interventions to Promote Well-Being for People Living with Dementia and Mild Cognitive Impairment. Gerontology 2019;65(4):430-440 [FREE Full text] [CrossRef] [Medline]
  17. Garrett B, Taverner T, Gromala D, Tao G, Cordingley E, Sun C. Virtual Reality Clinical Research: Promises and Challenges. JMIR Serious Games 2018 Oct 17;6(4):e10839 [FREE Full text] [CrossRef] [Medline]
  18. Smits M, Staal JB, van Goor H. Could Virtual Reality play a role in the rehabilitation after COVID-19 infection? BMJ Open Sport Exerc Med 2020;6(1):e000943 [FREE Full text] [CrossRef] [Medline]
  19. Kim W, Cho S, Ku J, Kim Y, Lee K, Hwang H, et al. Clinical Application of Virtual Reality for Upper Limb Motor Rehabilitation in Stroke: Review of Technologies and Clinical Evidence. J Clin Med 2020 Oct 21;9(10):3369 [FREE Full text] [CrossRef] [Medline]
  20. Clemenson GD, Stark CEL. Virtual Environmental Enrichment through Video Games Improves Hippocampal-Associated Memory. J Neurosci 2015 Dec 09;35(49):16116-16125 [FREE Full text] [CrossRef] [Medline]
  21. Choi JW, Kim BH, Huh S, Jo S. Observing Actions Through Immersive Virtual Reality Enhances Motor Imagery Training. IEEE Trans Neural Syst Rehabil Eng 2020 Jul;28(7):1614-1622. [CrossRef] [Medline]
  22. Kothgassner OD, Felnhofer A. Does virtual reality help to cut the Gordian knot between ecological validity and experimental control? Annals of the International Communication Association 2020 Jul 12;44(3):210-218. [CrossRef]
  23. Horan B, Heckenberg R, Maruff P, Wright B. Development of a new virtual reality test of cognition: assessing the test-retest reliability, convergent and ecological validity of CONVIRT. BMC Psychol 2020 Jun 12;8(1):61 [FREE Full text] [CrossRef] [Medline]
  24. Parsons TD. Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences. Front Hum Neurosci 2015;9:660 [FREE Full text] [CrossRef] [Medline]
  25. Hrgarek N. Certification and regulatory challenges in medical device software development. 2012 Presented at: 2012 4th International Workshop on Software Engineering in Health Care (SEHC); June 4-5, 2012; Zurich, Switzerland. [CrossRef]
  26. Gilman BL, Brewer JE, Kroll MW. Medical device design process. Annu Int Conf IEEE Eng Med Biol Soc 2009;2009:5609-5612. [CrossRef] [Medline]
  27. Kinsel D. Design control requirements for medical device development. World J Pediatr Congenit Heart Surg 2012 Jan 01;3(1):77-81. [CrossRef] [Medline]
  28. International Medical Device Regulators Forum. Software as a Medical Device (SaMD): Key Definitions. 2013.   URL: [accessed 2021-02-22]
  29. Sharples S, Cobb S, Moody A, Wilson JR. Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems. Displays 2008 Mar;29(2):58-69. [CrossRef]
  30. Serge SR, Moss JD. Simulator Sickness and the Oculus Rift: a First Look. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2016 Dec 20 Presented at: Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2015; Los Angeles, CA p. 761-765. [CrossRef]
  31. McGill M, Boland D, Murray-Smith R, Brewster S. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays. New York, NY: ACM Press; 2015 Presented at: CHI '15: CHI Conference on Human Factors in Computing Systems; April 2015; Seoul, Republic of Korea. [CrossRef]
  32. Baker S, Waycott J, Robertson E, Carrasco R, Neves BB, Hampson R, et al. Evaluating the use of interactive virtual reality technology with older adults living in residential aged care. Information Processing & Management 2020 May;57(3):102105. [CrossRef]
  33. Tuena C, Pedroli E, Trimarchi PD, Gallucci A, Chiappini M, Goulene K, et al. Usability Issues of Clinical and Research Applications of Virtual Reality in Older People: A Systematic Review. Front Hum Neurosci 2020;14:93 [FREE Full text] [CrossRef] [Medline]
  34. Global Harmonization Task Force. Report No.: GHTF/SG1/N071:2012. Definition of the Terms "Medical Device" and "In Vitro Diagnostic (IVD) Medical Device".   URL: http:/​/www.​​docs/​ghtf/​final/​sg1/​technical-docs/​ghtf-sg1-n071-2012-definition-of-terms-120516.​pdf [accessed 2021-02-22]
  35. Definitions; generally - 21 U.S. Code § 321(h). 2019.   URL: https:/​/www.​​content/​pkg/​USCODE-2019-title21/​pdf/​USCODE-2019-title21-chap9-subchapII-sec321.​pdf [accessed 2021-05-29]
  36. US Food and Drug Administration. Policy for device software functions and mobile medical applications. 2019.   URL: [accessed 2021-02-22]
  37. Quality System Regulation, 21 C.F.R. § 820. 2020.   URL: [accessed 2021-05-29]
  38. US Food and Drug Administration. Guidance for industry: Quality systems approach to pharmaceutical cGMP regulations. 2006.   URL: [accessed 2021-02-22]
  39. McHugh M, McCaffery F, Casey V, Pikkarainen M. Integrating Agile Practices with a Medical Device SDLC. European Systems and Software Process Improvement and Innovation Conference (EuroSPI, Vienna, Austria, 21-25 May). 2012.   URL: [accessed 2021-05-20]
  40. Spence J. There has to be a better way! software development. New York, NY: IEEE; 2005 Presented at: Agile Development Conference (ADC'05); Denver, CO; July 24-29, 2005 p. 272. [CrossRef]
  41. McHugh M, Cawley O, McCaffcry F, Richardson I, Wang X. An agile v-model for medical device software development to overcome the challenges with plan-driven software development lifecycles. 2013 Presented at: 5th International Workshop on Software Engineering in Health Care (SEHC); 2013; San Francisco, California. [CrossRef]
  42. Luczak J. Establishing a small company's medical device quality system. The TQM Journal 2012 Jun 08;24(4):363-373. [CrossRef]
  43. ISO 13485:2016 Medical devices — Quality management systems — Requirements for regulatory purposes. International Organization for Standardization.   URL: [accessed 2021-02-22]
  44. Forsberg K, Mooz H. The Relationship of System Engineering to the Project Cycle. In: INCOSE International Symposium. 1991 Oct Presented at: INCOSE International Symposium; 1991; Chattanooga, TN p. 57-65. [CrossRef]
  45. European Commission. Regulation (EU) 2017/745 of the European Parliament and of the Council. 2017.   URL: [accessed 2021-05-29]
  46. Design Controls, 21 C.F.R. § 820.30. 2020.   URL: [accessed 2021-05-29]
  47. ISO 14971:2019 Medical devices — Application of risk management to medical devices. International Organization for Standardization.   URL: [accessed 2021-02-22]
  48. Stadler J, Seidl N. Software failure modes and effects analysis. 2013 Jan Presented at: Proceedings Annual Reliability and Maintainability Symposium (RAMS); 2013; Orlando, Florida, USA. [CrossRef]
  49. IEC 60812:2018 — Failure modes and effects analysis (FMEA and FMECA). International Electrotechnical Commission.   URL: [accessed 2021-02-22]
  50. IEC 62366-1:2015 Medical devices — Part 1: Application of usability engineering to medical devices. International Electrotechnical Commission.   URL: [accessed 2021-02-22]
  51. Applying human factors and usability engineering to medical devices. US Food and Drug Administration. 2016.   URL: [accessed 2021-02-22]
  52. Wallace DR, Kuhn DR. Failure modes in medical device software: an analysis of 15 years of recall data. Int. J. Rel. Qual. Saf. Eng 2011 Nov 21;08(04):351-371. [CrossRef]
  53. Hall G, editor. Fundamentals of Medical Device Regulations, Third Edition. Rockville, MD: Regulatory Affairs Professionals Society (RAPS); 2020.
  54. European Commission. Directive 2007/47/EC of the European Parliament and of the Council. 2007.   URL: [accessed 2021-05-29]
  55. General Principles of Software Validation. US Food and Drug Administration. 2002.   URL: https:/​/www.​​regulatory-information/​search-fda-guidance-documents/​general-principles-software-validation [accessed 2021-02-22]
  56. IEC 62304:2006 Medical device software — Software life cycle processes. International Electrotechnical Commission.   URL: [accessed 2021-02-22]
  57. Regan G, Mc Caffery F, Mc Daid K, Flood D. Medical device standards' requirements for traceability during the software development lifecycle and implementation of a traceability assessment model. Computer Standards & Interfaces 2013 Nov;36(1):3-9. [CrossRef]
  58. AAMI TIR45(R2018) - Guidance on the use of AGILE practices in the development of medical device software. Association for the Advancement of Medical Instrumentation.   URL: [accessed 2021-02-22]
  59. Cohn M. User stories applied: For agile software development. Crawfordsville, Indiana: Addison-Wesley Professional; 2004.
  60. Martel D, Lauzé M, Agnoux A, Fruteau de Laclos L, Daoust R, Émond M, et al. Comparing the effects of a home-based exercise program using a gerontechnology to a community-based group exercise program on functional capacities in older adults after a minor injury. Exp Gerontol 2018 Jul 15;108:41-47. [CrossRef] [Medline]
  61. Valiani V, Lauzé M, Martel D, Pahor M, Manini TM, Anton S, et al. A New Adaptive Home-based Exercise Technology among Older Adults Living in Nursing Home: A Pilot Study on Feasibility, Acceptability and Physical Performance. J Nutr Health Aging 2017;21(7):819-824 [FREE Full text] [CrossRef] [Medline]
  62. Lauzé M, Martel DD, Aubertin-Leheudre M. Feasibility and Effects of a Physical Activity Program Using Gerontechnology in Assisted Living Communities for Older Adults. J Am Med Dir Assoc 2017 Dec 01;18(12):1069-1075. [CrossRef] [Medline]
  63. Hart E, Hayes A, Hodges L, Jett K, Finetto C, Hutchison S, et al. Feasibility of and Adherence to a Home-Based Video Game Rehabilitation Protocol. Am J Occup Ther 2020 Aug 01;74(4_Supplement_1):7411520504p1. [CrossRef]
  64. Prvu Bettger J, Green CL, Holmes DN, Chokshi A, Mather RC, Hoch BT, et al. Effects of Virtual Exercise Rehabilitation In-Home Therapy Compared with Traditional Care After Total Knee Arthroplasty: VERITAS, a Randomized Controlled Trial. J Bone Joint Surg Am 2020 Jan 15;102(2):101-109. [CrossRef] [Medline]
  65. Adams RJ, Lichter MD, Krepkovich ET, Ellington A, White M, Diamond PT. Assessing upper extremity motor function in practice of virtual activities of daily living. IEEE Trans Neural Syst Rehabil Eng 2015 Mar;23(2):287-296 [FREE Full text] [CrossRef] [Medline]
  66. Ellington A, Adams R, White M, Diamond P. Behavioral intention to use a virtual instrumental activities of daily living system among people with stroke. Am J Occup Ther 2015;69(3):6903290030p1-6903290030p8 [FREE Full text] [CrossRef] [Medline]
  67. Adams RJ, Lichter MD, Ellington A, White M, Armstead K, Patrie JT, et al. Virtual Activities of Daily Living for Recovery of Upper Extremity Motor Function. IEEE Trans Neural Syst Rehabil Eng 2018 Jan;26(1):252-260. [CrossRef] [Medline]
  68. Lee H, Chang W, Lee J, Hwang J. Therapeutic potential of the home-based exercise program with the augmented reality system on balance in stroke patients: A preliminary report. Annals of Physical and Rehabilitation Medicine 2018 Jul;61:e36. [CrossRef]
  69. Storz C, Schulte-Göcking H, Woiczinski M, Azqueta-Gavaldon M, Azad SC, Kraft E. [Exergames for patients with complex regional pain syndrome : A feasibility study]. Schmerz 2020 Apr 24;34(2):166-171. [CrossRef] [Medline]
  70. Salisbury J, Liu R, Minahan L, Shin H, Karnati S, Duffy S. Patient Engagement Platform for Remote Monitoring of Vestibular Rehabilitation with Applications in Concussion Management and Elderly Fall Prevention. 2018 Presented at: IEEE International Conference on Healthcare Informatics (ICHI); 2018; New York City, NY. [CrossRef]
  71. Salisbury J, Aronson T, Simon T. At-Home Self-Administration of an Immersive Virtual Reality Therapeutic Game for Post-Stroke Upper Limb Rehabilitation. In: Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play. 2020 Nov Presented at: 2020 Annual Symposium on Computer-Human Interaction in Play; November 2020; Virtual Event (Canada). [CrossRef]
  72. Zhang T, Booth R, Jean-Louis R, Chan R, Yeung A, Gratzer D, et al. A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality. JMIR Serious Games 2020 Oct 28;8(4):e18153 [FREE Full text] [CrossRef] [Medline]
  73. Patel NA, Butte AJ. Characteristics and challenges of the clinical pipeline of digital therapeutics. NPJ Digit Med 2020 Dec 11;3(1):159 [FREE Full text] [CrossRef] [Medline]
  74. Christodoulakis C, Asgarian A. Barriers to adoption of information technology in healthcare. 2017 Nov Presented at: Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering; 2017; Markham, ON, Canada p. 66-75.
  75. Medical Device Coordination Group. MDCG 2019-16 Guidance on Cybersecurity for Medical Devices. 2019.   URL: [accessed 2021-05-29]
  76. Content of Premarket Submissions for Management of Cybersecurity in Medical Devices. US Food and Drug Administration. 2014.   URL: [accessed 2021-02-22]
  77. Postmarket Management of Cybersecurity in Medical Devices. US Food and Drug Administration. 2016.   URL: [accessed 2021-02-22]
  78. Jump M, Finnegan A. Using Standards to Establish Foundational Security Requirements for Medical Devices. Biomed Instrum Technol 2017 Sep 02;51(s6):33-37. [CrossRef] [Medline]
  79. Association for the Advancement of Medical Instrumentation. AAMI TIR80001-2-2:2012 – Application of risk management for IT-networks incorporating medical devices – Part 2-2: Guidance for the disclosure and communication of medical device security needs, risks and controls.   URL: [accessed 2021-05-29]
  80. Manufacturer Disclosure Statement for Medical Device Security. National Electrical Manufacturers Association. 2015.   URL: [accessed 2021-02-22]
  81. Wu F, Eagles S. Cybersecurity for Medical Device Manufacturers: Ensuring Safety and Functionality. Biomed Instrum Technol 2016;50(1):23-33. [CrossRef] [Medline]
  82. Shum LC, Valdés BA, Van der Loos HM. Determining the Accuracy of Oculus Touch Controllers for Motor Rehabilitation Applications Using Quantifiable Upper Limb Kinematics: Validation Study. JMIR Biomed Eng 2019 Jun 06;4(1):e12291. [CrossRef]
  83. Hemphill S, Nguyen A, Rodriguez ST, Menendez M, Wang E, Lawrence K, et al. Mobilization and calibration of the HTC VIVE for virtual reality physical therapy. Digit Health 2020;6:2055207620950929 [FREE Full text] [CrossRef] [Medline]
  84. Harte R, Quinlan LR, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, et al. Human-Centered Design Study: Enhancing the Usability of a Mobile Phone App in an Integrated Falls Risk Detection System for Use by Older Adult Users. JMIR Mhealth Uhealth 2017 May 30;5(5):e71 [FREE Full text] [CrossRef] [Medline]
  85. Harte R, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, Quinlan LR, et al. A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology. JMIR Hum Factors 2017 Mar 16;4(1):e8 [FREE Full text] [CrossRef] [Medline]
  86. Held JPO, Yu K, Pyles C, Veerbeek JM, Bork F, Heining S, et al. Augmented Reality-Based Rehabilitation of Gait Impairments: Case Report. JMIR Mhealth Uhealth 2020 May 26;8(5):e17804 [FREE Full text] [CrossRef] [Medline]
  87. Salisbury JP, Keshav NU, Sossong AD, Sahin NT. Concussion Assessment With Smartglasses: Validation Study of Balance Measurement Toward a Lightweight, Multimodal, Field-Ready Platform. JMIR Mhealth Uhealth 2018 Jan 23;6(1):e15 [FREE Full text] [CrossRef] [Medline]
  88. International Electrotechnical Commission. IEC 60601-1 - Medical electrical equipment - Part 1: General requirements for basic safety and essential performance.   URL: [accessed 2021-05-29]
  89. ISO 9241-210:2019 Ergonomics of Human-System Interaction — Part 210: Human-Centred Design for Interactive Systems. 2019.   URL: [accessed 2021-02-22]
  90. Yuan Y. Paving the Road for Virtual and Augmented Reality [Standards]. IEEE Consumer Electron. Mag 2018 Jan;7(1):117-128. [CrossRef]
  91. Birckhead B, Khalil C, Liu X, Conovitz S, Rizzo A, Danovitch I, et al. Recommendations for Methodology of Virtual Reality Clinical Trials in Health Care by an International Working Group: Iterative Study. JMIR Ment Health 2019 Jan 31;6(1):e11973 [FREE Full text] [CrossRef] [Medline]
  92. AlMousa M, Al-Khalifa HS, AlSobayel H. Requirements Elicitation and Prototyping of a Fully Immersive Virtual Reality Gaming System for Upper Limb Stroke Rehabilitation in Saudi Arabia. Mobile Information Systems 2017;2017:1-12. [CrossRef]
  93. Tong X, Gromala D, Amin A, Choo A. The Design of an Immersive Mobile Virtual Reality Serious Game in Cardboard Head-Mounted Display for Pain Management. In: Pervasive Computing Paradigms for Mental Health. Cham, Switzerland: Springer International Publishing; 2016 Presented at: 5th International Conference, MindCare 2015; September 24-25, 2015; Milan, Italy p. 284-293. [CrossRef]
  94. Eisapour M, Cao S, Domenicucci L, Boger J. Participatory Design of a Virtual Reality Exercise for People with Mild Cognitive Impairment. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. New York, NY: ACM Press; 2018 Presented at: 2018 CHI Conference on Human Factors in Computing Systems; April 21–26, 2018; Montréal, QC, Canada p. 1-9. [CrossRef]
  95. Eisapour M, Cao S, Domenicucci L, Boger J. Virtual Reality Exergames for People Living with Dementia Based on Exercise Therapy Best Practices. 2018 Sep 27 Presented at: Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2018; Los Angeles, CA p. 528-532. [CrossRef]
  96. Shen J, Xiang H, Luna J, Grishchenko A, Patterson J, Strouse RV, et al. Virtual Reality-Based Executive Function Rehabilitation System for Children With Traumatic Brain Injury: Design and Usability Study. JMIR Serious Games 2020 Aug 25;8(3):e16947 [FREE Full text] [CrossRef] [Medline]
  97. Elor A, Teodorescu M, Kurniawan S. Project Star Catcher. ACM Trans. Access. Comput 2018 Nov 22;11(4):1-25. [CrossRef]
  98. Dias P, Silva R, Amorim P, Lains J, Roque E, Pereira ISF, et al. Using Virtual Reality to Increase Motivation in Poststroke Rehabilitation. IEEE Comput Graph Appl 2019;39(1):64-70. [CrossRef] [Medline]
  99. Ou Y, Wang Y, Chang H, Yen S, Zheng Y, Lee B. Development of virtual reality rehabilitation games for children with attention-deficit hyperactivity disorder. J Ambient Intell Human Comput 2020 Apr 10;11(11):5713-5720. [CrossRef]
  100. Rottier P, Rodrigues V. Agile development in a medical device company. New York, NY: IEEE; 2008 Aug Presented at: Agile 2008 Conference; August 4-8, 2008; Toronto, ON, Canada p. 218-223. [CrossRef]
  101. McHugh M, McCaffery F, Coady G. An Agile Implementation within a Medical Device Software Organisation. Cham, Switzerland: Springer International Publishing; 2014 Nov Presented at: International Conference on Software Process Improvement and Capability Determination; November 4-6, 2014; Vilnius, Lithuania p. 190-201. [CrossRef]
  102. Gordon WJ, Stern AD. Challenges and opportunities in software-driven medical devices. Nat Biomed Eng 2019 Jul;3(7):493-497. [CrossRef] [Medline]
  103. Kostić M. Challenges of Agile Practices Implementation in the Medical Device Software Development Methodologies. European Project Management Journal 2017;7(2):36-44 [FREE Full text]
  104. Laukkarinen T, Kuusinen K, Mikkonen T. Regulated software meets DevOps. Information and Software Technology 2018 May;97:176-178. [CrossRef]
  105. Laukkarinen T, Kuusinen K, Mikkonen T. DevOps in regulated software development: case medical devices. New York, NY: IEEE; 2017 Presented at: 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER); May 20-28, 2017; Buenos Aires, Argentina p. 15-18. [CrossRef]
  106. Lie M, Sánchez-Gordón M, Colomo-Palacios R. DevOps in an ISO 13485 regulated environment: A multivocal literature review. In: Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering Measurement (ESEM). New York, NY: ACM; 2020 Presented at: 14th ACM/IEEE International Symposium on Empirical Software Engineering Measurement (ESEM); October 2020; Bari, Italy p. 1-11. [CrossRef]
  107. Keutzer L, Simonsson US. Medical Device Apps: An Introduction to Regulatory Affairs for Developers. JMIR Mhealth Uhealth 2020 Jun 26;8(6):e17567 [FREE Full text] [CrossRef] [Medline]
  108. Lang M. Heart Rate Monitoring Apps: Information for Engineers and Researchers About the New European Medical Devices Regulation 2017/745. JMIR Biomed Eng 2017 Aug 18;2(1):e2 [FREE Full text] [CrossRef]
  109. Lewis TL, Wyatt JC. mHealth and mobile medical Apps: a framework to assess risk and promote safer use. J Med Internet Res 2014 Sep 15;16(9):e210 [FREE Full text] [CrossRef] [Medline]
  110. Vincent CJ, Niezen G, O'Kane AA, Stawarz K. Can standards and regulations keep up with health technology? JMIR Mhealth Uhealth 2015 Jun 03;3(2):e64 [FREE Full text] [CrossRef] [Medline]

AAMI: Association for the Advancement of Medical Instrumentation
CFR: Code of Federal Regulations
cGMP: current good manufacturing practice
DOF: degrees of freedom
EU: European Union
FDA: Food and Drug Administration
FMEA: failure modes and effects analysis
HMD: head-mounted display
IEC: International Electrotechnical Commission
ISO: International Organization for Standardization
MDCG: Medical Device Coordination Group
MDR: Medical Device Regulation
MDS2: Manufacturer Disclosure Statement for Medical Device Security
OTS: off-the-shelf
QMS: quality management system
QSR: quality system regulation
SaMD: software as a medical device
SPR: safety and performance requirements
TIR: technical information report
VR: virtual reality
VRaMD: virtual reality as a medical device

Edited by G Eysenbach; submitted 04.01.21; peer-reviewed by C Jones, B Kim, S van der Veen; comments to author 25.01.21; revised version received 22.02.21; accepted 17.04.21; published 03.06.21


©Joseph Peter Salisbury. Originally published in JMIR Biomedical Engineering (, 03.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Biomedical Engineering, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.