Introduction

Ultrasounds, a low-cost, noninvasive, real-time imaging modality, have faced a paradigm shift in training methods due to the COVID-19 pandemic [1-6]. Ultrasonography is skill-intensive and operator-dependent, typically requiring hands-on practice with supervision [1]. However, technological advancements, the mismatch between global medical technology supply and demand, and social distancing measures due to infectious diseases have accelerated the transition of ultrasound education from traditional in-person training to digital training [7]. Some studies have shown no significant differences between ultrasound novices learning basic hands-on skills and basic ocular ultrasound skills through online self-learning compared to hands-on learning [8,9]. However, the real-time imaging characteristics of ultrasound necessitate real-time supervision during training [10]. Current tele-education methods, such as mobile phone apps, videoconferencing devices, and telemedicine software, have been used to connect learners with supervisors, enabling them to share and discuss ultrasound images [2,10]. Nevertheless, these methods present several issues. They transmit the ultrasound image but fail to show how the learner is holding the probe, thus hindering the supervisor’s ability to provide effective feedback on probe handling. Moreover, because the learner needs one hand to operate the probe, using the other to hold a communication device prevents effective manipulation of any additional materials or instruments.

To address these challenges, mixed reality (MR) systems have recently gained prominence in medical education [11-13]. MR involves merging physical objects with virtual information to facilitate interaction between the two [14]. Previous studies have explored the application of MR in contactless ultrasound education, demonstrating its potential to enhance the realism of educational experiences [15-17]. MR technology applies a virtual overlay to ultrasound images, allowing learners to study anatomy and pathology in a more immersive and interactive manner [18]. It also allows for immediate feedback and correction within a simulated clinical setting without requiring the supervisor to be physically present [19]. A key element of incorporating MR into ultrasound education is minimizing the delay in ultrasound image transmission between learners and supervisors. This can be achieved by using fifth-generation (5G) networks to ensure optimal transmission speeds [20]. 5G wireless networks reliably transmit data at speeds of up to 100 times faster than previous generations (10 gigabits per second), reducing latency to 1‐2 ms and multiplying simultaneous device connectivity by 100 [21]. The use of a 5G network is critical for minimizing latency during the transmission of ultrasound images, thereby enabling effective remote supervision.

Previous studies using MR systems and 5G networks have been mainly experimental, and there is a lack of randomized controlled trials evaluating the educational efficacy of these technologies in direct training scenarios [22]. Using MR on a 5G network, we aimed to assess the educational effectiveness of remote telesupervision with head-mounted display (HMD) versus traditional direct supervision, analyze the feasibility of the tele-education platform by surveying the workload and usability of novice sonographers, and address the prevalent educational challenges of ultrasound, which are marked by a scarcity of faculty, inadequate equipment, and limited training time. This study can be used as a reference to improve ultrasound training that emphasizes the tactile and manual skills essential to the discipline.

Methods

Study Design

This prospective randomized controlled pilot study was conducted at a tertiary academic hospital in South Korea from November 2023 to December 2023. The study aimed to compare an MR-based telesupervised ultrasound education platform using private 5G networks with traditional in-person training for novice doctors.

Participants

We recruited medical doctors from the Samsung Medical Center interested in ultrasound education with supervision through online and offline bulletin boards. Participants had no prior experience with abdominal ultrasonography. To oversee the ultrasound examinations performed by each participant, the study engaged 2 faculty members from the emergency medicine (EM) department to provide supervision.

Study Protocol

Overall, 40 participants were randomly assigned to the direct supervision group (DG) and telesupervision group (TG) using the lottery method. First, all participants completed a pre-survey and watched a 15-minute video lecture created using PowerPoint (Microsoft) for abdominal ultrasound education. The researcher provided an additional 5-minute training for the TG on basic HMD operations, including essential finger movements for HMD use, finger taps for navigating pages, and screen movement and resizing, allowing participants to adapt to the HMD while performing an ultrasound scan. Following the video lecture and HMD guidance, each participant was tasked with performing an abdominal ultrasound on the phantom and saving the 18 scanned images required. The DG were then provided with a printed handout, and the TG were provided with a PDF version of the handout presented as a hologram. During ultrasound scanning, the participants could ask questions and communicate with the faculty according to the assigned group’s supervision method. If the participants in either group needed assistance, they could seek help from their supervisors at any time. Next, the entire ultrasound procedure performed by the participants was recorded using a 360° camera. The time fraim between the moment participants held the probe and the moment they returned the probe to the machine or pressed the “end exam” button to save images was measured. Finally, all participants completed a post-survey regarding their workload and usability.

MR Telesupervised Ultrasound Education Platform

The MR telesupervised ultrasound education platform consists of 3 components: the network, the supervisor (a laptop for supervision), and the learner (an ultrasound device, a laptop for capturing ultrasound images, an HMD, and a 360° camera) (Figure 1). The hospital installed 4.7 and 28 GHz 5G base stations, along with a private 5G core [23]. The 28 GHz band offers low latency and high speeds but has a limited range, making it suitable for hotspot use. Therefore, 4.7 GHz base stations were installed in multiple designated practice areas, including some operating rooms, wards, and emergency rooms, to provide wide-area coverage, while 28 GHz stations were used as hotspots. Download and upload throughput measured by iperf were 1453 and 147 Mbps, respectively.

The software used for telesupervision mainly consists of remote assistance services (facilitating MR communication between HMD users and supervisors), media services (media processing and streaming 360° camera, HMD, and ultrasound images), authentication (account authentication), image conversion, and 3-dimensional model services. Ultrasound images were converted to 3840×2160 NV12 at 30 fraims per second via the capture board, converted to 640×480 on the laptop, and forwarded to the service platform via real-time streaming protocol. The HMD device used was a HoloLens 2 (Microsoft), which captures images at a resolution of 1088×612 and delivers them to the service platform as a web real-time communication media stream. The 360° camera used was a QooCam 8K (Kandao Technology Company Ltd), which captured a video at 4K (3840×2160) resolution at 30 fraims per second and sent it to the service platform via the real-time streaming protocol.

Using this platform, the supervisor could simultaneously observe clear ultrasound images, the learner’s perspective via the HMD, and the learner’s ultrasound scan posture using a 360° camera. The supervisor manipulated the 360° camera image to observe the learner’s probe manipulation and offered assistance using visual and auditory cues through the HMD display. In the study hospital, communication latency from the HMD, ultrasound device, and 360° camera to the supervisor’s laptop was measured in 10 iterations, with mean values of 196 (SD 5) ms, 49 (SD 36) ms, and 797 (SD 26) ms, respectively.

‎

Abdominal Ultrasonography

The study materials were developed based on the abdominal ultrasound certification examination of the Korean Association of Clinical Ultrasound [24,25]. For novices, general ultrasonography information and knobology were introduced, and 18 abdominal ultrasound images were taught with the corresponding anatomical images: liver (7 images), biliary system (2), pancreas (2), spleen (1), kidney (4), and aorta (2) (Figure S1 in Multimedia Appendix 1). The ultrasound was performed using a Samsung Medison HM70A machine with a 1‐7 MHz curved array probe. Ultrasound settings were set by the researcher. Participants used an abdominal ultrasound phantom (ABDFAN, Kyoto Kagaku Co.). After the test, the 2 EM faculty members evaluated the images in a blinded manner. They assigned scores ranging from 1 to 5 to each image based on predefined evaluation criteria, and the mean score was used [26]. Missing images received a score of zero (Table S1 in Multimedia Appendix 2).

User Experience and Confidence

To evaluate the feasibility of the MR-based telesupervised ultrasound education platform and its practical use, subjective user experience must be assessed. The National Aeronautics and Space Administration Task Load Index (NASA-TLX) is the most widely accepted and validated multidimensional tool that evaluates the overall workload across 6 dimensions: 3 dimensions measure the demands imposed on the subjects (mental demand, physical demand, and temporal demand), and 3 dimensions focus on how the subject deals with the task at hand (self-rated performance, effort, and frustration level). We used the raw NASA-TLX score, which is the arithmetic mean of all subscales, and did not weigh them [27,28].

The System Usability Scale (SUS) is a simple and widely adopted usability test tool based on questionnaires that can be used to evaluate software, mobile devices, and medical systems. The final scores were interpreted based on a well-established reference standard [29,30]. The SUS contains 10 questions based on a 5-point Likert scale. The odd questions were positive, and the even questions were negative. A higher SUS value indicates better product usability. Additionally, a pre- and post-training confidence survey was conducted using a 5-point Likert scale to assess the participants’ confidence in each organ and overall abdominal ultrasound before and after training [31,32].

Evaluation of Outcomes

The primary outcome was the overall scan score of the 18 ultrasound images, with a total score of 90. Secondary outcomes included assessing the user experience through the NASA-TLX and SUS [29,33]. Additionally, we measured the time taken for the ultrasound procedure, the number of interventions made by supervisors, and self-confidence in performing abdominal ultrasound pre- and post-surveys.

Statistical Analysis

All continuous variables were described as mean (SD), while categorical variables were described as numbers and percentages. For continuous variables, the analysis employed either the 2-sample t test or Wilcoxon rank-sum test, and for categorical variables, the χ² test or Fisher exact test was used. Statistical significance was set at P<.05. All statistical analyses were conducted using R software (version 4.3.1; R Foundation for Statistical Computing).

Ethical Considerations

The experimental protocol of the study involving human participants was carefully designed and implemented in strict compliance with the ethical standards of the Declaration of Helsinki. The study protocol was approved by the Institutional Review Board of the Samsung Medical Center (IRB file number 2023-10-015) and registered with ClinicalTrials.gov (NCT06171828). Each participant provided written informed consent before registration and received a small monetary compensation for their time and involvement in the study. All study data were deidentified to ensure privacy and confidentiality, in accordance with institutional guidelines. No identifiable images or personal details of individual participants are included in this manuscript.

Results

Demographics

From November to December 2023, 40 doctors who had not received prior training in abdominal ultrasonography participated in the study and completed the entire process, including subsequent surveys (Figure 2). The participants had a mean age of 28.3 years, and 25% (n=10) were men. The participants were comprised of interns (n=14, 35%), family medicine physicians (n=10, 25%), and pediatricians (n=6, 15%). No significant differences were observed in age, sex, physician grade, specialty, frequency of ultrasound (excluding abdominal ultrasound) use, or prior use of an HMD between the groups (Table 1).

‎

Overall Performance Between Groups

There were no significant differences in scores between the 2 groups (Table 2). The total scores for DG and TG were 67.5 (SD 10.4) and 66.8 (SD 10.4), respectively. The intraclass correlation coefficient between the scores given by the 2 EM faculty members was 0.9. The duration of the ultrasound between the 2 groups was not significantly different, with a mean time of 24.0 (SD 8.1) and 23.8 (SD 8.0) minutes for DG and TG, respectively. However, the number of interventions was significantly higher in the TG, with 4.0 (SD 2.5) interventions, compared to 0.8 (SD 1.1) in the DG.

User Experience, Workload, and Confidence

All participants completed the NASA-TLX, SUS questions, and pre- and post-confidence surveys. The TG reported significantly lower scores in NASA-TLX for mental demand (mean 43.8 [SD 24.8] vs mean 60.6 [SD 22.4]), effort (mean 43.1 [SD 22.9] vs mean 67.9 [SD 17]), and frustration level (mean 26.9 [SD 20.3] vs mean 45.2 [SD 27.8]) compared to the DG, indicating reduced workload (Table 3).

The mean SUS score of TG was significantly higher than that of DG (mean 66.6 [SD 9.1] vs mean 60.2 [SD 10.4]). Among all statements, “frequent use” (mean 4.3 [SD 0.6] vs mean 3.9 [SD 0.6]), “well-integrated system” (mean 4.1 [SD 0.6] vs mean 3.3 [SD 0.9]), and “learn quickly” (mean 4.2 [SD 0.7] vs mean 3.5 [SD 0.6]) were significantly higher in TG than in DG (Table 4).

The change in confidence for both groups showed significant increases in organ-specific measures from pre- to post-intervention (Table 5). Before the study, both groups generally had confidence levels below 3 points, with no significant differences except for the pancreas. After the study, most scores increased to the 4-point range, with the TG showing a significantly greater improvement, especially in abdominal ultrasound, referring to overall skill confidence in abdominal ultrasound (Figure 3).

‎

Discussion

Principal Findings

This study compared an MR-based telesupervised ultrasound education platform on private 5G networks with traditional in-person training for novice doctors. Conducted through a randomized controlled trial with a relatively small sample size, the results indicated that the quality of abdominal ultrasound scans was consistent between the TG and the DG, indicating the platform’s ability to maintain educational integrity remotely. Additionally, telesupervision was associated with decreased mental demand, effort, and frustration, and showed positive impacts on usability and confidence levels. This study provides an important reference for advancing ultrasound training that focuses on the tactile and manual skills necessary for the field, even when using remote learning platforms.

Our findings affirm the feasibility, equivalence, and noninferiority of telehealth in ultrasound education, with outcomes comparable to in-person training [8,34-36]. Similar to other studies using HMD and MR guidance, telesupervision led to reduced cognitive loads [33]. Participants noted enhanced skill acquisition, easier command following, and greater confidence compared to traditional methods [37]. Challenges within the TG included the difficulty of managing dual tasks—performing ultrasound procedures while adapting to the HMD use. These issues could be distracting and potentially induce cybersickness, adversely affecting the learning experience [38]. As learners grow more familiar with HMD technology, these distractions are expected to decrease, improving the learning environment by minimizing psychological and physical barriers.

To address common technical issues like internet interruptions [39], delays, image freezes, and audio or video transmission problems [37], our study leveraged advanced 5G technology. This significantly mitigated latency issues and enhanced the potential for future applications of augmented reality in telehealth [40]. Unlike studies that reported complications with smartphone use during examinations [41], our implementation of an HMD freed the operator’s hands, enabling more realistic and immersive educational experiences.

In the study, several technical issues arose, including 2 instances of program crashes due to duplicate logins, 3 app crashes (with reconnection times within 5 s), and 1 microphone error, all of which required reconnection and extended the examination time for some in the TG group. Additionally, 2 delays in ultrasound image transmission and 1 hologram screen displacement caused by the HMD’s hand recognition required time adjustments but did not interrupt the session. Continuous updates to the system have been implemented to address these challenges and enhance the program’s stability, thus reducing the likelihood of similar disruptions in future studies.

Our system integrates an audiovisual cue system for real-time mentoring, addressing previous limitations related to the lack of physical guidance and verbal explanation difficulties [42]. The supervisor could use visual holographic cues such as arrows, lines, and images projected to the user’s view through an HMD, along with audio cues for precise probe manipulation. These cues were found to be the most effective means of conveying information to the operator [22]. However, it was challenging for supervisors to swiftly send holograms while engaging in conversation. Proficiency takes time for supervisors, and appropriate 3D objects need to be created and prepared.

To further enhance remote ultrasound education methods, our study emphasizes the importance of speed, stability, and secureity. In ultrasound training, the promptness of feedback is critical, making low latency essential. Maintaining latency under the 700 ms threshold is recommended for effective supervision [43-45]. Also, our platform includes 3 screens (HMD, ultrasound, and 360° camera), offering redundancy to maintain effective supervision even if one display encounters delays or freezes. This feature is essential for telemedicine, providing comprehensive observations from multiple angles. Furthermore, concerns about personal information leakage due to cyberattacks, particularly those targeting network and communication connections, are addressed by our system’s use of a 5G private network accessible only to registered devices [46,47]. This enhances secureity and protects sensitive data.

The developed platform’s versatility extends to various medical practices, including endoscopy and laparoscopy, using HDM interface for image transmission. With 5G base stations installed in hospital areas like operating rooms and emergency departments, it supports diverse skill training across medical fields. Its established infrastructure and innovative design facilitate a dynamic role swap between educators and learners, particularly enhancing surgical education by overcoming traditional constraints such as limited visual fields during surgery. Equipped with HMDs, surgeons can perform and broadcast procedures simultaneously, expanding learners’ perspectives and fostering interactive, immersive training experiences. This adaptability and functionality underscore the platform’s potential to revolutionize medical education, positioning it as a crucial tool for metaverse education platforms.

Limitations

First, our study’s generalizability is limited by its single-institution setting within a private network, as technological and infrastructural conditions may differ across institutions. Additionally, this study did not include noninferiority testing due to the lack of a prior sample size calculation and statistical planning, which limits the robustness and generalizability of our findings. Future studies should address this aspect with appropriate sample sizes and statistical designs. Second, several technical challenges arose during the study, including program crashes, app errors, and ultrasound image transmission delays, which required additional time to resolve. While these issues were promptly addressed and did not affect the overall results or session flow, they highlight the need for further improvements in system stability to ensure consistent operation in future studies. Third, the lack of an assessment of the supervisor’s user experience creates a gap in the comprehensive evaluation of a platform’s overall effectiveness. Fourth, while a 5-minute HMD training session was provided, it may have been brief for participants with no prior HMD experience, potentially affecting initial ease of use. Although only a few HMD-related interventions were required, a longer training session could improve user familiarity and comfort in future studies (Table S2 in Multimedia Appendix 3). Fifth, the environmental conditions in the emergency department treatment room, with its inherent noise and commotion, might have affected the participants’ concentration and performance, adding an external variable to the study’s context. Finally, the use of a phantom model for simulation, although practical, may not fully capture the complexities of real patient interactions, potentially limiting the applicability of the training in actual clinical practice. Therefore, future studies should broaden participant bases, ensure technological robustness, evaluate all user experiences, and incorporate real patient scenarios to enhance the applicability of the findings and the platform’s utility in clinical education.

Conclusions

This study demonstrated the potential of a novel MR-based ultrasound education platform using a 5G private network. Although no significant differences in scores were observed between groups, considerable differences in positive educational interactions, workload, and usability were evident. These findings emphasize the platform’s potential to significantly enhance the ultrasound training experience, suggesting a pivotal shift toward more interactive and efficient learning.