Improving the quality of breast ultrasound examination performed by inexperienced ultrasound doctors with synchronous tele-ultrasound: a prospective, parallel controlled trial
Article information
Abstract
Purpose
This prospective study explored the value of synchronous tele-ultrasound (US) to aid doctors inexperienced in US with breast US examinations.
Methods
In total, 99 patients were enrolled. Two trainee doctors who were inexperienced in US (trainee A [TA] and trainee B [TB]) and one doctor who was an expert in US completed the US examinations sequentially. TA completed the US examinations independently, while TB was instructed by the expert using synchronous tele-US. Subsequently, the expert performed on-site US examinations in person. Separately, they selected the most clinically significant nodule as the target nodule. Consistency with the expert and image quality were compared between TA and TB to evaluate tele-US. Furthermore, TB and the patients evaluated tele-US through questionnaires.
Results
TB demonstrated higher consistency with the expert in terms of target nodule selection than TA (93.3% vs. 63.3%, P<0.001). TB achieved good inter-observer agreement (ICC, >0.75) with the expert on five US features (5/9, 55.6%), while TA only did so for one (1/9, 11.1%) (P=0.046). TB’s image quality was higher than TA’s in gray value, time gain compensation, depth, color Doppler adjustment, and the visibility of key information (P=0.018, P<0.001, P<0.001, P=0.033, and P=0.006, respectively). The comprehensive assessment score was higher for TB than for TA (3.96±0.82 vs. 3.09±0.87, P<0.001). Tele-US was helpful in 69.7% of US examinations and had a training effect in 68.0%. Furthermore, 63.6% of patients accepted tele-US and 60.6% were willing to pay.
Conclusion
Tele-US can help doctors inexperienced in US to perform breast US examinations.
Introduction
Breast cancer is the most frequently occurring cancer in female patients worldwide [1]. Screening and early detection of breast cancer significantly reduce the risk of death from the disease [2]. There are several modalities of breast cancer screening, including mammography, ultrasonography (US), and magnetic resonance imaging [3]. Due to its advantages including non-invasiveness, radiation-free nature, convenience, and relatively low cost, US has been regarded as an effective complementary imaging modality adjunct to mammography in breast cancer screening [4,5]. However, because the generation and interpretation of US images are highly operator-dependent [6,7], doctors experienced in US are the key factor for obtaining high-quality breast US images. Furthermore, the quality of breast US examinations will determine the effectiveness and performance of breast US screening. Due to the current global shortage of medical resources, the quantity and quality of doctors who perform US are far below the demand, especially in rural or remote areas [8]. A meta-analysis showed that rural populations had significantly higher odds of having late-stage breast cancer at diagnosis than urban populations, leading to a worse prognosis for patients and increasing the burden on the healthcare system [9]. One of the major causes for this phenomenon is the insufficiency of effective breast US screening to detect breast cancer at an early stage in rural areas. Therefore, competent breast screening for people in rural or remote areas is very important. In order to ensure the quality of breast US screening, improving the quality of breast US examinations is one of the most critical steps. However, due to distance and resource constraints, it is not practicable for US experts to perform on-site breast screening for every patient face-to-face.
Telemedicine has been recognized as a tool to address barriers to access to health care and medical resources [10,11]. As a branch of telemedicine, tele-ultrasound (tele-US) might have the potential to solve the problems of insufficient medical resources for high-quality US examinations [12,13], including breast US screening.
Tele-US is generally divided into asynchronous and synchronous modes [14,15]. The synchronous mode enables remote doctors who specialize in US to monitor the entire process of on-site US scanning, while analyzing the US images simultaneously [16]. Because US examination is a real-time and operator-dependent process, the synchronous mode seems to be more suitable for tele-US than the asynchronous mode.
On-site monitoring and consultation by experts would support doctors inexperienced in US in breast US screening. Theoretically, remote synchronous monitoring and consultation may have similar effects as long as there is no time delay for US scanning and adequate image quality is obtained. However, the economic cost of synchronous tele-US is much higher than that of the asynchronous mode [17]. Therefore, before devoting large-scale resources to synchronous tele-US, its clinical feasibility and benefit should be fully evaluated. To the authors’ knowledge, solid evidence for the generalization of tele-US is lacking. Therefore, a prospective and parallel controlled trial was designed to explore the feasibility and value of synchronous tele-US.
Materials and Methods
Study Design and Setting
This prospective, parallel, controlled, inter-reader agreement trial was conducted at a university-affiliated hospital (Shanghai Tenth People’s Hospital). The study was approved by the institutional review board of the hospital (No. SHSY-IEC-4.1/20-65/01). At the beginning of each US examination, patients will sign a written informed consent agreeing to use their data for study purposes.
Synchronous Tele-US System
The tele-US system consisted of one data collector device (Shanghai Keyi Intelligent Technologies Co. Ltd., Shanghai, China), one US system (Toshiba Aplio 500, Toshiba Medical Systems Corp., Otawara, Tochigi, Japan), one web camera, two headsets, and one laptop. The data collector device is the key part of the system and has two main functions. First, the collector can capture lossless US videos in Digital Imaging and Communications in Medicine format from the US equipment and transfer them to the screen of the remote US expert’s laptop over the internet. Second, the collector can transmit real-time videos and voices from the web camera and headset to the remote site. With the help of the high-speed network (300 Mbps downstream and 50 Mbps upstream) of the specialized line, this system allows real-time monitoring of the entire US scanning procedure and high-quality communication between the on-site and remote doctors, without affecting the resolution of US images (Fig. 1).
Synchronous tele-ultrasonography system setup.
A web camera (A), an ultrasound equipment (Toshiba Aplio 500) (B), and a headset (C) at the patient’s side were connected to the data collector device (D). By the use of the data collector device, relative data collected by above mentioned devices were transmitted to the remote expert’s laptop (F) with the help of the high-speed private network (E). G. At the patient’s side, the doctor inexperienced in ultrasonography finishes the ultrasound examination under the synchronous guidance of the expert. H. The remote expert monitors the entire process of the ultrasound examination and provides synchronous guidance to the doctor inexperienced in ultrasonography.
Participant Recruitment
Five doctors who performed US were included. The US scanning procedure involved three doctors, including two trainee doctors and one expert (trainee A [TA], trainee B [TB], and the expert). The review procedure involved two other doctors experienced in US. Moreover, all data were collected and organized by one independent coordinator.
TA and TB were selected from residents in training. They both had limited (less than 30 exams) experience with US examinations and both had received certification from the courses of the authors’ department. The training course was specifically designed to select two US operators from six trainees for this trial. The training course took about 5 hours in total. In the first hour, one US expert instructor introduced the anatomy of the breast and knowledge of some common breast diseases. During the second hour, the instructor explained the principles of US and the method of US parameter adjustment. During the third to fourth hours, the instructor explained the US manifestations of some common breast diseases and the grading rules of American College of Radiology Breast Imaging Reporting and Data System (BI-RADS). Subsequently, the trainees were given a quiz (20 questions, 5 points each). The last hour was hands-on practice for these trainees. Only those who passed the quiz (≥80 points) and were recognized as sufficiently proficient by the instructor in the hands-on practice were certified. Finally, two trainees who finished the course and had similar US practice abilities were selected as the operators in this trial. The other three doctors who were experts in US (one expert and two reviewers) all had more than 10 years of experience in breast US examinations.
Consecutive patients who would receive breast US examinations were asked whether they were willing to participate in this trial. These patients were included in the trial after signing informed consent forms.
The inclusion criteria for patients were as follows: (1) age ≥18 years and ≤80 years, and (2) agreeing to participate in the trial and signing the consent form. The exclusion criterion was incomplete data.
Experimental Procedure
In each case, the patient was scanned by the three doctors in turn (TA, TB, and the expert) (Fig. 2). TA completed the US examinations independently, while TB completed the US examination under the monitoring and guidance of the remote expert using the synchronous tele-US system. The expert was located in another room when TB began the US examination. The expert was able to observe the synchronous retransmission of the US equipment screen and the examination by TB. Moreover, the expert could communicate with TB in real time and provide guidance to TB on image quality and diagnosis. TB's results were obtained through discussion with the expert. Finally, the expert entered the room and verified the diagnosis by performing an on-site US examination in person on the patient.
Process of the ultrasound examinations in this trial.
TA, trainee A; TB, trainee B; US, ultrasonography; BI-RADS, Breast Imaging Reporting and Data System.
These three doctors were asked to scan all the breasts through the conventional process using the same protocol. If no nodule was detected, images of the thickest part of the breast were recorded. If one or more nodules were found, then the only nodule or the nodule with the most clinical significance was chosen as the target nodule to be recorded. The clinical significance of the nodules was measured according to the possibility of malignancy. The findings of the doctors, including the size, location, US features and BI-RADS categories of the target nodule were recorded on a specific sheet (Supplementary Data 1). Each US feature of the target nodule was stored as one image separately for further analysis.
TB and each patient were asked to complete a questionnaire after each US examination. The questions for TB were as follows: (1) Was the guidance using the tele-US helpful for the diagnosis?, (2) Did the guidance using the tele-US have a training effect on me? The questions for each patient were as follows: (1) Would you accept the use of tele-US?, (2) Are you willing to pay for this new method of US examination? The possible responses to these questions were "yes," "no," and "uncertain." The total time for each US examination was recorded.
Image Quality Evaluation
The images obtained by TA and TB were reviewed by the other two doctors who were experienced in US. The reviewers evaluated the quality of images obtained by TA and TB based on the reference images obtained by the US expert. The final result was the consensus reached by the reviewers’ discussion.
The evaluation of image quality was divided into three steps (Fig. 3). The first step was the background image quality evaluation, including the suitability of gray values, depth, focus position and time gain compensation (TGC) adjustment. The second step was the target nodule image quality evaluation, including the suitability of grayscale US and color Doppler adjustment and the visibility of all key information (the US features that facilitate identification of a nodule as benign or malignant). The third step was to perform a comprehensive assessment using a 5-point Likert scale (1, meaningless, images were undiagnosable or not meaningful; 2, poor, poor image quality may affect the diagnosis; 3, adequate, acceptable for interpretation; 4, excellent, minor suggestions for improvement of image quality; 5, perfect, no suggestions for improvement of image quality). The reviewers evaluated all aspects of the first two steps regarding whether the images were suitable (the parameters were properly adjusted or all the key information was fully displayed); then, they provided a score to represent the overall image quality. For cases without target nodules or inconsistent target nodules, the second step was not completed.
Process of image analysis in this trial.
TA, trainee A; TB, trainee B.
The quality of the images obtained by TA and TB was compared according to the suitability coefficient and the score of the overall image quality. The suitability coefficient denotes the percentage of suitable adjustments within a given parameter.
Blinding and Parallel Control
Blinding took place as follows: (1) TA and TB were blind to each other’s scanning procedure, (2) the expert was blind to TA’s scanning procedure, and (3) the reviewers were blind to the source and date of the images being analyzed. However, the images from the expert could be referred to at any time during the process of analyzing image quality.
The parallel control was reflected in the following aspects: (1) TA and TB had a similar level of knowledge of US theory and limited experience with US examinations; (2) the US courses they completed were the same; and (3) the coordinator monitored the entire process of the trial, ensuring that TA, TB, and the expert did not interfere with each other.
Statistical Analysis
Quantitative data were first tested for normality, then the mean and standard deviation were used to describe data with a normal distribution, and the median and interquartile ranges were used to describe data with a non-normal distribution. They were compared using the Student t-test or the Mann-Whitney U test as appropriate. Categorical variables were expressed as numbers and percentages and were compared using the chi-square test or Fisher exact test as appropriate. Interobserver agreement was compared through the intra-group correlation coefficient (ICC) with a two-way random effect model. A value higher than 0.75 was considered to reflect good inter-observer agreement. The ICC was only calculated in cases in which the doctors inexperienced in US selected the same target nodules as the expert. The mean values of the rolling cohorts of 30 consecutive patients (i.e., cohort 1, patients 1-30 inclusive; cohort 2, patients 2-31 inclusive; cohort 3, patients 3-32, etc.) were used to analyze the results of the 5-point Likert scale in this trial, and a simple linear regression model was used to model these data. Trends in the change of image quality and TB’s experience with tele-US were reflected in the slope of the equation.
Statistical analysis was carried out using SPSS version 20.0 (IBM Corp., Armonk, NY, USA) and GraphPad Prism version 8.0.2 (GraphPad Software, La Jolla, CA, USA).
Results
Participants
During the study period (2020 April-2020 June), a total of 100 patients were recruited consecutively as volunteers for this trial. Among them, 32 (32%) patients were referred for a US examination due to breast pain, 43 (43%) for palpable masses, 11 (11%) for postoperative follow-up, and 14 (14%) for health checks. One patient was excluded because of incomplete images and data. Finally, 99 patients (average age, 53.4±13.4 years) were enrolled.
Consistency
Nodules were detected in 60, 60, and 62 patients by TA, TB and the expert, respectively (Table 1). However, the probability of choosing the same target nodule as the expert was higher for TB (56/60, 93.3%) than for TA (38/60, 63.3%) (P<0.001) (Table 2). Regarding the US features and BI-RADS categories, only the feature of calcification (ICC, 0.81) for TA had good inter-observer agreement with the expert. However, for TB, the BI-RADS categories, orientation, echo pattern, calcifications, and internal characteristics all showed good inter-observer agreement with the expert (ICC, 0.89, 1, 0.85, 0.84, and 0.85, respectively) (Table 3). Out of the eight US features and BI-RADS catergories US features, TB achieved good inter-observer agreement with the US expert on five (5/9, 55.6%), while TA only did so for one (1/9, 11.1%) (P=0.046) (Table 3).
Baseline characteristics of target nodules found by TA, TB, and the expert
Target nodules found by TA and TB
Inter-observer agreement for the target nodule features and measurements obtained by TA, TB, and the expert
Total Time of the US Examinations
TB spent more time in the US examinations than TA (TA, 355.63±116.65 seconds; TB, 397.07±192.34 seconds) (P=0.010).
Image Quality
TB’s image quality was better than TA’s in terms of gray value, TGC, depth, color Doppler adjustments, and the visibility of all key information (P=0.018, P<0.001, P<0.001, P=0.033, and P=0.006, respectively). The comprehensive assessment score was also higher for TB (3.96±0.82) than for TA (3.09±0.87) (P<0.001) (Table 4, Fig. 4). The mean values of the comprehensive assessment score in each rolling cohort and the corresponding linear regression equations are shown in Fig. 5. Linear functions with an increasing trend were found for both TA and TB (slope of TA, 0.01099, P<0.001; slope of TB, 0.01223, P<0.001).
Results of image quality analysis
Ultrasonography of a target nodule in a 43-year-old woman obtained by trainee A (TA), trainee B (TB), and the expert.
A. In the image obtained by TA, the adjustment of depth is not appropriate and the image cannot show the blood signals in the nodule correctly. B. In the image obtained by TB, the adjustment of depth is appropriate and the blood signals in the nodule are shown correctly. C. Image was obtained by the ultrasound expert.
Linear regression analyses of the comprehensive assessment score.
A. Circles represent the mean values of the comprehensive assessment score in each rolling cohort for trainee A. B. Triangles represent the mean values of the comprehensive assessment score in each rolling cohort for trainee B. The solid lines represent the curve fit to these results.
Questionnaire
Regarding questions focused on the value of tele-US in diagnosis, TB believed that it was helpful, not helpful, and of uncertain helpfulness in 69.7% (69/99), 29.3% (29/99), and 1.0% (1/99) of cases, respectively. For the questions focused on the training effect, TB believed that tele-US was helpful, not helpful, and of uncertain helpfulness in 68.0% (68/99), 29.3% (29/99), and 2.0% (2/99) of cases, respectively.
A total of 63.6% (63/99), 34.3% (34/99), and 2.0% (2/99) of the patients accepted, rejected, and were uncertain about whether they could accept tele-US, respectively. Furthermore, 60.6% (60/99), 28.3% (28/99), and 11.0% (11/99) of the patients accepted, refused and were uncertain about whether they were willing to pay for this new method of screening, respectively.
Discussion
This prospective, parallel controlled trial evaluated the value of synchronous tele-US in breast US examinations. Based on the results of this trial, expert guidance using the synchronous tele-US system enables doctors inexperienced in US to perform satisfactory breast US examinations of patients.
In the past few decades, an increasing number of studies have focused on synchronous tele-US [8,17-23]. However, due to limitations in image processing technology and communication technology, most of synchronous transmissions in these studies came at the expense of lower resolution. Therefore, most of them were only feasibility studies [8,17-21]. Few studies have investigated the diagnostic effect of synchronous tele-US systems. Kim et al. [22] demonstrated that synchronous tele-US can improve the sensitivity and specificity of residents’ diagnosis of appendicitis from 0.917 to 1.000 and from 0.899 to 0.975, respectively. However, the appendix is an organ in a fixed position and the symptoms of appendicitis are pronounced. Therefore, their results may not apply to breast US examinations, which involve a larger scanning area, random nodule location, inapparent symptoms, and more complex diagnoses. Our trial is a good complement to enlarge the range of applications of synchronous tele-US.
Jensen et al. [23] proved that the guidance of experts using synchronous tele-US could improve the image quality of focused cardiac US. Their results were consistent with the results of this trial. However, good image quality is only one of the necessities for a correct diagnosis. Our trial evaluated the value of synchronous tele-US not only in terms of image quality, but also from the standpoint of image information interpretation. The results of this trial provide solid evidence of the benefit of synchronous tele-US in a more comprehensive way.
The findings of the present study also imply that expert guidance using the synchronous tele-US system may be an effective modality to train doctors inexperienced in US. Other methods to train doctors inexperienced in US include simulator training and point-of-care US training programs [24,25]. Further studies should conduct a comparative evaluation of the training effect and cost-effectiveness of these various modes. The authors conclude that synchronous tele-US is a feasible and acceptable way to train doctors inexperienced in US in rural and remote areas.
Indeed, synchronous tele-US will increase the total time of breast US examinations and may also increase the financial burdens of patients. Based on these results, the increase in time and economic burden would be acceptable for most patients in China. Moreover, theoretically, through constant mentoring, doctors inexperienced in US will become increasingly skilled, and the total time of the US examination will decrease. Furthermore, synchronous tele-US can improve the quality of screening examinations performed by doctors inexperienced in US and may have the ability to improve the early detection rate of cancer and reduce referral rates, which will help to save medical resources from the standpoint of society as a whole. In addition, tele-US could reduce the transportation and accommodation costs when patients from remote areas travel to large cities to see doctors. However, it should be noted that this was only a survey of Chinese patients, and in other countries, acceptance may decline as the burden of health care increases.
The cost of synchronous tele-US guidance might be an important impediment to real-world clinical applications. Our study was a clinical trial, so patients did not pay for the tele-US guidance. However, considering the investment of infrastructure and participation of US experts, it will be necessary for patients to pay for tele-US. Currently, breast US in China costs about $20 and is covered by health insurance. According to the results of this trial, most patients in distant areas would be willing to pay the cost for tele-US. Therefore, an additional price of about $20 might be acceptable to patients. However, the price of tele-US might be different in different countries. Further studies are needed to evaluate the cost-effectiveness of synchronous tele-US systems.
For using tele-US, the protection of personal information is an important issue. In this trial, the data were transmitted through a special medical line that was isolated from the open public network. This is one of the methods to protect personal information. Other methods include the encryption of personal information (such as name, age, sex, telephone number, home address, etc.), blockchain, private network, data anonymization, and enforcement of related laws and regulations.
There were also some limitations in this trial. First, the single-center design and relatively small sample size limited the generalizability of these results. Further studies with a larger sample size and multicenter verification will be needed in the future. Second, in order to eliminate the effect of inconsistent clinical experience between on-site and remote experts, the expert who provided remote guidance was the same person who performed the on-site US examinations in the present trial. This may have caused some degree of overestimation in the comparison of consistency. Third, there was no pathological verification of the breast lesions, which would ultimately prove whether the diagnosis was correct. In a future study, with a longer follow-up time and a higher number of biopsies, more solid evidence will be obtained to confirm these results. Fourth, the questions designed for TB were aimed to evaluate the real practical value of tele-US guidance; however, a questionnaire-based evaluation after each US examination is prone to subjectivity. Additional studies yielding objective or quantitative results could mitigate this issue.
This prospective, parallel, controlled trial proved that synchronous tele-US can help remote doctors inexperienced in US improve the image quality and diagnostic accordance with a US expert. This method was also well-accepted by both patients and doctors. This is the first trial to explore the value of synchronous tele-US in breast US and lays a foundation for subsequent studies.
Notes
Author Contributions
Conceptualization: Sun YK, Li XL, Wang Q, Zhou BY, Zhu AQ, Qin C, Xu HX, Guo LH. Data acquisition: Sun YK, Li XL, Wang Q, Zhou BY, Zhu AQ, Xu HX, Guo LH. Data analysis or interpretation: Sun YK, Li XL, Wang Q, Guo LH. Drafting of the manuscript: Sun YK, Xu HX, Guo LH. Critical revision of the manuscript: Sun YK, Li XL, Wang Q, Zhou BY, Zhu AQ, Qin C, Xu HX, Guo LH. Approval of the final version of the manuscript: all authors.
No potential conflict of interest relevant to this article was reported.
Acknowledgements
This study received funding from the Shanghai Municipal Health Commission (Grants 2019LJ21 and SHSLCZDZK03502), the Science and Technology Commission of Shanghai Municipality (Grants 19441903200, 18441905500 and 19DZ2251100), and the Science and Technology Assistance Subject for Xinjiang (Grants 2020E0282).
Supplementary Material
Supplementary Data 1.
Record table of the US examination results (https://doi.org/10.14366/usg.21081).
References
Article information Continued
Notes
Key point
A prospective, parallel controlled trial showed that synchronous tele-ultrasound can help doctors inexperienced in ultrasonography to improve image quality and diagnostic accordance with an expert.