Effect of an annotation-free artificial intelligence system for simultaneous detection and diagnosis on breast ultrasonography: a multireader, multicase study across diverse healthcare professionals
Article information
Abstract
Purpose
This study aimed to evaluate how a deep learning–based artificial intelligence (AI) system that provides automatic lesion detection and diagnosis affects the diagnostic performance of readers from diverse professional backgrounds.
Methods
A total of 1,000 breast ultrasound (US) images (500 malignant, 250 benign, and 250 negative) were independently reviewed by 15 readers (six radiologists, three surgeons, three physicians, and three radiographers) in two sessions: session 1 without AI assistance and session 2 with AI assistance following a 2-week washout period. Reader performance across sessions was compared using the area under the localization receiver operating characteristic curve (AULROC), sensitivity, specificity, and accuracy.
Results
With AI assistance, the reader-averaged AULROC increased significantly from 0.864 (95% confidence interval [CI], 0.831 to 0.898) to 0.910 (95% CI, 0.888 to 0.931) (P=0.002). The standalone AI achieved an AULROC of 0.909 (95% CI, 0.889 to 0.930). Radiographers demonstrated a significant increase in AULROC, from 0.868 (95% CI, 0.840 to 0.897) to 0.922 (95% CI, 0.908 to 0.936) (P=0.032). Across all readers, sensitivity increased from 85.3% to 95.0% and accuracy from 82.3% to 85.7% (both P<0.001). Specificity did not change significantly among radiologists or surgeons (P=0.089 and P=0.955, respectively) but decreased among physicians (82.3% vs. 75.7%, P<0.001) and radiographers (81.7% vs. 76.7%, P<0.001). Radiologists without fellowship training or with <8 years of experience showed significantly improved sensitivity and accuracy, accompanied by decreased specificity after AI assistance (all P<0.05).
Conclusion
AI assistance significantly increased the average diagnostic performance of readers in breast US interpretation. Among less experienced professionals and radiologists, improvements in sensitivity and accuracy were accompanied by decreased specificity.
Introduction
Breast cancer is the most commonly diagnosed malignancy and remains the leading cause of cancer-related death among women worldwide [1]. Although mammography is the only screening modality proven to reduce mortality [2,3], it has limitations, particularly an increased risk of false-negative results in women with dense breasts [2,4–6]. To address this limitation, supplemental screening modalities—including ultrasonography (US), digital breast tomosynthesis, and breast magnetic resonance imaging—are increasingly used [5].
Among these modalities, breast US is a cost-effective, safe, and widely accessible imaging technique that avoids radiation exposure and enables the detection of asymptomatic cancers not visible on mammography. Despite its high sensitivity, the diagnostic performance of US is limited by substantial operator dependence in both lesion detection and image interpretation [7]. Furthermore, examinations may be performed by a wide range of healthcare professionals—including technologists, physicians, and general radiologists—many of whom may lack dedicated training in breast imaging. Recently, artificial intelligence (AI)–based computer-assisted diagnosis (CADx) systems for breast US have been developed to support differential diagnosis and clinical decision-making [8–11], with promising results in reducing inter-reader variability. However, in prior studies, AI outputs were heavily influenced by the user, as these systems required manual delineation or adjustment of regions of interest (ROIs) [8,9,12].
Advances in computer-assisted detection (CADe) have enabled the development of AI systems that automatically detect lesions and provide differential diagnoses simultaneously, thereby more closely reflecting real-world clinical workflows [10,13]. The contemporary AI system evaluated in this study [13] further extends these capabilities by providing annotation-free, real-time lesion detection with concurrent malignancy risk stratification, eliminating the need for manual ROI selection. To date, no pivotal study has evaluated the clinical efficacy of such annotation-free AI systems across readers with diverse professional backgrounds and varying levels of experience.
In this context, the authors conducted a multireader, multicase study to evaluate the effect of a deep learning–based AI system that autonomously performs lesion detection and differential diagnosis on the diagnostic performance of readers from diverse professional backgrounds interpreting breast US images.
Materials and Methods
Compliance with Ethical Standards
This study was approved by the institutional review boards (IRBs) of three institutions (Severance Hospital, IRB No. 1-2023-0073, Wonkwang University Hospital IRB No. 2023-07-017-002, Kyungpook National University Chilgok Hospital IRB No. KNUCH 2023-11-029) and the requirement for informed consent was waived.
Study Population and Image Collection
Between December 2011 and December 2023, 801 women who underwent breast US examinations for various indications at three institutions were included. The women’s mean age was 53.4±13.6 years (range, 19 to 89 years). Of the 801 women, 221 (27.6%) reported symptoms (e.g., a palpable mass, nipple discharge, or breast pain) as the reason for undergoing US evaluation.
The US examinations and clinical reports of all 801 women were retrospectively reviewed by three dedicated breast radiologists (H.W.K., W.H.K., and J.H.Y), each with more than 10 years of experience in breast imaging, to identify candidate lesions for the reader study. Representative grayscale US images of adequate diagnostic quality were selected for review according to the ground-truth categories: (1) negative US images depicting normal breast parenchyma without definable abnormalities; (2) benign US images showing breast lesions that were either pathologically confirmed (by core biopsy, vacuum-assisted biopsy, or surgical excision) or remained stable over more than 2 years of imaging follow-up; and (3) malignant US images with pathologically confirmed breast cancer. Multiple lesions were selected from a single patient when more than one lesion satisfied the ground-truth definitions described above. One representative image was selected for each lesion. A total of 1,000 breast US images were collected for the reader study. These images were further reviewed by an expert panel of three breast radiologists—each a staff radiologist (W.H.K., J.H.Y., and H.W.K.)—with 10, 16, and 17 years of experience in breast imaging. The reference-standard ROI for each lesion was determined by consensus among the three radiologists (Supplementary Fig. 1). The US images were obtained using equipment from various vendors, as summarized in Supplementary Table 1.
Reader Characteristics
Fifteen readers from different professions and clinical backgrounds were recruited for this study (Table 1). Of the 15 readers, 12 were physicians and three were radiographers. Among the 12 physicians, six (50.0%) were radiologists, three (25.0%) were surgeons, two (16.7%) were obstetrician-gynecologists, and one (8.3%) was a family medicine practitioner.
Demographics of the 15 readers recruited for this study
Two of the three surgeons were breast surgeons (one fellow and one surgeon with 3 years of experience). The family medicine practitioner routinely ordered and discussed the results of breast imaging examinations, including breast US, but did not perform or interpret US examinations. The physicians represented nine different centers; three (25.0%) were in private practice, whereas nine (75.0%) were affiliated with academic institutions. Among the six radiologists, three (50.0%) had completed fellowship training in breast imaging. The median duration of experience among the radiologists was 8 years. Specifically, two (33.3%) radiologists had ≥8 years of experience (15 and 22 years), whereas the remaining four (66.7%) had <8 years of experience (<1 year for two readers and 4 and 6 years for the other two). The three radiographers were all affiliated with academic centers and routinely performed mammography or automated breast US examinations. None of the radiographers performed hand-held US or interpreted breast US examinations.
AI System for Breast US
A deep learning–based, annotation-free dual detection-diagnosis AI system (CadAI-B for Breast, BeamWorks Inc., Daegu, Korea) designed to assist healthcare professionals in acquiring and interpreting breast US images was used in this study. The system employs deep convolutional neural networks trained in a weakly supervised manner [13,14]. Through this training approach, the model learns to distinguish malignant, benign, and normal US images by intrinsically focusing on regions characteristic of malignant or benign pathology. During scanning, the AI system processes each frame to highlight regions suspected of malignancy (CADe) and assigns a malignancy risk category on the basis of the corresponding classification output (CADx).
The AI system uses a jointly learned architecture that integrates CADe and CADx functions through weakly supervised learning. In the detection phase (CADe), the system assists in identifying suspicious regions within breast US images by automatically highlighting areas of interest using an abnormality map generated by a spatial attention mechanism. This abnormality map is normalized to produce a pixel-level abnormality score ranging from 0 to 1, with values closer to 1 indicating a greater likelihood of abnormality. In the diagnosis phase (CADx), the model uses a malignancy-classification head to provide a level of malignancy suspicion for a given B-mode image. This malignancy suspicion level is expressed as a probability score (CadAI-Score) ranging from 0 to 100 and undergoes a calibration process to align with the American College of Radiology (ACR) Breast Imaging Reporting and Data System (BI-RADS) categorization system, referred to as AI-BI-RADS (CadAI BI-RADS) (Fig. 1).
Representative image of the web-based system used for reader evaluation in the two sessions.
Grayscale ultrasound images were presented to the reader (left panel), followed by the artificial intelligence (AI)–assisted results (right panel). The AI output included a semitransparent colored heatmap delineating the breast lesion, a probability score (CadAI Score, 28% for this case), and a calibrated Breast Imaging Reporting and Data System (BI-RADS) category (CadAI BI-RADS, 4B for this case).
The AI model was trained on more than 670,000 breast US images, comprising both static and cine images. The trained model was calibrated using a separate tuning set with BI-RADS assessments assigned by radiologists as reference labels to align probability predictions with the ACR BI-RADS framework. The calibrated model classified findings into six BI-RADS assessment categories: BI-RADS 1/2, 3, 4A, 4B, 4C, and 5. Images used for system training or tuning were not included in this study.
Image Interpretation Sessions
All US images were analyzed by the AI system. The average detection time per US image was 0.02 seconds. The grayscale US images and their corresponding AI results, captured as single image files, were uploaded to a dedicated web-based platform developed for the reader study. Readers were provided with login credentials to access the platform and record their interpretation data. Before the reading sessions, each reader underwent a one-on-one online training session on how the grayscale US images and AI results would be displayed in the system. Pretest sessions were conducted using 10 cases that were not included in the reader study.
Image review consisted of two consecutive reading sessions. In session 1, readers interpreted only the 1,000 grayscale US images selected for the study. After a 2-week washout period, session 2 began, in which the same 1,000 images were presented together with their corresponding AI results (Fig. 1). In both sessions, readers were instructed to assess each case using (1) BI-RADS categories (1, 2, 3, 4A, 4B, 4C, and 5) and (2) the probability of malignancy (POM, 0–100%). For lesion localization, readers manually marked the center of the lesion with a single dot for suspected benign or malignant lesions (Supplementary Fig. 1).
All 15 readers interpreted the 1,000 US images in reader-specific random orders, and the image order was reshuffled for each reader in session 2. To minimize potential bias, readers were blinded to patient demographics and information on the US equipment vendors.
Data and Statistical Analysis
For the calculation of diagnostic performance, BI-RADS assessments of 1–3 were considered negative and those of 4A–5 were considered positive, and a 50% threshold was applied for the POM. Reader and standalone AI performance were calculated using the area under the localization receiver operating characteristic curve (AULROC), with mislocalized benign or malignant lesions penalized as false negatives. The AULROC for BI-RADS assessment and POM was compared between sessions 1 and 2 using the Obuchowski-Rockette multireader multicase model, with the reader specified as a random effect. The 95% confidence interval (CI) for the AULROC was estimated using the jackknife method. Sensitivity, specificity, and accuracy were calculated for both readers and AI and were compared using logistic regression with generalized estimating equations. To evaluate performance differences according to experience among the six radiologists, the median value of 8 years was used as the cutoff. Subgroup analysis according to radiologist experience was therefore based on a threshold of 8 years.
Statistical analyses were performed using R software version 4.3.3 (R Foundation for Statistical Computing, Vienna, Austria). Statistical significance was defined as P<0.05.
Results
Image Dataset Demographics
Table 2 summarizes the demographic characteristics of the 1,000 US images (370 from institution A, 250 from institution B, and 380 from institution C) obtained from the 801 women included in this study. Of these 1,000 US cases, 250 (25.0%) were negative, 250 (25.0%) were benign, and 500 (50.0%) were malignant. The mean diameter of the 500 malignant masses was 17.4 mm (standard deviation [SD], 8.3 mm; range, 4 to 41 mm), whereas that of the 250 benign masses was 11.4 mm (SD, 7.4 mm; range, 3 to 53 mm). Among the 500 malignant lesions, 199 (39.8%) were symptomatic. Of the 250 benign lesions, 163 (65.2%) were confirmed by biopsy or excision. High-risk lesions, such as atypical ductal hyperplasia, intraductal papilloma, and atypical lobular hyperplasia, were diagnosed by excisional biopsy. Of the 500 malignant lesions, 434 (86.8%) were invasive cancers and 44 (8.8%) were ductal carcinomas in situ.
Demographic features of the 1,000 US images used for the reader study
Comparison of AULROC Using BI-RADS Assessment in Readers between Sessions
The results of the AULROC analysis based on BI-RADS assessments by readers and AI are presented in Table 3. The mean AULROC for the 15 readers increased significantly from session 1 to session 2, from 0.864 (95% CI, 0.831 to 0.898) to 0.910 (95% CI, 0.888 to 0.931) (P=0.002) (Fig. 2). The standalone AI achieved an AULROC of 0.909 (95% CI, 0.889 to 0.930). Similarly, the mean AULROC based on POM increased from 0.867 (95% CI, 0.828 to 0.905) in session 1 to 0.923 (95% CI, 0.903 to 0.942) in session 2 (P=0.001) (Supplementary Table 2, Supplementary Fig. 2).
AULROC for readers and AI using BI-RADS assessment
Localization receiver operating characteristic (LROC) curves for readers and artificial intelligence (AI) using Breast Imaging Reporting and Data System (BI-RADS) assessment.
Thin curves represent LROC for each of the 15 readers. The bold red curve shows the average reader LROC for session 1, and the bold green curve shows the average reader LROC for session 2. In session 2, the average area under the localization receiver operating characteristic curve (AULROC) for BI-RADS was significantly higher than in session 1 (0.910 vs. 0.864, P=0.002). The bold blue curve represents the standalone AI AULROC (0.909).
When the readers were grouped by profession, radiographers showed a significant increase in mean AULROC based on BI-RADS assessment from session 1 to session 2, from 0.868 (95% CI, 0.840 to 0.897) to 0.922 (95% CI, 0.908 to 0.936) (P=0.032). Radiologists, surgeons, and physicians did not show significant differences in AULROC between sessions (all P>0.05).
Among the six radiologists, the mean AULROC did not differ significantly between sessions 1 and 2 when stratified by fellowship training (P=0.179 and P=0.550, respectively). Radiologists with ≥8 years of experience (n=2) demonstrated an increase in AULROC in session 2, to 0.940 (95% CI, 0.838 to 1.000), compared with 0.927 (95% CI, 0.819 to 1.000) in session 1 (P<0.001). A similar trend was observed for the mean AULROC based on POM in the subgroup analysis, with a significant increase in session 2 among surgeons (Supplementary Table 2).
Comparison of Sensitivity, Specificity, and Accuracy by Subgroups between Sessions
The diagnostic performance of the readers and AI in terms of sensitivity, specificity, and accuracy is summarized in Table 4. For all 15 readers, average sensitivity and accuracy were significantly higher in session 2 than in session 1: 95.0% (95% CI, 93.6 to 96.4) versus 85.3% (95% CI, 83.6 to 87.0), and 85.7% (95% CI, 83.9 to 87.5) versus 82.3% (95% CI, 80.8 to 83.8), respectively (all P<0.001). In contrast, average specificity was significantly lower in session 2 than in session 1: 76.5% (95% CI, 73.4 to 79.6) versus 79.3% (95% CI, 76.9 to 81.7) (P<0.001). The standalone AI achieved a sensitivity of 94.2% (95% CI, 92.2 to 96.3), a specificity of 76.4% (95% CI, 72.7 to 80.1), and an accuracy of 85.3% (95% CI, 83.1 to 87.5). Standalone AI detection sensitivity, specificity, and detection time are summarized in Supplemental Material 1.
Sensitivity, specificity, and accuracy of readers in the two sessions according to subgroups
Regardless of professional background, sensitivity and accuracy improved significantly from session 1 to session 2 (all P<0.05). Radiologists and surgeons showed no significant differences in average specificity between sessions (P=0.089 and P=0.955, respectively), whereas physicians (75.7% [95% CI, 72.2 to 79.3] vs. 82.3% [95% CI, 79.8 to 84.8], P<0.001) and radiographers (76.7% [95% CI, 73.4 to 80.0] vs. 81.7% [95% CI, 79.1 to 84.4], P<0.001) showed a significant decrease in specificity in session 2.
Among the six radiologists, those without fellowship training (n=3) showed significant improvements in sensitivity and accuracy in session 2 (all P<0.05). In contrast, they showed a significant decrease in specificity in session 2 (P<0.001). Radiologists with fellowship training did not show significant differences between sessions (all P>0.05). When stratified by experience using a median of 8 years, radiologists with <8 years of experience showed significant improvements in sensitivity and accuracy in session 2 (all P<0.05), along with a significant decrease in specificity (P=0.034). Radiologists with ≥8 years of experience showed higher average sensitivity in session 2 than in session 1, 95.6% (95% CI, 94.1 to 97.1) versus 93.4% (95% CI, 91.7 to 95.1) (P=0.006), without significant changes in specificity or accuracy.
Discussion
This study demonstrated that a deep learning–based AI system for breast US (CadAI-B for Breast), which provides autonomous lesion detection and simultaneous diagnosis, significantly improved the overall diagnostic performance of readers. Among professional groups, radiographers showed a significant improvement in AULROC with AI assistance, whereas other groups did not demonstrate statistically significant changes. Sensitivity and accuracy improved significantly across all professional groups, although these gains were accompanied by decreased specificity among radiographers and physicians. Among radiologists, those without breast imaging fellowship training or with less experience showed improved sensitivity and accuracy, with a corresponding reduction in specificity following AI assistance.
The reader-averaged AULROC based on BI-RADS assessment increased significantly after AI assistance, from 0.864 (95% CI, 0.831 to 0.898) to 0.910 (95% CI, 0.888 to 0.931) (P=0.002). These findings are consistent with prior studies reporting improved diagnostic performance with AI-assisted breast US interpretation [8–10,15], supporting the role of AI in enhancing reader performance. Existing AI systems typically require users to define or adjust ROIs, which can substantially influence both AI output and subsequent image interpretation. In contrast, the AI system used in this study autonomously determines ROIs without human input, thereby enabling a more objective assessment of AI-assisted interpretation. Notably, the standalone AI achieved an AULROC of 0.909 (95% CI, 0.889 to 0.930), which exceeds the previously reported range of 0.84–0.89 [9,10,16] and is higher than the performance of general radiologists without breast fellowship training in this study (0.889 [95% CI, 0.846 to 0.933]). These findings further highlight the potential of AI to support breast US interpretation at a level comparable to that of general radiologists.
AI for breast US has been shown to support inexperienced readers who lack dedicated training [9,15,17–19]. When stratified by professional background, AULROC based on BI-RADS assessment did not change significantly among radiologists, surgeons, or physicians after AI-assisted interpretation (Table 3). In contrast, radiographers demonstrated a significant increase in AULROC, from 0.868 (95% CI, 0.840 to 0.897) to 0.922 (95% CI, 0.908 to 0.936) (P=0.03). In clinical workflows involving technologist-performed breast US—particularly in supplemental screening for women with dense breasts—technologists typically perform image acquisition, document findings, and present images to an interpreting radiologist [20]. In this context, AI systems that assist with lesion detection and diagnosis may enhance detection and help prioritize suspicious cases for radiologist review. The system’s ability to generate intuitive heatmaps without requiring manual annotation may further improve detection consistency and facilitate efficient triage. Additionally, experienced radiologists showed improvement in AULROC with AI assistance, consistent with prior reports [17], suggesting that autonomous AI systems may also reinforce lesion detection in experienced readers.
In this study, sensitivity and accuracy increased significantly in session 2 across all reader groups (Table 4). A key finding was that specificity decreased among physicians, radiographers, and less experienced radiologists or those without breast imaging fellowship training. This pattern suggests that less experienced readers may rely more heavily on AI outputs, a phenomenon commonly referred to as automation bias. Although overall performance, as reflected by AULROC, improved, the performance curve in session 2 closely resembled that of standalone AI (Fig. 2, Supplementary Fig. 2). A recent study evaluating automation bias in mammography interpretation reported that less experienced radiologists were more likely to adopt higher BI-RADS categories suggested by AI, even when incorrect [21]. Similarly, in the present study, less experienced readers may have been reluctant to downgrade BI-RADS 4–5 assessments, leading to increased sensitivity at the expense of specificity.
Previous studies have reported improvements in specificity with AI-assisted or CADx systems [10,19,22]. In contrast, these findings demonstrate that AI assistance primarily improved sensitivity, with or without a tradeoff in specificity depending on reader experience. A key distinction between prior systems and the system evaluated here is that lesion detection is fully automated, whereas earlier systems required manual ROI selection. The autonomous generation of heatmaps highlighting suspicious regions may have contributed to increased sensitivity. These findings suggest that the AI system may be particularly useful in improving cancer detection, such as in technologist-performed supplemental screening US. However, the observed reduction in specificity indicates a potential increase in false-positive interpretations, which could lead to unnecessary biopsies. Accordingly, users should be aware that the impact of AI assistance may vary depending on user experience and clinical context. Further research on reader–AI interaction and targeted user training is warranted to optimize the clinical implementation of AI in breast US interpretation.
This study has several limitations. First, as a retrospective study, it is subject to potential selection bias in both the study population and the selection of US images. Second, non-mass lesions were excluded, which may limit generalizability. Third, the dataset was enriched with malignant cases, which may not reflect real-world prevalence and may affect generalizability. Fourth, only a single representative US image per lesion was provided, whereas clinical interpretation typically involves multiple imaging planes. In addition, statistical limitations exist in the subgroup analysis due to the small number of certain groups of readers (e.g., experienced breast radiologists). Finally, AI outputs were presented simultaneously in three formats—heatmaps, POM, and AI-derived BI-RADS categories—making it difficult to determine which component most influenced reader decisions.
In conclusion, AI assistance significantly improved the average diagnostic performance of readers in breast US interpretation. Among less experienced professionals and radiologists, improvements in sensitivity and accuracy were accompanied by decreased specificity.
Notes
Author Contributions
Conceptualization: Park H, Yoon JH. Data acquisition: Kim HW, Kim WH, Yoon JH. Data analysis or interpretation: Park H, Han K, Kim J, Yoon JH. Drafting of the manuscript: Park H, Yoon JH. Critical revision of the manuscript: Park H, Han K, Kim HW, Kim WH, Kim J, Yoon JH. Approval of the final version of the manuscript: all authors.
Conflict of Interest
Won Hwa Kim and Jaeil Kim are CEO/CTO of BeamWorks, Inc. Role of the two authors were to provide the AI system for AI analysis of the US images and the web-system for the reader study. Authors who are not employees of the commercial entity had full control of the data and information submitted for publication.
Acknowledgments
This study was supported by a Korea Medical Device Development Fund grant from the Korean government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711197554, RS-2023-00227526).
Supplementary Material
Standalone artificial intelligence (AI) detection performance (https://doi.org/10.14366/usg.26033).
Ultrasound machines used for breast ultrasonography examinations (https://doi.org/10.14366/usg.26033).
AULROC for readers according to subgroups using POM for diagnosis (https://doi.org/10.14366/usg.26033).
Criteria for successful AI-based lesion detection (https://doi.org/10.14366/usg.26033).
AULROC of readers and AI according to POM (https://doi.org/10.14366/usg.26033).
References
Article information Continued
Notes
Key points
The annotation-free artificial intelligence (AI) system significantly improved the sensitivity and overall diagnostic accuracy of readers across diverse professional backgrounds interpreting breast ultrasonography. Among less experienced professionals and radiologists, these gains in sensitivity and accuracy were accompanied by a reduction in specificity. These findings support the integration of annotation-free AI tools to enhance diagnostic performance and reduce inter-reader variability in breast ultrasonography interpretation.