Artificial intelligence (AI) in breast imaging is a growing field, with numerous studies having demonstrated its value for imaging interpretation [1]. Indeed, multiple AI applications have been developed for breast imaging, spanning mammography, magnetic resonance imaging, and ultrasound modalities. Regarding the role of AI applications in breast ultrasound specifically, studies have shown that AI applications have a high sensitivity in predicting malignancy [2-4].

Koios decision support (DS) for Breast (Koios DS Breast; Koios Medical, Inc., New York, NY, USA) is a proprietary, Food and Drug Administration (FDA)–cleared AI software program designed to aid radiologists’ interpretation of lesions identified on breast ultrasound. Koios DS generates a Breast Imaging Reporting and Data System (BI-RADS)–aligned probability of malignancy for any lesion indicated by a region of interest (ROI) on the static breast ultrasound image drawn manually by the radiologist. Koios DS has been reported to improve radiologists’ accuracy in diagnosing breast cancer [3,5].

Breast ultrasound is an essential tool in the detection and diagnosis of breast cancer. The sonographic characteristics of a lesion help drive the level of suspicion and therefore the predicted likelihood of benignity or malignancy [6]. In some cases, a sonographic lesion is given a probably benign (BI-RADS category 3) assessment and short-interval follow-up is recommended for the patient to ensure stability. While most BI-RADS category 3 sonographic lesions will prove benign on follow-up, follow-up remains essential to identify early changes in the lesion that may indicate malignancy. Studies have shown that BI-RADS category 3 sonographic lesions have a malignancy rate from 0.2%-2.2%, which varies based upon the presence or absence of a mammographic finding [7,8]. The malignancy rate increases to 4.9% in lesions that demonstrate interval growth on follow-up ultrasound [8]. Moreover, morphologic changes in growing lesions are more highly associated with malignancy [8]. Appropriate follow-up can detect these early changes and allow for timely diagnosis and treatment, which are essential to improve patient outcomes.

Follow-up imaging for an individual patient may be delayed for a variety of reasons, but there are also environmental constraints and resource limitations that may impact the follow-up of a large population of patients at the same moment in time, such as experienced during the coronavirus disease 2019 (COVID-19) pandemic [9]. During the COVID-19 pandemic, our institution, like many across the country, had to meet the challenge of scheduling breast imaging follow-up appointments for many patients with delayed follow-up. Notably, triaging care may be particularly challenging when assessing patients with a similar risk profile, as with patients with BI-RADS category 3 sonographic lesions.

To our knowledge, no prior study has assessed the use of using breast ultrasound AI DS to triage care once follow-up is delayed. We evaluated our institution's experience in using Koios DS as part of the clinical workflow to triage patients with BI-RADS category 3 sonographic lesions whose follow-up was delayed during the COVID-19 pandemic, against subsequent imaging and/or pathologic follow-up results. While our institution’s process of using Koios DS for triaging was set during the COVID-19 pandemic, the process and findings may be extrapolated to breast ultrasound follow-up in scenarios where resources are limited.

Our study demonstrates that AI DS may be a helpful triaging tool when large groups of patients have simultaneously delayed follow-up, such as experienced during the COVID-19 pandemic. In our study, we evaluated the output of the Koios DS software, which was obtained as part of the clinical workflow to triage in patients with probably benign sonographic lesions and delayed follow-up during the COVID-19 pandemic, against the true presence or absence of breast cancer defined by stability/resolution on follow-up imaging for at least 2 years or pathology results if the lesion was biopsied. We found that Koios DS had a sensitivity and NPV of 100% while the false-positive rate was 30%.

The false-positive rate is higher than the acceptable rate if the AI software were acting in isolation to diagnose breast cancer; however, in this context, we used the AI software to assess those lesions already interpreted as probably benign by an experienced radiologist in order to help triage probably benign lesions that had delayed follow-up. The NPV of 100% reassures us that in those patients with a benign assessment by the AI system, the likelihood of malignancy is essentially zero (Fig. 5). A sensitivity of 100% is also helpful in this setting of triage, ensuring us that breast ultrasounds with cancer are captured appropriately by the AI system as suspicious or probably malignant. All three of the lesions that were assessed as probably malignant by the AI system were benign, two of which were likely fat necrosis at a site of prior lumpectomy. Of note, the AI system is not specifically trained to evaluate fat necrosis or postoperative changes.

Conversely, two cancers assessed as probably benign by the radiologist were accurately assessed as suspicious by Koios DS. On retrospective review, one case assessed as probably benign fat necrosis does not meet classic criteria for fat necrosis. However, at our institution we see a large volume of postoperative patients, in whom early evolving fat necrosis fail to meet typical probably benign or benign sonographic appearance. Given the patient’s recent left breast mastopexy, and based upon real time appearance, which the radiologist described as mixed echogenicity with central areas of hypoechogenicity, the interpreting radiologist assessed the finding as probably benign fat necrosis. The other case misinterpreted as a probably benign lymph node, even on retrospective review does appear to have morphology suggestive of a lymph node.

The potential of AI DS in breast ultrasound imaging has been demonstrated in a variety of publications [2-5,11,12]. But, to our knowledge, none have trialed such software for patient triaging purposes. Mango et al. [5] demonstrated that Koios DS improves the accuracy of breast ultrasound interpretation in a retrospective reader study that included 900 breast ultrasounds lesions. They also found that Koios DS has an extremely high sensitivity of 98%. The study included ultrasound findings assessed as BI-RADS category 2-5 by an interpreting radiologist, unlike our cohort of solely BI-RADS category 3 lesions. Shen et al. [11] highlighted the utility of an AI system developed and evaluated in a New York University ultrasound dataset. In their retrospective reader study of 600 breast ultrasound exams, the AI system improved the diagnostic accuracy of breast ultrasound interpretation by the radiologist, by 4.4% (90.1% to 94.5%). In Browne et al.’s study [2], which used Koios DS, 50 BI-RADS category 3 lesions were biopsied, with seven demonstrating malignant pathology. These lesions underwent cytology assessment at the radiologist’s discretion with subsequent core biopsy based on pathology results, which differs from BI-RADS atlas recommendations for BI-RADS category 3 lesions which typically involve short-interval imaging follow-up [6]. In our practice, if the radiologist deems that histologic assessment warranted, then a BI-RADS category 4 assessment is given. This difference in practice may account in part for the difference in malignancy rate within the BI-RADS category 3 study samples (0.9% malignancy rate in our study compared with 15.4% in Browne’s study [2]). Coffey et al. [4] used Koios DS to analyze 345 triple-negative breast cancers. The authors found that Koios DS misclassified 12 of the 345 cancers as probably benign or benign, but correctly classified six out of six cancers that were assessed as probably benign or benign by the radiologist. Similar to our findings, Koios DS had an extremely high sensitivity of 96%. The difference in performance metrics between our study and the previous studies of Coffey et al. [4] and Browne et al. [2] (in addition to those listed above) is likely, at least in part, a result of the low absolute number of cancers in our study. And lastly, Lee et al. [12] retrospectively reviewed 492 breast masses and found that while specificity, positive predative value, and accuracy significantly improved with the use of AI computer-assisted diagnosis, the area under the curve only improved with inexperienced radiologists.

In breast cancers that initially present with probably benign appearing features, timely follow-up is essential to identify changes and ensure prompt diagnosis [8]. However, as experienced during the COVID-19 pandemic, there are external elements that may impact the follow-up for many patients at the same moment(s) in time [13]. Resource limitations and environmental safety considerations may occur with minimal warning or ability to prepare. As a safety measure during the COVID-19 pandemic, in March 2020, the Society of Breast Imaging recommended that 6-month follow-up breast imaging exams are delayed until risks from the pandemic have decreased [14] and patients report experiencing those delays in breast cancer imaging [15]. As imaging facilities and institutions quickly reopened, there was pressure to accommodate an increased demand of breast imaging appointments for those with delayed screening and diagnostic appointments, while simultaneously mitigating transmission risk, ensuring social distancing in waiting rooms, and limiting the number of patients in clinic [13]. The ability to triage patients in such circumstances became essential and our results support that AI DS may be able to identify which patients to prioritize for follow-up. Of note, the potential utility of AI DS to triage extends beyond the circumstances of the COVID-19 pandemic. Such a tool may prove beneficial in number of situations in which patient care triaging needs are present, such as natural disaster, equipment, or material supply chain challenges, or in communities in which there is a struggle to adequately meet healthcare needs, both within the United States and abroad.

There are several limitations to our study. First, this is a retrospective single-institutional study. Further studies, particularly prospective trials, assessing change in management and clinical outcomes based upon integrated breast imager and AI DS assessment are needed to fully evaluate the potential of AI DS in triaging patients with breast ultrasound findings. Second, we are a dedicated cancer institute with breast imaging subspecialists. As such, the impact of a breast ultrasound AI DS tool on patients with BI-RADS category 3 assessments may vary across institutions and have a more powerful impact in settings with fewer resources. Third, we may also have a larger patient population of patients with prior lumpectomies or surgical breast excisions. As Koios DS is not trained to evaluate fat necrosis and postoperative changes, Koios DS may perform superiorly in a population of patients without prior breast surgery. Fourth, 74 patients were lost to follow-up. We recognize that some patients may have incomplete follow-up secondary to factors independent from the COVID-19 pandemic, such as seeking healthcare services at another institution. However, we also know that the pandemic did not affect people uniformly. Data have shown that the pandemic disproportionally affected minorities and people of lower socioeconomic status [16]. Our final dataset excluding those lost to follow-up is therefore biased toward a patient population that was able to return for their imaging appointments and does not include those who lost health insurance and/or other resources directly or indirectly because of the pandemic. Lastly, in our sample of 221 lesions with completed follow-up, there were two cancers. Our malignancy rate is appropriate for BI-RADS category 3 lesions at less than 2%; however, the low absolute count of cancers is a limitation when calculating statistical performance metrics. Specifically, positive predictive value (PPV) is not reported, as it is very dependent on the prevalence of cancer. The low prevalence of cancer in this population could lead to unreliable estimates of PPV.

In conclusion, breast ultrasound AI DS can be used to triage probably benign breast ultrasound findings by identifying those cases that are more likely to be malignant. This may be particularly useful when large numbers of follow-up appointments are simultaneously delayed, such as during the COVID-19 pandemic, and when radiologists are prioritizing patient care with limited resources.