The emergence of large language models (LLMs) has marked a significant advancement in artificial intelligence and generated considerable interest due to their potential to transform medical practice [1,2]. LLMs, such as generative pretrained transformers (GPT) like ChatGPT (OpenAI), exhibit exceptional capabilities in natural language processing tasks, including clinical question answering, text summarization, and contextual analysis [3-9]. These models are trained on comprehensive datasets that incorporate scientific literature, medical publications, and diverse digital resources across multiple disciplines and languages [2,10,11].

Recent developments have introduced multimodal LLMs that extend beyond text analysis to include the interpretation of audio, visual, and video data [12,13]. These advanced systems have shown promising results in various medical applications, including diagnostic assessment, clinical documentation, and disease identification, achieving these outcomes without additional medical-specific training [12-16].

In radiology, LLMs have been successfully applied to data mining and structured reporting [17-19]. Recent technological progress has led to enhanced multimodal LLMs, including OpenAI's GPT, Anthropic's Claude, and Google's Gemini [20]. These updated versions demonstrate improved diagnostic accuracy compared to their previous iterations, underscoring the importance of continuous model development [21,22]. However, as new systems continue to emerge, a systematic evaluation of their performance and clinical utility remains essential to ensure proper implementation and to minimize potential risks of misuse. Notably, a previous study using New England Journal of Medicine Image Challenge cases suggested that LLMs could provide correct answers even without image inputs, and that their performance was influenced more by text input length than by image interpretation [21].

Therefore, a study was designed to examine the radiological imaging interpretation capabilities of three widely used multimodal LLMs using cases with multiple imaging inputs while minimizing clinical text information. The study utilized image-based diagnostic challenges with multiple-choice questions from the publicly available educational repository of the Korean Society of Ultrasound in Medicine (KSUM), which provides bi-monthly content to subscribers. Additionally, this study examined various factors affecting model performance, including the effects of prompt engineering, question types, case rarity, difficulty levels, and the knowledge cutoff dates of the LLMs.

This study evaluated the diagnostic performance of three multimodal LLMs—Claude 3.5 Sonnet, GPT-4o, and Gemini-1.5-Pro-002—in interpreting radiological cases from the KSUM case discussion repository. Among all tested models and prompt engineering strategies, Claude 3.5 Sonnet with AI-generated prompts achieved the highest numerical accuracy. Nevertheless, the performance of all models remained below human diagnostic benchmarks, highlighting the current limitations of autonomous radiological image interpretation by LLMs. The integration of descriptive text inputs led to significant improvements in diagnostic accuracy across most scenarios. These findings underscore the critical role of contextual information in enhancing interpretative accuracy [28,29].

Multiple factors influenced diagnostic accuracy, including the prompt engineering methodology, various input factors, and model pretraining characteristics [29-31]. Original prompts yielded superior performance compared to basic prompts, aligning with previous research [30,32]. However, chain-of-thought prompts produced similar accuracy to original prompts. GPT-4o, known for its capabilities in specialized scientific reasoning [33], along with the other evaluated LLMs, naturally employed a step-by-step approach in their diagnostic processes during the conversational testing. This suggests that explicit chain-of-thought instructions in system prompts may be redundant [26], as these advanced models have already integrated analytical reasoning into their base responses [34].

Analysis of reflection prompts revealed distinct behavioral patterns among models. While GPT-4o and Gemini-1.5-Pro-002 exhibited minimal answer revision, Claude 3.5 Sonnet demonstrated a 17.9% revision rate. However, its revised responses showed decreased accuracy (41.8%, 28/67) compared to the initial answers (49.3%, 33/67). Only two cases improved from incorrect to correct answers, whereas seven cases changed from correct to incorrect following revisions, typically due to recalibration between general and atypical scenario interpretations. Notably, with the integration of descriptive text, the revised accuracy remained essentially unchanged (64.2%, 43/67), with two cases improving and two cases worsening. This pattern suggests that diagnostic uncertainty in imaging analysis may stem from insufficient pretraining on radiological images relative to textual data [35]. Moreover, the implementation of multiagent prompts did not replicate previously reported performance enhancements, possibly due to lengthy prompt structures (293 words) and limitations imposed by single-session chat and single-system prompts, which may have constrained effective agent collaboration [25,26,32]. This finding indicates opportunities for future research and development in optimizing multiagent architectures.

The AI-generated prompt, developed as an enhancement of the original prompt structure, consistently demonstrated superior performance despite its substantial length (295 words), indicating a reduced dependence on human-engineered prompts. This effect was particularly pronounced with Claude 3.5 Sonnet, which exhibited enhanced performance with Anthropic-developed technology. This suggests that prompts generated using the same underlying architecture may be more effective for that specific model. Furthermore, as shown in Supplementary Table 5, in the analysis excluding AI-generated prompts, performance differences between models were not statistically significant for imaging-only input, although Claude maintained numerically higher accuracy (44.8%). This finding implies that the superior performance initially observed may be partially attributed to the alignment between Claude’s architecture and its AI-generated prompts, highlighting the importance of platform-specific prompt optimization strategies.

Regarding case rarity, GPT-4o initially performed poorly on rare cases with the original prompt (6.7%, 1/15) compared to non-rare cases (53.9%, 28/52; P=0.001). However, the implementation of AI-generated prompts substantially improved rare case performance (40.0%, 6/15), making it comparable to non-rare cases (48.1%, 25/52; P=0.796). Additionally, the multivariable analysis revealed that while the correction rate was consistently positively associated with diagnostic accuracy (OR, 4.3 to 6.4; P<0.05), the negative impact of case rarity (OR, 0.1; P=0.014) was effectively mitigated through AI-generated prompts (OR, 1.0; P=0.984). This suggests that although the model’s pretraining and supervised fine-tuning data likely emphasized typical cases, optimized prompt engineering can help overcome this limitation. For future development, post-training strategies should deliberately incorporate more atypical and rare cases to improve model performance across a broader spectrum of radiologic presentations.

Temporal analysis revealed varying patterns in model performance relative to knowledge cutoff dates. GPT-4o and Gemini-1.5-Pro-002 showed no significant differences between pre- and post-cutoff cases. In contrast, Claude 3.5 Sonnet demonstrated significantly higher accuracy for pre-cutoff cases (64.0%, 32/50) compared to post-cutoff cases (23.5%, 4/17; P=0.005). However, this finding should be interpreted cautiously due to the inherent complexity in determining effective knowledge cutoffs. Recent research has suggested that reported cutoff dates may not accurately reflect the temporal alignment of various resources within LLM training data, implying that the actual temporal boundaries affecting model performance may differ from the reported dates [36]. These limitations, combined with proprietary restrictions on accessing detailed pretraining data, underscore the need for more comprehensive investigations into the temporal effects on LLM performance in medical imaging applications.

The present study demonstrated that adding descriptive text inputs improved model performance, which aligns with previous studies in radiological image analysis [37]. They confirm that radiologic image descriptions and medical history are strong contributors to LLM performance in imaging analysis. It is acknowledged that the quality and specificity of these text descriptions may influence model performance. Standardized methods for investigating how different levels of text input quality—regarding imaging descriptions, medical history, and structure—affect model accuracy are needed in future studies to provide valuable insights for optimizing multimodal LLM applications in radiology.

This study had several limitations. First, the relatively small sample size may limit the generalizability of the findings and may have affected the ability to detect statistically significant differences between individual models after multiple comparison corrections. Future studies with larger datasets across various radiological conditions would be valuable to validate the findings and potentially identify additional patterns in model performance. Second, the analysis did not include a detailed evaluation of the LLMs’ reasoning processes behind their answer selections, making it difficult to determine whether correct responses resulted from genuine understanding or mere pattern recognition. Third, the study’s reliance on multiple-choice questions for performance evaluation may not fully represent real-world clinical scenarios that typically require free-text responses. Future studies should incorporate free-text radiological interpretations and reporting that more closely reflect clinical practice. Fourth, methodological constraints prevented the evaluation of text-only inputs (description-only) due to variations in user and system prompts, which would have compromised direct comparisons. Fifth, human performance metrics from the KSUM website may be subject to reporting bias, as they only reflect responses from users who voluntarily submitted answers online, potentially affecting the representativeness of the comparative analysis. Lastly, due to the lack of open technologies for prompt generation across different platforms, it was not possible to test how various model architectures might respond to platform-specific prompt optimization techniques. Future studies should explore this aspect to better understand the relationship between model architecture and prompt engineering effectiveness.

Claude 3.5 Sonnet, when utilizing AI-generated prompts, demonstrated the highest diagnostic accuracy among the evaluated multimodal LLMs, although it did not reach human performance benchmarks. Consequently, autonomous radiological image interpretation by LLMs remains limited for direct clinical implementation. Given the significant enhancement in performance achieved through the integration of descriptive text inputs, combining radiologist-generated descriptive content with LLM analysis holds potential as a supportive diagnostic tool.