ChatGPT-o1 Preview Outperforms ChatGPT-4 as a Diagnostic Support Tool for Ankle Pain Triage in Emergency Settings
学术急诊医学档案,
卷 13 编号 1 (2025),
6 九月 2025
,
第 e42 页
https://doi.org/10.22037/aaemj.v13i1.2580
摘要
Introduction: ChatGPT, a general-purpose language model, is not specifically optimized for medical applications. This study aimed to assess the performance of ChatGPT-4 and o1-preview in generating differential diagnoses for common cases of ankle pain in emergency settings.
Methods: Common presentations of ankle pain were identified through consultations with an experienced orthopedic surgeon and a review of relevant hospital and social media sources. To replicate typical patient inquiries, questions were crafted in simple, non-technical language, requesting three possible differential diagnoses for each scenario. The second phase involved designing case vignettes reflecting scenarios typical for triage nurses or physicians. Responses from ChatGPT were evaluated against a benchmark established by two experienced orthopedic surgeons, with a scoring system assessing the accuracy, clarity, and relevance of the differential diagnoses based on their order.
Results: In 21 ankle pain presentations, ChatGPT-o1 preview outperformed ChatGPT-4 in both accuracy and clarity, with only the clarity score reaching statistical significance (p < 0.001). ChatGPT-o1 preview also had a significantly higher total score (p = 0.004). In 15 case vignettes, ChatGPT-o1 preview scored better on diagnostic and management clarity, though differences in diagnostic accuracy were not statistically significant. Among 51 questions, ChatGPT-4 and ChatGPT-o1 preview produced incorrect responses for 5 (9.8%) and 4 (7.8%) questions, respectively. Inter-rater reliability analysis demonstrated excellent reliability of the scoring system with interclass coefficients of 0.99 (95% CI, 0.998–0.999) for accuracy scores and 0.99 (95% CI, 0.990–0.995) for clarity scores.
Conclusion: Our findings demonstrated that both ChatGPT-4 and ChatGPT-o1 preview provide acceptable performance in the triage of ankle pain cases in emergency settings. ChatGPT-o1 preview outperformed ChatGPT-4, offering clearer and more precise responses. While both models show potential as supportive tools, their role should remain supervised and strictly supplementary to clinical expertise.
- Large Language Models
- ChatGPT
- Emergency Medicine
- Triage
- Artificial intelligence
- Ankle
##submission.howToCite##
参考
Hadi MU, Qureshi R, Shah A, Irfan M, Zafar A, Shaikh MB, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints. 2023.
Fui-Hoon Nah F, Zheng R, Cai J, Siau K, Chen L. Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Taylor & Francis; 2023. p. 277-304.
Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine. 2023;388(13):1233-9.
Yao JJ, Aggarwal M, Lopez RD, Namdari S. Current Concepts Review: Large Language Models in Orthopaedics: Definitions, Uses, and Limitations. JBJS. 2021:10.2106.
Shah NH, Entwistle D, Pfeffer MA. Creation and adoption of large language models in medicine. JAMA. 2023;330(9):866-9.
Merrell LA, Fisher ND, Egol KA. Large language models in orthopaedic trauma: a cutting-edge technology to enhance the field. JBJS. 2023;105(17):1383-7.
Kaboudi N, Firouzbakht S, Shahir Eftekhar M, Fayazbakhsh F, Joharivarnoosfaderani N, Ghaderi S, et al. Diagnostic Accuracy of ChatGPT for Patients' Triage; a Systematic Review and Meta-Analysis. Arch Acad Emerg Med. 2024;12(1):e60.
Walker HL, Ghani S, Kuemmerli C, Nebiker CA, Müller BP, Raptis DA, Staubli SM. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. Journal of Medical Internet Research. 2023;25:e47479.
Chlorogiannis DD, Apostolos A, Chlorogiannis A, Palaiodimos L, Giannakoulas G, Pargaonkar S, et al. The Role of ChatGPT in the Advancement of Diagnosis, Management, and Prognosis of Cardiovascular and Cerebrovascular Disease. Healthcare (Basel). 2023;11(21).
Samaan JS, Yeo YH, Rajeev N, Hawley L, Abel S, Ng WH, et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obesity surgery. 2023;33(6):1790-6.
Shah S, Thomas AC, Noone JM, Blanchette CM, Wikstrom EA. Incidence and cost of ankle sprains in United States emergency departments. Sports health. 2016;8(6):547-52.
Shah S, Thomas A, Noone J, Blanchette C, Wikstrom E. Emergency department visits associated with ankle sprains in the United States. Value in Health. 2016;19(3):A12.
Doherty C, Delahunt E, Caulfield B, Hertel J, Ryan J, Bleakley C. The incidence and prevalence of ankle sprain injury: a systematic review and meta-analysis of prospective epidemiological studies. Sports medicine. 2014;44:123-40.
Halabchi F, Hassabi M. Acute ankle sprain in athletes: Clinical aspects and algorithmic approach. World journal of orthopedics. 2020;11(12):534.
Kerkhoffs GM, van den Bekerom M, Elders LA, van Beek PA, Hullegie WA, Bloemers GM, et al. Diagnosis, treatment and prevention of ankle sprains: an evidence-based clinical guideline. British journal of sports medicine. 2012;46(12):854-60.
Court-Brown CM, McBirnie J, Wilson G. Adult ankle fractures—an increasing problem? Acta Orthopaedica Scandinavica. 1998;69(1):43-7.
Renström PA, Lynch SA. Ankle ligament injuries. Revista Brasileira de Medicina do Esporte. 1998;4:71-80.
Polzer H, Kanz KG, Prall WC, Haasters F, Ockert B, Mutschler W, Grote S. Diagnosis and treatment of acute ankle injuries: development of an evidence-based algorithm. Orthop Rev (Pavia). 2012;4(1):e5.
Manske RC, Wolfe C, Page P, Voight M. Enhancing the Diagnosis of Lateral Ankle Sprains: The Role of MSK Diagnostic Ultrasound in Evaluating ATFL and CFL. International Journal of Sports Physical Therapy. 2024;19(2):245.
Cavazos Jr GJ, Harkless LB. The epidemiology, evaluation, and assessment of lateral ankle sprains in athletes. Journal of Sports Medicine and Therapy. 2021.
Alowais SA, Alghamdi SS, Alsuhebany N, Alqahtani T, Alshaya AI, Almohareb SN, et al. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC medical education. 2023;23(1):689.
HaghighiKian SM, Shirinzadeh-Dastgiri A, Vakili-Ojarood M, Naseri A, Barahman M, Saberi A, et al. A Holistic Approach to Implementing Artificial Intelligence in Lung Cancer. Indian Journal of Surgical Oncology. 2024:1-22.
Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of diagnostic and triage accuracy of Ada health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: clinical data analysis study. JMIR mHealth and uHealth. 2023;11(1):e49995.
Le K, Chen J, Mai D, Le KDR. An Evaluation on the Potential of Large Language Models for Use in Trauma Triage. Emergency Care and Medicine. 2024;1(4):350-67.
Hoppe JM, Auer MK, Strüven A, Massberg S, Stremmel C. ChatGPT with GPT-4 outperforms emergency department physicians in diagnostic accuracy: retrospective analysis. Journal of Medical Internet Research. 2024;26:e56110.
Mika AP, Martin JR, Engstrom SM, Polkowski GG, Wilson JM. Assessing ChatGPT responses to common patient questions regarding total hip arthroplasty. JBJS. 2023;105(19):1519-26.
Wedmore I, Young S, Franklin J. Emergency department evaluation and management of foot and ankle pain. Emergency Medicine Clinics. 2015;33(2):363-96.
Kunze KN, Varady NH, Mazzucco M, Lu AZ, Chahla J, Martin RK, et al. The large language model ChatGPT-4 exhibits excellent triage capabilities and diagnostic performance for patients presenting with various causes of knee pain. Arthroscopy: The Journal of Arthroscopic & Related Surgery. 2024.
Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. Journal of medical Internet research. 2023;25:e50638.
Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-80.
- 摘要 ##plugins.themes.ojsPlusA.frontend.article.viewed##: 429 ##plugins.themes.ojsPlusA.frontend.article.times##
- pdf (English) ##plugins.themes.ojsPlusA.frontend.article.downloaded##: 1108 ##plugins.themes.ojsPlusA.frontend.article.times##
##plugins.themes.ojsPlusA.frontend.article.downloadstatastics##
##plugins.block.makeSubmission.linkLabel##
SJR
COPE
