Assessing the Knowledge of ChatGPT in Answering Questions Regarding Female Urology
Urology Journal,
Vol. 21 No. 06 (2024),
27 Azar 2024
,
Page 410-414
https://doi.org/10.22037/uj.v21i.8194
Abstract
Purpose: With the recent increase in the use of artificial intelligence in the medical field, this study aimed to evaluate the accuracy and adequacy of ChatGPT's responses to questions related to female urology.
Methods: Intensive internet research was performed to prepare a frequently asked question (FAQs) list. Scientific
questions were created in accordance with the European Urology Association (EAU) Non-neurogenic Female
Lower Urinary Tract Symptoms Guidelines, EAU Chronic Pelvis Pain Guidelines, and EAU Neuro-Urology
Guidelines. All answers by ChatGPT were analysed by two experienced urologists and each answer was scored
between 1 and 4 by the physicians. A score of 1 was the highest and showed that the answer was completely true
and sufficient. The reproducibility of ChatGPT answers was evaluated by asking each question twice using two
different computers.
Results: A total of 96 (97.0%) ChatGPT answers about female urology were accurate and sufficient, and categorized
as grade 1. Additionally, two (2.0%) answers were scored as grade 2, and one answer (1.0%) was scored as
grade 3. None of ChatGPT's responses about female urology were classified as grade 4. In total, 83 questions were
prepared according to EAU guidelines recommendations, and ChatGPT gave complete accurate and satisfactory
answers for 68 (82.9%) questions. The reproducibility rate was highest for ChatGPT answers for questions related
to urinary incontinence, pelvic organ prolapses, and pelvic pain syndromes, and reproducibility rate was 100% for
each subgroup. The reproducibility rate for ChatGPT answers was lowest for CPG questions (84.1%).
Conclusion: For the first time our study revealed that ChatGPT had an excellent accuracy rate in answering questions related to female urology with 97% success rate. In addition, the outcomes of this study showed that ChatGPT accurately and satisfactorily answered 82.9% of questions about female urology based on EAU guidelines.
- artificial intelligence
- guideline
- female urology
How to Cite
References
Anger JT, Saigal CS, Pace J, Rodríguez LV, Litwin MS, Urologic Diseases of America Project. True prevalence of urinary incontinence among female nursing home residents. Urol. 2006;67:281-7.
Chen A, Shahiyan RH, Anger JT. Interstitial cystitis/bladder pain syndrome treatment: a systematic review of sexual health outcomes. Sex Med Rev. 2022;10:71-6.
Tasnim N, Bangash K, Amin O, Luqman S, Hina H. Rising trends in iatrogenic urogenital fistula: A new challenge. Int J Gynaecol Obstet. 2020;148:33-6.
Abufaraj M, Xu T, Cao C, Siyam A, Isleem U, Massad A, Soria F, Shariat SF, Sutcliffe S, Yang L. Prevalence and trends in urinary incontinence among women in the United States, 2005–2018. Am J Obstet Gynecol. 2021;225:166-e1.
Chen J, Wang Y. Social media use for health purposes: systematic review. J Med Internet Res. 2021;23:e17917..
Samaan JS, Yeo YH, Rajeev N, et al. Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obes Surg. 2023 Apr 27:1-7.
Caglar U, Yildiz O, Meric A, et al. Evaluating the performance of ChatGPT in answering questions related to pediatric urology. J Pediatr Urol. 2023:S1477-5131(23)00318-2.
Rahimli Ocakoglu S, Coskun B. The Emerging Role of AI in Patient Education: A Comparative Analysis of LLM Accuracy for Pelvic Organ Prolapse. Med Princ Pract. Published online March 25, 2024. doi:10.1159/000538538
Choueka D, Tabakin AL, Shalom DF. ChatGPT in Urogynecology Research: Novel or Not?. Urogynecol. Published online March 25, 2024. doi:10.1097/SPV.0000000000001505
Zhou Z. Evaluation of ChatGPT's Capabilities in Medical Report Generation. Cureus. 2023;15:e37589.
Yuksel B, Cakmak K. Healthcare information on YouTube: Pregnancy and COVID-19. Int J Gynaecol Obstet. 2020;150:189-93.
Alsyouf M, Stokes P, Hur D, Amasyali A, Ruckle H, Hu B. 'Fake News' in urology: evaluating the accuracy of articles shared on social media in genitourinary malignancies. BJU Int. 2019;124:701-6.
Van Bulck L, Moons P. Response to the Letter to the Editor on: Dr. ChatGPT in Cardiovascular Nursing: A Deeper Dive into Trustworthiness, Value, and Potential Risks . Eur J Cardiovasc Nurs. 2023;zvad049.
Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmol Sci. 2023;3:100324.
- Abstract Viewed: 809 times
- 8194/pdf Downloaded: 529 times