ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study
Gautham Arun
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Singapore Polytechnic, Singapore, Singapore
Search for more papers by this authorVivek Perumal
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorFrancis Paul John Bato Urias
Singapore Polytechnic, Singapore, Singapore
Search for more papers by this authorRanganath Vallabhajosyula
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorEmmanuel Tan
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorOlivia Ng
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorKian Bee Ng
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorCorresponding Author
Sreenivasulu Reddy Mogali
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Correspondence
Dr. Sreenivasulu Reddy Mogali, Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, 11 Mandalay Road, Singapore 308232, Singapore.
Email: [email protected]
Search for more papers by this authorGautham Arun
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Singapore Polytechnic, Singapore, Singapore
Search for more papers by this authorVivek Perumal
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorFrancis Paul John Bato Urias
Singapore Polytechnic, Singapore, Singapore
Search for more papers by this authorRanganath Vallabhajosyula
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorEmmanuel Tan
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorOlivia Ng
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorKian Bee Ng
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Search for more papers by this authorCorresponding Author
Sreenivasulu Reddy Mogali
Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore
Correspondence
Dr. Sreenivasulu Reddy Mogali, Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, 11 Mandalay Road, Singapore 308232, Singapore.
Email: [email protected]
Search for more papers by this authorAbstract
Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT-generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI-driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context-aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy (n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5-point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher (U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.
CONFLICT OF INTEREST STATEMENT
The authors declare that they have no competing interests.
Open Research
DATA AVAILABILITY STATEMENT
Materials and findings from the study have been reported and/or supplemented in the manuscript. Raw data about the Likert scale ratings of the panel and third-party raters of this study are available from the corresponding author on request.
Supporting Information
Filename | Description |
---|---|
ase2502-sup-0001-Supplementary_materials_1.docxWord 2007 document , 18.7 KB |
Supplementary materials S1.. |
ase2502-sup-0002-Supplementary_materials_2.docxWord 2007 document , 57.7 KB |
Supplementary materials S2.. |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- 1Bawack RE, Wamba SF, Carillo KDA, Akter S. Artificial intelligence in e-commerce: a bibliometric study and literature review. Electron Mark. 2022; 32: 297–338. https://doi.org/10.1007/s12525-022-00537-z
- 2Guo X, Shen Z, Zhang Y, Wu T. Review on the application of artificial intelligence in smart homes. Smart Cities. 2019; 2: 402–420. https://doi.org/10.3390/smartcities2030025
10.3390/smartcities2030025 Google Scholar
- 3Webster C, Ivanov S. Robotics, artificial intelligence, and the evolving nature of work. Springer eBooks; 2019. p. 127–143. https://doi.org/10.1007/978-3-030-08277-2_8
- 4Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019; 6(2): 94–98.
- 5Liu S, Ko QS, Heng KQA, Ngiam KY, Feng M. Healthcare transformation in Singapore with artificial intelligence. Front Digit Health. 2020; 2:592121. https://doi.org/10.3389/fdgth.2020.592121
- 6Miao F, Holmes W, Huang R, Zhang H. AI and education: a guidance for policymakers. Palgrave Macmillan, Cham; 2021. https://doi.org/10.54675/PCSP7350
- 7Abou Hashish EA, Alnajjar H. Digital proficiency: assessing knowledge, attitudes, and skills in digital transformation, health literacy, and artificial intelligence among university nursing students. BMC Med Educ. 2024; 24: 508. https://doi.org/10.1186/s12909-024-05482-3
- 8Gordon M, Daniel M, Ajiboye A, Uraiby H, Xu NY, Bartlett R, et al. A scoping review of artificial intelligence in medical education: BEME Guide No. 84. Med Teach. 2024; 46: 446–470. https://doi.org/10.1080/0142159x.2024.2314198
- 9Xu Y, Jiang Z, Ting DSW, Kow AWC, Bello F, Car J, et al. Medical education and physician training in the era of artificial intelligence. Singapore Med J. 2024; 65: 159–166. https://doi.org/10.4103/singaporemedj.smj-2023-203
- 10Chan LK, Pawlina W. Artificial intelligence or natural stupidity? Deep learning or superficial teaching? Anat Sci Educ. 2020; 13(1): 5–7.
- 11Abdellatif H, Al Mushaiqri M, Albalushi H, Al-Zaabi AA, Roychoudhury S, Das S. Teaching, learning and assessing anatomy with artificial intelligence: the road to a better future. Int J Environ Res Public Health. 2022; 19(21):14209.
- 12Adams LC, Busch F, Truhn D, Makowski MR, Aerts HJWL, Bressem KK. What does DALL-E 2 know about radiology? J Med Internet Res. 2023; 25:e43110.
- 13Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol Sci. 2023; 3(4):100324.
- 14Mogali SR. Initial impressions of ChatGPT for anatomy education. Anat Sci Educ. 2023; 17: 444–447. https://doi.org/10.1002/ase.2261
- 15Moldt JA, Festl-Wietek T, Mamlouk AM, Nieselt K, Fuhl W, Herrmann-Werner A. Chatbots for future docs: exploring medical students' attitudes and knowledge towards artificial intelligence and medical chatbots. Med Edu Online. 2023; 28(1):2182659.
- 16Ng O, Han SP. Addressing digital inequities in the age of large language models (LLMs). Med Educ. 2024. https://doi.org/10.1111/medu.15446
10.1111/medu.15446 Google Scholar
- 17Wilson AB, Brown KM, Misch J, Miller CH, Klein BA, Taylor MA, et al. Breaking with tradition: a scoping meta-analysis analyzing the effects of student-centered learning and computer-aided instruction on student performance in anatomy. Anat Sci Educ. 2019; 12(1): 61–73.
- 18Barry DS, Marzouk F, Chulak-Oglu K, Bennett D, Thierney P, O'Keeffe GW. Anatomy education for the Youtube generation. Anat Sci Educ. 2016; 9(1): 90–96.
- 19Bolek KA, Jong GD, Henssen D. The effectiveness of the use of augmented reality in anatomy education: a systematic review and meta-analysis. Sci Rep. 2021; 11:15292.
- 20Sinou N, Sinou N, Filippou D. Virtual reality and augmented reality in anatomy education during COVID-19 pandemic. Cureus. 2023; 15(2):e35170.
- 21Mogali SR, Chandrasekaran R, Radzi S, Peh Z, Tan G, Rajalingam P, et al. Investigating the effectiveness of three-dimensionally printed anatomical models compared with plastinated human specimens in learning cardiac and neck anatomy: a randomized crossover study. Anat Sci Edu. 2022; 15(6): 1007–1017.
- 22Ye Z, Jiang H, Bai S, Wang T, Yang D, Hou H, et al. Meta-analyzing the efficacy of 3D printed models in anatomy education. Front Bioeng Biotechnol. 2023; 11:1117555.
- 23Lazarus MD, Truong M, Douglas P, Selwyn N. Artificial intelligence and clinical anatomical education: promises and perils. Anat Sci Educ. 2020; 17: 249–262. https://doi.org/10.1002/ase.2221
10.1002/ase.2221 Google Scholar
- 24Jacobs E, Wainman B, Bowness J. Applying artificial intelligence to the use of ultrasound as an educational tool: a focus on ultrasound-guided regional anesthesia. Anat Sci Educ. 2023; 17: 919–925. https://doi.org/10.1002/ase.2266
- 25Li YS, Lam CSN, See C. Using a machine learning architecture to create an AI-powered chatbot for anatomy education. Med Sci Educ. 2021; 31(6): 1729–1730.
- 26Abd-alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S, et al. Large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023; 9:e48291. https://doi.org/10.2196/48291
- 27Collins BR, Black EW, Rarey KE. Introducing AnatomyGPT: a customized artificial intelligence application for anatomical sciences education. Clin Anat. 2024; 37: 661–669. https://doi.org/10.1002/ca.24178
- 28 OpenAI. ChatGPT. 2023 [cited 2023 Jul 27]. Available from: https://chat.openai.com
- 29Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. 2023; 17: 926–931. https://doi.org/10.1002/ase.2270
- 30Lucas HC, Upperman JS, Robinson JR. A systematic review of large language models and their implications in medical education. Med Educ. 2024. https://doi.org/10.1111/medu.15402
- 31Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023; 11(6): 887.
- 32Deng J, Lin Y. The benefits and challenges of ChatGPT: an overview. Front Comput Intell Syst. 2020; 2(2): 81–83.
10.54097/fcis.v2i2.4465 Google Scholar
- 33Holzinger A, Keiblinger K, Holub P, Zatloukal K, Müller H. AI for life: trends in artificial intelligence for biotechnology. N Biotechnol. 2023; 74: 16–24.
- 34Briggs J, Ingham F. LangChain AI handbook; n.d. [cited 2023 Jul 26]. Available from: https://www.pinecone.io/learn/series/langchain/
- 35Betts JG, Desaix P, Johnson E, Johnson JE, Korol O, Kruse D, et al. Anatomy and physiology 2e. OpenStax; 2022. Available from: https://openstax.org/details/books/anatomy-and-physiology-2e
- 36Sullivan GM, Feinn R. Using effect size-or why the P value is not enough. J Grad Med Educ. 2012; 4(3): 279–282. https://doi.org/10.4300/JGME-D-12-00156.1
- 37Berry DM. ELIZA and the end of human reason. In: M Baranovska, S Höltgen, editors. Hello, I'm Eliza: fifty years of conversations with computers. Berlin: Projekt Verlag; 2018. p. 53–70.
- 38Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, et al. ChatGPT and other large language models are double-edged swords. Radiology. 2023; 307(2):e230163.
- 39Stokel-Walker C. ChatGPT listed as author on research papers: many scientists disapprove. Nature. 2023; 613: 620–621.
- 40Fijacko N, Gosak L, Štiglic G, Picard CT, Douma MJ. Can ChatGPT pass the life support exams without entering the American Heart Association course? Resuscitation. 2023; 185:109732.
- 41Patel SB, Lam K, Liebrenz M. ChatGPT: friend or foe? Lancet Digit Health. 2023; 5: e112–e114.
- 42Cahan P, Treutlein B. A conversation with ChatGPT on the role of computational systems biology in stem cell research. Stem Cell Rep. 2023; 18: 1–2.
- 43Kıyak YS, Kononowicz AA. Case-based MCQ generator: a custom ChatGPT based on published prompts in the literature for automatic item generation. Med Teach. 2024; 46: 1018–1020. https://doi.org/10.1080/0142159x.2024.2314723
- 44Lund B, Wang S. Chatting about ChatGPT: how may AI and GPT impact academia and libraries? Lib Hi Tech News. 2023; 40: 26–29. https://doi.org/10.1108/LHTN-01-2023-0009
10.1108/LHTN-01-2023-0009 Google Scholar
- 45Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber Phys Syst. 2023; 3: 121–154. https://doi.org/10.1016/j.iotcps.2023.04.003
10.1016/j.iotcps.2023.04.003 Google Scholar