Volume 17, Issue 7 p. 1396-1405
RESEARCH REPORT

ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study

Gautham Arun

Gautham Arun

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Singapore Polytechnic, Singapore, Singapore

Search for more papers by this author
Vivek Perumal

Vivek Perumal

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Search for more papers by this author
Francis Paul John Bato Urias

Francis Paul John Bato Urias

Singapore Polytechnic, Singapore, Singapore

Search for more papers by this author
Yan En Ler

Yan En Ler

Singapore Polytechnic, Singapore, Singapore

Search for more papers by this author
Bryan Wen Tao Tan

Bryan Wen Tao Tan

Singapore Polytechnic, Singapore, Singapore

Search for more papers by this author
Ranganath Vallabhajosyula

Ranganath Vallabhajosyula

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Search for more papers by this author
Emmanuel Tan

Emmanuel Tan

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Search for more papers by this author
Olivia Ng

Olivia Ng

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Search for more papers by this author
Kian Bee Ng

Kian Bee Ng

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Search for more papers by this author
Sreenivasulu Reddy Mogali

Corresponding Author

Sreenivasulu Reddy Mogali

Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore

Correspondence

Dr. Sreenivasulu Reddy Mogali, Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, 11 Mandalay Road, Singapore 308232, Singapore.

Email: [email protected]

Search for more papers by this author
First published: 21 August 2024
Citations: 1

Abstract

Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT-generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI-driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context-aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy (n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5-point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher (U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.

CONFLICT OF INTEREST STATEMENT

The authors declare that they have no competing interests.

DATA AVAILABILITY STATEMENT

Materials and findings from the study have been reported and/or supplemented in the manuscript. Raw data about the Likert scale ratings of the panel and third-party raters of this study are available from the corresponding author on request.