Recent scientific report This study investigated the proficiency of ChatGPT in answering questions related to colorectal cancer (CRC).
study: Evaluating AI in Healthcare: Comparative Analysis of Expert and ChatGPT Responses to Colorectal Cancer Questions. Image credit: Miha Creative/Shutterstock.com
background
Colorectal cancer is one of the leading causes of cancer death worldwide. Despite advances in the medical field, survival rates for colorectal cancer patients are low.
To reduce the mortality associated with CRC, it is essential to diagnose this disease early and provide comprehensive treatment and care.
One of the main factors hindering early detection of colorectal cancer is lack of awareness about the symptoms of colorectal cancer. Many patients miss serious signs of colorectal cancer due to a lack of accessible knowledge, delaying the opportunity to seek timely help from a doctor.
It should also be noted that much of the online information about CRC is misleading. Most peer-reviewed resources, such as UpToDate and the MSD manual, are designed for medical professionals rather than the general public.
It is important to develop a reliable platform that patients can easily navigate to obtain reliable information about their disease. These platforms should provide easy-to-understand medical information and provide important guidance on when to see a doctor.
ChatGPT is a free AI system based on deep learning and large-scale language models (LLM). ChatGPT responds to a wide range of prompts, making it applicable to many fields, including healthcare.
This technology can be used to educate the public about various diseases. Therefore, this AI-based platform allows patients to make informed health decisions. This technology could then lead to earlier diagnosis of the disease and better treatment outcomes.
About research
Although many studies have shown the effectiveness of ChatGPT in medicine, more research is needed to evaluate its accuracy. To this end, the current study investigated the proficiency of ChatGPT for CRC diagnosis and treatment.
A book on “Colorectal Cancer: Answering Your Questions” published in China was referenced to evaluate the accuracy of ChatGPT (GPT-3.5 version) in answering questions about CRC.
ChatGPT had a total of 131 questions covering various aspects of CRC, including surgical management, radiotherapy, medical treatment, stoma treatment, interventional treatment, pain management, and deep vein treatment.
To test ChatGPT’s accuracy, the test questions have already been answered by experts. His ChatGPT answers to each question were evaluated and scored by a clinician specializing in CRC.
research result
The reproducibility of ChatGPT results was examined and showed a high level of uniformity in precision and comprehensiveness. The consistent reproducibility of ChatGPT’s responses demonstrates the reliability of this system in providing accurate medical information.
ChatGPT showed promising accuracy but fell short in terms of comprehensiveness. This shortcoming of ChatGPT may be related to training the AI model using extensive and non-specific data.
Therefore, updating an AI model like ChatGPT with specific or specialized data can significantly improve the depth and breadth of the model’s response.
ChatGPT’s overall score showed that the model performed very well, especially regarding radiotherapy, stoma care, pain management, and vein care.
Although ChatGPT showed good performance in providing valid answers to CRC-related questions, it still lacked expertise, especially regarding surgical management, basic information, and internal medicine. This performance degradation is expected to hinder the adoption of AI models like ChatGPT in clinical practice.
conclusion
The current study has limitations, including an insufficient number of valid questions related to CRC. Additionally, this study used the CRC Public Health Book as a data source, which fundamentally limited the types of questions based on the CRC.
The questions were carefully selected by the authors and may not include all questions that patients and their families might actually have. Another limitation of this study is that it uses book responses as a benchmark for scoring his ChatGPT responses to CRC-based questions.
The current study highlighted the potential and limitations of ChatGPT for CRC questions.
The insights presented here may form the basis for future improved versions of ChatGPT to help accurately diagnose CRC and facilitate early treatment.
Future studies should use larger sample sizes to investigate the real-world effectiveness of ChatGPT. Additionally, researchers should explore ways to integrate personal health information with AI models to provide personalized information to patients.