Assessing the accuracy and consistency of responses from Chat Generative Pre-trained Transformer in human papillomavirus infection and protection

Abstract

Chat Generative Pre-trained Transformer is an artificial intelligence chat system that is pre-trained on a large dataset of language data, providing detailed answers to user inquiries. The aim of this study was to evaluate the accuracy and consistency of the model's answers to frequently asked patient questions about human papillomavirus infection. METHODS: The questions prepared for input into Chat Generative Pre-trained Transformer were sourced from globally recognized associations and organizations and were divided into four categories: basic knowledge, diagnosis, treatment, and preventive medicine. The evaluation levels consisted of comprehensive, correct but inadequate, some correct and some incorrect, and completely incorrect. Three independent reviewers evaluated the responses. RESULTS: A total of 123 questions about human papillomavirus infection were prepared. Of the questions prepared for entry into Chat Generative Pre-trained Transformer related to human papillomavirus infection, 52% were categorized as preventive medicine, 28.5% as basic knowledge, 13.8% as diagnosis, and 5.7% as treatment. The model provided "comprehensive" answers to 115 out of 123 (93.5%) questions. When examined by category, the model provided comprehensive answers to 94.3% of the questions related to "basic knowledge," 88.2% of the questions about "diagnosis," 100% of the questions about "treatment," and 93.8% of the questions about "preventive medicine." Additionally, the model provided reproducible answers to 120 (97.6%) questions. CONCLUSION: Chat Generative Pre-trained Transformer performed well by providing accurate and consistent answers to frequently asked questions about human papillomavirus infection risk factors, transmission, diagnosis, impacts, associated cancers, prevention methods, and vaccination. The model may be considered a valuable informational resource for patients when used in conjunction with care provided by healthcare professionals.