Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions

"Overall, ChatGPT...has the potential to be a valuable resource both for providing immediate medical information to patients and for improving healthcare efficiency and decision-making for healthcare professionals. Indeed, if evaluated and trained by experts on controlled medical information,...ChatGPT could rapidly transform the communication of medical knowledge."

Given the importance of accurate information regarding vaccines, this study aimed to determine the correctness, clarity, and exhaustiveness of the artificial intelligence (AI) tool ChatGPT's responses to misleading questions about vaccines and immunisation, in order to: (i) evaluate how these new information tools may be able to provide relevant and correct information with regard to vaccination adherence; (ii) evaluate if GPT-3.5, being free, has significant differences from the more advanced, paid version; and (iii) evaluate whether the use of AI could help increase health literacy and reduce vaccine hesitancy.

The study was based on the answers given by ChatGPT to the list of the 11 questions concerning "Vaccines and Immunization: Myths and Misconceptions", published on October 19 2020, taken into consideration alongside those given by the World Health Organization (WHO). This list, originally written by the U.S. Centers for Disease Control and Prevention, addresses common misconceptions about vaccination that are often cited by concerned parents as reasons to question the wisdom of having their children vaccinated. Thus, the WHO responded to the listed questions in order to provide an information tool for the general population and health professionals charged with carrying out vaccination. In order to assess whether the answers provided by the chatbot were equally accurate, the listed questions were administered in an individual chat by an investigator to both the free (GPT-3.5) and paid (GPT-4.0) versions of ChatGPT.

Overall, the study's raters perceived that the ChatGPT findings provided accurate and comprehensive information on common myths and misconceptions about vaccination in an easy-to-understand, conversational manner, without providing misinformation or harmful information. In particular, the determinants that had the greatest impact on the scores assigned were: scientific veracity, appropriateness of vocabulary, and the logical order chosen for the description of the contents with regard to clarity and to the completeness of the answer for the exhaustiveness item.

Nevertheless, in some cases, aspects of the description of the contents could be improved. For example, in the raters' opinion, the answers given by both versions of ChatGPT to Question 2 (Which Disease Shows the Impact of Vaccines the Best?) were misleading. In particular, by citing immunisation against smallpox as the only example of the significant impact of vaccination, the chatbot suggested that the eradication of the disease they prevent is the only tangible benefit. From ChatGPT, it is not clear why the implementation of mass vaccination is not directly followed by a dramatic drop in the disease incidence. Indeed, the AI tool appears to disregard the benefits offered by vaccination in the short term (e.g., the management of infection clusters and management of the disease as demonstrated with the COVID-19 vaccination) and in the long term (e.g., the impact of vaccination on economic growth and on the sustainability and efficiency of health systems). This finding is worrying, in the researchers' estimation, considering that "complacency" is one of the main determinants of vaccine hesitancy, and any perception that the vaccine may not be essential in the prevention of infectious diseases may discourage citizens from adhering to vaccination programmes.

Qualitatively, the GPT-4.0 responses were superior to the GPT-3.5 responses in terms of correctness, clarity, and exhaustiveness. The fact that ChatGPT is available for free allows even the most economically disadvantaged patients to access reliable and personalised medical information; that said, many cannot access it for economic or cultural reasons. The availability of a better-performing version (GPT-4.0) only for paying users poses the problem of equality in accessing information, thus raising ethical issues.

Reflecting on the findings, the researchers note that a lack of the basic knowledge necessary to discern between what is correct, clear, and exhaustive versus what is not must be taken into account when referring to how the general public can question an AI. This observation means that the use of these tools in healthcare settings will require careful consideration in order to prevent potentially detrimental uses, such as bypassing professional medical advice, and ethical issues, including the potential risk of bias and factual inaccuracies. In addition, since these AI tools are only as trustworthy as the data they are trained on, it is important to consider that the system does not clarify the sources from which it draws the information. This "black box" could constitute a problem, especially for those aiming to address or investigate scientific issues. However, ChatGPT's own answers underlined the importance of reliable and in-depth sources of information, as well as the use of terms associated with uncertainty, emphasising that the results generated are no substitute for clinical consultation of healthcare professionals.

In conclusion, considering these technologies' "wide availability and potential societal impact, it is critical to exercise caution, acknowledge their limitations and develop appropriate guidelines and regulations with the involvement of all the relevant stakeholders. In particular, the quality of this innovative approach depends, and will continue to depend, more and more on the ability to ask the correct questions as well as on the critical ability of those who use it and will use it, as possible ethical and legal issues could limit potential future applications."