By Leanna Coy, FHP-BC
When looking for information, the internet long ago replaced the encyclopedia. Now, with growing popularity, artificial intelligence (AI) is becoming the go-to for information. About one-third of people now use AI as a first source for information. This shouldn’t come as a surprise in our society where people increasingly want quick, clear answers. But should this be where you go for health information?
AI is still in early stages compared to other sources that have been around for years. This means companies are still working out bugs and flaws with these systems. Over and over, AI has been found to provide incorrect health information. This matters because your life may depend on it.
What’s wrong with this picture
Generative AI chatbots are large language models (LLMs). They work by referencing large volumes of text and articles about a topic. They then measure the odds of the words appearing next to each other, trying to generate normal-looking sentences. They don’t aim for accuracy, but what appears as “normal.” This is where the problems begin.
Accuracy is one of the most important aspects of health information. It can literally mean a difference between life and death. But AI doesn’t know that. They are only creating a response as a language model. They don’t understand the difference between “data” and “facts.” This means they have “hallucinations” where they can provide information that sounds true, but isn’t practical. The hallucinations can run from ridiculous to harmful. Some examples include:
- Telling people to eat rocks as a way to add minerals and vitamins to their diet
- Advising the addition of glue into pizza sauce to keep cheese from sliding off
- The National Eating Disorders Association removed its chatbot after the AI repeatedly gave information that could worsen eating disorders
- Incorrectly listing a pap smear as a screening test for vaginal cancer
AI and people don’t communicate well together
Researchers are looking at the AI to find where the problems exist. One thing that’s become clear is that people aren’t speaking in the same manner as the AI. People tend to provide information in pieces or omit it. When talking with a person, the listener can infer or read between the lines. An AI can’t. The AI can only work with the information it is given and not infer anything.
People often don’t know what information to give the AI to get accurate advice. In a study conducted by Oxford, LLMs provided medical information that people didn’t find useful. The AI would give a list of possible conditions, leaving the people guessing which would fit their symptoms. Also, when slight variations were made to questions, the LLMs gave drastically different responses.
You can mislead the AI with your questions
When patients ask, “I think I have this diagnosis. What can I do to treat it?” they are leading the AI on a problematic path. First, you’ve self-diagnosed, and the AI doesn’t question that. The chatbots are trained to be people pleasers, but aren’t people. They are unlikely to disagree with you. The diagnosis may be wrong, leading to incorrect medication and treatment recommendations from the AI. These recommendations may be harmful.
The AI can offer answers that are technically correct but inappropriate due to a lack of context. The problem is the disconnect between how patients ask questions and how the chatbots are evaluated. Patients often ask emotion-backed questions that lack information. This can mislead chatbots that are trained on exam-style questions.
The machine cannot diagnose and treat illness
LLMs are providing facts based only on the words in the questions asked. They cannot interpret the context of the question or review your related health history. A health professional can and knows when to stop providing information to patients.
One study found that when a chatbot was asked how to perform a medical procedure at home, it correctly warned that it should only be done by a health professional. But instead of ending the conversation, it then gave the user step-by-step instructions anyway. A doctor or nurse knows the safety issues behind why not to give those instructions. LLMs don’t have the ability to reason like humans. They cannot explain how they reach conclusions about diagnoses or treatment options. Reasoning is key in making safe and accurate medical decisions.
Another study looked at the popular AI chatbots – Gemini, DeepSeek, Meta AI, ChatGPT, and Grok. They tested them with questions on cancer, vaccines, stem cells, nutrition, and athletic performance. The majority of the responses had problems ranging from mild to problematic. The worst responses were with athletic performance and nutrition. But the AI sounded confident with their answers.
Understand the limits of information
For anyone using LLMs for health information, understanding the limits of the AI is important. Consider any AI response a quick, snapshot response rather than a fully informed one. Always go deeper by looking for cited sources and reliable sources you trust. This includes your healthcare provider.
When you ask questions of a health professional, they know how to read between the lines to understand the subtext and emotion behind your questions. A health professional will know which follow-up questions to ask you before making recommendations, diagnoses, and treatment plans. They can also physically assess you, which the AI cannot.
Know that the information from an AI is likely biased because healthcare has been biased for decades. Health studies were historically done on young, white, healthy men. This can lead to a whole slew of information that doesn’t relate to the average person asking the questions.
AI is also spreading misinformation. The LLMs pull sources from anywhere on the web – whether or not they are reputable. The sources may come from false or misleading stories that are shared over and over again on the internet. The more of this type of information that’s available online, the greater the chance that the AI will repeat the false information.
Using AI safely
Health information can be confusing. Ask the AI to clarify medical terms. For example, “What does triple-negative breast cancer mean in plain language?” Use the AI to help simplify studies and medical information. Try uploading articles and information on treatment guidelines. Then ask the chatbot very clear, specific questions about the article to help you better understand the topic.
Remember to protect your privacy. Don’t upload or share your personal information with an AI. This includes your full name, birth date, medical records, or insurance details. Most AI chatbots collect, store, and potentially use the information you type into them. How the information is used depends on the AI company’s rules and your account settings.
The tech behind AI chatbots is in a constant state of improvement. Right now, AI isn’t ready to replace your trusted doctor or nurse practitioner. Health professionals and researchers are advocating for the development of safer, more accurate AI that the general public can use. This development should follow the rigorous studies medications and treatments undergo before being presented to the public. Right now, that isn’t happening. So, if you choose to use AI for your health info, stay mindful of its limitations. Consider following up with your healthcare provider, who can talk through your questions.
