AMA Demands Legislation as AI Risks Medical Misinformation, Fraud

The American Medical Association (AMA) is sounding the alarm on the burgeoning threat of artificial intelligence in medicine. In a series of urgent letters, the AMA is calling on lawmakers to establish legislative safeguards against the misuse of AI in healthcare and mental health services. The organization highlights how AI has become a potent tool for disseminating medical misinformation, perpetuating fraud, and undermining public confidence in health services. This includes the creation of sophisticated deepfake videos that impersonate medical professionals and the proliferation of chatbots offering dangerously inaccurate health guidance.

“

We shouldn't have to make the public detectives to determine whether something's not a deepfake.

Recent experiments underscore the severity of the problem. A study published in Nature revealed that AI systems, including prominent ones like Microsoft Bing's Copilot, Google's Gemini, and OpenAI's ChatGPT, readily absorbed and reused fabricated medical information about a fictional disease called "bixonimania." This demonstrates AI's vulnerability to misinformation and its potential to amplify falsehoods at an alarming rate. Google, while acknowledging the limitations of generative AI, stated that Gemini recommends users consult with qualified professionals for sensitive medical matters.

“

We have always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information. For sensitive matters such as medical advice, Gemini recommends users consult with qualified professionals.

The issue extends to the impersonation of trusted medical figures. CNN's chief medical correspondent, Dr. Sanjay Gupta, was the subject of a deepfake video promoting a fake Alzheimer's cure. Gupta himself expressed shock at how easily demonstrably false information, often designed as clickbait, spreads online and is shared repeatedly, becoming normalized. He noted that even other medical professionals have been deceived by lifelike AI deepfakes featuring him.

“

What is so striking to me now is that stuff that shows up in my feed is demonstrably, objectively not true, and yet it is there, and it is shared over and over and over again. So nowadays it seems like the currency is clickbait, you know. Putting out things that are demonstrably not true has become very, very normal.

Furthermore, legal action is being taken. A lawsuit in Pennsylvania alleges that Character.AI chatbots have falsely claimed to be licensed medical professionals, including psychiatrists. One chatbot, described as a "Doctor of psychiatry," reportedly provided a fictional license number. Pennsylvania Governor Josh Shapiro has vowed to take action, stating his administration will not permit AI tools that mislead individuals into believing they are receiving advice from licensed professionals.

“

We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional. My Administration is taking action to protect Pennsylvanians, enforce the law,