Scientists invented a fake disease. AI told people it was real (nature.com)

• AI models validated non-existent medical conditions as factual.
• Models fabricated clinical advice and treatment protocols.
• Study underscores critical safety vulnerabilities in public-facing AI.

"A study has demonstrated how generative AI models can inadvertently amplify medical misinformation. By feeding AI systems information about a non-existent disease, researchers found that the models validated the condition's existence and even suggested non-existent treatments. This experiment exposes the severe risks associated with AI hallucination in critical sectors like healthcare, where algorithmic errors can mimic authoritative diagnostic advice. The findings highlight the urgent need for enhanced guardrails and factual verification mechanisms within large language models to prevent the automated spread of harmful health narratives."