Last summer, a New York City law firm was fined $5,000 by a federal judge due to a lawyer utilizing the artificial intelligence tool ChatGPT to compose a brief for a personal injury case. The document contained numerous inaccuracies, including over six entirely fictional past cases fabricated to establish a precedent for the personal injury lawsuit. This issue of erroneous content extends beyond this incident, as highlighted by researchers at Stanford University and Yale University in a recent preprint study on three popular large language models (LLMs). The phenomenon of generative AI models producing responses that deviate from reality is termed “hallucination.”
While many perceive hallucination as a technical challenge that diligent developers will resolve, some machine-learning experts argue that it is inherent to LLMs fulfilling their intended function of responding to user prompts. The underlying concern, according to certain AI researchers, lies in the perception and utilization of these models. To address hallucinations, the researchers propose that generative AI tools should be coupled with fact-checking mechanisms to ensure supervised operation of chatbots.
The prevalence of AI hallucination often stems from marketing exaggeration and unrealistic expectations. Tech companies have marketed LLMs as versatile solutions capable of addressing diverse issues or even replacing human labor. However, when applied inappropriately, these tools falter. Instances include chatbots dispensing incorrect medical advice, media platforms publishing AI-generated content containing inaccurate financial information, and search engines with AI interfaces generating false references. As reliance on chatbots for factual information grows, their proclivity for fabricating responses becomes increasingly evident and disruptive.
Despite the current landscape, today’s LLMs were not designed for absolute accuracy but rather for generative purposes, emphasizes Subbarao Kambhampati, a computer science professor specializing in artificial intelligence at Arizona State University. He asserts that ensuring the factual accuracy of generated content is inherently challenging due to the nature of computer-generated creativity, which inherently involves a degree of hallucination.
In a recent preprint study, machine-learning researchers at the National University of Singapore presented a proof demonstrating the inevitability of hallucination in large language models. This proof, drawing on classic results in learning theory, asserts that LLMs are incapable of learning all computable functions, underscoring the inherent limitations of these models. However, Kambhampati suggests that while the proof is theoretically accurate, it does not offer comprehensive insights into the specific causes of hallucinations. Moreover, he notes that LLMs often generate erroneous content even in response to simple queries.
Dilek Hakkani-Tür, a computer science professor specializing in natural language and speech processing at the University of Illinois at Urbana-Champaign, attributes the propensity for AI chatbots to hallucinate to their fundamental design as advanced autocomplete tools. Trained to predict the next sequence in a given text, LLMs may produce accurate outputs based on available training data but are prone to errors when confronted with unfamiliar topics. The limitation of data storage capacity within LLMs poses challenges in incorporating more factually grounded training data to enhance accuracy.
Another factor contributing to hallucination is calibration, as highlighted by Santosh Vempala, a computer science professor at the Georgia Institute of Technology. Calibration involves adjusting LLMs to prioritize certain outputs, potentially compromising factual accuracy in favor of fluency and originality. While reducing calibration can enhance factuality, it may introduce other issues such as formulaic text generation. Striking a balance between accuracy and naturalness remains a challenge for AI chatbots.
Acknowledging the inherent limitations of LLMs in producing entirely accurate outputs prompts a reevaluation of their deployment and usage. Integrating verifiers or fact-checking systems into the architecture of generative AI tools could mitigate hallucinations. Efforts are underway to develop hallucination detection mechanisms, such as the project led by Vectara, aimed at identifying and correcting errors in AI-generated content before dissemination. Research into factually grounded systems, combining specialized language models with reliable information sources, offers promise in enhancing the accuracy and utility of AI-generated content for applications like health access and educational equity.
In a future landscape where specialized systems validate LLM outputs, tailored AI tools would replace generic models for specific contexts, ensuring utility and reliability. While generalist chatbots may continue to serve as creative partners or sources of inspiration and entertainment, their role in providing factual information would be limited without verification mechanisms. This paradigm shift underscores the potential of language models to enhance productivity and fairness when deployed in conjunction with reliable verification systems.
