An analysis has revealed that the main cause of hallucinations in artificial intelligence (AI) chatbots like ChatGPT is the 'AI model training and evaluation method that encourages guessing until the correct answer is found.' On 5th, OpenAI disclosed this in a research report titled 'Why Language Models Hallucinate.' It is notable that OpenAI is the first AI corporation to release materials researching the reasons large language models (LLM) hallucinate.
The report stated, 'The main reason language models hallucinate is that the (model) training and evaluation processes encourage guessing rather than acknowledge uncertainty.' The OpenAI researchers compared the basic design and training methods of language models to multiple-choice exams. If a student does not know the answer to a question and guesses, they may get some answers right; however, leaving the answer blank will guarantee a score of zero. The issue is that AI model evaluations are conducted based on accuracy, which means that guessing will yield a higher score than responding 'I don't know' to a question.
Researchers also elaborated that the AI model pretraining process also encourages hallucinations. During the training process, the AI model learns simply by 'guessing the next word' without being shown whether specific information or statements are true or false. For instance, if given the sentence 'Today, the weather is __,' the model predicts words like 'nice' or 'cloudy' to fill in the blank. The AI model focuses on generating sentences that 'look natural' rather than determining 'whether this statement is true or not.' The researchers noted that 'language models learn to mimic overall patterns by only viewing positive examples of fluent language when creating sentences.'
As a result, it easily learns rules such as grammar and spelling, but can only guess facts that do not frequently appear in the training data. In fact, the researchers asked the AI chatbot about their birthday three times, and the chatbot provided a different answer each time and failed to guess correctly. OpenAI noted, 'In cases of rare information, such as an individual's birthday, which does not appear in the training data and cannot be predicted based on patterns, the model ends up focusing solely on guessing the next word,' which leads to confident but incorrect answers manifesting as hallucinations.
As a solution to reduce hallucinations, the researchers suggested 'deducting points for incorrect answers and awarding partial credit if the model admits it doesn't know the answer' to prevent the AI from randomly guessing answers. OpenAI expressed that the background for presenting these research results is 'to clarify the nature of hallucinations and help eliminate misunderstandings surrounding them.'
Hallucinations are one of the biggest challenges that the AI industry seeks to resolve. This is because hallucinations can sharply decrease the trustworthiness of AI in specialized fields such as law, healthcare, finance, and life sciences, where even small errors or inaccuracies are not permissible. In the meantime, AI experts have identified hallucinations as one of the key obstacles to achieving general artificial intelligence (AGI). Demis Hassabis, CEO of Google DeepMind, has pointed out that 'currently, AI has too many gaps and provides incorrect answers even to basic questions.'
OpenAI, Anthropic, Google DeepMind, Amazon, Mistral, and Cohere are major generative AI corporations taking various measures to reduce hallucinations in AI models. They are focusing on improving the quality of data used for AI model training and establishing fact-checking and verification systems to lower the rate of false answers.
A representative example is retrieval-augmented generation (RAG), which reveals external sources for answers. Most AI corporations are responding to hallucinations by applying RAG technology to modern AI models to provide sources for answers. Recently, Mistral, a representative AI corporation from France that raised 1.7 billion euros (approximately 2.76 trillion won), announced that it signed a news exchange contract with the news agency AFP earlier this year to strengthen the fact-checking capabilities of its AI chatbot. It stated that it will integrate thousands of articles held by AFP into its chatbot to enhance answer accuracy.
Amazon Web Services (AWS) has introduced a safety mechanism called 'automated reasoning checks' that verifies the accuracy of answers generated by the AI model using mathematical logic and reasoning. AWS explained that this is particularly helpful in fields like security, finance, and life sciences, where the margin of error is low.
However, AI experts pointed out that even if the performance of AI models improves and verification systems get better, it will be impossible to eliminate hallucinations 100%. The OpenAI researchers stated, 'Regardless of the size of the AI model or its search and reasoning capabilities, there will always be questions in the real world that inherently do not have answers, so accuracy cannot converge to 100%.' Amr Awadallah, founder of a generative AI startup and former Google executive, said, 'Since the way AI models function is inherently probabilistic, it's difficult to completely eliminate hallucinations.'
Concerns have also been raised that efforts to increase the accuracy of answers may conflict with another pillar of AI models, which is performance based on 'creativity.' When using AI to write novels, poetry, or scripts, maximum creativity must be exerted, and there is a possibility that focusing too much on accuracy may weaken this ability.
Some industry insiders believe that as the performance of AI models improves and hallucinations are alleviated, hallucinations themselves will not be a major hurdle to achieving AGI. Dario Amodei, CEO of Anthropic, evaluated that despite the hallucination issue, most generative AI models produce less nonsense than humans. He stated at an event last May, 'It depends on how you measure it, but AI models should exhibit fewer hallucinations than humans. However, AI hallucinations appear in (unexpected) surprising ways.'