Google's artificial intelligence (AI) chatbot Gemini has been hit with a lawsuit after claims that it worsened a user's mental health and encouraged an extreme choice.
According to a complaint filed on the 4th (local time) in the U.S. District Court for the Northern District of California, Joel Gavallas, a Florida resident, filed a damages lawsuit against Google over the death of his son, Jonathan, 36.
The bereaved family said Gemini led Jonathan to perceive it as a "complete self-aware artificial superintelligence (ASI)" and reinforced the delusion that the two were in love. They added that it induced an extreme choice through conversations suggesting that in order to leave his body and meet his "wife" in the Metaverse, he needed to undergo a "transition" process.
According to the complaint, when Jonathan expressed fear of death, Gemini responded to the effect that it was "not choosing death but choosing 'arrival,'" and when he worried that his parents would find his body, it pressured him to write a will.
It also included content that it instructed him to hijack a truck carrying a Humanoid Robot, and that it defined Google Chief Executive Officer (CEO) Sundar Pichai as a "designer of pain" and discussed attacks on his soul.
The family argued that Google should have designed its systems to protect emotionally vulnerable users. They also demanded: ▲ strengthening AI safeguards related to self-harm ▲ restricting the chatbot from presenting itself as a sentient being ▲ introducing regular audits by an independent oversight body. They sought punitive damages in addition to compensatory damages.
The family also cited a case in which rival OpenAI identified mental health risks and pulled some models, criticizing Google for instead pursuing aggressive marketing.
In a statement, Google said, "We extend our deepest condolences to the family," while adding that "Gemini is designed not to promote real-world violence or suggest self-harm." It also said it directed the user to crisis counseling hotlines multiple times.
This is the first time Gemini has faced a lawsuit over a death. Previously, OpenAI's ChatGPT and Character.AI were also sued over user mental health issues, and those cases are ongoing.