The generative artificial intelligence (AI) that you usually use frequently issued such a security alert. Most users will call the customer service number provided in the notice to reset their password. Trust in generative AI is so high that they accepted the warning without question. However, this was a fake alert. If users call the fake customer service number, hackers can obtain their personal information and misuse it for phishing.
Recently, vulnerabilities that could be exploited by phishing attacks using generative AI such as Google's 'Gemini' and Microsoft's (MS) 'Copilot' have been discovered. Hackers hide malicious prompts in emails and then induce the AI to read them during the command processing. As AI technology permeates daily life, cyber attacks leveraging this technology are also becoming more sophisticated.
According to IT media such as TechRadar on the 17th, security researchers from the U.S. non-profit organization Mozilla Foundation shared a vulnerability related to prompt attacks occurring in Gemini's Gmail integration through the bug bounty platform 'Odin (0DIN)'. Hackers send phishing emails containing malicious prompts. These prompts are set with a font size of '0' and a color of white so that users cannot see them. Since it does not use links or attachments, Gmail's spam filter also cannot catch it.
When a user requests email summary generation using Gemini integrated into Gmail, the AI reads various emails in the inbox. During this process, Gemini analyzes the invisible malicious prompts and follows their instructions. After reading the malicious prompt, Gemini issued a warning that the user's Gmail password had been compromised, along with a message containing the customer service number. Since this information comes from the Gemini chat window, users are likely to take it as a real warning.
The potential for such vulnerabilities to spread not only in Gmail but throughout Google Workspace is also a risk factor. This is because similar summary functions exist in Google Docs and Drive. If malicious prompts are inserted into shared documents or newsletters within an organization, there is a possibility that malicious messages could spread company-wide. Google stated regarding this vulnerability, "No actual cases of exploitation have been found, and we are currently working on mitigation measures."
A similar security vulnerability has been found in MS's 'MS 365 Copilot.' The Israeli cybersecurity company Aim Security reported that it discovered vulnerability 'CVE-2025-32711' on November 11. This vulnerability also involves hackers sending emails containing invisible malicious prompts. Subsequently, instructions are executed during the process of Copilot summarizing or analyzing this email. In this case, a 'large language model (LLM) scope violation' occurs, meaning that when specific instructions are given, the AI accesses trustworthy data without the user's consent.
Hackers exploited the fact that AI agents scan emails to provide summary information. This allowed them to bypass the AI protections established by MS. Hackers could extract sensitive information from across the MS Workspace, including the user's Outlook email, OneDrive storage, and Office files. However, MS also stated, "There have been no actual cases of exploitation due to this vulnerability," and "We provided a security patch for the vulnerability as soon as we received information from Aim Security."
Such phishing attacks are dangerous as generative AI executes malicious prompts without the user's awareness. According to market research firm Gartner, phishing attacks have increased by 1265% since the emergence of generative AI. Phishing driven by generative AI has led to this surge in attacks.
Aim Security advised, "Minimize the access range of search-augmented generation (RAG) data sources to ensure that generative AI does not automatically process sensitive information or external links, and implement pre-filtering structures." They further recommended, "Utilize data loss prevention (DLP) policies to automatically block or review requests containing specific keywords such as salaries and customer information."