At the "Transparency and Accountability Center (TAC)" inside TikTok's Singapore headquarters visited on the 11th. When a TikTok official held a cigarette in hand in front of a camera consolidated to a monitor, the "smoking index" shown on the screen rose into the 50% range. Then, when the person put the cigarette to the lips and pretended to smoke, the figure instantly shot up to 99% and a warning light came on.
Regulations to reduce the negative impact of social media (SNS) have been spreading around the world recently. Protecting teenagers, who are vulnerable to harmful content, has emerged as a key agenda item. After Australia became the first in the world in Dec. last year to ban SNS use for those under 16, Indonesia and Malaysia also restricted the creation of accounts for those under 16. France passed a bill last month restricting SNS use for those under 15, and the United Kingdom, Spain, Greece, Canada and the United Arab Emirates (UAE) are also strengthening age-verification requirements.
Although countries differ in the method and degree of regulation, a shared sense of concern underlies them. There is worry that exposure to harmful content on SNS without restraint, or the repeated recommendation of similar information that strengthens confirmation bias, can have a negative impact on users. Accordingly, for global platform corporations, filtering harmful content quickly and accurately while also creating a safe content environment free from confirmation bias has emerged as a major task.
TikTok's TAC is a space that shows efforts to create a safe content environment. Here, the "Trust and Safety (T&S) team" reviews how TikTok screens content for harmfulness before it is shown to users. Korean-language content is mainly reviewed by evaluators who are native Korean speakers. According to TikTok, about 185 million pieces of content were deleted in the first quarter of this year alone.
◇ "AI filters first, then humans decide"… 185 million deletions in the first quarter alone
TikTok's content review is carried out in three broad stages. When a user uploads a video, about 99.5% of content deemed not harmful is posted immediately, while the remaining roughly 0.5% is classified as potentially violating community guidelines and receives additional review. The first review is handled by AI-based Machine Learning, and most harmful content is filtered at this stage. According to TikTok, of the content deleted in the first quarter of this year, about 96%, or 176 million items, were automatically removed through Machine Learning.
At the TAC, this process could be observed directly. Machine Learning did more than simply identify specific objects in a video; it grasped context by synthesizing human actions and surroundings. When the person in front of the monitor put a cigarette to the lips and mimed smoking, the smoking-related metric surged into the 90% range, while there was little change when the same action was done with a lollipop. The analysis targets even include text and audio. When a phrase symbolizing the Islamic State (IS) appeared, the metric related to "extremist flag" rose sharply.
A TikTok official said, "Machine Learning synthesizes various signals—images, text, audio, and behavior—to judge the likelihood of a violation," and added, "Rather than identifying a person's face, it analyzes behavioral patterns such as body joints and posture, and no personally identifying information is collected in this process."
However, content that is difficult to judge by AI alone undergoes a second review by human moderators in T&S. Targets include so-called "gray zone" content, such as wearing revealing clothing for a protest performance rather than sexual activity, or hate speech with the intent to bully even without direct scenes of violence. Moderators comprehensively examine frame-by-frame screenshots, comments, hashtags and posting context to make a final determination on whether the content violates the guidelines.
At the TAC, visitors could also take on the role of a human moderator and experience a mock review process. On the screen appeared a video with the floor covered in red liquid. At first glance, it looked like a scene of someone bleeding from an accident. But upon closer inspection, the liquid that looked like blood was engine oil that had leaked during the accident. Selecting "no violation" produced a correct result. It was a case in which AI would find it hard to distinguish real blood from engine oil based on color and scenes alone.
◇ "Sixty percent preferences, 40 percent new content"… TikTok's disclosed recommendation algorithm
As important as content safety screening, another task TikTok treats as critical is the "filter bubble" that fuels user confirmation bias. The TAC includes a space not only for content review but also explaining how the recommendation algorithm works and the mechanisms that prevent users from getting trapped in particular interests.
TikTok's recommendation system operates by showing users videos in bundles of eight and learning from their responses. It analyzes viewing time and user reactions such as likes, shares and comments, and then repeats the process of composing the next eight recommended videos. The explanation is that after watching about 64 videos, the direction of recommendations tailored to the user's preferences is formed to some extent. For users who have just signed up and lack preference information, the first recommendation bundle is composed based on region, the smartphone's language setting, and content that is currently popular and safe. As user reactions accumulate, it refines interests and adjusts recommended content.
However, to keep interest-based recommendations from turning into a filter bubble that confines users to certain topics, different content is intentionally mixed in. It avoids showing the same type or the same creator's videos in excessive succession, and it continuously recommends content that users do not usually consume. While no exact ratio is fixed, TikTok said it typically shows about 60% content the user prefers and about 40% new content mixed in.
This is not only to broaden user interest but also to prevent excessive immersion in a particular topic or viewpoint. Even if a user shows interest in topics such as depression, mental illness or dieting, the platform avoids repeatedly exposing only similar content and recommends new content alongside it. For political content as well, TikTok said that even if a user mainly consumes content with one leaning, it intentionally mixes in content with opposing viewpoints in the recommendation feed.
At the same time, control over the recommendation feed is opened to users, and exposure opportunities are opened to creators. In "content preferences" settings, users can filter specific keywords or directly control the exposure level of topics of interest. Signals users send, such as "not interested" or reports, are also reflected in future recommendations. The number of a creator's followers is not an absolute criterion that determines recommendations.
TikTok is also strengthening its response to the recently emerging problem of "AI slop (low-quality AI-generated content)." It is tightening sanctions on accounts that mass-produce low-quality AI content without meaningful human involvement, and it labels AI-generated or edited content that appears realistic. It applies automatic labeling to content produced with TikTok's own editing tools, and identifies AI content created on external platforms through content authentication technology.
A TikTok official said, "We are also strengthening our response to accounts that mass-produce low-quality AI content without meaningful human involvement," adding, "However, AI slop often struggles to garner user reactions such as likes or comments and thus frequently falls out of the recommendation algorithm naturally."