A technology using artificial intelligence (AI) that automatically corrects colors and lighting in photos or videos based on text input like "freshly cooked chicken" and "cold blue color" has been developed. It allows for easy and quick creation of desired effects without using complex editing tools during film production and photo/video editing.
Ulsan National Institute of Science and Technology (UNIST) announced on the 23rd that a research team led by Professor Baek Seung-ryeol of the Graduate School of Artificial Intelligence developed an AI model called "Text2Relight" that changes lighting effects for portraits and videos based on creative text commands. This research was conducted with the support of Adobe, a graphic and video editing software company, and the Ministry of Science and ICT. Researcher Cha Jun-ok from UNIST participated as the first author.
The research conducted jointly with Adobe has been accepted by the Association for the Advancement of Artificial Intelligence (AAAI), one of the three major AI conferences. Research results will be presented at the regular conference scheduled to be held in Philadelphia, U.S., from the 25th of this month.
Existing text-based image editing AI models were not specialized in lighting effects, leading to distortions of the original images or limited lighting adjustments.
The newly developed AI model can express various lighting effects suited to emotional atmospheres, as well as colors and brightness, through creative natural language text. It adjusts the colors of both the subject and the background simultaneously without distorting the original image.
To develop the Text2Relight model, the joint research team created a large-scale synthetic dataset so that the AI could learn the correlation between creative text and lighting. By utilizing ChatGPT and text-based diffusion models to generate lighting data, they applied techniques such as One-Light-at-a-Time (OLAT) which uses one light source at a time and Lighting Transfer to facilitate learning under various lighting conditions.
Additional training with auxiliary data such as shadow removal and lighting position adjustments strengthened visual consistency and the realism of the lighting.
Professor Baek Seung-ryeol noted that "Text2Relight technology reduces the time spent in photo and video editing," adding that "it has great potential in the content sector by enhancing the immersion of virtual and augmented reality."