Illustration by Dixie Leota
Gabby Nguyen | 3min read
Thanks to Samsung's AI, we now have magical moving portrait
In Harry Potter, Hogwarts’ portraits are able to talk and move around from picture to picture. Nowadays, thanks to the remarkable progress of artificial intelligence, living portraits or paintings are no longer a movie scene. The technology was developed by a group of researchers at Samsung AI Center, and Skolkovo Institute of Science and Technology in Moscow. Using the latest trend in artificial intelligence – adversarial learning – Samsung's new AI has demonstrated that AI code can breathe life into portrait paintings, photos of dead celebrities and transform single still images into moving and talking heads.
In this video, the Mona Lisa, who was famous for its ambiguous expression, is animated like one of the moving paintings in the Harry Potter series. She turns her head, mouths words, and even blinks. Surprisingly, AI only needs a single picture to bring Mona Lisa to life.
The question is, how can AI transform a still image or an old painting into a living picture? First, an embedded network maps information like the size of the eyes, nose, and mouth in an input image and converts it into vectors. A generator network will copy the facial expression of someone in a video by plotting the person’s facial landmarks. Afterwards, a discriminator network will paste the embedded vectors from the input image onto the landmarks in the target video so that the input image mimics the motion in the video.
At the end, a “realism score” is calculated. The score is used to inspect how closely the source image matches the poses in the target video. Before the system is good enough to work on examples with very few input samples like the Mona Lisa, it requires extensive pre-training, via VoxCeleb2 dataset.
The pre-training phase allows the model to work on inputs where there are very few examples. These networks start out really bad at their jobs, but as they perform their jobs millions of times, they begin to improve, and the competition between the two networks is what drives both to continue getting better.
Samsung’s new AI technology can be a stunning tool in filmmaking and painting, as well as other artistic fields. However, concerns have been raised about the potential uses of such technology in the wrong hands.
Last year US lawmakers warned that such faked videos could be damaging to national security. Researchers and even journalists have created convincing fake videos of figures including former US president Barack Obama and Russian President Vladimir Putin. In this case, a speech given by Mr Obama was "mapped" onto images of Mr Putin, making it look like they were his words. In the future, when this technology is popular and complete, and living portraits are more real than ever, can we afford to distinguish between real videos and fake ones?
Reference: Pottermore, New Atlas, The Register UK, Technology Intelligence, TechCrunch, Fast Company