Eventually, we may be able to get people from the past to tell us their own stories.

The Heart of Machine, by Nan Ze and Qian Zhang.

In recent weeks, an AI-inspired “Let’s Move” trend has swept the Internet, with photo-to-video animations, sometimes set to music, popping up on social networks’ timelines.

The Deep Nostalgia service from online genealogy service MyHeritage, which creates the best motion effects for still photos, was one of the hottest events of the weekend. The tool is a bit like the Live Photos feature on the iPhone, which automatically generates a few seconds of video to help smartphone photographers find the best angles.

But unlike the iPhone’s composit-and-selection approach, Deep Nostalgia can bring content taken with any camera to life. The tool creates short videos of people posing in the same manner as example people during AI training, and is designed to help people see photos of past loved ones in a new light.

Unlike other products that exist as apps, using MyHeritage images is all you need to do is sign up for a free account on their website and upload photos (the first few are free). The image processing process is completely automated. It’s not a problem if your old photos are a little low resolution — Deep Nostalgia automatically enhances the images with super resolution and other enhancements before processing them.

Tool link: www.myheritage.com/deep-nostal…

There are also privacy concerns. MyHeritage says it will not share any data uploaded by users with any third parties, and images uploaded before registration are deleted immediately after processing to protect your privacy.

The sudden appearance of such a simple and easy-to-use tool became a tool of choice for twitter and other communities, as people quickly pushed their imaginations to the limit. Since it is the product of artificial intelligence, we naturally must first use it to “resurrect” a grandfather — the pioneer of modern computer science Alan Turing. Try it out with this classic photo:

The AI perfectly reflected Turing’s intelligent eyes.

And Lu Xun, who said everything:

Lin Huiyin, The first female architect in China:

Is there anything more dramatic? If rounded, Roman statues are people. To confuse AI, some archaeologists have used photos of statues to create giFs:

He was Antinous of the Roman Empire in the first century A.D. Should we thank AI for its technical prowess or the reductiveness of classical sculpture?

Despite its amazing imagination, Deep Nostalgia has its limitations: it can only work with the face of a single image, so you can’t expect Deep learning to create a walking mummy for you. If you have already tried more than five images, you must sign up for an account to continue “creating”.

In any case, that doesn’t stop the imagination from running wild.

Of course, the netizens also expressed their gratitude for the original intention of Deep Nostalgia.

“My father died 29 years ago when I was just a few months old. It was the first time I saw him in action, the first time I saw him blink, smile…”

In the future, can we expect people in museums to tell us their own stories?

Abraham Lincoln in Motion video produced by MyHeritage (audio version). Perhaps in the near future, we can expect historical figures in museums to tell us their stories “in person”.

Technology that might be used

As MyHeritage doesn’t reveal exactly what techniques were used in the deep nostalgia project, do-it-yourself researchers can only make their own guesses.

Among them, Gilles Louppe, a professor at the University of Liges in Belgium, guessed that they had implemented a 2019 paper from the Samsung AI Center called “Fee-Shot Adversarial Learning of Realistic Neural Talking Head. Models “.

In the paper, researchers at Samsung and the Skolkovo Institute used only a few images or even a single painting to create a head animation of a person speaking.

Specifically, the researchers used techniques such as fee-shot learning to synthesize mainly head images and facial landmarks. Fee-shot learning means the model simulates faces using only a Few or even one image. Meta trainning was performed using the VoxCeleb2 video dataset. In the process of meta-learning, the system creates three neural networks: an embedder network to map frames to vectors, a generator network to map facial feature points in the composite video, and a discriminator network to evaluate the authenticity and posture of the generated image.

Combining the three networks, the system can perform a long meta-learning process on large video datasets. After meta-learning converges, close-up neural head models with few-shot or one-shot can be constructed. The model treats the unseen target task as an adversarial learning problem so that the learned high quality generators and discriminators can be utilized.

The authors of the paper, said: “the crucial point is that although the need to adjust the tens of millions of parameters, the system can vary from person to person to initial im out and discriminant parameter, so the training can be quickly finish with only a few images. This method can quickly learn new faces even portraits and personalized image feature model.”

Address: arxiv.org/pdf/1905.08…

Of course, other ideas have been proposed and a paper from NeurIPS 2019 (” First Order Motion Model for Image Animation “) is close.

The task of this paper is: given a picture and a driving video containing a series of actions, then generate a new video, the people in the new video are the people in the source picture, and the actions are the actions in the driving video.

The whole model can be divided into two modules: motion estimation module and image generation module. In the motion estimation module, the model separated the appearance and motion information of the target object through self-supervised learning, and carried out feature representation. In the image generation module, the model will model the occlusion during the target movement, and then extract the appearance information from the given celebrity picture and combine the previously obtained feature representation to synthesize the video.

  • Paper links: aliaksandrsiarohin. Making. IO/first – order…

  • Project link: github.com/AliaksandrS…

Of course, the above are just two different guesses, which model is more stable and closer to MyHeritage’s implementation effect needs to be tried by yourself.

Content of the reference: www.theverge.com/2021/2/28/2…

Mp.weixin.qq.com/s/b8MjrsSt4…