FantasyTalking

Overview

FantasyTalking is a tool that turns a photo and voice clip into a talking person on video. You get real lips moving with the voice natural gestures and smooth movement in the background. It’s solid for creators avatars and digital people stuff.

The crew behind it is from Alibaba’s map team working with some folks from Beijing University of Posts and Telecom. Maps to video talking heads? Yeah kinda wild. But that’s what they did.

Here's how it works:

You upload a photo
You upload a voice clip
And if you want you can throw in a short prompt

It spits out a video where the person talks with synced lips eyebrow movement shoulder shifts and background motion. Not just the mouth everything’s moving like a real human. It can't lip-sync a video with an already speaking person, replacing the audio while keeping everything else in the video, except for the lip movements. For that, turn to LatentSync.

And here’s how it pulls that off:

• It lines up audio with body movement first then sharpens the lip sync frame by frame
• They don’t just copy full-face images like others do. They use a face attention trick that keeps the face looking like the same person all the way through

There’s a slider for “motion intensity.” You want your person calm or super expressive? You pick. It adjusts how big the gestures and facial expressions are.

In tests against other tools like Hallo3 and Sonic FantasyTalking came out on top. Lip sync felt real body movement wasn’t stiff and the face stayed the same. Even in scenes with lots going on in the background it held up.