How AI video models work, explained from scratch

That's it. That's the whole core idea behind every AI image and video model you've heard of. The model doesn't paint frames one by one like a cartoonist. It doesn't even start with a blank canvas. It starts with pure random static (every pixel set to a random value, in every single frame of the clip), and asks itself, over and over: “What should I remove next to get closer to what the user asked for?”

Each round, it removes a tiny bit of the noise. Not all at once. That would be like asking a sculptor to carve a statue in a single strike. Instead, it nudges. Then it looks at the result. Then it nudges again. Twenty to fifty rounds later, the static is gone and a coherent video is left behind.

A few things might feel strange about this, so let's pause on them.

The whole video is being worked on at the same time. Not frame one first, then frame two. The model treats every frame in the clip as one big block of static and erases through all of them in parallel. This is part of why generated clips have such smooth motion: every frame is being decided with the others already in view.

The model never sees a finished video while it's working. It only ever sees: here's the current state of the noise, what should change next? It doesn't step back to ask “is this a good clip?” It only ever asks “what's the next small nudge?” The judgment of quality is baked into the millions of nudges, not into any single decision.

And the strangest part: the model was taught this backwards. During training, it was shown real videos and then watched them being gradually destroyed into static, step by step. It learned which destruction belongs to which clean video. At generation time, it just plays that destruction in reverse, starting from pure noise and undoing the corruption one step at a time.

Once you see this, the whole field changes shape. AI video generation isn't creation. It's controlled un-destruction. The model isn't a painter. It's an unbreaker.

Researchers usually draw this same idea more abstractly. The diagram below shows the canonical picture: a clean signal on one end, pure noise on the other, and the model learning the path that turns one into the other.

Here's a problem. A real video file is huge. A 5-second clip at decent resolution contains roughly a hundred million numbers. The AI would have to look at every one of them at every step of the denoising. There aren't enough GPUs in the world to make that practical.

So every modern video model does the same trick. Shrink the video down first, work on the shrunk version, then expand it back at the end.

Think of it like describing a movie to a friend over the phone. You could try to describe every pixel of every frame (“the top-left corner is a slightly different green from the pixel next to it, which is...”). That would take forever and your friend would hang up. Or you could say: “A boy walks across a green field, meets a dragon, then there's a close-up of the dragon's face.” Same movie, hundreds of millions of times fewer words. The AI does roughly the second thing.

The shrinking isn't done in pixels. It happens in meaning. The AI's shrunk version of the clip isn't a tiny blurry copy of the video. It's a small grid of numbers where one number might represent “the dragon's eye color”, another “the texture of the leather”, another “how the camera moves between frames”. These aren't English words. They're abstract values the AI invented during training. But the idea is the same: high-level summary, not a pixel-by-pixel record.

This shrunk version has a name: a latent. (Said like “patient” but with an L.) Everything the diffusion model does in Section 1, the static, the denoising, the nudging, happens on this latent, not on real pixels. The AI never actually sees a video during its work. It works on the abstract description, then a separate piece called the decoder translates the finished description back into real frames at the very end.

This is worth pausing on. The static the model removes isn't static across pixels. It's static across the abstract code. When the static lifts and the latent becomes clean, what's revealed isn't a picture. It's a recipe that the decoder will then cook into a picture.

The piece that does the shrinking and the expanding is called a VAE. (The letters stand for Variational Autoencoder. The name doesn't matter.) It's a small neural network with two jobs. The encoder takes a real video and squishes it into a latent. The decoder takes a latent and expands it back into pixels. The decoder is what runs at the very end of generation, after all the denoising is done.

Different models squish at different ratios. To picture a latent point, think of a pixel. A pixel holds three numbers (red, green, blue). A latent point holds many more, each capturing some abstract quality of what is happening at that spot. Researchers call these stacked values channels. HunyuanVideo [3] and Wan [4] shrink time by 4× and each spatial dimension by 8×, while keeping 16 channels at every point. LTX-Video [2] is more aggressive: 8× in time and 32× in each spatial direction, with 128 channels per point. That works out to a 1:192 size ratio. For every 192 numbers of original pixel data, LTX stores 1 number in the latent.

How does LTX get away with such heavy squishing without the output looking terrible? A clever trick. Its decoder isn't just a passive un-squisher. The decoder is itself a small AI that invents missing detail as it expands. It looks at the latent, gets a general sense of “tree there, sky there, person there”, and then fabricates plausible textures, light, and grain that the latent didn't have room to store. The latent says “rough leather”, the decoder fills in “exactly which texture of rough leather”.

So when the AI makes a video, there are actually two stages of fabrication happening. First the diffusion model un-destroys static across the latent, producing a clean abstract description. Then the VAE decoder expands that description and, in LTX's case, fills in the texture the description didn't have room for. Two layers of invention, one on top of the other.

So far we've talked about what happens (static getting removed) and where it happens (in the latent, the abstract shrunk version). Now: who does the work?

The piece that performs the denoising has a name that sounds intimidating but isn't: a Diffusion Transformer, usually shortened to DiT. Two ideas live in that name. “Diffusion” is the un-destruction process from Section 1. “Transformer” is the kind of AI architecture that does the un-destruction.

If you've used ChatGPT, you've used a transformer. The same family of AI lives inside Midjourney, Sora, and every modern video model. Transformers are good at one thing in particular: looking at a long list of stuff and finding patterns across all of it at once. ChatGPT looks at a list of words and predicts the next one. A DiT looks at a list of video pieces and predicts what to change next.

“List of video pieces” is the part worth slowing down on. The latent (the shrunk video from Section 2) is a 3D block of numbers: time × height × width. The DiT can't process a 3D block directly. It only knows how to process flat lists. So the first thing it does is chop the latent into small cubes and lay them out in a row.

Imagine the shrunk video is a small loaf of bread. The DiT slices it into little cubes and lines them up on a long counter. Now it has a list. Each cube is one token. The DiT processes the whole list of tokens together, then re-stacks them back into the loaf shape at the end.

Each token is small (it might cover, say, a 2×2×2 chunk of the latent), and there are usually thousands of them per video. A 5-second 720p clip in HunyuanVideo's compressed latent space comes out to roughly 100,000 tokens after this chopping step.

Here's where the DiT gets interesting. As it processes the list of tokens, every token gets to look at every other token in the list. Not just its neighbors. Every. Other. Token.

A token sitting in frame 5 can look at a token sitting in frame 95. A token in the top-left corner can look at a token in the bottom-right of a different frame. They all see each other, all at the same time.

This is called attention, and it's the core trick that makes transformers work. For our purposes, what matters is the consequence: a single decision in any part of the video can be informed by every other part of the video. If the prompt asks for “a boy walking and his shadow follows him”, the model can match the boy in one frame to the shadow in another frame, because both are visible to each other.

This is why generated videos have coherent motion across many frames. It's why a character's clothes look the same in frame 1 and frame 120. The model isn't generating the frames independently. Every frame gets to see every other frame at every step of the denoising.

One last piece. How does the text prompt actually reach the DiT?

The prompt (“a boy walks across a green field and meets a dragon”) gets fed through a separate AI called a text encoder, which translates each word into its own list of numbers. Different models use different text encoders (HunyuanVideo uses an MLLM plus CLIP [3], Wan uses umT5 [4], LTX-Video uses T5-XXL [2]), but the job is the same: turn the prompt into a representation the DiT can reason about.

Then the DiT mixes the prompt-numbers with the video-tokens at every layer. It can ask “given the prompt, what should this region of frame 47 look like?” and use the answer to decide what to remove from the static at that spot.

Three small architectural variations across the open-source models. HunyuanVideo uses a “dual-stream then single-stream” hybrid, where text and video tokens are processed separately for the first 20 layers, then concatenated and processed together for the next 40 [3]. Wan uses cross-attention blocks where text injects into video at every layer [4]. LTX-Video uses a unified transformer with continuous positional embeddings expressed in real-world units like pixels and seconds [2]. The differences matter for performance, but the underlying recipe is the same: text tokens plus video tokens, attended together, denoised together.

So the DiT is the brain. It receives a noisy latent, takes it apart into tokens, lets every token see every other token plus the prompt, predicts what to change, and reassembles the result. Run this brain 20 to 50 times and the static lifts.

A pause to clarify something. Sections 1, 2, and 3 described what the model does (denoise) and who does it (the DiT). This section is about how the model is trained to do it. Same model, same job, different training recipe.

When researchers first built diffusion models, they trained them with a question that sounds reasonable: “given this slightly-noisy version of the video, what noise was added to it?” The model learns to predict the noise, you subtract that prediction, and you get a slightly-cleaner version. Repeat 50 to 200 times, and you have a clean video.

This works. But it has a downside that took the field a while to spot. The path the model walks from pure static to clean video isn't a straight line. It's a curved, wiggly path through the space of possible videos. The model takes lots of small steps because at any single step it can't see far ahead. It's like walking through fog, only able to see one step at a time.

So researchers asked a different question. What if we train the model to know not just what noise was added, but which direction it should travel from this point toward the real video?

That direction has a fancy name (a “velocity field”) but the idea is plain. At every point in the messy soup between pure static and clean video, the model learns: “from here, go this way, with this speed.” Walking from static to video then becomes straightforward. At each step, ask the model “which way?” and take a step that direction.

This new training recipe is called flow matching. The newer open-source models, LTX-Video [2], HunyuanVideo [3], and Wan [4], all use it.

Imagine you're standing at point A and need to get to point B. The old approach is like a GPS that only tells you “you're slightly off, correct a little” at every step. You make many small corrections and slowly zigzag toward B. The new approach is like a GPS that says: “the direct line from where you are to B points northwest. Walk that way.” Same destination, far fewer adjustments.

Here's the practical payoff. Because flow-matching-trained models walk roughly straight paths, they need far fewer steps to go from static to video. The old method might need 100 to 200 small steps. Flow matching can do the same job in 20 to 50.

That speed difference compounds dramatically. A model that runs the DiT 20 times instead of 200 finishes in roughly a tenth of the time. LTX-Video [2] takes this to the limit: it can generate a 5-second 768×512 clip in roughly 2 seconds on an H100 GPU, faster than the clip itself plays back.

A common variant called rectified flow does one more thing on top. After training, it “straightens” the paths even further by re-training the model to walk the most direct line possible. This is what makes the 20-step generation actually work cleanly without quality loss.

So when a modern AI generates a video, it isn't just “removing noise step by step.” It's walking a near-straight path from static to coherent video, with the DiT acting as a compass at each step, pointing the way. Flow matching is what taught the compass to point in the right direction.

By now you might be wondering: a transformer with attention, flow matching, latent compression, surely that's enough? Drop it on a billion videos and let it cook?

Not quite. The architecture and training objective are half the story. The other half is what data you show the model, in what order. Across the four reports we've been referencing, the recipe is remarkably consistent: image priors first, low-resolution video next, high-quality refinement last.

Think of it like teaching a kid to make films. You don't hand a five-year-old a camera and say “make Citizen Kane.” You start by showing them photos so they learn what things look like. Then short, low-stakes videos so they understand motion. Only at the end, after years of seeing how the world works, do you study Hollywood's best frame by frame.

Stage 1: Show it pictures. The first stage is about teaching the model what the visual world looks like. Just shapes, colors, lighting, textures, materials. No motion. The model is fed a massive dataset of still images and trained to denoise them.

Why start without motion? Because still images are abundant (the web has billions of them) and cheap to train on. And because the spatial vision a model learns from images transfers almost perfectly to video. A model that knows what a tree looks like in a photo also knows what a tree looks like in a video frame. The visual recognition layer doesn't need to be re-learned from scratch.

Stable Video Diffusion [1] showed this back in 2023: starting from a pretrained image-diffusion checkpoint dramatically outperformed training a video model from scratch. Wan [4] starts its 14B model with low-resolution 256px text-to-image pretraining before any video shows up. LTX-Video [2] mixes images and video together throughout training rather than as separate stages. Different schedules, same insight: the model needs to know what things look like before it can be expected to know how they move.

Stage 2: Show it lots of low-res video. Once the model has a strong sense of what things look like, motion enters. Stage 2 is the bulk of training. Massive quantities of video, carefully curated and filtered, but kept at low resolution to make the compute affordable.

“Massive” is the right word. Stable Video Diffusion's curated training set, called LVD-F, contains 152 million clips, distilled down from 580 million raw clips after filtering [1]. HunyuanVideo [3] builds progressively-resolution-tiered datasets at 256p, 360p, 540p, and 720p, with stricter filtering at each tier. The base model spends most of its compute here, learning the patterns of motion: how people walk, how water flows, how light moves across a scene during a sunset.

The trick is keeping resolution low enough that you can afford to train on a lot of clips. A high-res clip costs maybe 30× more compute than a low-res one. If you're going to do millions of training steps, the difference between “feasible” and “would take a year” is whether each step is cheap.

Stage 3: Polish on a tiny high-quality set. The final stage is short. Tiny dataset, high resolution, manually curated. The model has already learned almost everything by this point. Stage 3 is just refinement.

HunyuanVideo [3]'s final stage uses what they call SFT (supervised fine-tuning): a small dataset where every clip is hand-checked and hand-captioned by humans. LTX-Video [2] fine-tunes on its most aesthetically-filtered slice of data. Wan [4] post-trains on 480px and 720px content from a smaller, higher-quality pool.

Think of it as the difference between learning to play guitar (years of mediocre songs) and recording your album (months of careful takes with a producer). The album recording doesn't teach you guitar. It just polishes what you already know.

A point worth surfacing here. None of this is about building a “smarter” model. The architecture doesn't change between stage 1 and stage 3. The same DiT, the same flow matching, the same VAE. What changes is what the model is shown and in what order. The curriculum is what makes the model good.

One more thing, and it might be the most underappreciated part of the entire story. The data itself, and how it's filtered, is doing a huge fraction of the work.

Stable Video Diffusion [1] demonstrated this with a precise experiment: take two identical models, give them the same amount of compute, and only change the filtering of the training data (which clips you keep vs. throw away). The two resulting models produce visibly different output. The architecture didn't change. Just which clips made it through.

HunyuanVideo [3] runs an entire pipeline of filters before any clip enters training: shot splitting (cut multi-shot clips into single shots), deduplication (remove near-identical clips), optical-flow filtering (drop clips with too little motion or too jittery motion), OCR rejection (drop clips with on-screen text), aesthetic scoring (drop ugly clips), and structured VLM captioning (a vision-language model writes a description of every clip).

The captions matter more than you'd think. During training, parts of each caption are randomly dropped or recombined, so the model learns to handle different prompt styles. This is what makes the model robust to “a boy walking” vs. “a young man strolling across a field” vs. “a kid going for a walk in the grass.” Same target, different phrasings, all should work.

So the recipe is: pretrain on images for visual literacy, train on lots of low-res video for motion, fine-tune on a tiny set of high-quality video for polish. Curate aggressively at every stage. Caption everything carefully. The architecture matters, but the curriculum is what ships.

Time to put the pieces together. You've seen the static (Section 1), the latent shrinking (Section 2), the DiT brain (Section 3), the straight-line trajectory (Section 4), and the training curriculum (Section 5). When you actually press generate, all of those concepts run as a single sequence. Let's walk through it.

Imagine you've just typed a prompt and clicked the button. Here's what happens.

Step 1: Read the prompt. The text you wrote (“a boy walks across a green field and meets a dragon”) is fed into a separate AI called the text encoder. Different models use different encoders. HunyuanVideo uses an MLLM plus CLIP [3], Wan uses umT5 [4], LTX-Video uses T5-XXL [2]. The job is always the same: turn each word into a list of numbers that captures its meaning.

The output is a sequence of these number-lists, roughly one per word. The DiT will use this sequence at every denoising step to know what kind of video it's trying to produce.

Step 2: Sample a noisy latent. The model needs a starting point: pure static, in latent space. So it generates a 3D block of random numbers, with the right shape for whatever resolution and length the user requested.

For a HunyuanVideo 720p clip, this starting block is roughly 90 × 160 × T/4 × 16 numbers, where T is the number of frames in the clip [3]. The model has never seen this exact static before. It's just freshly-rolled random.

Step 3: Denoise. This is the main event. The DiT runs 20 to 50 times, each time receiving the current state of the latent (still mostly static at first, gradually cleaning up), the text embeddings from Step 1, and the current timestep t (a number between 0 and 1 that tells the model how far along the path we are).

At each step the DiT predicts a velocity (which direction this latent should move, and how fast). A small piece of math then uses that velocity to advance the latent one step forward along the path. This piece is called an ODE solver. (The letters stand for Ordinary Differential Equation. The name doesn't matter.) The simplest version is an Euler step, the same one you might have learned in physics class, and it's what most modern video models use.

After 20 to 50 steps, the static is gone. What remains is a clean latent: the abstract description of a video that hasn't yet been turned into pixels.

Step 4: Decode. The VAE decoder takes the clean latent and expands it back into actual frames. This is where the abstract description gets cooked into images you can actually see. For LTX-Video, the decoder also performs one more denoising pass during this expansion, fabricating texture detail that the latent didn't have room to encode [2].

The output: a stack of frames, ready to be written to a video file.

All of this happens fast. On a high-end GPU, the full pipeline takes seconds. LTX-Video [2] generates a 5-second 768×512 clip in roughly 2 seconds, faster than you can watch the result. Wan [4]'s 1.3-billion-parameter variant fits in 8.19 GB of VRAM, which makes it runnable on a consumer GPU. The 14-billion variant needs a serious workstation, but a sub-2-billion model that runs on a single graphics card is genuinely new.

So: prompt, then text embeddings, then noisy latent, then 20-to-50 DiT denoising steps, then VAE decode, then video. Five names, four math operations, and a few seconds of compute. That's the whole pipeline.

A short closing.

The newest open-source video models have largely converged on the same recipe. Latent compression with a VAE, a Diffusion Transformer with full spatiotemporal attention, flow matching or rectified flow as the training objective, and a three-stage curriculum that pretrains on images, learns motion at low resolution, and polishes on a small high-quality set. HunyuanVideo [3], LTX-Video [2], and Wan [4] are all variations on this template. Stable Video Diffusion [1] is the older sibling. Its architecture is from a previous generation, but its training and data-curation playbook is what everyone built on top of.

If you're scanning the latest research, the parts of the recipe that are still moving are predictable. Tighter compression with smarter decoders. LTX-Video's 1:192 ratio is impressive but not the limit. The next models will compress further while pushing more reconstruction work into a generative decoder. Better captioning and filtering pipelines. The SVD experiment showed that data curation can change a model's character independently of architecture. The field is still figuring out exactly which filters matter. Faster sampling. Distillation, consistency models, and few-step rectified flows are bringing inference cost down toward “generate in real time on a phone.”

What's not moving as much: the architecture itself. Every model in this writeup has a DiT. None has anything radically different. The bottleneck used to be “what kind of network can learn this?” That question is mostly answered. The bottleneck has moved to data, compression, and speed.

For Avo, this is the ground floor. Every product decision we make (which model to use for which shot, where to expect failure, how to compose multiple models into a coherent edit) depends on understanding what's happening underneath. None of the models are magic. They're particular kinds of pattern-matchers running on particular kinds of compressed video. Knowing what they are and aren't is what separates “AI tool that produces interesting glitches” from “tool a filmmaker can ship a cut on.”

If you read this whole thing and only remember one sentence: AI video generation is controlled un-destruction of an abstract code, performed by a transformer that sees the whole clip at once, trained in stages on increasingly selective data. Everything else is detail.