Motion is the Message

The first thing to forget is the phrase "compressing a video file." It sounds like the codec receives a finished object and squeezes it. That is the wrong picture. A modern video encoder watches a moving world and writes a screenplay for reconstructing it: reuse this old patch, shift it two pixels right, brighten the edge, smooth the mosquito noise, and trust that the viewer will not miss the rest.

If this post works, you will not come away knowing every AV1 tool. That would be too much and not enough at the same time. You will come away with the spine that lets those tools make sense later: prediction, residue, quantization, entropy. Prediction says "here is what the next block probably looks like." Residue says "here is how the prediction was wrong." Quantization decides which mistakes are too small to pay for. Entropy coding turns the remaining symbols into bits.

AV1 is a good companion because it is modern, open, and genuinely intricate. It improves on VP9 with more ways to split blocks, predict pixels, describe motion, filter reconstructed images, and code symbols. But under the pile of tools, it still plays the same old game as H.264 and HEVC: send the decoder enough instructions to rebuild a plausible sequence of pictures without sending every pixel in every picture.

§1Why raw video is impossible

Start with an ordinary 1080p video. One frame is 1920 by 1080 pixels. If each pixel used 24 bits of RGB color, a single frame would be about 6.2 MB. At 60 frames per second, that becomes roughly 373 MB every second, or about 3 gigabits per second before audio, headers, retransmits, and the small matter of the internet having opinions.

1920 * 1080 * 24 bits * 60 fps
= 2,985,984,000 bits per second
= about 3 Gbps

Netflix, YouTube, FaceTime, and Discord do not get to spend 3 Gbps on every 1080p stream. They live in the single-digit megabits. The codec has to remove more than 99% of the raw description while leaving behind something your visual system accepts as the same scene.

There are only two honest sources of that savings. The first is redundancy: the sky is blue in many neighboring pixels, the wall remains a wall from one frame to the next, and a face moves smoothly rather than teleporting randomly. The second is irrelevance: human vision is less sensitive to some color detail, less offended by small high-frequency errors in textured areas, and often too busy following motion to audit every coefficient.

Compression is not magic. It is the art of not paying twice for what stayed predictable, and not paying at all for what the viewer will forgive.

§2The four-part trick

Most modern video codecs are called hybrid block codecs. "Block" because they divide pictures into rectangular regions. "Hybrid" because they combine several different compression ideas rather than trusting one silver bullet. AV1 still belongs to this family, even though its blocks can be split in many more shapes than the old 16 by 16 macroblock world.

The sneaky part is that the encoder must think like the decoder. After it quantizes the residue, it reconstructs the block locally using the same rounded data the decoder will receive. That reconstructed block, not the original, becomes the reference for later frames. Otherwise the encoder would make future predictions from a world the decoder has never seen, and errors would drift until the video dissolved.

So the encoder is doing two jobs at once. It searches for the cheapest story to tell, and it simulates the listener who will hear that story. This is why encoding is often much slower than decoding: the decoder follows orders; the encoder has to choose them.

§3Frames become guesses

Image compression can exploit space: neighboring pixels tend to resemble each other. Video compression gets a second dimension for free: neighboring frames tend to resemble each other too. If a camera pans right, much of frame N+1 is already visible in frame N. If a person walks across the room, their face is not new information every frame. It is old information in a new place.

This is motion-compensated prediction. The encoder divides the current frame into blocks, searches reference frames for similar patches, and sends a motion vector: "copy a block from over there." The decoder has the same reference frame, so it can perform the same copy. Then the encoder sends the residue for what the copy failed to capture.

That sounds object-aware, but the codec does not need to know that a face is a face or a car is a car. It searches for blocks that explain pixels cheaply. A single object may be described by many block vectors. A single block may mix foreground, background, shadow, and camera noise. The codec is not doing semantic understanding; it is doing ruthless accounting.

There are frames that cannot cheaply depend on older frames. A scene cut, a flash, a title card, or the first frame of a stream needs to be coded mostly from itself. These are intra frames: they predict from neighboring pixels within the same picture. Inter frames predict from other pictures. Real streams use both, because random access and error recovery require occasional self-contained anchors, while bandwidth demands that most frames borrow heavily from the past and sometimes from the future.

§4The residue is the bill

After prediction, the encoder subtracts the predicted block from the original block. What remains is the residue. If prediction was excellent, the residue is mostly zeros and tiny values. If prediction was bad, the residue is expensive: lots of nonzero detail has to be transformed, quantized, and coded.

The residue is usually transformed before it is written. Instead of storing raw per-pixel errors, codecs convert a block of errors into frequency-like coefficients: broad smooth changes in one place, fine detail in another. This matters because human vision and natural images are both biased. Smooth low-frequency structure is often important. Tiny high-frequency coefficients are often negotiable. Quantization then rounds coefficients, sometimes all the way to zero.

This is the most emotionally honest part of lossy compression: it destroys information on purpose. The trick is not to destroy randomly. The encoder asks a rate-distortion question over and over: "If I spend these bits here, how much visible harm do I avoid?" High quality settings make smaller sacrifices. Low bitrate settings become more aggressive, especially in motion, texture, shadows, and color detail.

Entropy coding is the final bookkeeping layer. It does not know what a face is, and it does not forgive visual errors. It only knows probability. If zeros are common, code zeros cheaply. If a motion vector near the previous vector is common, code the difference cheaply. AV1 uses context-adaptive entropy coding so the meaning of "likely" changes with neighborhood and syntax context. That is one reason modern bitstreams look like alien legal documents: every bit has been negotiated against local probability.

§5Where AV1 gets fancy

AV1's core achievement is not that it discovered an entirely new religion of compression. It gave the encoder a larger, sharper menu of decisions while keeping the decoder implementable in hardware. More block partitions. More intra prediction modes. Richer compound inter prediction. Better tools for warped motion and global motion. In-loop filters such as CDEF and loop restoration. Film grain synthesis. Chroma tools. A more modern entropy coder.

That list can be numbing until you attach it to the four-part loop. Each tool mostly answers one of three questions: Can I predict this block better? Can I make the residue less visible? Can I code the final symbols with fewer bits?

The phrase "better compression" often hides a trade. Meta's early production-style AV1 tests found large bitrate savings against x264 and VP9, but also much higher encoding time for the tested encoders and settings. The AV1 technical overview paper reports more than 30% bitrate reduction versus VP9 for the same decoded quality. Those are real gains, but they do not arrive for free. Someone pays: the encoder, the decoder, the battery, the hardware designer, or the latency budget.

This is why codec adoption is never only about math. A codec can be elegant and still arrive slowly because phones need hardware decoders, browsers need support, streaming services need encoding farms, real-time products need low-latency presets, and every patent story has to be boring enough for lawyers to sleep through. AV1 is interesting precisely because it lives at the crossing of algorithms, hardware, web politics, and human perception.

§6What to build first

The best toy project is not "write AV1." That path leads directly into a swamp of syntax elements and regret. The best toy project is a tiny block-motion codec for grayscale frames. It will be crude, but it will teach the load-bearing idea.

Take two small images. Split the second into 8 by 8 blocks. For each block, search a small window in the first image for the closest matching patch. Store the motion vector and the pixel-wise residue. Then reconstruct the second image from the first image plus those vectors and residues. Finally, quantize the residues and watch what breaks.

for block in current_frame.blocks(8, 8):
    best = None

    for candidate in reference_frame.nearby_patches(block.position, radius=12):
        error = sum_absolute_difference(block.pixels, candidate.pixels)
        if best is None or error < best.error:
            best = candidate

    motion_vector = best.position - block.position
    prediction = reference_frame.copy_patch(best.position)
    residue = block.pixels - prediction
    quantized_residue = round(residue / q) * q

    bitstream.write(motion_vector, quantized_residue)

That toy already contains the seed of the real thing. Make the block size variable and you have partition search. Try predicting from the left and top neighbors and you have intra prediction. Transform the residue before quantization and you have transform coding. Count symbol frequencies and you have the beginning of entropy coding. Add a quality metric and you have rate-distortion thinking. The mountain becomes climbable because you are walking up the same path the real codecs elaborate.

The little grid above is deliberately primitive. The red shape is the actual block position. The green shape is the prediction after shifting the old block. Gold cells are the bill: places where prediction and reality disagree. Slide until the bill shrinks. That is motion search in nursery form.

§7What to read next

If this becomes a series, I would split it like the Python concurrency posts: each chapter owns one mechanism deeply enough to leave a scar, then hands the next chapter a clean unresolved question.

For outside reading, use these in order rather than all at once. The technical overview of AV1 is the best map once the mental model is in place. Meta's production-style AV1 comparison is useful because it talks about real content, bitrate savings, and encoding complexity. The AV1 bitstream specification is not a tutorial, but it is the source of truth when you want to see what a decoder is actually required to do. The Xiph test media archive is a good source of sample clips if you build the toy project.

§8Take with you

• Raw video is absurdly large. The codec's job is not a small optimization; it is survival.
• Modern codecs are prediction machines. They send decisions the decoder can replay, not just compressed pixel dumps.
• The residue is the cost of being wrong. Better predictions leave cheaper leftovers.
• Quantization is where loss enters. It rounds away detail according to bitrate, quality, and perceptual priorities.
• AV1 is an elaboration of the same loop. Its many tools make more predictions possible, make artifacts less painful, or code symbols more efficiently.