![[The AI That Rewrote a Single Line of Video Code -- and Won 0.18 Percent]](/images/the-ai-that-rewrote-a-single-line-of-video-code-and-won-018-percent.webp)
The Gap Between a Paper and a Product
In 2026, a team of researchers at City University of Hong Kong and Lingnan University published a case study asking a deceptively narrow question: can a large language model design a video coding tool? [1] The answer, according to their experiments, is a qualified yes. An AI-generated replacement for the Planar prediction mode in the Fraunhofer Versatile Video Encoder achieved 0.18 % bitrate savings with 0.4 % complexity overhead on the standard benchmark. [1] Those numbers are small. They are also real, measured, and reproducible — and they show an LLM producing a working, decoder-compliant coding tool that beats the hand-crafted original in a standardized encoder.
The distance between such a result and a shipping codec is measured in years, not months. But the paper’s contribution is not the magnitude of the gain. It is the demonstration that the gain exists at all, and that it emerged from a process no human engineer directed step by step.
What Planar Mode Does, and Why It Matters
To understand what the LLM accomplished, one has to understand what it replaced. Planar prediction is one of the oldest and most reliable tools in the intra-prediction toolbox. Introduced in 2010 (Lainema & Ugur, 2010), it approximates smoothly varying two-dimensional surfaces by interpolating from reconstructed boundary samples — the pixels immediately above and to the left of the block being encoded. The source cites the original Planar paper as Lainema and Ugur, 2010, in the 28th Picture Coding Symposium, pp. 198-201. Rather than signaling explicit control points, it extrapolates the lower-right texture region from the top-right and bottom-left corners. The result is a prediction that handles homogeneous regions efficiently and serves as a foundational member of the most-probable-mode list.
The Planar mode’s simplicity is precisely what made it a good test case. Its core logic is a linear interpolation from reconstructed boundary samples, making it simple and interpretable. An LLM given the task of improving it would not have to navigate a sprawling codebase or reason about complex syntax signaling. It would have to design a better prediction function, a mathematical mapping from boundary pixels to a two-dimensional block.
That constraint was deliberate. The researchers limited the LLM’s search target to the core predictor body while leaving the surrounding codebase — reference preparation, syntax, rate-distortion optimization — untouched. The AI was not asked to redesign the codec. It was asked to redesign one function inside it.
The Generation-and-Evaluation Loop
The workflow that produced the winning predictor is an evolutionary loop. The LLM generates candidate predictors in C++, each accompanied by a short design idea. Each candidate is syntactically validated, compiled into the codec, and evaluated through actual encoding runs. The measured coding performance is then fed back into the next prompting iteration, allowing the workflow to accumulate empirical history and iteratively search for better formulations.
This is not a one-shot query. It is a search process, and the LLM is the mutation operator. The researchers implemented two strategies: exploration instructions, which direct the model to synthesize predictors structurally distinct from existing candidates, and modification instructions, which guide localized structural or numeric variations on a single parent.
The prompt engineering enforces operational boundaries. Each prompt contains five components: a task description, historical feedback from parent candidates (including their design ideas, raw C++ code, and per-sequence BD-rate), strategy-specific instructions, an interface specification defining accessible input variables, and validity constraints requiring deterministic integer arithmetic and decoder reproducibility. Candidates that fail parsing, compilation, or decoding are rejected and not retained.
The loop is disciplined. It is not a creative free-for-all. The LLM operates inside a box defined by codec conventions, and the box is what makes the output usable.
The VVenC Result: Two Predictors, Two Philosophies
The first experiment replaced the default Planar mode in VVenC-1.14.0 under its faster preset, a lightweight configuration where most VVC intra tools are disabled. Training used the first frame of 22 natural sequences from the common test conditions, and after 20 iterations, the final candidates were tested on full-length sequences.
Two evolved predictors emerged, selected by different regularization strengths. Both achieved consistent bitrate savings. The first, selected with a lower regularization weight, delivered an average BD-rate reduction of 0.18 % with only a 0.4 % encoding-time increase. The second, selected with a higher weight, achieved 0.16 % savings.
What is striking is how differently the two predictors work. The first improves Planar through adaptive boundary filtering. It computes the gradient of the top and left reference lines, applies a three-tap smoothing filter to both, and applies the filter twice to whichever side is much less smooth. It then tunes the final averaging step with a gradient-dependent offset, slightly shifting the predicted surface according to the imbalance between top and left variation. The planar philosophy is preserved, but local reference fluctuations are suppressed before prediction.
The second predictor changes how the Planar surface is formed. It constructs a Planar-like surface using the same two interpolation components but normalizes their sum by the combined width and height rather than the original weights. It then blends in center-oriented references, with smaller blocks receiving a stronger center contribution, and softly pulls out-of-range predictions toward a boundary-derived midpoint.
Neither of these is a trivial tweak. Each represents a coherent design idea, expressed in code that compiles and runs inside a production encoder.
The ECM Extension: Gains in a Saturated Field
![[The AI That Rewrote a Single Line of Video Code -- and Won 0.18 Percent] (Image 1)](/images/the-ai-that-rewrote-a-single-line-of-video-code-and-won-018-percent_post1.webp)
The second experiment moved to the Enhanced Compression Model, the state-of-the-art reference platform with far more competitive coding tools. Here the baseline already includes directional Planar variants that retain only one interpolation direction, and the LLM’s predictors were tested against that ECM anchor. The LLM was tested in two integration strategies: directly replacing the two directional predictors, and inserting a new Planar-like mode as an additional option with its own entropy context.
The evaluation was constrained to eight 416 by 240 sequences, four from Class D and four from Class C downsampled to the same resolution. Training used one frame per sequence; testing reported performance over ten frames. All experiments ran 30 iterations.
Both strategies yielded gains. Direct replacement produced 0.0271 % BD-rate reduction with 0.42 % encoding-time increase. Insertion produced 0.0316 % reduction with 0.90 % increase. The paper notes that the reported BD-rate of the directional Planar modes for Class D is around negative 0.05 %, which makes these improvements meaningful in a setting where the easy gains have already been taken.
The evolved replacement predictor turned out to perform operations similar to the VVenC winner: each directional branch applies filtering to its one-sided boundary before prediction, with the smoothed boundary conditionally blended with the original sample according to local gradient. The inserted mode is more aggressive, blending global and smooth gradients using cosine similarity confidence when sign consistency fails. One is a refinement of boundary stability; the other is a more complex gradient-conditioned surface.
The Generalization Test
A predictor that works on natural video might fail on screen content, which has different statistical properties — sharp edges, text, flat regions. The VVenC experiments trained on natural sequences from Classes A through E. When tested on Class F, the screen-content class, the evolved predictors still delivered savings: 0.17 % for the first candidate and 0.16 % for the second. The paper attributes this to the diversity of the training set.
This matters because it suggests the LLM is not overfitting to a narrow distribution. It is finding structural improvements that transfer. The same cannot be said for resolution scaling: the paper notes that a predictor found at low resolution does not necessarily transfer well to higher resolutions, and a predictor effective under the simpler faster preset may fail under more saturated tool configurations. Generalization is partial, not universal.
Why This Is Harder Than It Looks
Designing a coding tool is not like designing a quantization parameter rule. A QP rule operates at the pre-encoding stage, adjusting decisions before the encoder runs. A coding tool is embedded in the core codec pipeline. Modifications propagate: a change to prediction affects transform, quantization, and rate-distortion optimization downstream. Any logic adjustment must account for syntax signaling overhead and strict encoder-decoder matching. The decoder must reproduce exactly what the encoder predicted, or the bitstream is invalid.
These constraints make the search space treacherous. The LLM cannot propose a predictor that uses encoder-only state, randomness, dynamic allocation, I/O, or external helper logic. It must write deterministic integer arithmetic with valid boundary indexing. The prompt enforces these rules, and candidates that violate them are discarded.
The fact that any useful predictor emerged from this process reflects the prompt design as much as the model. The researchers constrained the problem until it was solvable, then let the LLM search within the constraints.
The Cost of Discovery
The workflow works, but it is expensive. In the ECM experiment, nearly 2,000 new candidates were evaluated before a useful predictor was observed. [1] With parallel evaluation on 256 logic cores, one iteration still takes about one hour. The search is brute-force in character: generate many candidates, test them all, keep the best.
This cost structure limits what can be explored. The paper suggests that a low-complexity in-domain exploration strategy is critical. For prediction tools, expert workflows often use residual signals to pre-select candidates before full encoding. A similar procedure might benefit the LLM workflow, filtering candidates before they consume encoding time.
There are other directions for improvement. The feedback context to the LLM could be richer: residual maps, content features, mode competition statistics, failure cases. The current loop returns code, design ideas, and sequence-level BD-rate, but not the texture of why a candidate failed. A model that understood the failure modes might search more efficiently.
Coding tool design also couples tool logic, syntax, and RDO algorithms. The current experiments isolate the predictor body, leaving syntax and RDO untouched. A joint optimization workflow would extend the scope, but also the complexity.
What the LLM Actually Learned
The two VVenC predictors and the ECM replacement share a family resemblance: they all modify boundary handling. The first smooths references before interpolation. The second blends center references and clamps outliers. The ECM replacement filters one-sided boundaries conditionally. The inserted ECM mode is the outlier, using cosine similarity to blend gradients.
This convergence appears not to be accidental. The Planar mode’s weakness is its sensitivity to noisy boundary samples. A single outlier pixel in the reference line can distort the entire predicted block. Across experiments, the LLM repeatedly arrived at filtering or conditioning the boundary to improve prediction. That suggests a real insight about the problem domain, not a random mutation.
![[The AI That Rewrote a Single Line of Video Code -- and Won 0.18 Percent] (Image 2)](/images/the-ai-that-rewrote-a-single-line-of-video-code-and-won-018-percent_post2.webp)
The inserted mode goes further, treating the prediction as a gradient-aware surface rather than a simple interpolation. It is more complex, and it costs more encoding time, but it also delivers slightly larger savings. The trade-off between complexity and gain is visible in the numbers.
The Barrier Is Not Technical
The paper’s conclusion is careful. The observation is encouraging: LLMs can discover meaningful and competitive Planar predictors under nontrivial coding settings. But the current workflow faces a clear obstacle, and that obstacle is computational cost. Finding an effective predictor depends heavily on exploring a large function space, and the exploration is slow.
This is not a problem of model capability. The LLM generated code that compiled, ran, and improved compression. It did so within constraints that would challenge a human engineer. The bottleneck is the evaluation loop: every candidate must be encoded, measured, and fed back. That takes time, and time limits the search.
If the workflow becomes substantially more efficient and reliable, it may provide a new source of candidate tools for future standardization exploration. That is the paper’s forward-looking claim, and it is modest. It does not promise that LLMs will replace codec engineers. It suggests they might become a useful generator of candidates, a way to explore design space that human expertise has not covered.
The 0.18 % gain is small. But it was found by a machine, inside a standardized encoder, under constraints that ensure it works. That is the concrete achievement, and it is enough to justify the next attempt.
What Comes Next
The paper outlines several paths forward. Richer feedback could make the LLM more informed. Joint optimization of tool logic, syntax, and RDO could extend the scope. Low-complexity pre-selection could reduce the evaluation burden. None of these is trivial, but all are engineering problems, not fundamental barriers.
The deeper question is whether the approach scales beyond Planar mode. Planar is simple, interpretable, and isolated. Other coding tools are more entangled: motion compensation, transform selection, loop filtering. Each interacts with more of the pipeline. Each would require more careful constraint design.
The paper does not answer this question. It establishes a proof of concept on the easiest possible target and reports the result honestly. The cost is high but manageable. The method works, but only within narrow boundaries.
That is arguably how most useful technologies begin. Not with a breakthrough that changes everything overnight, but with a demonstration that something previously impossible is now possible, if only in a limited setting. The LLM did not design a better video codec. It designed a slightly better Planar mode. That is enough to suggest the next attempt might do more.
The paper frames its work as a first step toward examining where LLM-based coding tool design shows promise and what remains open. The promise is visible in the numbers. The open questions are visible in the cost, the limited generalization, and the narrow scope. Both are part of the same story: a machine learned to improve a piece of software that humans have been refining for years, and the improvement, however small, is real.
The barrier to the next step is not whether the AI can design a tool. The evidence suggests it can. The barrier is whether the evaluation loop can be made fast enough, and the feedback rich enough, to let the search go further. That is a problem of infrastructure, not intelligence. And infrastructure, unlike insight, can be built.