The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation

Axogion Yin^1* Kai Shen^2* Yichong Leng² Xu Tan^2† Xinyu Zhou² Juncheng Li^1† Siliang Tang¹
¹Zhejiang University ²Moonshot AI
^*Equal contribution ^†Corresponding author

Abstract

Recent advancements in text-to-video (T2V) generation have been driven by two competing paradigms: autoregressive language models and diffusion models. However, each paradigm has intrinsic limitations: language models struggle with visual quality and error accumulation, while diffusion models lack semantic understanding and causal modeling. In this work, we propose LanDiff, a hybrid framework that synergizes the strengths of both paradigms through coarse-to-fine generation. Our architecture introduces three key innovations: (1) a semantic tokenizer that compresses 3D visual features into compact 1D discrete representations through efficient semantic compression, achieving a $\sim$ 14,000 $\times$ compression ratio; (2) a language model that generates semantic tokens with high-level semantic relationships; (3) a streaming diffusion model that refines coarse semantics into high-fidelity videos. Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the VBench T2V benchmark, surpassing the state-of-the-art open-source models Hunyuan Video (13B) and other commercial models such as Sora, Kling, and Hailuo. Furthermore, our model also achieves state-of-the-art performance in long video generation, surpassing other open-source models in this field.

Overview

The architecture of LanDiff. Given text inputs, we first extract text embeddings and employ an LLM to generate semantic tokens in the first stage. Subsequently, we utilize a diffusion model to synthesize perceptual features conditioned on these semantic tokens, followed by a VAE decoder that transforms these features into the final video frames.

Video Generation Models Performance Comparison

Radar chart visualization of performance comparison across different dimensions on VBench. The plot compares LanDiff against five competitive baselines: Sora, Hailuo, HunyuanVideo, Kling, and CogVideoX-5B. For better readability, the values in the radar chart have been normalized to a scale ranging from 0.3 to 0.8. The normalization was performed using the min-max scaling formula: $n o r m a l i z e d = 0.3 + 0.5 \times \frac{v a l u e - m i n_v a l u e}{m a x_v a l u e - m i n_v a l u e}$

Text-to-Video Model Comparison

CogVideoX-5B

LanDiff

"A colossal, human-shaped cloud towers over the earth, its massive form casting a shadow across the landscape. The cloud man's features are distinct, with a stern expression and outstretched arms. Suddenly, the cloud man releases a barrage of lightning bolts, illuminating the sky as they streak towards the earth. The scene is set against a backdrop of a stormy sky, with dark clouds and distant thunder adding to the dramatic atmosphere."

"A life-sized ice sculpture of a playful dog, with intricate details and a joyful expression, stands in the middle of a sunlit, grassy field on a sweltering summer day. The ice dog, initially solid and vibrant, begins to melt under the relentless heat, with droplets of water forming on its surface. As the day progresses, the ice dog's form gradually diminishes, with its once sharp features becoming blurred and distorted. The melting process accelerates, and the ice dog's body starts to collapse, pooling into a puddle of water on the ground. By the end of the day, all that remains is a shallow puddle, reflecting the cloudless sky, with the memory of the once majestic ice dog now just a memory."

"Two vibrant hot air balloons, one red and the other blue, are seen soaring through a clear blue sky, their baskets gently bumping against each other mid-air. The red balloon features intricate gold patterns on its surface, while the blue balloon boasts a white and silver design. As they collide, the passengers in the baskets, dressed in casual attire, react with surprise and excitement. The scene is set against a backdrop of a picturesque landscape, with lush green hills and a sparkling river below. The balloons' vibrant colors contrast beautifully with the azure sky, creating a visually stunning and dynamic scene."

CogVideoX-5B

LanDiff

"Wind turbines stand in a vast, open field, their blades spinning gracefully in the breeze. The elegant motion of the turbine blades is captured against a clear blue sky. The wind turbines appear serene in the landscape, highlighting the vastness of the open field as they become smaller in the distance."

"A breathtaking view of a majestic snowy mountain peak is reflected in a pristine alpine lake, creating a flawless mirror image. The sun's rays cast a subtle shimmering effect on the water's surface, enhancing the serene ambiance. The camera captures the scene from a distance, highlighting the symmetry and beauty of nature's perfect reflection. The mountain's snow-capped peak, the lake's clear waters, and the shimmering effect all contribute to a mesmerizing and tranquil atmosphere."

"A close-up view of a Christmas tree reveals a variety of decorations including a purple ornament with a gold pattern, a gold textured ornament, a small white house-shaped ornament with red roof and gold details, and a brown pine cone. The tree branches are dense and green, providing a natural backdrop for the ornaments. The camera pans slightly across the scene, maintaining focus on the ornaments while subtly shifting the perspective."

CogVideoX-5B

LanDiff

"A sleek, modern train glides effortlessly over a towering steel bridge. The polished exterior of the train reflects the golden hues of the setting sun. The bridge, an architectural marvel, spans a deep, verdant valley, with lush forests and a winding river far below. As the train moves, its rhythmic clatter harmonizes with the distant calls of birds and the gentle rustling of leaves. The majestic bridge stands silhouetted against a vibrant, twilight sky, as the train continues its journey into the horizon."

"A sleek white sailboat glides gracefully across a calm, azure sea, its sails billowing gently in the breeze. Above, a silver airplane soars through a clear blue sky. The boat's hull reflects the sunlight, creating a shimmering effect on the water's surface. The airplane, seen in a high-altitude flyover, casts a shadow that momentarily aligns with the boat's path, creating a fleeting connection between sea and sky. The scene is captured in a wide shot, ensuring both the boat and airplane are prominently centered, emphasizing their contrasting yet harmonious presence in the vast expanse."

"A group of silver-colored fish with darker fins swim among green aquatic plants in an aquarium setting. The fish move gracefully through the water, navigating around the plants, which are of various sizes and shades of green. The aquarium environment is designed to mimic a natural habitat, with rocks and shadows in the background contributing to the underwater scene."