Most AI “watches” a video with the sound off. Then it reads a transcript on the side. It misses tone, sound effects and timing.
Tencent says WeMM-Video fixes that. And you can download it. Here’s what’s real.
Quick answer
WeMM-Video is a family of four open models from Tencent’s WeChat Vision team. Sizes: 2B, 4B and 9B dense models, plus a 35B-A3B Mixture-of-Experts model. They take text, images, audio and video, and they can use a video’s original soundtrack together with its frames. They output text. The code is Apache 2.0. The paper went on arXiv on November 19, 2026, and coverage followed on November 20. The benchmark wins are Tencent’s own claims.
What is WeMM-Video?
An omni-modal model family. “Omni-modal” means it handles several input types at once: text, images, audio and video.
It’s Chinese-English bilingual. Tencent says it was trained on about 1.2 trillion multimodal tokens. It has two modes: Instruct for fast direct answers, and Thinking for step-by-step reasoning.
How is it different from older video AI?
Older setups often split the job. One model looks at frames. Another turns speech into text. Then you stitch the results.
WeMM-Video is built for joint audio-visual understanding. It takes the soundtrack and frames together, so it can in theory pick up things like tone, background sounds and what happens when. That’s the pitch. Real-world quality on your videos still needs testing.
What exactly did Tencent release?
- Weights on Hugging Face, in formats like BF16, FP8, AWQ 4-bit, GGUF and NVFP4 (availability varies by size).
- Code: examples, Docker images, serving recipes for patched vLLM and SGLang, and a native runtime for GGUF.
- Evaluation pipeline: the paper and press coverage say it’s released. The GitHub page I checked doesn’t describe it, so check the repo yourself.
License note. The source code is Apache 2.0. The model license is set per checkpoint on each Hugging Face model card. Read it before commercial use.
How good is it?
Tencent’s claims:
- The 35B-A3B sets a new open-source best on omni-modal video understanding.
- It beats Qwen3.5-Omni-Flash and Seed2.0-Mini-0428, and narrows the gap with Gemini-3.0-Flash. That last bit means it still trails Gemini.
- The 9B reportedly beats the larger Qwen3-Omni-30B-A3B-Instruct.
- Tested on 120+ benchmarks.
Big caveat. These come from Tencent’s paper and announcement. I found no independent tests yet, and the repo shows results as charts, not numbers I could check.
What do you need to run it?
- An NVIDIA GPU with a CUDA 12.9-compatible driver.
- Python 3.12 and FFmpeg.
- Patched vLLM or SGLang builds, not stock releases, for serving.
- Docker setups need the NVIDIA Container Toolkit.
No VRAM numbers are given per model size. The small 2B and 4B models are the ones to try first on modest hardware. Also, Docker serving limits each request to up to two videos and two audio items.
Security tip: the repo says to review the custom code before turning on trust_remote_code, and to pin a commit.
What could you build with it?
Ideas, not Tencent promises:
- Searching video libraries by what’s said and what’s shown.
- Summarizing lectures, interviews and YouTube videos.
- Flagging content for review, with a human in the loop.
Whether it works in real time depends on your hardware. Nothing I read promises live performance.
What we still don’t know
- Independent benchmark results.
- Exact model license terms per checkpoint.
- GPU memory needs by size.
- How it handles Urdu, Hindi and other languages. Tencent describes it as Chinese-English bilingual.
FAQ
What is WeMM-Video?
A family of four open omni-modal models from Tencent that understand video together with audio, images and text.
What sizes does WeMM-Video come in?
2B, 4B and 9B dense models, plus a 35B-A3B Mixture-of-Experts model.
Is WeMM-Video open source?
The code is Apache 2.0, and weights are on Hugging Face. Check each model card for the model license.
Can it hear audio in videos?
Yes. It can use the original soundtrack alongside the frames. Output is text only.
Does it support Urdu or Hindi?
Not stated. It’s described as Chinese-English bilingual. Test before relying on it.
Is it better than Gemini?
No. Tencent says the 35B-A3B narrows the gap with Gemini-3.0-Flash, which means it still trails.
Bottom line
WeMM-Video is a serious open release. Weights, code, small and large sizes.
But the “best open model” claim is Tencent’s own. Download the 2B or 4B, run your own videos, and judge for yourself.