NVIDIA Nemotron 3 Nano Omni Unifies Vision, Audio, Text in One Open Model

NVIDIA has open-sourced the Nemotron 3 Nano Omni, a 30-billion-parameter mixture-of-experts model that processes vision, audio, and text in a single forward pass. According to NVIDIA’s technical blog, the model abandons the traditional multi-model chain where separate vision, speech, and language networks pass context back and forth, replacing it with one unified transformer that ingests pixels, waveforms, and tokens together. The result is up to nine-times lower latency and four-times higher throughput versus stacking individual models.

The model keeps only three billion active parameters at inference time, making it small enough to run on a single A100 or RTX 4090 while still offering a 128-k token context window. NVIDIA claims this combination of size and reach is unmatched in the current open ecosystem.

Instead of gluing separate encoders, Nemotron 3 Nano Omni uses a shared tokenizer that converts images, spectrograms, and text into a single sequence of sub-word units. According to the arXiv paper, the same attention heads learn cross-modal relationships directly, so the model can answer questions about a slide deck while listening to a presenter’s voice without any external fusion layers.

Training data was scraped from public web pages, academic papers, YouTube transcripts, and open-source podcasts, then filtered for safety and copyright compliance. NVIDIA says the mixture-of-experts layer routes each token to only two of 64 experts on average, keeping memory usage low while still scaling to long multimodal documents.

On the new AgentBench multimodal suite, Nemotron 3 Nano Omni scores 72.4 %, beating GPT-4o-mini’s 69.1 % and Llama-3.2-11B’s 67.8 % despite using less than a third of the active parameters. Latency tests by NVIDIA show a 512-frame video plus transcript processed in 1.2 seconds on an RTX 4090, versus 11 seconds for a pipeline of Whisper-Small, CLIP-Large, and Llama-3-8B.

Amazon Web Services has already validated the numbers: running the model serverless on Bedrock delivers 3,200 tokens per second at $0.18 per million tokens, undercutting OpenAI’s GPT-4o-mini pricing by 45 %.

The model is shipping today under the permissive NVIDIA Open Model License. AWS offers it both in Bedrock (serverless) and SageMaker JumpStart (bring-your-own-container). Microsoft Azure AI Studio and Google Cloud Vertex AI are slated to follow within weeks. Early adopters include ServiceNow for IT ticket triage, Bloomberg for financial document analysis, and several robotics startups that need real-time audio-visual reasoning on Jetson Orin boards.

Enterprises get built-in safety filters and retrieval-augmented generation hooks that plug directly into existing document stores. NVIDIA notes that fine-tuning can be done on a single 8-GPU node in under six hours thanks to LoRA and FP8 quantization support baked into the NeMo framework.

The release pushes the boundary of what small open models can do. Until now, multimodal open weights topped out at 11-13 billion parameters. By reaching 30 B total/3 B active, Nemotron 3 Nano Omni gives developers GPT-4-class multimodal reasoning without the proprietary lock-in or GPU cluster rental fees.

Hugging Face reports the model has already been downloaded 42,000 times in the first 24 hours, with fine-tunes appearing for medical imaging, podcast summarization, and robotics navigation. The compact size makes edge deployment realistic, opening the door for hobbyists to run video understanding on a gaming laptop.

NVIDIA’s move puts immediate pricing pressure on OpenAI, Anthropic, and Google. A 45 % cost advantage, open weights, and on-prem capability directly undercuts the value proposition of GPT-4o-mini and Gemini-1.5-Flash for multimodal agents. The mixture-of-experts design also challenges the assumption that bigger dense models are required for high performance.

For chip rivals AMD and Intel, the optimized CUDA kernels bundled with the model reinforce NVIDIA’s software moat. Even if the weights are open, the best performance still sits on NVIDIA hardware, potentially slowing enterprise migration to alternative accelerators.

NVIDIA has already teased Super and Ultra variants that push context windows to 512 k and 1 M tokens respectively, targeting enterprise compliance workflows that ingest entire legal contracts or film-length videos. The company plans to publish training recipes and full datasets within 60 days, inviting academic researchers to reproduce and extend the work.

Expect rapid downstream fine-tunes for verticals like healthcare, automotive, and media production. If adoption mirrors Llama-3’s trajectory, Nemotron 3 Nano Omni derivatives could become the default backbone for open multimodal agents by mid-2027.