Home / Blog / The Design Space of Tri-Modal Masked Diffusion Models
Tech News

The Design Space of Tri-Modal Masked Diffusion Models

Apple Machine Learning Research introduced the first tri-modal Masked Diffusion Models from scratch, reporting strong results at 3B scale on 6.4T tokens.

By Dillip Chowdary • Oct 14, 2026 • Source: Apple Machine Learning Research

The Design Space of Tri-Modal Masked Diffusion Models
Illustration · Pexels

At a glance

Venue
NeurIPS paper, October 2026
Modalities
Text, image-text, audio-text
Model scale
3B model, preliminary
Training data
6.4T tokens
Tasks
Text generation, T2I, T2S

Apple Machine Learning Research introduced the first tri-modal Masked Diffusion Models, pretrained from scratch on text, image-text, and audio-text data, in a NeurIPS paper published in October 2026. Apple Machine Learning Research's report sets that pretraining against recent work that initialized and finetuned a base unimodal model for bi-modal generation.

This account covers multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects, optimized inference sampling defaults, and the stochastic differential equation reparameterization of batch size. The preliminary model, at 3B model scale on 6.4T tokens, is shown for text generation, T2I tasks, and T2S tasks. It is for researchers and engineers who train discrete diffusion models.

01What actually changed

They present that model as a unified design. The demonstration is a preliminary model at 3B model scale and 6.4T tokens, and the authors report strong results in text generation, T2I tasks, and T2S tasks. Apple Machine Learning Research places the paper in Methods and Algorithms.

They present that model as a unified design. The demonstration is a preliminary model at 3B model scale and 6.4T tokens, and the authors report strong results in text generation, T2I tasks, and T2S tasks. Apple Machine Learning Research places the paper in Methods and Algorithms.

The authors systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, and batch-size effects, and they provide optimized inference sampling defaults. The batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization, which they say eliminates the need for tuning the optimal batch size as reported in recent work. They describe the work as "the largest scale systematic open study of multimodal discrete diffusion models conducted to date" and say it provides valuable insights into scaling behaviors across multiple modalities.

02How it works

Task coverage in the paper is text generation, T2I tasks, and T2S tasks. Discrete diffusion is the frame they set beside autoregressive language models. The study includes modality mixing ratios, noise schedules, and optimized inference sampling defaults among the settings for that model.

Task coverage in the paper is text generation, T2I tasks, and T2S tasks. Discrete diffusion is the frame they set beside autoregressive language models. The study includes modality mixing ratios, noise schedules, and optimized inference sampling defaults among the settings for that model.

The batch-size analysis yields a novel stochastic differential equation (SDE) based reparameterization. The authors say it eliminates the need for tuning the optimal batch size reported in recent work.

The reparameterization decouples physical batch size from logical batch size. Physical batch size is often chosen from compute: GPU saturation, FLOP-efficiency, and wall-clock time.

Logical batch size is chosen to balance the variance of gradients during stochastic optimization. Recent work is the reference they give for the tuning step this split removes.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

03Why it matters now

This study pretrains the tri-modal MDM from scratch and publishes scaling behavior for that choice. The authors report strong results at 3B model scale on 6.4T tokens in text generation, T2I tasks, and T2S tasks. They say the study supplies valuable insights into scaling behaviors across multiple modalities.

This study pretrains the tri-modal MDM from scratch and publishes scaling behavior for that choice. The authors report strong results at 3B model scale on 6.4T tokens in text generation, T2I tasks, and T2S tasks. They say the study supplies valuable insights into scaling behaviors across multiple modalities.

Recent work, on their account, still required tuning an optimal batch size. The SDE reparameterization is how they remove that tuning, and they call the reparameterization novel.

Physical batch size follows GPU saturation, FLOP-efficiency, or wall-clock time. Logical batch size follows the variance of gradients during stochastic optimization.

The design space also records modality mixing ratios, noise schedules, and optimized inference sampling defaults. Those settings are reported with the preliminary 3B model trained on 6.4T tokens of text, image-text, and audio-text.

04Who is affected

Bruno Kacper Mlodozeniec is noted at the University of Cambridge, Theo X. Olausson at MIT, and Jason Ramapuram at Google DeepMind. Footnotes say that work was done while at Apple.

Bruno Kacper Mlodozeniec is noted at the University of Cambridge, Theo X. Olausson at MIT, and Jason Ramapuram at Google DeepMind. Footnotes say that work was done while at Apple.

Labs that initialize a unimodal model and finetune it for bi-modal generation are the prior approach named in the study. People who still tune an optimal batch size are the audience for the SDE split between a compute-based physical batch size and a logical batch size set by gradient variance.

Teams pretraining on text, image-text, and audio-text use the mixing ratios, noise schedules, and multimodal scaling laws. Reported tasks are text generation, T2I, and T2S at 3B model scale and 6.4T tokens.

05What to watch

The authors call the showcased model preliminary and report strong results at 3B model scale (6.4T tokens) in text generation, T2I tasks, and T2S tasks. It illustrates a unified design pretrained from scratch on text, image-text, and audio-text.

Published beside it are multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects, and optimized inference sampling defaults. The SDE reparameterization separates physical batch size from logical batch size and, the authors say, removes the optimal-batch tuning step reported in recent work.

Related papers appear with it: Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion (July 11, 2025; ICML; Methods and Algorithms; Speech and Natural Language Processing) and How to Scale Your EMA (December 4, 2023; NeurIPS; Methods and Algorithms; equal contributors). TCSM is a novel and versatile objective for training and fine-tuning discrete diffusion models, a general framework with broad applicability that supports pre-training directly from data samples.

Preserving training dynamics across batch sizes enables a trade-off between batch size and wall-clock time. Stochastic gradient descent scales the learning rate linearly with batch size, and a model EMA is a functional copy of a target model.

Frequently asked questions

What is a tri-modal Masked Diffusion Model?

It is a Masked Diffusion Model pretrained from scratch on text, image-text, and audio-text. Apple Machine Learning Research presents it as the first tri-modal MDM in a NeurIPS paper published in October 2026.

How large is the preliminary model?

The authors report a preliminary model at 3B model scale, trained on 6.4T tokens, with strong results in text generation, T2I tasks, and T2S tasks.

What does the SDE batch-size reparameterization change?

It decouples physical batch size from logical batch size. Physical batch size is chosen from GPU saturation, FLOP-efficiency, and wall-clock time. Logical batch size balances the variance of gradients during stochastic optimization. The authors say this removes the need to tune the optimal batch size reported in recent work.

Which authors have outside affiliations noted?

Bruno Kacper Mlodozeniec is noted at the University of Cambridge, Theo X. Olausson at MIT, and Jason Ramapuram at Google DeepMind. Footnotes say that work was done while at Apple.

What are Target Concrete Score Matching and How to Scale Your EMA?

They are related papers listed with this study. TCSM, dated July 11, 2025, is an ICML framework for training and fine-tuning discrete diffusion models and supports pre-training directly from data samples. How to Scale Your EMA, dated December 4, 2023, is a NeurIPS paper on preserving training dynamics across batch sizes, including linear learning-rate scaling in stochastic gradient descent.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →