Home / Blog / DeepInfer: High-performance inference framework for…
Tech News

DeepInfer: High-performance inference framework for DeepSeek models

DeepSeek open-sourced DeepInfer, the production inference framework, with offline serving for DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950.

By Dillip Chowdary • Oct 20, 2026 • Source: github.com

DeepInfer: High-performance inference framework for DeepSeek models
Illustration · Pexels

At a glance

Product
DeepInfer inference framework
Supported model
DeepSeek-V4.1-Flash
License
Apache License 2.0
Hardware
NVIDIA SM100 GPUs, Huawei Ascend 950 NPUs
Requires
Linux, GCC >= 13.0, Python 3.12
Draft length
5 by default (DEEPINFER_USE_NEXTN_DECODE)

DeepSeek has released DeepInfer, the high-performance inference framework that powers DeepSeek's large-scale production services. github.com's report says this open-source release provides "offline inference with the core features needed for serving at scale." The only supported model listed is DeepSeek-V4.1-Flash. The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run.

This piece covers the feature list, sampler flags, accelerators, build prerequisites, benchmark hardware, and the parts a complete service still needs. It is for engineers running DeepSeek-V4.1-Flash offline on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, and for readers following the vLLM, SGLang, and TensorRT-LLM projects named in the acknowledgments.

01What actually changed

The release packages optimized kernels with the model and sampler runtime. Parallelism is listed as Attention DP / balanced CP and MoE EP + EPLB.

That same supported-feature block names DSpark and confidence-scheduled verification, SWA bounded replay, and asynchronous scheduling. The repository describes the scheduler as a "zero-overhead batch scheduler." Batch-invariant inference, deterministic sampling, and structured outputs are the other three entries.

The block is the offline-inference feature set, and the only model named with it is DeepSeek-V4.1-Flash. DeepSeek's production services run on this framework.

DeepInfer is released under the Apache License 2.0, and the repository places the text in the LICENSE file. Acknowledgments say the framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.

That note thanks their developers. The bibtex entry type is misc and the key is deepinfer2026.

Its author is DeepSeek-AI, its year is 2026, and its publisher is GitHub. The title reads "DeepInfer: High-performance, large-scale inference framework for DeepSeek models." The howpublished field points at the DeepInfer GitHub repository.

02How it works

The optimized kernels are FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. DSpark and confidence-scheduled verification accelerate decoding with draft tokens.

DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length, and the default length is 5. The repository examples contain the sampler configuration that goes with those two variables. Both variables belong to the DSpark and confidence-scheduled verification feature listed with the optimized kernels and the model and sampler runtime.

DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length, and the default length is 5. The repository examples contain the sampler configuration that goes with those two variables. Both variables belong to the DSpark and confidence-scheduled verification feature listed with the optimized kernels and the model and sampler runtime.

Decoder SWA bounded replay reduces decoder prefill work by replaying only a bounded suffix. Default V4.1 settings are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140.

The retained suffix must cover the SWA window plus the maximum backtracking length. Asynchronous scheduling overlaps CPU batch scheduling with device execution and is enabled by default.

It sets DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, and it passes a fixed seq_seed to SamplingParams. Structured outputs constrain generation to a regex, a JSON schema, or an EBNF grammar. Callers pass so_args with schema_type and schema to SamplingParams.

It sets DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, and it passes a fixed seq_seed to SamplingParams. Structured outputs constrain generation to a regex, a JSON schema, or an EBNF grammar. Callers pass so_args with schema_type and schema to SamplingParams.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

03Why it matters now

The second is VE/Prefill/Decode disaggregation, where "most required engine functionality is already implemented." The third is load balancing across data-parallel replicas for both prefill and decode. The getting-started path included here is offline inference, launched from examples/offline_inference.py. This release provides that offline path, and it leaves the KV cache, disaggregation, and data-parallel load balancing out of the release.

The second is VE/Prefill/Decode disaggregation, where "most required engine functionality is already implemented." The third is load balancing across data-parallel replicas for both prefill and decode. The getting-started path included here is offline inference, launched from examples/offline_inference.py. This release provides that offline path, and it leaves the KV cache, disaggregation, and data-parallel load balancing out of the release.

Performance benchmarks are grouped under Prefill and under Decode. Prefill was measured on 4 SM100 GPUs.

Decode was measured on 64 SM100 GPUs. Each of those measurements stayed within one scale-up domain.

The script that reproduces the benchmarks is run_sweep.sh. Evaluation benchmarks are written up in eval/README.md.

On NVIDIA, DEEPINFER_BATCH_INVARIANT=1 keeps model results identical across different batch compositions. Deterministic sampling, also NVIDIA only, produces identical outputs when the input tokens and the sampling parameters match and a fixed seq_seed is set.

04Who is affected

Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Every build requires Linux, GCC >= 13.0, and Python 3.12.

Before building, the instructions say to create and activate an empty Python 3.12 environment. They also say to keep DEEPINFER_ACCELERATOR exported in the environment where DeepInfer runs. People who need a KV cache system, VE/Prefill/Decode disaggregation, or data-parallel load balancing still have to supply those pieces, which this release does not include.

Before building, the instructions say to create and activate an empty Python 3.12 environment. They also say to keep DEEPINFER_ACCELERATOR exported in the environment where DeepInfer runs. People who need a KV cache system, VE/Prefill/Decode disaggregation, or data-parallel load balancing still have to supply those pieces, which this release does not include.

The NVIDIA build exports DEEPINFER_ACCELERATOR=nvidia. The Ascend build sets that same variable to ascend, and the instructions say to use ascend for the Ascend platform.

A local copy is chosen by setting DEEPINFER_HF_MODEL_DIR to /path/to/DeepSeek-V4.1-Flash-1017. The command is python examples/offline_inference.py. Build notes tell operators to keep DEEPINFER_ACCELERATOR exported wherever DeepInfer runs, and they list Linux for both the NVIDIA and Ascend paths.

A local copy is chosen by setting DEEPINFER_HF_MODEL_DIR to /path/to/DeepSeek-V4.1-Flash-1017. The command is python examples/offline_inference.py. Build notes tell operators to keep DEEPINFER_ACCELERATOR exported wherever DeepInfer runs, and they list Linux for both the NVIDIA and Ascend paths.

05What to watch

Batch-invariant inference and deterministic sampling are both marked NVIDIA only. Deterministic sampling means identical outputs for identical input tokens and sampling parameters.

Those two settings of DEEPINFER_ENABLE_MTP_VARLEN are opposites. Ascend 950 is specified with DEEPINFER_ACCELERATOR=ascend and CANN 9.2, apart from the NVIDIA-only pair. Batch-invariant mode keeps model results identical across different batch compositions, and the repository marks that mode NVIDIA only.

Those two settings of DEEPINFER_ENABLE_MTP_VARLEN are opposites. Ascend 950 is specified with DEEPINFER_ACCELERATOR=ascend and CANN 9.2, apart from the NVIDIA-only pair. Batch-invariant mode keeps model results identical across different batch compositions, and the repository marks that mode NVIDIA only.

Default V4.1 SWA replay keeps DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE at compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS at 140, and the retained suffix must still cover the SWA window plus the maximum backtracking length. Asynchronous scheduling stays enabled by default, overlapping CPU batch scheduling with device execution.

Structured outputs still require so_args schema_type and schema for a regex, a JSON schema, or an EBNF grammar. A complete service still needs the missing KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode.

Frequently asked questions

What is DeepInfer?

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale, under the Apache License 2.0.

Which DeepSeek model does DeepInfer support?

The supported model is DeepSeek-V4.1-Flash. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, downloaded into the Hugging Face cache on first run, or loaded through DEEPINFER_HF_MODEL_DIR from a local copy.

What hardware and software does DeepInfer require?

DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The build needs Linux, GCC >= 13.0, and Python 3.12, plus CUDA 13 on NVIDIA or CANN 9.2 on Ascend.

How does DeepInfer speed up decoding with draft tokens?

DSpark and confidence-scheduled verification accelerate decoding with draft tokens and use confidence estimates to schedule verification. DEEPINFER_ENABLE_MTP_VARLEN=1 enables that path, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, defaulting to 5.

What is not included in the DeepInfer open-source release?

A complete inference service still needs a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode. The project says most required engine functionality for disaggregation is already implemented.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →