The ERNIE team at Baidu has announced the release of ERNIE-5.5-Flash-Base, which is an unrefined pre-training checkpoint of the ERNIE 5.5 Flash model. This release was posted on the Hugging Face platform on November 5th.
The model, which utilizes a sparse Mixture-of-Experts (MoE) architecture, has a total of 280 billion parameters, with 14 billion parameters activated per token. It was trained from scratch on 33.5 trillion tokens, with a maximum training context size of 128 thousand tokens, although support for context in fine-tuning and inference reaches 1 million tokens.
Baidu intentionally released the model in an unfinished state. The weights are obtained from the completion phase of pre-training and have not yet undergone intermediate or subsequent training stages. The company states that this makes it a general starting point for research in domain adaptation, instruction tuning, and reinforcement learning. The team reported that a post-trained version of ERNIE 5.5 Flash will be released later.
The available weights are provided in FP8 and BF16 formats according to the Baidu ERNIE 5.5 Flash Base Model v1.0 License Agreement. The model can be used via SGLang and vLLM, and is accompanied by a fine-tuning example in the NVIDIA Megatron-Bridge repository.
This 43-layer model is designed with an emphasis on efficiency. Most layers use what is called Baidu Sparse Attention, which merges every 128 tokens into a single cache entry. Only one of the seven layers retains access to the full token-level context through multi-head latent attention with top-k sparse sampling. In total, 37 out of 43 layers operate with compressed sequences, allowing for a small KV cache. Furthermore, the low-rank gating attention design reduces the gate projection parameters by 75%, according to the developers.
In terms of MoE, Baidu replaced the standard single router with multiple independently initialized routers that jointly evaluate experts. The company claims that this improves the alignment between routing scores and actual expert utility without changing the number of active experts, computational power, or communication. Each MoE layer selects 10 out of 512 routed experts plus one shared expert, operating in a compressed latent space, which roughly halves the expert communication and computation. A component called ECHO reuses the last layer of the model to predict future tokens instead of adding a separate multi-token prediction module and serves as a draft model for speculative decoding.
The Muon optimizer with mixed-precision FP8 was used for pre-training across thousands of accelerators, along with the new FlashMask-4 attention kernel. Baidu asserts that this kernel operates up to 175% faster in the forward pass than FlashAttention-4 on SM100 class GPUs at large head sizes. Data mixing focused on repository-level code, including commits, merge requests, code reviews, CI-verified fixes, and long documents.
Baidu tested the model on 30 benchmarks, comparing it with DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base using a unified evaluation system. Reported results include 78.4 points on MMLU-Pro versus 69.2 for DeepSeek-V4-Flash-Base, and 82.3 points on HumanEval+ versus 69.5. However, the compared models outperformed in some tests, such as ChineseSimpleQA, BBH with chain-of-thought prompting, and InfiniteBench.
