SIM-CoT: Supervised Implicit Chain-of-Thought

Abstract

Implicit Chain-of-Thought (CoT) methods offer a token-efficient alternative to explicit CoT reasoning in Large Language Models (LLMs), but a persistent performance gap has limited their adoption. We identify a core latent instability issue when scaling the computational budget of implicit CoT: as the number of reasoning tokens increases, training often becomes unstable and collapses. Our analysis shows that this instability arises from latent representations becoming homogeneous and losing semantic diversity, caused by insufficient step-level supervision in current implicit CoT methods. To address this, we propose SIM-CoT, a plug-and-play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space. SIM-CoT employs an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information. The auxiliary decoder is removed at inference, preserving the efficiency of implicit CoT with no added overhead. It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis. SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. It further surpasses the explicit CoT baseline on GPT-2 by 2.1% with 2.3× greater token efficiency, while closing the performance gap on larger models like LLaMA-3.1 8B. Code: https://github.com/InternLM/SIM-CoT

Author abstract · arXiv abstract · v2

Publication

International Conference on Learning Representations (ICLR), 2026

Paper and resources

• Paper · arXiv v2 (PDF)
• arXiv:2509.20317
• Google Scholar
• Code · InternLM/SIM-CoT

Research topics

SIM-CoT · Implicit chain-of-thought · Latent reasoning · Step-level supervision · Training stability · Reasoning interpretability · Token efficiency · Auxiliary decoder

Research problem and approach

Longer implicit reasoning chains can collapse when latent states become homogeneous. SIM-CoT aligns each implicit token with an explicit reasoning step to preserve semantic diversity and enable step-level inspection. arXiv abstract · v2

Main contributions

• Adds a plug-in training module that provides supervision for individual latent reasoning states. arXiv abstract · v2
• Projects latent tokens into an explicit reasoning vocabulary without adding inference-time decoder overhead. arXiv abstract · v2

Method comparison

arXiv abstract · v2

Selected results

• Adding SIM-CoT to CODI raises GSM8k-Aug accuracy from 52.7% to 56.1% while the reported token count remains 13.2. The out-of-domain average rises from 55.8% to 56.8% with the same 13.4-token average. Table 2 · LLaMA-3.2-1B implicit reasoning · arXiv v2
• For the 1B model, a 1B decoder yields 56.1% GSM8k-Aug accuracy, while 3B and 8B decoders yield 50.4% and 50.0%. Larger auxiliary decoders do not improve this setting. Table 4a · decoder-size ablation · arXiv v2

Cite this paper

Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, Dahua Lin. SIM-CoT: Supervised Implicit Chain-of-Thought. International Conference on Learning Representations, 2026, 2026, pp. 56721–56742.

@inproceedings{arxiv250920317,
  title     = {{SIM-CoT: Supervised Implicit Chain-of-Thought}},
  author    = {Xilin Wei and Xiaoran Liu and Yuhang Zang and Xiaoyi Dong and Yuhang Cao and Jiaqi Wang and Xipeng Qiu and Dahua Lin},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  volume    = {2026},
  pages     = {56721--56742},
  url       = {https://proceedings.iclr.cc/paper_files/paper/2026/hash/5d087955ee13fe9a7402eedec879b9c3-Abstract-Conference.html}
}