Paper finds zeroing under 0.05% of weights adds LLM capabilities
First seen Nov 10 Updated Nov 11, 23:33 UTC Heat 40 high evidence
A paper by Jack Lu and colleagues proposes Deletion Tuning, which learns a binary mask over the gate projection of transformer blocks. The authors say zeroing fewer than 0.05% of weights matches full fine-tuning across RL with verifiable rewards, SFT and on-policy distillation.
- Who should care
- Researchers and engineers doing LLM post-training and continual learning.
Why it matters: Deletion Tuning stores a mask hundreds to thousands of times smaller than a full fine-tuning checkpoint and, the authors say, forgets less, so it could cut storage for per-task adapters.
Key facts
- Fewer than 0.05% of weights zeroed
- FFT, LoRA and random sparse fine-tuning lose up to 5.4 points
- Submitted 7 Nov 2026
Context
The paper compares against LoRA and random sparse fine-tuning, and says Deletion Tuning's weight changes are closest to full fine-tuning.
Timeline
Coverage
-
The Surprising Effectiveness of Deleting Weights in LLM Post-Traininghuggingface.co · HF Papers · 2 comments · 41 stars
Discussion
-
The Surprising Effectiveness of Deleting Weights in LLM Post-Trainingarxiv.org · HN · 30 pts
Evidence
High evidence: the claim comes from an official source (the company or project itself), or at least two independent sites reported it.
Details written Nov 11, 17:32 UTC from the sources listed above.