Loopback Transformer paper unifies residual looped transformers
First seen Nov 16 Updated Nov 16, 11:44 UTC Heat 14 medium evidence
A paper proposes the Loopback Transformer, a looped transformer variant that reads each block's individual updates instead of its full output. Geometric decay sets the total update weight, and attention learns how to allocate it across updates. The authors report lower validation loss than standard Transformer baselines on ClimbMix.
- Who should care
- Researchers working on parameter-efficient transformer architectures.
Key facts
- Uses 40-75% of baseline Transformer parameters (excluding token embeddings, per the snippet)
Timeline
Coverage
-
Loopback Transformer: Towards a Unified View of Residual Looped Transformershuggingface.co · HF Papers · 1 comments · 3 stars
Evidence
Medium evidence: several community reports point the same way, but there is no official confirmation yet.
Details written Nov 16, 06:25 UTC from the sources listed above.