What to know
- Moonshot AI’s Kimi team introduced Attention Residuals, a new architecture concept for Transformer models.
- It replaces fixed residual connections with attention-based mixing across layers.
- This allows models to selectively use earlier representations instead of blindly combining them.
- The result can improve scaling, training stability, and information flow in large AI systems.
Moonshot AI, the company behind the Kimi chatbot and large language model family, has introduced a new architectural concept called Attention Residuals (AttnRes) aimed at improving how Transformer-based AI models process information across layers.
In most modern Transformer models, each layer uses residual connections to stabilize training. This means every layer adds its output to a shared hidden state, allowing deep networks to train without losing information. While this mechanism has been essential for scaling models, researchers increasingly see limitations in how residuals combine information from earlier layers.
Kimi’s new Attention Residuals approach attempts to rethink that mechanism.
Instead of simply adding outputs together with equal weight, the new method allows each layer to choose how much information it takes from earlier layers. This selection is done using attention across depth—similar to how Transformers already use attention across tokens in a sentence.
In practical terms, the model calculates attention weights over previous layers. Each layer then builds its input as a weighted combination of the token embedding and earlier layer outputs, rather than inheriting a single blended hidden state.
This change may sound subtle, but it addresses several structural problems in standard Transformers.
One major issue with traditional residual accumulation is that all previous layer outputs are merged with equal importance, causing hidden states to grow larger as models get deeper. Over time, this makes it harder for the model to preserve specific information from individual layers.
Another limitation is lack of selective access. Different parts of a model—such as attention layers or feed-forward networks—may benefit from different types of earlier representations. With fixed residuals, every layer receives the same blended signal, even if it is not optimal for that layer’s task.
Attention Residuals aim to solve these problems by letting layers actively retrieve useful information from earlier points in the network. This approach treats model depth somewhat like a sequence dimension: the network can attend to earlier “positions” in its layer stack.
For developers and AI users, the impact is indirect but potentially significant. Improvements in architectural efficiency can help AI systems scale to larger models, maintain stronger reasoning over long contexts, and reduce training inefficiencies. In other words, techniques like Attention Residuals are part of the ongoing effort to make large language models more stable, scalable, and adaptable as they grow in size and capability.
The research also reflects a broader trend in AI architecture design: applying attention mechanisms not only to tokens and sequences, but also to other dimensions of neural networks, such as layers and memory structures.
Moonshot AI has been actively experimenting with new architectures through its Kimi model family, which includes large-scale mixture-of-experts systems with up to a trillion parameters designed for advanced reasoning and agent tasks.
Attention Residuals represent another step in that experimentation—showing that even long-established components of Transformer models, like residual connections, may still evolve as AI research continues pushing toward more capable systems.
