Description
The paper introduces Attention Residuals, a new approach to replacing the traditional residual connections used in Transformer models, which typically add layer outputs together in a fixed and uniform way. Instead of this rigid accumulation, the authors propose using an attention mechanism across layers, allowing each layer to selectively retrieve and combine information from earlier layers based on the input. This helps address issues like information dilution and inefficient scaling in deep networks, enabling models to better preserve useful representations as they grow deeper. To make the method practical, the paper also introduces a more efficient block-based variant, and experiments show that this approach can improve performance and training behavior with minimal overhead, making it a promising drop-in improvement for modern large language models.
