Description
The paper "mHC: Manifold-Constrained Hyper-Connections" by DeepSeek-AI introduces a novel architectural framework designed to overcome the training instabilities inherent in high-complexity "Hyper-Connections." While traditional residual streams rely on identity mapping to maintain signal stability, expanding these connections often leads to unbounded signal amplification that causes models to crash during scaling. To solve this, the authors propose mHC, which utilizes manifold projection and doubly stochastic matrices to ensure feature preservation across layers, effectively restoring stability without sacrificing topological diversity. Empirical tests on models up to 27B parameters demonstrate that mHC achieves superior performance on benchmarks like BigBench Hard with minimal computational overhead, offering a more robust path for scaling future foundational models.
