ABOUT
HOW IT WORKS
FOLLOW US
FEEDBACK
Discover useful tools, tutorials, and helpful resources organized by category
The paper LeWorldModel introduces a new way to train AI “world models” directly from raw pixels in a simple and stable manner, addressing a key problem in prior approaches where models collapse or require complex tricks to work. It proposes a streamlined architecture using just two loss terms—one to predict future states in a latent space and another to keep those representations well-structured—eliminating the need for heavy tuning or pre-trained components. Despite being lightweight (about 15M parameters and trainable on a single GPU), the model achieves competitive performance across various control tasks while planning much faster than larger foundation-model-based systems, and it also learns meaningful physical structure, allowing it to detect unrealistic or physically implausible scenarios.
The paper introduces Attention Residuals, a new approach to replacing the traditional residual connections used in Transformer models, which typically add layer outputs together in a fixed and uniform way. Instead of this rigid accumulation, the authors propose using an attention mechanism across layers, allowing each layer to selectively retrieve and combine information from earlier layers based on the input. This helps address issues like information dilution and inefficient scaling in deep networks, enabling models to better preserve useful representations as they grow deeper. To make the method practical, the paper also introduces a more efficient block-based variant, and experiments show that this approach can improve performance and training behavior with minimal overhead, making it a promising drop-in improvement for modern large language models.
This paper explores how cooperation can emerge among self-interested agents in multi-agent reinforcement learning without hardcoded assumptions about how other agents learn; instead of using specialized meta-learning or explicit timescale separations, the authors show that training sequence-model agents against a diverse mix of co-players naturally leads them to infer and adapt to their partners’ strategies within an episode, producing in-context best-response behaviors that, under mutual pressure to shape each other’s learning dynamics, lead to cooperative outcomes, suggesting a scalable way to achieve cooperation using standard decentralized RL with sequence models.
The paper introduces Deep Delta Learning (DDL), a new neural network architecture that extends traditional deep residual connections by replacing the fixed identity shortcut with a learnable, data-dependent geometric transformation called the Delta Operator. By parameterizing this operator with a direction vector and a gating scalar, the model can smoothly interpolate between identity mapping, projection, and reflection behaviors, giving the network fine-grained control over how information is erased and written at each layer. This enables it to represent more complex, non-monotonic dynamics than standard residual networks while retaining stable training, effectively broadening the expressive power of deep models without losing the benefits of residual learning.
The paper "mHC: Manifold-Constrained Hyper-Connections" by DeepSeek-AI introduces a novel architectural framework designed to overcome the training instabilities inherent in high-complexity "Hyper-Connections." While traditional residual streams rely on identity mapping to maintain signal stability, expanding these connections often leads to unbounded signal amplification that causes models to crash during scaling. To solve this, the authors propose mHC, which utilizes manifold projection and doubly stochastic matrices to ensure feature preservation across layers, effectively restoring stability without sacrificing topological diversity. Empirical tests on models up to 27B parameters demonstrate that mHC achieves superior performance on benchmarks like BigBench Hard with minimal computational overhead, offering a more robust path for scaling future foundational models.
In this work, the authors propose a model called Rex‑Omni — a 3-billion-parameter multimodal large language model (MLLM) — which reframes a wide range of visual-perception tasks (object detection, spatial grounding, key-pointing, OCR) as a “next-point prediction” problem. Instead of the traditional regression-based coordinate prediction widely used in detectors, Rex-Omni quantises spatial coordinates into discrete tokens and asks the model to predict the next token (or point) in a sequence, allowing detection and localization to be treated in the same token-prediction paradigm as language modelling. The paper argues that this unification leads to both stronger language-vision alignment and improved localization performance. They support their claims with evaluations on standard benchmarks (such as COCO) and show that Rex-Omni outperforms prior methods when given the same or less supervision. Beyond just detection, the design opens up a path toward more general “detect-anything” models where a single architecture handles many tasks.
The paper Recurrent Neural Network Regularization by Zaremba, Sutskever, and Vinyals (2014) introduces a novel approach to applying dropout in Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Traditional dropout techniques, effective in feedforward networks, often fail in RNNs due to the recurrent connections amplifying noise, which can hinder learning. The authors propose a method to apply dropout selectively within LSTMs, demonstrating its efficacy in reducing overfitting across various tasks, including language modeling, speech recognition, image caption generation, and machine translation.
The paper introduces DeepSeek-OCR, a vision-language model designed to go beyond traditional OCR by extracting text from images and understanding visual context such as layout, semantics, and multimodal relationships. It presents a mixed-expert architecture where only relevant sub-modules activate, enabling efficient processing of high-resolution document images and complex visuals. The authors describe their training system, datasets spanning screenshots, PDFs, and scene text, and evaluate the model on benchmark tasks where it demonstrates strong performance in text extraction, document understanding and layout reasoning. The paper concludes that the unified vision + language approach offers improved accuracy and flexibility for modern OCR applications.
In this paper, the authors challenge the prevailing assumption that poisoning large language models (LLMs) becomes harder as model size and training data scale up. Through large-scale experiments on models ranging from 600 million to 13 billion parameters, they show that injecting only a fixed, near-constant number of malicious training samples (e.g. ~250) can successfully implant backdoors regardless of how much clean data the model sees. They further demonstrate that this phenomenon holds both in pretraining and fine-tuning settings, suggesting that backdoor attacks could be more broadly feasible than previously thought. The findings raise serious implications about how we assess and defend against data poisoning in future, larger models.
The Transformer uses self-attention and feed-forward layers in stacked encoder and decoder blocks to process sequences, unlike previous models relying on RNNs or convolutions, achieving parallel processing and reducing computational complexity.
The authors explore how a modern generative video model, Veo 3, demonstrates a surprising level of zero-shot capability across many visual tasks it never explicitly trained for. Tasks include segmentation, edge detection, image editing, understanding physical dynamics, affordance recognition, and even reasoning puzzles like maze solving and symmetry completion. They argue that these emergent abilities suggest video models are evolving toward generalist vision foundation models, much like how LLMs transformed language understanding. Their experiments support a hierarchy of visual competence — from perception to modeling to manipulation to reasoning — showing that Veo 3 can “see, model, act, and reason” with visual content in ways that hint at future unified vision systems.
DeepSeek-R1 demonstrates that large language models can be trained to develop advanced reasoning abilities using reinforcement learning (RL) rather than depending on human-annotated reasoning traces. By building on DeepSeek-V3 Base and using a method called Group Relative Policy Optimization (GRPO), the authors use only final prediction correctness as the reward signal—without instructing the specific reasoning path. The result is that the model learns emergent behaviors like self-reflection, verification steps, and dynamic adaptation of strategy. DeepSeek-R1 outperforms models trained with supervised learning using human demonstrations in tasks involving mathematics, coding, and STEM reasoning. The supplementary materials (ESM1) provide detailed experimental setups, dataset details (including how they handle decontamination of evaluation data), and additional evaluation metrics that support the main claims.
Mixtral 8x7B introduces a Sparse Mixture of Experts (SMoE) language model built on the Mistral 7B architecture, but enhanced with eight feedforward “experts” per layer. A lightweight router activates just two of these experts per token during inference, giving each token access to a massive 47 billion parameters while only actively using 13 billion—striking an effective balance between capacity and efficiency (47B vs 13B). Trained with a context size of 32K tokens, Mixtral outperforms or matches the performance of Llama 2 70B and GPT-3.5 across benchmarks—especially excelling in math, code generation, and multilingual tasks. Its instruction-tuned variant, Mixtral 8x7B – Instruct, surpasses leading models like GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B-chat on human evaluation tests. Both models are released under an Apache 2.0 license and accompanied by open-source tooling integrations.
The paper empirically uncovers robust power-law relationships between a language model’s performance (measured via cross-entropy loss) and three core factors: model size, dataset size, and compute budget. Remarkably, these trends hold across more than seven orders of magnitude, while architectural variations like layer depth or width exert minimal impact. The authors further derive simple formulas that predict overfitting behavior and optimal compute allocation—revealing that larger models are significantly more sample-efficient, and that achieving compute-efficient training often means using very large models trained on modest data volumes and stopping well before full convergence.
The paper introduces the Transformer, an innovative neural network architecture that relies entirely on attention mechanisms—discarding both recurrence and convolution—delivering faster training and superior performance. Tested on machine translation tasks, the model achieves a remarkable 28.4 BLEU score on English-to-German and sets a new state-of-the-art 41.8 BLEU on English-to-French translation, all while training with significantly less compute time. The authors also demonstrate the model’s versatility by applying it successfully to constituency parsing, showcasing its efficiency, parallelism, and adaptability across diverse tasks.