A Retrospective on Our World Records

Over the past few weeks, our work has led to a series of records across the Modded-NanoGPT Main and Optimization Tracks, OpenAI's Parameter Golf, and the SlowRun Tiny Track. Today, we're sharing how we achieved them.

Modded-NanoGPT challenges participants to train GPT-2-scale models to a fixed validation loss in the shortest wall-clock time. Our entry attacks the MLP down-projection, one of the most expensive matmuls in the block, by moving it to FP8.

We move the MLP down-projection to FP8, but the main challenge is not the matmul itself. A naïve implementation has to materialize the full post-activation tensor, scan it once to compute the exact amax, then read it again to quantize it before the down-projection. For this shape, those extra memory passes are enough to erase most of the theoretical speedup from the FP8 GEMM.

Our implementation changes the dataflow. While the fused up-projection kernel has the post-activation values in registers, it writes the BF16 tensor needed by the backward pass, emits an FP8 version for the forward down-projection, and accumulates a tiny partial-amax tensor. Instead of rereading the full activation tensor, we only reduce those partial amax values later.

To avoid blocking the current forward pass on the exact current amax, we use delayed scaling: the FP8 activation scale at training step t is derived from the measured amax at step t-1, multiplied by a safety margin. This makes quantization a streaming operation inside the producer kernel rather than a separate memory-bound pass.

We also keep a transposed FP8 cache of the down-projection weight W2, refreshed after optimizer updates. This lets the down-projection run as an FP8 scaled matmul on both operands. The weight-cache path is less important than the activation path, because W2 is much smaller than the post-activation tensor, but it completes the FP8 forward path.

In the end, this gives us a forward-only FP8 down-projection that is profitable end-to-end: the large activation tensor is never reread just to compute scales or quantize, while the existing BF16 backward path is left unchanged.

A few implementation notes:

• The W2 cache refresh does not fundamentally need delayed scaling. Since W2 is much smaller than the post-activation tensor, an exact, non-delayed refresh would likely be viable as well.
• We intentionally keep materializing the BF16 post-activation tensor for backward, matching the current master behavior. Avoiding this materialization and reconstructing the needed values during backward is orthogonal to this PR, and has already been explored in a separate submission.
• We use amax(t-1) with a safety margin instead of a rolling window, as it was sufficient to avoid meaningful clipping for this activation.

We tracked our experiments on Flywheel. Below, you can find a visual illustration of our kernel implementation.

While the Main Track races against wall-clock time, the Optimization Track fixes the training setup and rewards fewer optimizer steps. The constraint is therefore narrow: same data, same batch size, same architecture, one forward-backward pass per step. Under that contract, the only way forward is to change the optimizer.

Our entry uses a late-stage vector-extrapolation method from the Anderson/RRE family in parameter space. The idea behind it is to look at the last few iterates of an optimization process and ask whether their recent changes reveal a better point than the raw next step: it builds a small linear model from the recent trajectory and takes a controlled extrapolated move.

In our case, the iterates are not losses or gradients, but the model weights themselves. Near the end of training, we store a short history of recent parameter vectors, form the differences between them, and solve a tiny least-squares-style system to find a combination that cancels the slowest-moving component of the trajectory. If the local dynamics are approximately linear, this can move the weights closer to where the optimizer was already heading, but in fewer steps.

Because training is noisy and the optimizer state is not a clean fixed-point iteration, a naïve extrapolation can easily jump too far. We therefore make the acceleration deliberately conservative: it only starts late in training, uses a history of four checkpoints, runs every five steps, applies damping, and caps the extrapolated move to at most 0.1% of the parameter-vector norm.

For this record, we tracked our experiments on Flywheel.

OpenAI's Parameter Golf competition, instead, challenges teams to train a language model to the lowest possible loss while keeping the checkpoint under 16MB on disk and the whole run under 10 minutes. It was our first outing as a team on this kind of competitive track, and by April 16th we landed on two small, nearly free additions that compose well on top of the current best public submission.

The first is an attention gate placed right after the attention mechanism, just before the output projection—the placement identified by the Qwen team's work on gated attention. Sitting there, the gate adds a non-linearity between the V and O projections and gives the model a direct way to suppress parts of the attention output, which helps mitigate attention-sink behavior. We restrict it to a small slice of the input channels (12 of 512), which keeps the parameter cost close to zero while matching results seen on the Modded-NanoGPT challenge, where the same trick has been adopted.

The second is a Smear Gate, originally introduced by Larry Dial on the Modded-NanoGPT challenge: a single gated sum between each token's embedding and the one right before it, applied once before the model's layers. We revive it in an input-dependent form and, after ablating channel counts, settle on 12 channels, enough to nudge the model toward distant, non-local relationships instead of settling for the easy local ones.

Together, these two additions were enough to get a submission approved onto the official Parameter Golf leaderboard. We discussed the submission extensively on X.

Where the Main and Optimization Tracks race against the clock, SlowRun changes the constraint: the data budget is fixed—every track trains on the same 100M-token FineWeb corpus—and the only way to win is to squeeze more signal out of it.

We introduce a learnable implementation of Exclusive Self-Attention (XSA) in the first six transformer layers. XSA removes from an attention output the component aligned with the token's own value vector, encouraging the layer to represent contextual rather than self-position information. Instead of applying a fixed correction, the implementation gives every layer and attention head a learnable strength, allowing the model to adopt the mechanism only where it is useful.

Pushing further, we extended XSA to all transformer layers and coupled it with fused FP8 multi-token prediction, setting a second, better record. We train a single language-model head against three successive targets at each position. The custom fused FP8 soft-capped cross-entropy kernel avoids materializing the full vocabulary-logit tensor during training, providing both a denser learning signal and faster execution.

Working across several challenges at once also changed how we collaborated with coding agents. With several agents exploring in parallel, it quickly became difficult to remember what had been tried and why it had failed.

We began each challenge with a deliberately broad phase of exploration. Agents could take different branches, run experiments, and return with evidence without losing the path that led there. As promising trajectories emerged, we became progressively more involved, first selecting and refining directions, then proposing new ideas, and finally working directly with the code and the agents operating closest to it.

Each record followed a different path. While SlowRun converged quickly around a strong idea and the Optimization Track was solved through several rounds of autonomous iteration and refinement, the Modded-NanoGPT Main Track demanded much closer human involvement, down to the dataflow, kernels, and low-level implementation choices.

In our iterations, we used Flywheel both as a shared project memory and an epistemic map: hypotheses could be connected to the experiments that tested them, the evidence they produced, and the conclusions we drew from them. The lesson here was not that autonomy can't perform, but that different levels of autonomy are required for different problems. In this quest toward autonomous research, a skill to develop became clear: understanding when to steer agents or intervene in the working implementation, and at what level.

Many branches of the research graphs remain unexplored, and that's where we are headed next.