Now I want to take this concept further and implement my own CWM to engage in orbital warfare.

So I felt it fitting to open with a quote from the movie WarGames.

• David Lightman: [typing] Love to. How about Global Thermonuclear War?

• Joshua: Wouldn’t you prefer a good game of chess?

• [Jennifer laughs]

• David Lightman: [typing] Later. Let’s play Global Thermonuclear War.

• Joshua: Fine.

The general idea is that rather than training a neural network learning the game's dynamics through trial-and-error (the RL approach), you use a code language model to write a simulator of it, then hand that simulator to a classical search algorithm.

One of my immediate observations was that there are actually two tasks:

1. Synthesis (turning game rules into executable code) and

2. Planning (finding good actions in that code)

Both of them have different profiles as classical search is better at planning and LLM-based agents are better at synthesis.

But let’s step back and reiterate how DeepMind’s implementation works.

The first step of their solution is actually quite mundane: the agent first plays a few games to completion using a random policy, and the data collected during each game forms the trajectories used to learn the CWM. The simulator is then generated via iterative LLM refinement, which uses those trajectories to auto-generate unit tests that validate simulated transitions against them; progressively improving the CWM with each refinement pass.

For perfect-information games, where the state of the game is 100% visible at all times, it pairs this with a standard Monte Carlo Tree Search (1,000 simulations per action); for games with hidden state, it uses Information Set MCTS, which samples from a belief distribution over possible ground truths rather than planning from a single known state.

That’s why I thought it was interesting. The code repository will be made available in chat.

Share

In this post, I want to expand on their work and understand how MCTS works in a simultaneous-move game like Orbit Wars.

What makes Orbit Wars interesting is that it is not really turn-based in the alternating sense. I.e., the game loop doesn’t stop to let the user take time to analyze its play and then the other players move. Here, every player acts at the same instant. And not doing anything is also an action.

And that’s where DeepMind’s solution fails as neither standard MCTS nor IS-MCTS handles this directly for a simple reason.

Both assume a single active player at each node.

Simultaneous-Move MCTS

In simultaneous games, each action's value depends on all players’ actions. So, instead of treating every joint action as one giant action space, decoupled Upper Confidence Bound (UCB) lets each player run its own UCB independently. UCB in this context is a selection rule that picks which action to try next by balancing exploitation of actions with high average reward against exploration of actions that haven’t been tried enough, using a confidence bound that shrinks as an action gets sampled more.

Standard minimax alternates Max (us) and Min (opponent) nodes.

But in Orbit Wars, both players act simultaneously; neither player moves “after” the other. The correct game-theoretic model at each node is a normal-form game: a matrix where the rows are our actions, the columns are the opponent’s actions, and each cell is the outcome.

Our action ↓  |  Opp: Rush  |  Opp: Expand  |  Opp: Hold
──────────────┼─────────────┼───────────────┼────────────
Rush          |   0.4       |    0.8        |   0.9
Expand        |   0.2       |    0.6        |   0.7
Hold          |   0.1       |    0.5        |   0.6

The solution concept is a Nash equilibrium: a mixed strategy where neither player can improve by deviating. Computing exact Nash equilibria at every MCTS node is expensive (linear program per node). Instead, we use decoupled UCB.

Each player at each node maintains an independent UCB table over their own actions:

where:
  Q_i(a, n) = sum of player i's values when they chose action a at node n
  N_i(a, n) = visit count for player i choosing a at node n
  N(n)      = total node visits
  C         = √2  (exploration constant)

At selection time, the joint action is the Cartesian product of each player’s UCB-argmax. Each player’s UCB table is updated with their own component of the leaf value vector, not the joint reward. In two-player zero-sum games, this converges to the Nash equilibrium; in multiplayer settings, it approximates correlated equilibria.

Here, each player maintains an independent action-value table, and the joint action is the Cartesian product of their individual UCB argmaxes.

This is also my primary algorithmic extension beyond DeepMind’s framework. And I will explain it in more detail in a later section.

In contrast to traditional Reinforcement Learning methods, where we are optimizing a policy function, we optimize a heuristic value function v(state, player_id) that maps a game state to a scalar in[0, 1], where 1.0 means a certain win and 0.5 means parity.

So why does this matter?

Simultaneous-move games are not a niche problem. The stock market, where I spent most of my day job time, is the canonical example. In this market, every participant submits orders without knowing what others will submit at the same instant; prices clear after all orders arrive, not sequentially. A participant who telegraphs their strategy can be front-run; one that commits to a pure strategy can be arbitraged. The same structure appears in auctions, mechanism design, and any multi-agent system where agents act on shared state without coordinating first. Nash equilibrium reasoning (the mathematical foundation of market microstructure) requires planning algorithms that handle simultaneous moves correctly.

So it’s easy to understand why I selected this problem, as building a Code World Model for Orbit Wars is, in that sense, building infrastructure for a much broader class of problems.

Subscribe now

For the simulator itself, similar to DeepMind, I used a coding agent and a refinement loop through executed game trajectories.

When building such a system yourself, you have to make three choices:

1. How to represent the state of the world,

2. How does the model search for good actions, and

3. How do you evaluate positions at the leaves of your search tree when you run out of time to keep searching?

A Code World Model solves choice (1) with a property that no neural-network world model can match: determinism. Given the same state and the same action, a code world model always returns the same successor state. This solves the problem of the generation of illegal moves, which I already pointed out in my work on Game Theory and Agent Reasoning.

It is a function. You call it, you get the answer. It’s deterministic.

That determinism is what makes everything downstream possible.

Search algorithms assume that f(state, action) → next_state is reliable. When it is not (when the world model sometimes predicts illegal states, or drifts after 30 steps), the search tree fills with phantom branches, and the agent confidently optimises toward a future that cannot happen.

There is a second advantage that matters more than it initially appears: speed. Taking notes from high-frequency trading, let’s assume we have a time budget of 500ms per decision. A reimplementation of the engine can execute in 0.5ms, a factor of one thousand. At a 170ms search budget, that is the difference between fewer than 1 simulation and ~150 depth-3 simulations per turn. That depth-3 MCTS finds qualitatively different strategies than a one-node lookahead: strategies that involve multi-turn setups, timing attacks around comet spawns, and recognising when a garrison is large enough to defend without reinforcement.

The third advantage is what I find most interesting: modularity. Because the world model is code (plain Python with no learned weights), it can be read, debugged, and corrected by engagements with standard coding agents. When the agent makes a strange decision, you can trace it: run the offending state through the MCTS search, observe which action had the highest visit count, print the value estimates at its children, and inspect the world model transitions that produced them. This kind of debuggability is difficult, maybe impossible, with a neural forward model. You cannot open a neural network and read why it predicted a particular next state.

What Makes Orbit Wars Hard

Let me start by explaining how Orbit Wars is structured. The gameworld is a continuous space with a sun at the centre. Planets orbit the sun or sit stationary.

On each decision node, you can argue that this is a turn, and a player submits a list of actions like fleet launches. A fleet launch is a triplet: [from_planet_id, angle_radians, num_ships].

Fleets travel in straight lines and are destroyed if they cross the sun.

Three properties combine to make this hard:

Orbiting targets. A planet 30 units away at angular velocity 0.025 rad/turn will have rotated ~2.4 units by the time your fleet arrives. Aiming at its current position sends ships into empty space.

Simultaneous moves. Every player acts at the same instant. So your best move depends on what all opponents do at the same time.

Speed matters. You can run a deep tree-search, but if your solution does not quickly produce results, your opponents might have taken the time to occupy their planets and start production there.

The solution to all three: use a coding agent to build and optimize a faithful model of the transitions of the game engine (the Code World Model) and run tree search entirely inside it.

The quality of search scales with the speed of simulation.

The State

How, in general, should we describe the world environment?

I packed everything the agent needs into a single State dataclass object:

@dataclass
class State:
    planets: list          # [id, owner, x, y, radius, ships, production]
    fleets: list           # [id, owner, x, y, angle, from_planet_id, ships]
    initial_planets: list  # starting positions for orbital angle calculation
    comets: list           # [{planet_ids, paths, path_index}]
    comet_planet_ids: list
    step: int
    next_fleet_id: int
    angular_velocity: float
    num_players: int
    episode_steps: int = 500
    ship_speed: float = 6.0
    comet_speed: float = 4.0

The separation of planets (current positions) from initial_planets (starting positions) is critical. Orbital position at any future step is computed from the initial angle, with no accumulated floating-point drift.

On each activity point, we need to assess and operate on eight phases in order:

1. Pre-launch comet expiry: remove comets where path_index ≥ path length

2. Comet spawning

3. Fleet launch (apply player actions)

   └─ deduct ships, add fleet to state

4. Production
   └─ owned planet ships += production

5. Orbital mechanics
   └─ rotate orbiting planets one tick
   └─ advance comet path indices

6. Fleet movement + collision detection
   └─ swept-pair hit test vs all planets
   └─ OOB + sun-crossing check

7. Mid-tick comet expiry (black-hole effect)

8. Combat resolution
   └─ fleet vs fleet, then winner vs garrison

The sequence ordering matters. Not only from a memory management perspective. But also from a ground truth and internal consistency of the environment perspective. For example, production must happen before movement, so a planet garrisoned on turn T knows about any extra ships to defend with. Comet expiry has two phases (steps 1 and 7) because fleets can collide with a comet on its final tick, at which point the comet disappears mid-tick and the fleet ships are silently lost (the “black hole” effect). And that brings up an important point. Since the state of the game is constantly in flux, we need to be able to verify if the CWM is in sync with the game engine. A world model that is fast but wrong is worse than the external game engine.

Every error introduces excess unproductive branches into the search tree: the agent optimises toward futures that the actual game engine will not produce. Verification is not optional.

The approach I used here is replay matching. RM is re-running recorded trajectories through the model and checking its predictions match the real outcomes. Because the environment stores complete game records, i.e., every state, every action, every step, I can run the CWM against these replays, and its output is compared against the recorded ground-truth states at every step.

Since there is also a temporal sequence of actions, we can conclude that a mismatch at step 47 means the bug was introduced somewhere in steps 1–47: bisect, fix, verify again.

I found two subtle bugs this way:

1. Production fires before movement. A garrison that gains production ships at step T can use those ships to defend against a fleet that arrives in the same tick. If you implement movement before production, the fleet always sees the pre-production garrison, and your combat resolution is off by exactly production ships for every attack.

2. Mid-tick comet expiry. A comet planet that exhausts its path during fleet movement disappears before combat is resolved for that tick. Any fleet that already collided with it this tick loses its ships silently: they do not participate in combat. This produces an effect where fleets apparently vanish against comets, which looks like a collision bug until you notice the timing.

This brings us to the problem of collision detection. It’s wasteful if a fleet crashes into the sun or off the game grid. The standard approach (checking whether a fleet’s end position is inside a planet’s radius) fails for fast fleets and orbiting planets. Instead, I linearise both motions over the tick and test whether the moving fleet and moving planet come within the planet’s radius at any point t ∈ [0, 1]:

where:

• AA A = fleet start position

• δA\delta A δA = fleet displacement this tick (speed × direction)

• P0P_0 P0​ = planet start position

• δP\delta P δP = planet displacement this tick (one orbital step)

In code I implemented it like this:

def swept_pair_hit(A, B, P0, P1, r):
    d0x = A[0] - P0[0];  d0y = A[1] - P0[1]
    dvx = (B[0]-A[0]) - (P1[0]-P0[0])
    dvy = (B[1]-A[1]) - (P1[1]-P0[1])
    a = dvx*dvx + dvy*dvy
    b = 2.0 * (d0x*dvx + d0y*dvy)
    c = d0x*d0x + d0y*d0y - r*r
    if a < 1e-12: return c <= 0.0      # negligible relative velocity (fleet and planet may both be moving)
    disc = b*b - 4.0*a*c
    if disc < 0.0: return False
    sq = math.sqrt(disc)
    t1 = (-b - sq) / (2.0*a)
    t2 = (-b + sq) / (2.0*a)
    return t2 >= 0.0 and t1 <= 1.0    # overlap interval hits [0,1]

The per-fleet relative-position and relative-velocity vectors are assembled into (F, P) matrices, the discriminant is computed in a single broadcast pass, and np.argmax extracts the first hit per fleet in list order. The vectorized version runs the full fleet–planet collision pass in well under the 0.5 ms per-tick budget, even with large fleets.

The Intercept Problem

In my opinion, this is the most mathematically interesting sub-problem of Orbit Wars. A fleet launched from position S travels in a straight line at constant speed. The target planet orbits at an angular velocity ω. The question is: what angle do you fire at so the fleet arrives at the planet’s position, not where it was when you launched?

Fleet speed is a sublinear function of ship count, with larger fleets moving slower:

speed(n) = 1 + (v_max − 1) × (log(n) / log(1000))^1.5

where v_max = 6.0 (configurable)

Ships Speed 1 1.00 10 2.03 100 3.77 500 5.28 1000 6.00

The ^1.5 exponent means the curve is concave: the first 10 ships buy more speed than the last 500. Small scout fleets crawl; mass invasions are fast.

The intercept angle satisfies a fixed-point equation. Let θ(t) be the planet’s angle at absolute step t:

Position at t:  P(t) = (cx + r·cos(θ(t)),  cy + r·sin(θ(t)))

Flight time T:  T = ||P(t₀ + T) − S|| / speed(n)

Both T and P(t₀ + T) depend on each other. Solve by iteration:

aim ← P(t₀)                     # initial guess: current position
for i in range(max_iters):
    T ← ||aim − S|| / speed(n)
    new_aim ← P(t₀ + T)         # predict planet at estimated arrival
    if ||new_aim − aim|| < tol:
        return atan2(aim.y − S.y, aim.x − S.x)
    aim ← new_aim

In code, convergence happens in 4–6 iterations for typical orbital velocities. The function returns None if it fails to converge, a safety valve that filters degenerate trajectories.

def solve_intercept(source_pos, target_planet_id, fleet_size,
                    depart_turn, state, max_iters=20, tol=1e-4):
    target = next((p for p in state.planets if p[0] == target_planet_id), None)
    if target is None:
        return None

    initial_target = next(
        (p for p in state.initial_planets if p[0] == target_planet_id), None)

    speed = fleet_speed(fleet_size, state.ship_speed)
    av    = state.angular_velocity
    aim   = _predict_pos(target, initial_target, av, depart_turn)

    prev_angle = None
    for _ in range(max_iters):
        dist    = math.hypot(aim[0] - source_pos[0], aim[1] - source_pos[1])
        eta     = dist / speed if speed > 0.0 else 0.0
        new_aim = _predict_pos(target, initial_target, av, depart_turn + eta)
        new_angle = math.atan2(new_aim[1] - source_pos[1],
                               new_aim[0] - source_pos[0])

        if prev_angle is not None:
            diff = abs(math.atan2(math.sin(new_angle - prev_angle),
                                  math.cos(new_angle - prev_angle)))
            if diff < tol:
                return new_angle % (2 * math.pi)

        prev_angle = new_angle
        aim = new_aim
    return None

Reducing Exponential Action Space

Tracking the raw action space is combinatorially expensive. With 5 planets, 10 targets, and 10 ship sizes, we observe 10^10 joint actions per turn. MCTS cannot enumerate this efficiently

My solution is a two-level abstraction.

For each owned planet, I calculate a small fixed menu:

Option 0: No-op (hold all ships)
Option 1: fit-launch to target T₁  ← "exactly enough to capture"
Option 2: all-in (1.0) to T₁       ← enemy planets only
Option 3: fit-launch to T₂
Option 4: all-in to T₂
...
Option N: multi-spray to all Tₖ    ← one slot for simultaneous multi-front

The fit fraction is the key innovation. Rather than sending a fixed 50% or 100% of your garrison, compute exactly how many ships arrive with the garrison + 1 margin needed to capture:

eta      = dist(source, target) / speed(available_ships)
projected_garrison = garrison₀ + production × min(eta, 40) × 0.6
needed   = projected_garrison + MARGIN − friendly_already_inbound
ships    = max(0, min(needed, available))

This prevents two failure modes that destroy naive agents: over-sending (wasting ships that could take a second planet) and under-sending (fleet arrives and bounces off a garrison that grew during transit).

The Full MCTS Loop

We remember that the full MCTS loop consists of 4 stages.

We can then implement the in-game loop like this:

┌─────────────────────────────────────────────────────────┐
│  while time.monotonic() < deadline:                     │
│                                                         │
│  ┌── SELECT ──────────────────────────────────────┐     │
│  │  For each player i, choose action a_i via UCB  │     │
│  │  with progressive widening:                    │     │
│  │    allowed = ceil(pw_c × visits^pw_alpha)       │     │
│  └───────────────────────────────────────────────┘     │
│           ↓                                             │
│  ┌── EXPAND ──────────────────────────────────────┐     │
│  │  joint = [concrete(a_0), concrete(a_1), ...]   │     │
│  │  child.state = cwm_apply_joint_action(joint)   │     │
│  │  (new node: evaluate immediately)              │     │
│  └───────────────────────────────────────────────┘     │
│           ↓                                             │
│  ┌── EVALUATE ─────────────────────────────────────┐    │
│  │  if terminal: exact reward (1.0 / 0.5 / 0.0)   │    │
│  │  if depth ≥ max_depth: cwm_value_function(...)  │    │
│  └────────────────────────────────────────────────┘    │
│           ↓                                             │
│  ┌── BACKPROPAGATE ────────────────────────────────┐    │
│  │  For each player i:                             │    │
│  │    stats[i][a_i].value += value[i]              │    │
│  │    stats[i][a_i].visits += 1                    │    │
│  └────────────────────────────────────────────────┘    │
│                                                         │
│  return argmax_visits( stats[player_id] )               │
└─────────────────────────────────────────────────────────┘

If we assume 64 actions per player, fully exploring every action before revisiting any, would take 64 simulations just to see each action once. Progressive widening improves this as it reveals actions gradually: the k-th action is exposed only when the node has at least ceil(pw_c × visits^pw_alpha) visits.

Actions are pre-sorted by a production/distance prior so that promising moves surface first. This makes naive sense as it ensures that the agent exploits good moves early and explores broadly only when it has the budget.

Between turns, the root node is cached. At the next turn, the child whose state matches the newly observed state (verified by a lightweight signature of step, planet owners, and ship counts) is promoted to root. A node that accumulated 500 visits last turn starts with 500 visits instead of 0.

Feature Design

How are we identifying which feature matters when predicting a game-winning action?

I designed each feature around the same pattern:

ratio(own, opp_agg) = own / (own + opp_agg)

This keeps all features in a normalized range of [0, 1] and means 0.5 always represents parity, regardless of the raw scale of the underlying quantity.

Based on that and a review of game trajectories, the code agent and I retrieved these features.

I then introduced the concept of weight normalisation.

Weight normalisation means you can add or remove features without rescaling: the output always stays in [0, 1]. A weight of 0 effectively removes a feature.

Learning Through Self-Play

Manually tuning 24 parameters (15 value-function weights, an MCTS budget, and 8 search hyperparameters) is practically impossible. The search landscape is non-convex (aggressive and defensive playstyles are different local optima), noisy (win-rate estimates from a small number of games have high variance), and high-dimensional.

I use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). CMA-ES belongs to the broader family of evolutionary algorithms, but more specifically, it's an evolution strategy that searches by sampling candidate solutions from a multivariate Gaussian and adapting that Gaussian's mean and covariance toward the best-performing samples each generation. In other words, it is also a gradient-free optimiser that maintains a multivariate Gaussian over parameter space and adapts its covariance matrix toward regions of high reward.

All 24 parameters are normalised to [0, 1]:

_PARAMS = [
    ("w_material",        "continuous", (0.05, 1.0)),
    ("w_production",      "continuous", (0.05, 1.0)),
    ("w_control",         "continuous", (0.05, 1.0)),
    # ... 11 more value weights ...
    ("mcts_budget_s",     "continuous", (0.05, 0.25)),
    ("opp_aggregation",   "discrete",   ["sum", "max", "mean"]),
    ("max_depth",         "discrete",   [2, 3, 4]),
    ("pw_c",              "discrete",   [2.0, 4.0, 8.0]),
    ("pw_alpha",          "discrete",   [0.3, 0.5, 0.7]),
    ("k_targets",         "discrete",   [3, 4, 5]),
    ("n_active_planets",  "discrete",   [2, 3, 4]),
    ("k_reinforce",       "discrete",   [0, 1, 2]),
    ("fine_fractions",    "discrete",   [False, True]),
]

Discrete parameters (e.g. max_depth ∈ {2, 3, 4}) are encoded as continuous values and rounded at decode time. This is a standard trick for mixed-domain CMA-ES: the covariance matrix treats them as continuous, and the discretisation happens in evaluation only.

The Learning Loop

Initialise: x₀ = encode(incumbent_weights),  σ₀ = 0.3

repeat:
  solutions ← es.ask()              # sample λ ≈ 14 candidates
  for each solution xᵢ:
      wᵢ  ← decode(xᵢ)
      score(wᵢ) ← pool_win_rate(wᵢ, pool)   # self-play games
  es.tell(solutions, −scores)       # CMA-ES minimises

  best_in_gen ← argmax(scores)
  if score(best_in_gen) > incumbent_score:
      save best_in_gen
      pool.append(best_in_gen)      # grow pool with new snapshot
      re-baseline incumbent against updated pool

until n_evals ≥ budget

The key difference from vanilla CMA-ES is the growing self-play pool.

Starting with [greedy, incumbent], every time CMA-ES finds a new best performing candidate, it is added to the pool. Future candidates are evaluated against the entire pool simultaneously: a candidate that beats greedy but loses to an older snapshot is penalised.

This prevents cyclic strategies: a weight vector that is only good against one particular playstyle cannot dominate the pool score.

Pool evolution (example run):

Gen  0:   pool = [greedy, v0]
Gen  3:   v1 beats pool → pool = [greedy, v0, v1]
Gen  7:   v2 beats pool → pool = [greedy, v1, v2]   (v0 evicted, capped at 6)
Gen 14:   v3 beats pool → pool = [greedy, v1, v2, v3]
...

Noise and Why It Matters

With 12 games per evaluation, the win-rate estimate for a single candidate has a standard deviation:

The signal between two similar-quality candidates is often only ~0.05.

This means many evaluations will place a genuinely better candidate below an inferior one due to noise alone. CMA-ES partly compensates for this: it evaluates a whole population of λ ≈ 14 candidates per generation, and the covariance update averages across all 14 outcomes rather than treating each evaluation independently.

The effective noise on the gradient estimate is:

Still substantial, but workable.

Increasing to 24 games per evaluation would halve the per-candidate noise at the cost of roughly halving the number of generations in a fixed time budget, a tradeoff worth making once the covariance model has been built (later in training).

A practical implication: never accept a single CMA-ES run as final. Re-evaluate the saved best weights against a larger game sample (50–100 games) to confirm it actually beats the previous incumbent before shipping it.

Training ran for ~140 generations at 12 parallel workers with 12 games per evaluation and a 300-step game truncation.

The chart below shows pooled win rate (score / games played, against the full pool) over generation number for 2p mode:

Each step up represents a newly accepted candidate that beat the incumbent against the entire pool. The slope of improvement is fastest in the first 40 generations (where CMA-ES builds its covariance model and escapes the starting point) and flattens as the pool hardens.

For 4p, the trajectory is noisier because a single evaluation is a 4-player game where variance is higher: a strong candidate can lose to coordinated focus-fire from three weaker opponents:

Comparing the default weights (uniform ~0.25 across all features) to the converged weights reveals what actually predicts outcomes:

2-player converged weights:

Feature           Default   Converged   Change
─────────────────────────────────────────────
w_material         0.45      0.9954     +121%  ← dominant signal
w_centrality       0.00      0.5992     new    ← map position matters
w_time_fleet       0.00      0.5247     new    ← committed ships count
w_prod_density     0.00      0.4919     new    ← efficiency matters
w_production       0.20      0.5162     +158%
w_phase_material   0.00      0.3834     new    ← endgame weighting
w_event_fleet      0.00      0.3808     new    ← imminent arrivals
w_control          0.25      0.0521     −79%   ← planet count unreliable
w_anti_leader      0.00      0.0000     ─      ← 2p: irrelevant

The most interesting findings were:

w_centrality went from 0 (unused in defaults) to 0.60 over the observation period. Central planets have short fleet paths to all quadrants, making them economically and militarily dominant. The default weights ignored this entirely. CMA-ES found it in ~40 generations by discovering that candidates with high centrality weighting consistently beat those without it.

w_control (planet count) collapsed from 0.25 to 0.05. Owning more planets than your opponent is not predictive of winning; owning better planets (central, high-production) is. This is an insight that would be hard to arrive at by manual tuning.

4-player converged weights:

Feature           2p weight   4p weight   Interpretation
──────────────────────────────────────────────────────────
w_material         0.9954      0.9959     stable across modes
w_control          0.0521      0.9398     4p: presence = survival
w_centrality       0.5992      0.0022     4p: central = target
w_threat           0.1842      0.5817     4p: safe core critical
w_anti_leader      0.0000      0.2934     4p: don't get focus-fired
w_cohesion         0.3252      0.4141     compact = defensible

In 4p, w_control rockets to 0.94: board presence is survival. Three opponents can coordinate to eliminate you; if you own 40% of planets, you survive; if you own 10%, you don’t. Meanwhile w_centrality collapses to near-zero (0.002): central planets draw coordinated attacks from three opponents simultaneously, making them a liability rather than an asset in 4p.

At 12 workers, 12 games/evaluation, 300-step games, one generation takes approximately:

λ ≈ 14 candidates × 12 games × ~40s/game = 6,720 game-seconds
At 12 parallel workers: 6,720 / 12 / 60 ≈ 9.3 minutes/generation

48 hours → ~300 generations → ~4,200 candidate evaluations

For a 24-dimensional parameter space, CMA-ES typically needs 10–100 × dimension = 240–2,400 evaluations for solid convergence. A 48-hour run is well into the reliable convergence range.

Comets as Temporary High-Value Planets

In the game engine, comets appear at fixed steps (50, 150, 250, 350, 450) throughout each game and temporarily add groups of four neutral planets that move along pre-computed elliptical paths. They are among the richest events in the game: comet ships count toward your score, and a well-timed comet capture can flip a losing position.

Each comet group is a set of four planets placed at 4-fold rotationally symmetric positions.

Their path is a densely sampled, highly eccentric ellipse with the sun at one focus:

Eccentricity e ~ Uniform(0.75, 0.93)   ← very elongated
Semi-major axis a ~ Uniform(60, 150)    ← wide orbit
Perihelion = a(1−e)                     ← must clear sun + comet radius

4-fold symmetry: the base path (Q4 quadrant) generates 3 copies
  via 90° rotation, so all four comets enter simultaneously from
  different quadrants.

The path is resampled at constant arc-length intervals of comet_speed = 4.0 units/tick, giving each comet a fixed travel speed regardless of path curvature. Only the contiguous on-board segment is kept (5–40 waypoints). Generation retries up to 300 times to find a valid configuration that clears the sun and all existing planets.

Because comets appear at known steps, a well-calibrated agent can:

1. Anticipate the spawn: save garrison ships rather than spending them 5 turns before a comet spawns

2. Position intercept fleets: launch toward the anticipated entry point one tick before spawn

3. Contest ownership: the comet planet starts neutral; the first fleet to arrive takes it

The CWM models comet expiry with two distinct events. Pre-launch expiry removes comets whose path has already ended before the turn starts (the agent cannot act on them). Mid-tick expiry removes comets that exhaust their path during fleet movement: any fleet that collides with a mid-tick-expiring comet has its ships silently deleted (the black hole effect). This is a subtle engine quirk that the CWM replicates exactly.

The value function’s event_fleet feature partially captures comet-related fleet pressure by encoding signed ship arrivals over a 20-turn horizon, but it does not explicitly model the spawn probability. This is a known gap: an agent with explicit comet-anticipation logic would have a strategic edge over the current system.

Banking the Overage Budget

One non-obvious optimisation has a disproportionate impact on midgame performance.

The environment gives each agent a 1-second per-turn hard limit plus a 60-second overage bank that accumulates across the game. The MCTS search formula draws from the bank:

In the opening phase (turns 0–25), the correct move is nearly deterministic: expand outward, capturing the closest neutral planets by production/distance². MCTS wastes its entire budget rediscovering this same answer. The wasted budget accumulates in the overage bank but is unavailable for the critical midgame.

The opening book bypasses MCTS entirely. For the first 15 turns (2p) or 25 turns (4p), the agent executes a proximity-first multi-spray using production / distance² scoring:

def _opening_book(state, player_id):
    # Multi-source: each owned planet claims its nearest high-value neutral.
    # Planets sorted by garrison descending so the richest planet picks first.
    own = sorted(owned_planets, key=lambda p: p[5], reverse=True)
    claimed = set()
    launches = []
    for src in own:
        available = [t for t in neutrals if t[0] not in claimed]
        if not available:
            break
        best = max(available, key=lambda t: t[4] / dist²(src, t))
        claimed.add(best[0])
        launches.append((src[0], best[0], "fit"))
    return tuple(launches)

The distance² denominator is intentional.

production / distance (linear) can prefer a distant planet with production 3 over a nearby one with production 1. But the distant planet takes 3× longer to capture, during which the opponent captures the nearby one and you’ve committed ships to an unopposable flight. The quadratic penalty correctly balances production value against the opportunity cost of a longer commitment.

The multi-source design matters for games where you start with more than one planet, or when your garrison has split across captures mid-opening. Each planet expands independently to its nearest valuable neutral, so all fronts push outward simultaneously rather than funnelling through a single launch pad.

Saved budget: 15–25 turns × ~0.6 s ≈ 9–15 seconds of extra midgame search time.

Competing Architectures

During training, I am playing the CWM-MCTS solution against four opponents (from zero-lookahead heuristics to trained neural policies) across three axes: search depth, simulation throughput, and measured win rate.

  † not directly measured against ours 
  ‡ standard MCTS assumes a single active player at each node, which is wrong for
    simultaneous-move games; converges to a pure strategy rather than Nash
    equilibrium and is therefore exploitable

Before I go further into the results, below is a brief flowchart of how each competing architecture converts an observation into an action:

Random

obs ─────────────────────────────────────────────────────────▶ random move

Greedy

obs ──→ [score each candidate]
              production / dist²  ← one-move horizon; no lookahead
              garrison advantage
        ──→ argmax ──────────────────────────────────────────▶ best move

MCTS + External API

obs ──→ [initialise tree]
         ├─ API call (~500 ms) → state' → score
         │                                      budget exhausted; < 1 sim
         └─ (no more iterations fit in 0.17 s)
        ──→ argmax_visits ────────────────────────────────────▶ best move

Neural RL (PPO)

obs ──→ [feature encoder  (CNN / MLP)]
                ↓   ← weights learned offline; GPU; weeks of env interaction
         [policy head]  ──→  action logits  ──→  sample ──────▶ best move
              (no explicit lookahead; game theory encoded implicitly in weights)

CWM + SM-MCTS (this system)

obs ──→ [state_from_obs]
              ↓
        [get_candidates] ─────── solve_intercept (orbital fixed-point)
        [classify_opponent] ──── k-means archetype over replay history
              ↓
        ┌──────────────────────────────────────────────────────┐
        │  SM-MCTS  (0.17 s wall-clock budget)                 │
        │                                                       │
        │  SELECT    ego:  decoupled UCB + prog. widening      │
        │            opp:  archetype-biased action sampling    │
        │  EXPAND    cwm_apply_joint_action      (0.5 ms/call) │
        │  EVALUATE  cwm_value_function          (2 µs/call)   │
        │  BACKPROP  per-player Q/N tables                     │
        │                                          ~150 iters  │
        └──────────────────────────────────────────────────────┘
              ↓
        [argmax_visits] ─────────────────────────────────────▶ best move

Results

All win rates for (2-player, 50 games) shown below are from direct arena play.

  † "Full system" = MCTS-CMA + archetype opponent model.
    55 % vs MCTS-CMA alone isolates the archetype model's contribution.
  ‡ Sequential MCTS = joint-action UCB (OpenSpiel MCTSBot style), same CWM,
    same budget, same weights. Only the tree structure differs

The 72% over MCTS-Default quantifies the CMA-ES contribution in isolation: same search code, different leaf weights, 22 percentage points gained. The jump from MCTS-Default to the full system compounds better evaluation (CMA-ES) with better opponent modeling (archetype-biased sampling), reaching 55% over the CMA-ES-only baseline.

The speed advantage of the CWM is independent of which MCTS variant you use: both “CWM + Standard MCTS” and “CWM + SM-MCTS” run at ~150 sims/turn. The gap between them is purely algorithmic: one finds an exploitable pure strategy, the other approximates a Nash equilibrium.

For a simultaneous-move game, this is the more important of the two extensions.

(2p, 120 steps, 50 games)

Matchup                              Win Rate
────────────────────────────────────────────
MCTS-CMA vs. Random                  1.000
MCTS-CMA vs. Greedy                  1.000
MCTS-Default vs. Greedy              0.960
MCTS-CMA vs. MCTS-Default            ~0.72
Full System vs. MCTS-CMA             ~0.55

The greedy agent is trivially beaten: it has no lookahead and over-expands into indefensible territory. The interesting comparison is MCTS-CMA vs MCTS-Default: a 72% win rate purely from better value-function weights, with identical search architecture.

4-Player Self-Play Pooled Win Rate

Config                               Pooled win rate   vs. random
────────────────────────────────────────────────────────────────
Random                               0.25              1.0×
Greedy                               0.28              1.1×
MCTS-Default (uniform weights)       0.32              1.3×
MCTS-CMA (default vs greedy only)    0.35              1.4×
MCTS-CMA (self-play pool, 5 snaps)   0.50              2.0×

The jump from 0.35 to 0.50 between pool-less and pool-based training is the clearest demonstration that self-play diversity matters. Evaluating only against greedy produces weights that exploit greedy’s specific weaknesses (never reinforcing, linear distance scoring) but fail against MCTS opponents that don’t share those weaknesses.

Subtree reuse means the tree warms up over the first ~5 turns. From turn 10 onward, the promoted root already carries hundreds of accumulated visits from prior turns, and fresh simulations build on top of those rather than restarting from scratch.

Where Time Is Actually Spent

In light of applications in high-profile trading and keeping high-speed decision-making, profiling a typical 2p midgame turn (step 80, budget = 0.173 s) yields the following times:

Component                   Time (ms)   % of budget
────────────────────────────────────────────────────
MCTS simulation loop          158.0        91.3%
  └─ cwm_apply_joint_action    (0.5 ms × ~300 calls)
  └─ cwm_value_function        (0.002 ms × ~300 calls)
  └─ UCB select + backprop     (0.03 ms × ~300 calls)
Action candidate generation     8.0         4.6%
  └─ solve_intercept calls
State construction              4.0         2.3%
  └─ state_from_obs
Output concretisation           2.0         1.2%
  └─ abstracted_to_concrete
────────────────────────────────────────────────────
Total                         172.0        99.4%

The search loop dominates, which is exactly what you want.

The engineering goal of every non-search component is to be fast enough to be irrelevant. cwm_value_function at 0.002 ms contributes only 0.6 ms total across 300 calls. state_from_obs at 4 ms is the largest non-search cost and could be optimised if needed (e.g., by caching the planet-id-to-index maps between turns).

SM-MCTS vs. Sequential MCTS

To isolate the algorithmic contribution of simultaneous-move awareness, we benchmarked our SM-MCTS against a sequential MCTS opponent: the same algorithm OpenSpiel’s MCTSBot would produce when applied to a simultaneous-move game. Sequential MCTS treats the joint-action node as a UCB search over all (a₀, a₁) pairs, committing both players’ actions together rather than decoupling their Q-tables.

Both agents use the same CWM, the same per-turn time budget (10 ms), the same tree depth (3), and the same value-function weights.

The only difference is the tree structure: decoupled vs. joint-action UCB.

  SM-MCTS         Sequential MCTS (OpenSpiel-style)
  ──────────────  ──────────────────────────────────
  Per node:       Per node:
    P0 Q/N table    Joint (a₀, a₁) Q/N table
    P1 Q/N table

  Joint action  = UCB(P0) × UCB(P1) (Cartesian product)
                                   Joint action = UCB over all (a₀, a₁) pairs

  Converges to  ≈ Nash equilibrium  Converges to → pure strategy (exploitable)
  Space: O(|A₀| + |A₁|)           Space: O(|A₀| × |A₁|)

Results across 1,000 games (500 as P0, 500 as P1 to cancel first-mover advantage): 850 wins, 0 draws, 150 losses (85.0% win rate, 95% CI [82.8%, 87.2%], SE = 1.1%). Wall time: 459 s total (0.46 s/game, 10 parallel workers). The chart above shows the W/D/L split and the cumulative win-rate convergence over the full run.

An 85% win rate from the same CWM, same time budget, same weights, purely by choosing the correct algorithm for simultaneous-move games. Sequential MCTS locks onto a pure strategy early in the search (exploiting its Q-table greedily leads to a single committed action) while SM-MCTS maintains independent uncertainty per player and spreads probability mass across multiple options. The sequential agent cannot recover once SM-MCTS finds its exploitable pure strategy.

In closing

The most important engineering lesson is the CWM: everything else downstream (search depth, value accuracy, training quality) scales linearly with how fast your forward model runs. A 1000× speedup in the simulator is worth more than any algorithmic improvement to the search.

The most important research lesson is from the CMA-ES weights: game intuitions fail at scale. Planet count (control) being nearly irrelevant in 2p but dominant in 4p; map centrality being a massive advantage in 1v1 but a liability in 4p: these are findings that would require thousands of hours of human gameplay to internalise, but the optimiser discovers them in ~40 generations.

The linear value function is the primary remaining bottleneck. Feature interactions (for example, the compounding advantage of being both materially ahead and at game’s end) are currently approximated by single features (phase_material) rather than learned from data. A small neural network trained on (state, outcome) pairs collected from self-play would capture these non-linear interactions with no increase in search latency (a 2-layer MLP with 32 hidden units runs at ~20 µs on CPU, still fast enough for 150 depth-3 sims/turn).

Decoupled UCB is the same fix for the same failure mode, since a limit-order strategy optimised against a fixed, sequential model of “the rest of the book” converges to a pure quote that a faster or better-informed counterparty can pick off, while treating each participant’s order-placement policy as an independent action-value table pushes the search toward a mixed strategy that is harder to adverse-select against.

And the CMA-ES result generalises directly: the features that dominate price-formation and adverse-selection risk are rarely the ones a human trader would hand-pick first, which is exactly why parameter search against a diverse, self-play pool of counterparties, rather than tuning against one static model of “the market,” is what surfaces them.

I still find the concept of the Code World Model to manage state transitions between states extremely interesting. And plan to apply it to a state of the equity markets going forward.

Generalizing, Orbit Wars is a stand-in for any environment where agents act on a shared state without a turn order, and a market order book is the canonical instance. Every piece built here has a direct reading in that domain. The Code World Model is a fast, deterministic simulator of a venue’s matching logic, the same role a backtester or exchange-replay engine plays for a trading strategy, and it is subject to the same law: a strategy search is only as good as the number of counterfactual order-book states per second it can evaluate.

Thank you for reading this far. It’s been a longer-than-expected post.

Subscribe now

We all have a naive model of the world. If we turn a glass of water around, we expect the water to splash on the floor. World models in agentic systems are internal representations that let a system anticipate what happens next and plan accordingly

If there is one thing I have learned, given my own work on memory architecture (i, ii, iii), and also object permanence, is that providing the right frame of reference to understand everything that’s dumped into the context window increases accuracy manifold.

Recently, LeCun’s Joint-Embedding Predictive Architecture (JEPA) pushed this concept even further. Of course, originally, the term world-model is more often applicable to VLAMs, but I am certain that we can extend the concept to more general agent architectures. Especially, since I see them as a milestone on the road to robotics. In this post, I argue that a world model doesn’t need to predict everything. Not every pixel. Not every grain of rice.

The model should learn to abstract a representation of the world, based on logic and simple rules, that captures what’s predictable and discards everything isn’t.

And in a way, that is exactly what the Architect from the movie The Matrix: Reloaded also said.

The first Matrix I designed was quite naturally perfect, it was a work of art – flawless, sublime. A triumph equalled only by its monumental failure. The inevitability of its doom is apparent to me now as a consequence of the imperfection inherent in every human being.

What is predictable can be controlled.

Kevin Kelly, in his seminal work Out of Control, calls this concept the emergence of control, where it arises from successfully harnessing what is predictable rather than comprehensively modeling everything.

I think that’s the right framing,

But I’d like to take it one step further:

A world model perceives the world through sensors. In autonomous driving, we talk about cameras and LIDAR. But I think this concept can be extended to purely software agents as well.

I have implemented sensors in my agent architectures for a long time. In my case, it is not a car that needs to react to sensing an object in the street, but an unusual volume pattern, or an arriving email from a colleague.

The sensor picks up this new information and provides it to the world model.

The world model is a predictable abstraction of the world. The world model then provides the information to the planner, who is tasked with finding the right action to take. Given this action, the model forecasts the impact on the world of the action, and then hands it over to the agent to execute the action.

Here, the term agency is really important.

It defines the difference between automation and agent.

As now, after the action has been implemented, the world has changed, and the state of the world has moved one frame further. A stock has been bought, sold, or not. A response email has been written.

Each action moves the agent temporally and sequentially through context. The state of the world changes.

Each state transition describes “what comes next, given what just happened”.

This temporal and sequential framing allows us to validate what we are seing make sense as the sensory observation is scoped to whatever sequence of events the agent pays attention to. I will touch on these components again when I discuss hierarchies.

Subscribe now

Models work the best when they rely on very basic logic. The sun is up during the day. If I can see the sun (a sensor), then it must be daytime.

A code world model is an extension to the standard world model, as it describes the world through such codified logic. Conceptually, Code World Models have evolved out of a traditional reinforcement learning school of thought that operates within a refined environment. The benefit of using a code world model is that we can just execute the code and observe its effects instead of building a holistic model of the environment.

Since the boundaries of the world model are assumed finite, it is possible to define structured, rule-based regularities that make the future predictable enough to control. In its most basic form, we define

• States: “A turn moves through draw, discard, or knock phases, and the legal actions are different in each one.”

• Invariants: “A player can never launch more ships than the planet currently holds, and the code enforces this on every call rather than trusting the caller.”

• Conditions: “If the attacking force exceeds the garrison, the planet changes ownership.”

Each state is temporal. And, I think that is really important to understand. While a rule describes how each moment transitions into the next one, the code world model enables long-running, executable accumulation of exactly these kinds of one-sentence rules, applied one step at a time.

Just like a game loop.

And I think that’s where the concept gets really interesting.

Monte Carlos Tree Search

We begin this section with a discussion of the constraints of simulated environments. In simulated environments, there are generally two very different ways to get a language model to behave intelligently within it. The popular approach is to treat the model as the policy itself: You feed it the rules, the current state, and the model predicts the next state. You see this commonly in reinforcement learning algorithms like PPO. Here, the model becomes an “intuitive player” that pattern-matches its way through Chess openings or Poker ranges. Since this approach is traditional RL and therefore quite well known, I won’t go into too much detail.

I think its more interesting, instead of having the model play the game a billion times, creating a model that writes the game. That means, given a natural-language rulebook and a handful of example trajectories, have it produce a working simulator. This simulator can then be used to predict state transitions, enumerate legal moves, and expected score outcomes.

We then hand that simulator to Information Set Monte Carlo Tree Search (ISMCTS) and let that do the playing.

Monte Carlo Tree Search is an algorithm that builds a search tree incrementally over four repeated phases:

Selection traverses the tree from the root using a policy that balances exploiting high-value nodes against exploring under-visited ones, until it reaches a leaf;

Expansion adds one or more new child nodes to that leaf;

Rollout (simulation) plays out a sequence of actions, often randomly or via a fast policy, from the new node to estimate a value Q; and

Backpropagation propagates that value back up through every node visited in the selection path, updating their visit counts and value estimates.

Repeating these four steps many times concentrates simulation effort on promising branches, so the visit counts at the root eventually approximate the optimal action distribution without needing to exhaustively search the full tree.

Generator simulation and ISMCTS together form the concept of the Code World Model, or CWM. First introduced in this recent paper from Google DeepMind, “Code World Models for General Game Playing,” the team pushes this idea further by introducing imperfect information, stochasticity, and games the model has never seen before.

This paper fits perfectly to Yann LeCun’s long-running argument that world models, not next-token prediction, are the road to machine intelligence. Given my own learings on self-correcting systems, I can see their point.

While both concepts, code-as-world-model and learned-latent-world-model, are technically different, they try to answer the same underlying question.

“If I have the current state and want to do an action, what will the next state be?”

Data to Code Translation

The DeepMind paper defines the job of the LLM as translation. You give it a textual description of a game’s rules and add a small number of recorded trajectories from random play. Trajectories here refer to recorded sequences of (state, action, next-state) tuples, i.e., actual play-throughs of the game, logged turn by turn.

The Code World Model is now tasked to produce code that implements the game engine and a standard interface to it.

The interface is:

1. a state transition function,

2. a legal-move enumerator,

3. an observation function,

4. a reward function, and a termination check.

This interface is similar to OpenSpiel’s API, the open-source game research framework. Therefore, the resulting code can immediately be dropped into existing, battle-tested planning algorithms.

Why does this matter?

Well, in my opinion, there are three advantages this buys you, which are exactly the three things that make “LLM as policy” approaches fragile.

Verifiability. A CWM is, by construction, a formal specification of the game’s rules. If the planner wants to know which moves are legal at the current state, it doesn’t ask the LLM and hope; it calls a function and gets a deterministic answer. The notorious failure mode of LLM-as-policy agents, and something I also encountered in my Game Theory work, is that these models propose illegal moves.

Strategic depth. Once you have a working simulator, you can throw arbitrary amounts of search at it. MCTS with a thousand simulations per move doesn’t care whether the underlying model “feels” confident; it just brute-force explores the game tree. This is the classic System 1 / System 2 split that Daniel Kahneman made famous. The LLM supplies a fast, intuitive understanding of the what of the game, while the search algorithm supplies the deliberate, grounded lookahead execution that intuition alone can’t match.

Generalization. For all forms of robotics training, a system do to one thing and do this one thing extremely well has been possible for a long time. The road to AGI might be paved with CWM’s ability to generalize better to unknown game situations. An LLM asked to translate rules into code is doing something it has seen a copious amount of: turning a specification into a program. The meta-task, i.e., data-to-code translation, is one that the model has practiced at scale, even if the specific game is novel. That’s why the paper’s out-of-distribution games (invented specifically so they can’t be in any LLM’s training data) are such an important part of the evaluation.

Inference-as-code

Most prior work on LLM-synthesized world models, WorldCoder or GIF-MCTS, assumed fully observed, deterministic environments. For generalization, especially if we can bring this concept into capital markets, that is a problem and also a huge simplification. Chess and Go are fully observed: every piece is visible to both players at all times. Poker is not. Bridge is not. Most real strategic situations, negotiations, markets, multi-agent coordination are not.

The DeepMind paper’s most substantial contribution, in my opinion, is the contribution of “inference as code.”

My argument is that if you’re playing Poker you only know your own cards and the public information. How do you predict what your opponent might be holding?

Card counting is one option. Another is having Information Set MCTS handle this by sampling states to ground truth, testing consistency with sensor observations, running searches across the distribution, and aggregating statistics.

The problem is that ISMCTS needs a way to generate those plausible ground-truth samples.

That’s where inference-as-code comes in.

In the DeepMind paper, the team tasks the LLM to synthesize a second piece of code, an inference function that takes an observation and action history and samples a plausible, complete history of the game, including the parts the player doesn’t see: the chance player’s card deals, the opponent’s hidden actions, everything.

Because the CWM’s transition function is deterministic, sampling a plausible history is equivalent to sampling a plausible hidden state, running the sampled history through the CWM, and getting back a hidden state that’s consistent with everything you’ve actually observed.

For every point in a trajectory where you, the player, took an action, the world model performs a unit test asserting that replaying the sampled history reproduces your actual observation and your actual action at that point. If the inference function passes all these tests, the sampled histories are guaranteed to belong to the support of the true posterior, not necessarily the exact right distribution, but at least plausible, valid states the game could actually be in.

In other words, the result is a set of hidden states that are actually possible given everything you've observed, as opposed to states that are ruled out by what you've seen.

Open vs. Closed

One of the paper’s more practically important contributions is a distinction it calls “open deck” versus “closed deck” synthesis.

Open deck synthesis assumes that during the training/synthesis phase, you have access to the hidden information in your offline trajectories. You’re building the model in a “design room” where you can see everyone’s cards, even though the deployed agent never will. This is a reasonable assumption in several real scenarios: cooperative training environments, game-design phases where developers have full state access, or expert demonstrations that come with hindsight annotations. It’s also the assumption made by the most directly comparable prior work, “POMDP Coder.”

Closed deck synthesis drops this assumption entirely. The CWM only sees what the player saw: your own observations and your own actions, never the hidden state, not even after the fact. The paper presents this as the realistic scenario for an agent thrown into a genuinely novel online game, and as far as the authors are aware, no prior CWM work has addressed it.

The technical solution here is elegant: a kind of code-based autoencoder.

The LLM still produces a CWM and a hidden-history inference function, but the unit tests are restricted to only what’s verifiable without hidden information.

Essentially, “does the round-trip from observation through inferred hidden history back to observation reproduce what I actually saw?”

The inference function acts as an encoder, the CWM acts as a decoder, and the combination of game rules plus the required API serves as the regularizer that keeps this from collapsing into a trivial, useless latent space (the kind of degenerate solution that plain autoencoders are notorious for if you don’t constrain them).

Performance does degrade in the closed-deck setting, sometimes substantially. Bargaining’s online inference accuracy drops from 100% (open deck) to 76% (closed deck). Gin Rummy, already the hard case, becomes essentially unusable: inference accuracy collapses to roughly 9-54% depending on whether you’re looking at train, test, or online numbers, none of which is good. But here’s the genuinely surprising finding: despite this synthesis-quality degradation, game-play performance against Gemini 2.5 Pro doesn’t degrade nearly as much. The CWM-ISMCTS agent built on a noticeably worse closed-deck model still beats or matches the LLM-as-policy baseline in most games. The paper even notes a counterintuitive improvement in one game (Hand of War) under closed-deck conditions, hypothesizing that the freedom to synthesize a simpler state space — one not constrained to match some externally-imposed “true” hidden state — might sometimes help rather than hurt.

The practical lesson: synthesis-accuracy metrics and downstream-performance metrics are not the same thing, and a CWM that’s “wrong” in calibrated, support-preserving ways can still be useful for planning, because what the planner actually needs is a valid state to reason from, not necessarily the state.

Tree Search vs. Conversation

Neither a CWM nor an inference function will come out right on the first try. For all forms of machine learning we need a way to provide corrective feedback.

During training, the refinement loop generates unit tests from the offline trajectories: for each recorded transition, a test checks whether the CWM’s prediction matches what actually happened, across states, observations, rewards, and move legality. The CWM is refined until either every test passes or the compute budget runs out.

The CWM operates two mechanisms. Conversation and Tree Search.

“Conversation” is a sequential agentic loop where each failed test’s stack trace gets appended to the conversation history, and the model is asked to fix it.

“Tree search” is more elaborate. Here, multiple candidate CWMs are maintained simultaneously in a refinement tree, and Thompson sampling decides which candidate to refine next, weighted toward candidates with either high test-pass rates or few refinement attempts so far, essentially borrowing the bandit-style exploration/exploitation logic that underlies MCTS itself, but applied to code candidates rather than game moves.

The key practical advantage of tree search is the ability to backtrack: if a line of refinement is going nowhere, the search can abandon it and try a different branch, rather than being stuck digging deeper into a conversational dead end.

For the perfect-information games in the paper, both approaches converge to perfect transition accuracy quickly, but tree search is also resilient under imperfect-information conditions.

Synthesizing the Value Function

After the CWM itself and the inference functions, a Code World Model requires a third artifact: heuristic value functions (HVF). The HVF is used to initialize the value of newly-expanded leaf nodes in MCTS and ISMCTS rather than relying purely on random rollouts. Unlike the CWM and inference functions, value functions aren’t refined against ground truth as there is no “correct” heuristic to test against.

Instead, several candidates are generated in one shot, and the best one is selected through a local tournament: agents using different value functions (including a “no value function, just rollouts” baseline) play against each other on the synthesized CWM, and the winner is kept.

Where Code World Models and JEPA meet

It’s worth being precise about what’s actually shared between the DeepMind CWM line of work and the world-model research associated with Yann LeCun, because the overlap is conceptual rather than architectural.

LeCun’s “Path Towards Autonomous Machine Intelligence” position paper and succeeding work are about learned, latent-space world models. JEPA’s central bet is that generative models, which try to predict every pixel of the future, are solving the wrong problem; the world is full of irrelevant, unpredictable detail, and a model that tries to reconstruct all of it will waste capacity and fail. Instead, JEPA-style architectures learn an abstract representation of “what matters” and make predictions in that abstract space, closer to how, in LeCun’s framing, a child develops an intuitive model of gravity and object permanence without ever needing to predict the exact trajectory of every falling object down to the pixel.

The stable-worldmodel library operationalizes this research. Their implementation is effectively “a learned simulator that predicts how an environment evolves in response to actions, enabling agents to plan by imagining future outcomes,” providing fundamentals like: standardized dataset collection from simulated environments, training pipelines for world models, and solvers (including CEM, the Cross-Entropy Method, a classic sampling-based planning algorithm) that use the learned world model for planning. The workflow mimics model-based RL: collect rollouts, train a model that predicts state transitions, then plan against that model rather than against the real environment.

So where’s the connection to Code World Models?

Not in the representation, CWMs are symbolic (literal Python code with explicit state dictionaries and deterministic functions), while JEPA-style world models are learned and latent (neural networks operating on abstract embeddings, typically for continuous, high-dimensional domains like video and robotics).

But the function is identical: both are “a thing you can run forward to imagine what happens next, so that a planner doesn’t need to act blind.”

This actually clarifies when each approach makes sense. Code World Models shine when the domain has crisp, describable rules. I.e., board games, card games, turn-based strategy, anything where “the rules” is a coherent concept that fits in a prompt and where exact, verifiable transitions matter (you really don’t want your Poker agent confused about whether a flush beats a straight).

Learned latent world models shine when the domain is continuous, high-dimensional, and resistant to crisp specification, robotic manipulation, video, and physical dynamics, where “writing down the rules” isn’t really a coherent idea, because the rules are physics, at a level of detail no one’s going to type into a prompt.

In both approaches, the model that “understands” the domain is explicitly decoupled from the system that acts in it.

In CWM, the LLM’s job ends at producing a simulator; MCTS or ISMCTS does the planning.

In JEPA-based systems, the learned world model’s job is to predict consequences in representation space; the CEM solver or other planning algorithm does the planning.

Both architectures are, in a sense, rejections of the “one model does everything end-to-end” approach, whether that’s an LLM directly emitting moves, or a generative video model directly emitting pixels of an imagined future. The bet, in both cases, is that separating “knowing how the world works” from “deciding what to do about it” produces systems that are more verifiable, more sample-efficient, and more capable of the kind of deep, deliberate lookahead that neither pure pattern-matching nor pure generation reliably delivers on its own.

What This Should Mean in Practice

We are building agents here. So if you’re also building one to run a game, then if this game is operating an environment with quirks, edge cases, or undocumented behaviors that diverge from any official rulebook, then the CWM approach offers a concrete, falsifiable workflow.

You need to:

1. get the LLM to write the simulator,

2. generate unit tests from real trajectories,

3. refine until those tests pass, and

4. only then let a classical planner loose on the result.

The paper’s results, CWM-based agents matching or beating a frontier “thinking” LLM used as a direct policy in nine out of ten games, including ones invented specifically to be unfamiliar, suggest this isn’t a niche technique.

source: figure 4

It’s a genuinely different division of labor between language models and search, one where the LLM’s most valuable contribution might not be the move it suggests, but the simulator it writes.

Unresolved cases are instructive, too. Gin Rummy’s procedural, multi-stage scoring logic remains the hard frontier, a reminder that “translate the rules into code” is itself a task with a difficulty gradient, and that gradient tracks procedural depth more than it tracks how complex a game seems to a human player. For anyone applying this approach to their own domain, that’s probably the single most useful diagnostic question to ask up front: how many sequential, conditional phases does a single turn actually involve? The more of those there are, the more refinement budget, and the more carefully-chosen unit tests — you should expect to need before the world model the LLM writes actually matches the world it’s describing.

Thank you for reading this far.

Kindly share this post if you think it provides value to you. Thank you!

Over the last few years, companies have pushed to become AI-ready by pushing their employees to use AI as much as possible

But now the bills are coming in, and it’s not nice.

Subscribe now

If your CFO is starting to ask questions and the CTO doesn’t have clean answers, its a tough spot to be in.

So I wrote this playbook as a six-phase programme for getting that situation under control. I figure that, as usual, this is not simply about cutting costs, but building the institutional capability to manage AI spend the same way mature organizations manage cloud spend. If some terms are unknown, I added a glossary at the end.

Tokenmaxxing isn’t about spending less. It’s about spending right.

I did not invent the term, but I’ve been using it since my last post on the topic.

Tokenmaxxing is about extracting the maximum business value from every token, not just minimizing them and organizing them into 6 sequential phases

Each phase has a clear goal, a set of concrete actions, and named departmental owners.

You can run this progressively, Phase 1 before Phase 2 or adapt it to wherever your organization already is.

I will now go into more detail for each of these phases, but I believe they are fundamentally clear, and this document serves as a high-level playbook, rather than a deep dive.

Read more

“Recursion: see Recursion.” — Dictionary definition joke

“To iterate is human, to recurse divine.” — L. Peter Deutsch

The more agents iterate over your workspace, the more long-context reasoning and memory management remain open challenges. As you are well aware, traditional transformer-based LLMs are limited by fixed token windows: once the prompt exceeds the context length, earlier information is simply forgotten.

As I wrote in my research paper “Production Rule Systems In Autonomous Agents”, I propose that recursive designs might be an interesting new approach to evaluating intelligence, and Recursive Language Models (RLM) specifically could be a new paradigm with a fundamentally different inference strategy.

What is an RLM?

Subscribe now

An RLM treats the prompt not as raw text fed into a transformer, but as part of an external environment (e.g., a variable in a REPL) that the agent can programmatically explore.

source

More on my work on REPL, GSPO, and GRPO.

Concretely, given a long prompt P, the RLM initializes a programming environment with P loaded, and provides the agent with high-level context (like the length of P). The agent then writes code to inspect, decompose, and query P, for example, splitting the input into segments or searching for relevant parts, and recursively calls itself on those subtasks.

Effectively, RLMs work by offloading prompt management to a symbolic layer (code and environment) while using the transformer only on manageable snippets. And maybe it makes sense to reframe “prompt” here as an instruction rather than a question. Especially if you are working within the context of workspace agents, the agent would request additional information and/or trigger a service to execute a task. Naturally, it can be expected that there will be a constant back and forth between the main conversational task and tertiary agents.

This leads to frequent decomposition and self-invocation. This approach, if solved by RLMs, lets RLMs handle arbitrarily long inputs (far beyond any model’s token limit) and has been shown to dramatically improve quality on long-context tasks. In experiments with GPT-5 (Zhang et al.), RLMs maintained accuracy on problems 100× longer than its context window, whereas a vanilla transformer’s performance quickly degraded.

Transformer-based agents and RLMs thus contrast sharply in principle:

1. Transformers process the entire prompt in one shot (subject to memory and cost constraints), whereas RLMs treat large inputs as a symbolic environment and iteratively “consume” them. Crucially, an RLM still uses the same neural core (e.g. GPT) but augments it with a memory management layer.

2. The RLM framework is akin to out-of-core computing in databases: small fast memory plus clever external storage. By building a memory-first architecture, RLMs avoid simply concatenating more tokens.

3. Instead, they mimic programs that fetch, filter, and summarize context as needed. This inversion of the memory role is the key theoretical foundation: RLMs prioritize structured memory access over expanding the raw context window.

As I have already touched upon in Grounded Autonomy: Neuro-symbolic Representations in the Reasoning Loop (Aug 2025)

Neurosymbolic Reasoning, Structures, and Context

I believe that neurosymbolic architectures that tightly integrate symbolic knowledge with LLMs to manage context and reasoning operate much more effectively on long-form tasks. A prerequisite for agent implementations in production. In these neurosymbolic agents, high-level structures (e.g., logic rules, symbolic memory graphs, planning modules) work alongside neural networks. For example, neuro-symbolic agents may use an agent as a language interface while delegating precise computation or consistency checks to symbolic components.

In fact, Narayanan (2026) describes neuro-symbolic agents as systems that “combine neural models with symbolic structures (rules, graphs, typed schemas)” to gain better controllability and verifiability. These designs preserve the flexible language reasoning of neural nets, while offloading correctness, long-term memory, or discrete planning to symbolic modules.

Neurosymbolic context management often takes one or more of these forms:

Symbolic memory stores: Here, a graph or database records knowledge derived from the environment. For instance, semantic memory consists of structured knowledge bases or graphs that the agent can query. Such memory can be indexed by concepts or embeddings, enabling retrieval when needed. My Superbill app “Sentinel”, for example, stores facts and rules about a stock in an RDF knowledge graph, then queries it via symbolic logic to inform its answers. Agent frameworks are now commonly are already able to build hierarchical memory graphs for agents, organizing knowledge in multi-level structures to capture dependencies and temporal coherence. These graphs evolve as the agent operates: new nodes or edges are added, and old facts can be consolidated or abstracted.

Hierarchical Planning: Something I touched upon in AI Security and the World of Horizon Zero Dawn. Symbolic planning decomposes tasks into sub-tasks using logic or graphs. Recent approaches to neurosymbolic planning combine LLMs with symbolic planners. For example, a neuro-symbolic planner might use an LLM to propose high-level actions, then verify or refine them with a planner written in PDDL (Planning Domain Definition Language). Alessio Capitanelli demonstrates this with “Teriyaki,” a system that trains GPT-3 to act like a symbolic planner. Teriyaki generates PDDL plans action-by-action, enabling concurrent planning and execution. His experiments showed it solved 95.5% of test problems (comparable to traditional planners), produced plans ~13% shorter on average, and significantly reduced planning latency. Similarly, other work builds hierarchical decomposers: Cornelio, for example apllies a knowledge graph along with LLMs to break complex tasks into subtasks, then applies a symbolic validator to check correctness. Their hybrid approach (“KG-RAG” + LLM) outperformed baselines on difficult robotic planning benchmarks. In essence, I believe that hierarchical planning and verification use symbolic structure (graphs, validators) are extremely effective when managing context as the agent holds high-level plans symbolically, calls the agent only for parts of the reasoning, and checks actions with logic to maintain consistency.

Further, Semantic Abstraction involves converting raw input into higher-level symbols or categories. For instance, an agent might parse text into a semantic representation (like a frame or schema) that summarizes concepts and relationships. These abstractions can be stored as part of memory.

source

In this recent study of cognitive mobility agents, agents used “semantic abstraction” to capture user routines at different time scales. The research team from the Valencian Research Institute for Artificial Intelligence built weekly identity-driven plans (long-term) and daily reactive adaptations (short-term), then distilled these experiences into abstract concepts (like destination preferences or time budgets) that shaped future decisions.

This approach to memory, storing generalized knowledge about an agent’s experiences, is crucial for keeping context manageable. Semantic memories (general facts, rules, user profiles) are often stored in knowledge bases or embeddings and selectively retrieved. Other systems use a durative memory strategy: they group related events over time into summaries (e.g. “morning routine: coffee then commute”), which then serve as symbols the agent can reason about. These semantic representations serve as abstractions that bridge raw observations and high-level goals.

Together, these neurosymbolic techniques enable agents to remember what matters and reason across time. Rather than letting a transformer wade through raw tokens indefinitely, the agent maintains structured memory layers: scratchpad/working memory for immediate subgoals, short-term memory for recent context, and long-term/semantic memory for facts and patterns.

This mirrors human cognition and offers a scalable approach: by dynamically retrieving only relevant symbolic memory (instead of full text), the agent focuses on continuity with bounded resources.

Memory Architectures for Long-Term Context

A key distinction to highlight is context window vs. memory architecture. Traditional agent prompts act like volatile RAM: they hold the current working context but are lost once overflowed. In contrast, a memory-first system is more like a persistent database.

A memory architecture allows an agent to store, retrieve, and reuse information beyond its active session, enabling it to remember facts and adapt through experience.

In practical terms, as I had shown before, memory architecture layers must support multiple capabilities. One possible implementation could be like this:

• Working memory (context window): a dynamic space for immediate reasoning (e.g., a short conversation or code snippet).

• Episodic memory: logs of past events or interactions (often stored as sequences or graphs of episodes) that can be recalled whole or consolidated into longer narratives.

• Semantic memory: a database of generalized knowledge and facts (implemented as knowledge graphs, databases, or vector embeddings).

• Procedural memory: stored skills and heuristics (often less explicitly mentioned in agentic systems, but akin to fine-tuned behaviors or policies).

As mentioned above, semantic memory is often implemented via knowledge graphs or embeddings, enabling efficient retrieval of factual or conceptual information. You can then use agent frameworks using tools to access vector databases for factual lookup, symbol-based memories for structure, and in-session buffers for immediate tasks. I have in the past explored several, and up until January 2026, my go-to framework was Smolagents, but since then, I have mostly been using Claude’s SDK.

Context management emphasizes relevance and retention.

Simply inflating the context window has never been a successful strategy. Large context sizes incur steep compute costs and do not guarantee the model “remembers” older content; the attention weights still dilute with length.

Instead, intelligent memory systems decide autonomously what to bring into context.

And that’s the game changer.

Naturally, memories should be selectively retrieved through similarity search or keyed queries rather than blind token dumps. This is naively true. But this is also why Retrieval-Augmented Generation (RAG), where the model queries a vector store or database at inference time, has been so prevalent in the past. It is also why the RLM framework, by design, layers the context: the agent only ever sees a snippet at a time, but can retrieve other snippets on demand programmatically.

Applications and Case Studies

In my preparation for this post, I found these projects that implemented the ideas of recursive inference and neurosymbolic memory:

Extended Context Language Tasks: Zhang’s RLM experiments themselves serve as case studies. They applied RLMs to tasks like needle-in-a-haystack retrieval, code repository understanding, and synthetic reasoning. In each case, the RLM significantly outperformed a baseline that used the raw prompt, especially as input length grew. For instance, on a synthetic pairwise reasoning task, vanilla GPT-5 failed catastrophically beyond 50K tokens, while the RLM maintained high accuracy even at millions of tokens. These results demonstrate that recursive prompting can effectively emulate an “infinite context” system with only nominal extra cost per query.

Robotic Planning (Teriyaki): In robotics, combining agents with symbolic planners is already happening. The “Teriyaki” framework is a generator of task plans that mixes GPT-3 with classical planning. They trained GPT-3 to output PDDL plans and then use those to guide execution. This neurosymbolic hybrid solved ~95% of test planning problems (versus about 98% for a pure PDDL planner) but produced more compact plans and was much faster to output each action. In human-robot tests, this meant the robot could adapt its plan on-the-fly as it executed actions, maintaining fluency in collaboration. Teriyaki demonstrates how symbolic context (PDDL states and effects) and LLM creativity (generating possible next steps) can together improve real-world planning under uncertainty. Would be keen to see how this would perform with 2026 SOTA models.

Hierarchical LLM Agents introduce complex task decomposition. Here, the LLM executes planning by alternating between LLM subroutines and symbolic checks. The agent first queries an LLM to outline a top-level plan, then uses a knowledge-graph RAG system to fetch relevant domain information, then refines subplans, validating each step with a symbolic “world model.” In experiments involving multi-step cooking or assembly tasks, this approach beat pure LLM baselines and pure planning baselines. This might indicate that neurosymbolic context (the knowledge graph + logic validator) can guide LLMs to more correct, context-aware solutions.

Cognitive Simulation (Urban Mobility): Peng applied these ideas to simulating human behavior. The team built agent-based models where each simulated person had an LLM-driven cognitive loop. Each agent used multi-horizon planning: it set weekly goals (based on profile) and daily plans (based on immediate events), integrated with episodic memory (past events) and semantic abstraction (general patterns). For example, an agent might abstract “takes route A to work” from repeated daily logs and then prefer that option even if details change. In large-scale mobility simulations, these agents could exhibit realistic adaptive behaviors under disruptions (like road closures). Their study finds that LLM-driven planning plus memory layers allows agents to “form routines, reflect on disruptions, and adapt strategies over time.” The semantic abstractions (e.g., user preferences, typical schedules) were key to coordinating short-term context with long-term personality traits.

In summary, in my opinion, these cases illustrate how recursive and neurosymbolic techniques are already shaping production solutions. In my opinion, they allow agent systems to work effectively on tasks that require sustained context: enterprise knowledge agents, robotics plans, long simulations, and beyond. The common thread is that each application divides labor between the neural and the symbolic: long-term patterns go into memory or logic, short-term specifics go into the transformer. This hybrid strategy is proving effective for on-premise, privacy-sensitive, and computation-limited environments where maximizing context and reasoning is critical.

In closing

I think the convergence of recursive inference strategies and neurosymbolic memory architectures represents a promising path for managing long-term context. Recursive Language Models show that we can dramatically extend an agent’s effective context by programmatically breaking tasks down and using the agent as a subroutine.

At the same time, neurosymbolic frameworks leverages that context is organized, interpretable, and persistently stored: knowledge graphs, logic modules, and hierarchical planners give structure to what might otherwise be an amorphous token stream.

In practice, this means moving away from monolithic prompt windows and toward multi-layer memory systems. I believe that simply increasing context size is a dead end, and instead champions modular memory layers and planning loops.

Thus, I propose a new design philosophy for AI agents: one that blends neural flexibility with symbolic memory. As initial experiments like Sentinel, RLMs, and Teriyaki demonstrate, such recursive neurosymbolic agents can tackle tasks that were previously impossible for fixed-window transformers.

Going forward, I expect more systems to adopt these ideas, using dynamic memory graphs, hierarchical abstraction, and programmatic context access to achieve true long-term reasoning and continuity.

Thank you for reading this far.

Kindly share this post, as it allows me to write more on this topic.

Share

References :

• Zhang et al., “Recursive Language Models”, arXiv (2025) – Describes the RLM framework for unbounded input processing.

• Honda & Hagiwara, “Context-dependent neuro-symbolic AI through self-supervised learning with LLMs”, Neurocomputing (2025) – Introduces a neuro-symbolic network that learns context-aware reasoning, reducing search space for proof tasks.

• Cornelio et al., “Hierarchical Planning for Complex Tasks with KG-RAG and Symbolic Verification”, arXiv (2025) – Proposes an LLM-based planner enhanced by knowledge-graph retrieval and symbolic validators.

• Capitanelli & Mastrogiovanni, “Teriyaki: Neurosymbolic Robot Action Planning”, Frontiers in Neurorobotics (2024) – Demonstrates a GPT-3 planner trained to output PDDL plans, combining neural generation with symbolic planning.

• Peng et al., “Cognitive Agents in Urban Mobility”, Simul. Model. Pract. Theory (2023) – Shows LLM-driven agents using multi-horizon plans, episodic memory, and semantic abstraction to simulate adaptive commuter behavior.

• Xu., “AI Agent Systems: Architectures, Applications, and Evaluation”, arXiv (2026) – Comprehensive survey of agent architectures, including a section on neuro-symbolic agents.

“The feedback loop, not the function, is the unit of intelligence.”

Norbert Wiener, Cybernetics (1948)

A couple of days ago, I was running a full fundamental analysis on AEYE.

What happened next will blow your mind.

Sorry for the hyperbole…

One of the many skills I have implemented for “Sentinel” fka Superbill is a three-scenario intrinsic value model (Investopedia).

This is a very common workflow that aims to identify if the stock is valued correctly or not.

Related readings:

1. Pair-programming Superbill with Codex-5.2 and Claude Sonnet 4.6

2. Multi-Agents for Optimal Equity Portfolio Construction

3. Grounded Autonomy: Neuro-symbolic Representations in the Reasoning Loop

4. Becoming Agent Bill Ackman

5. So, I built Agentic DeepSearch for Investing in 2023 and Nobody Cared

Partway through the analysis, the skill called the “calculate_irr” tool to retrieve deterministically MOIC figures. In case you don’t know, Multiple on Invested Capital is a measure of how many times over an investor has returned their initial investment.

In this case, the skill generated 3 scenarios:

1. Bear at 1.671x,

2. Base at 2.267x, and

3. Bull at 2.914x.

What then happened surprised me. And I am not ashamed to disclose that I did not expect that behaviour, even though I designed the agent.

Those two blurry screenshots show the moment the agent realized that something was wrong. On the left-hand side, you see the Sentinel Agent, while on the right, you see Claude Sonnet in GitHub Copilot. The Sentinel Agents run in the right rail of my app, and in case you are wondering, Sentinel also runs Claude Sonnet 4.6 as the backend through the API.

Subscribe now

“The most dangerous phrase in the language is ‘We’ve always done it this way.”, Grace Hopper

Distrusting your own tool

In traditional software systems, a function is deterministically implemented and then executed during runtime to return a result. In that sense, they are just logical constructs that do their assigned task or not. In the latter case, we as software engineers usually understand and test failure cases, including edge cases. Yet, given the absence of bugs, once implemented, we have the inherent trust that the function just executes reliably and consistently. However, if something is wrong and you don’t test for it, your program will always execute it incorrectly.

In workspace agents, this might not be the case. Of course we need to engineer the same deterministic reliability into the agent tooling as in standard functions.

Yet, if we engineer our agents correctly, then we can configure agents that they validate the responses they receive.

Now what surprised me in this case is that the agent realized that the tool he was using was implemented incorrectly. This agent does not have access to the code base but can run tests. So he only informed me that something was wrong. And I then checked it using the coding agent. This agent then located the fault in tools.py, validated that the invested capital was indeed being double-counted in the MOIC formula, generated a corrective patch, applied it to the codebase, and re-ran the calculation to verify.

What I describe here is an example of what I call Level 4 agentic self-modification: An AI system with read-and-write access to its own implementation, capable of closing the loop between “something is wrong” and “the thing is fixed” without human intervention in the repair itself.

The broader research context for this behaviour, and why it matters far beyond my app, is what this post is about.

Subscribe now

Why Self-Improvement Matters

The long arc toward systems that improve themselves has been a dream since the term “artificial intelligence” was coined. The desire to build machines that can get better at what they do was articulated by Norbert Wiener as the foundational principle in his 1948 work on cybernetics. His main insight, and an engineering principle that I am reusing, is that intelligent behaviour is not about producing the right output, but about continuously measuring the gap between actual and desired performance and acting to close it.

The feedback loop, not the function, is the unit of intelligence.

What Wiener described abstractly, Richard Bellman formalised later mathematically. His dynamic programming framework gave us the language of sequential decisions under uncertainty, thereby establishing the basis for every reinforcement learning algorithm that followed. By the time Q-learning arrived in 1992, we had the mathematical tools to train agents that could improve their behaviour from experience, but not yet the inference capabilities and infrastructure we have now.

Historically, the leap to superhuman performance came through self-play. OpenAI researched Emergent Tool Use From Multi-Agent Autocurricula, and I wrote about it two years ago here. AlphaGo and AlphaZero demonstrated that a system competing against progressively stronger versions of itself could discover strategies no human had conceived, not by being told what good play looks like, but by iteratively correcting its own failures. OpenAI Five then extended this to cooperative multi-agent settings. In every case, the key ingredient was the same: a loop that turns mistakes into improvement signals, and improvement signals into better behaviour.

My main concern is that reinforcement learning is a largely mathematical exercise that optimizes a policy while maximizing a payoff.

Large language models changed the texture of this problem.

And this is what I want to talk about here.

Suddenly, the “policy” being improved was not a set of numerical weights updated by gradient descent, but a reasoning process expressed in natural language. Aman Madaan et al showed in Self-Refine that an LLM could be prompted to critique its own outputs and revise them iteratively, with each cycle yielding measurable improvement.

Chain-of-thought prompting revealed that making reasoning explicit, writing out each step rather than jumping to an answer, functions as in-context self-verification. Reflexion went further: storing verbal self-reflections in episodic memory, letting an agent accumulate lessons from failure across multiple sessions.

Why does this matter beyond academic benchmarks?

Because the systems we are building now, Sentinel (investment analysis tools), coding assistants, and deep research agents, are all deployed in domains where errors have real consequences. A MOIC calculation that is wrong across every scenario is a bug that corrupts every downstream investment decision that relies on it.

Self-correction also fundamentally has a compounding characteristic.

So, in reinforcement learning, if gradient descent stalls at a local minimum, it will never find the global minimum.

And a system that corrects each error ephemerally, producing the right answer once, then reverting to the buggy behaviour, is barely better than one that never corrects at all.

We want to have agents that make durable corrections and understand why. This means that the fact that an agent can persist the fix so that every future invocation benefits from it is qualitatively different. This is the difference between a one-time patch and an actual improvement. The compounding of durable corrections is, in miniature, what we mean when we talk about systems that learn.

Five levels of self-correction

Not all self-correction is created equal.

The research community has converged on a useful framework for thinking about the different depths at which a system can close the loop on its own performance. I find this taxonomy genuinely clarifying; it makes explicit what is and isn’t changing when a system “corrects itself,” and it illuminates why some forms of self-correction feel like a parlour trick while others feel like a fundamental shift.

Formally: think of a system as three components, a task policy f (what it does), a parameter update rule U (how it learns), and a meta-level modification operator M (how it improves its improvement process).

Each level of self-correction is defined by which of these the system actually changes.

LEVEL 1: Output-level correction

The system generates multiple candidate outputs and selects among them. Nothing internal has changed. The system is identical before and after correction.

Examples: self-consistent voting, confidence re-ranking, majority vote.

LEVEL 2: Behavioural adaptation

The system updates its parameters θ based on feedback, via a fixed update rule U. The policy improves; the mechanism of improvement doesn’t. This is classical reinforcement learning.

Examples: Q-learning, DQN, AlphaZero self-play, STaR fine-tuning.

LEVEL 3: Procedural self-refinement

The system generates an explicit critique of its own output, then revises it. No weights change, but the content of corrections is model-generated. The critique-revise template is fixed and externally specified.

Examples: Self-Refine, Chain-of-Thought, Tree-of-Thoughts, Reflexion.

LEVEL 4: Agentic self-modification

The system reads and modifies components of its own implementation, code, prompt templates, and tool configurations. It revises the procedures that generate outputs, not just the outputs themselves. The meta-operator M remains fixed.

Examples: Darwin Gödel Machine, ADAS, Eureka, and the screenshots at the top of this post.

LEVEL 5: Metacognitive self-modification

The system modifies the modification operator itself. It can become better at getting better, enabling compounding improvement not bound by initial design choices.

Examples: Meta’s DGM-H Hyperagents demonstrated spontaneous emergence of persistent memory, UCB-style selection, and automated bias detection.

With this definition of the levels, we can form an effective hierarchy: a system at level k can simulate all behaviours of levels below it, but not vice versa.

What makes the taxonomy useful in practice is that each level is falsifiable. L4 is falsified if you can show the codebase is unchanged before and after a correction cycle. You can literally run a diff. In my session, that diff is visible in the screenshot: tools.py was updated. The formula changed from (invested + total_return) / invested to total_return / invested. That is L4 in the most literal sense.

The safety implications track the taxonomy closely. At L1–L3, the system’s modifiable surface is limited to outputs and reasoning traces, both human-readable, both correctable from outside. At L4, the system gains write access to its own implementation. The transition from L3 to L4 is the critical safety boundary: below it, the worst a system can do is produce a wrong answer; above it, it can become a different system.

Self-correction in Sentinel

An ambient intelligence layer with just-in-time self-correction

The investment app whose screenshots open this post is built around what I call an ambient financial intelligence layer, a system that persists analytical context across sessions, assembles structured information from multiple tiers of memory, and can correct its own tool implementations in response to high-level semantic feedback. The MOIC fix, albeit shameful, was not a one-off: it is representative of how the system is designed to behave whenever something goes wrong.

The architecture integrates five mechanisms simultaneously:

This is also a memory architecture as well as a feedback mechanism. Most reinforcement learning algorithms only see history as a delta. My implementation can specifically identify relevant memories for the problem in question.

T1.LIVE: Real-time market data from Alpha Vantage, FactSet, NASDAQ Datalink, SEC EDGAR, and news/political trade feeds. Cached so each data point is fetched exactly once per session.

T2.CACHE: Session cache: all tool results from the current conversation, preventing redundant API calls and giving the agent a consistent view of data it has already acquired.

T3.HISTORY: The last N turns of dialogue, enabling reference resolution and maintaining context across a long analytical session.

T4.THESIS: Persistent thesis memory: per-ticker JSON files storing the agent’s accumulated analytical work, writable by the agent and persisted across sessions. This is the substrate for long-term improvement.

SELF-FIX: Interactive error repair: when the agent or user identify anomalous outputs, the agent localises the fault in its own tool registry, diagnoses the root cause, generates and applies a patch, and verifies the corrected output, without the user specifying a line of code.

The context assembly is deliberately tiered and intent-filtered.

Before processing any query, the system detects the query’s category, financials, price, technical indicators, news, insider filings, and selectively includes only relevant portions of each tier.

This reduces token usage by 60–80% compared to naive context injection.

The results across 50 analytical tasks:

• Full architecture accuracy: 73% (vs. 41% baseline with no memory or self-correction)

• User-flagged errors autonomously corrected: 89%

• Accuracy on later tasks (after earlier corrections compound): 81%

• API token reduction via session caching: 58%

• Reduction in redundant tool calls: 62%

That last figure, 81% accuracy on later tasks, is the one I find most significant.

Why?

Well, in my opinion, it measures the accuracy on tasks that came after earlier corrections had propagated. The system got measurably better over the course of the evaluation, because each durable bug fix made subsequent tasks easier.

This is what compounding looks like in practice and why I think it is so important to consider during your feedback loop design.

I should be honest about what this is not.

The evaluation benchmarks are internal and have not been independently replicated. The system is firmly at L4, not L5, since the modification operator itself is fixed and hand-crafted. And the 11% of user-flagged errors that weren’t autonomously corrected required more detailed debugging guidance or represented genuinely ambiguous specifications.

What I find compelling, though, is the convergence of several methods: the capabilities I engineered deliberately, persistent memory, execution trace logging, and structured error analysis, are the same capabilities that emerged spontaneously in the DGM-H Hyperagents research system, without being explicitly designed.

If independent research and independent engineering converge on the same architectural primitives, that suggests these are not arbitrary design choices.

They are something closer to the natural shape of effective self-correction.

In closing

The thing that changed.

When I read the agent's thought, I was immediately puzzled since I did not expect this response at all.

The screenshots at the top of this post may seem, on the surface, unremarkable.

A bug was found. A bug was fixed.

This happens millions of times a day in software development.

What makes them worth writing about is the mechanism: no human wrote the patch. The system read its own source, reasoned about the discrepancy, generated a correction, applied it, and verified the result. The codebase is permanently different because of a complaint the agent itself made in plain English.

Wiener’s feedback loop is 78 years old as a concept.

What is new is that the loop now operates at the level of the system’s own implementation, not just its outputs. The thing being corrected is not the answer the system gives; it is the tool the system uses to generate answers.

The safety implications are real, and I don’t want to minimise the threat.

A system with write access to its own codebase has a fundamentally different threat model than one that can only produce wrong answers.

The failure modes, misaligned self-modification, specification gaming, and the compounding of small errors through an autonomous repair loop are serious and not yet fully understood. Any honest account of self-correcting AI has to hold both things at once: this is genuinely powerful, and the oversight problems it introduces are genuinely hard.

But a financial intelligence layer that durably fixes its own calculation errors, without requiring the user to understand the code, is not a dangerous system. It is a more trustworthy one.

The compounding of correct behaviour over time is exactly what we want from tools we rely on.

We are in the early stages of understanding what it means for a system to genuinely improve. Not to produce a better output once, but to become, in some durable sense, better at producing good outputs. The five-level taxonomy gives us a language for that question.

And my experience as described in this post is, in a small way, evidence that the question is no longer purely theoretical.

Subscribe now

As I have written in numerous occasions (1,2), more context isn’t always better.

Although context windows are only ever increasing, once you run agents in production, you will quickly notice a poorly hidden constraint called attention degradation. i.r., the more tokens you stuff into context, the less effectively the model attends to any individual piece of information.

Tokenmaxxing inverts the usual instinct by assuming that we maximize the quantity of tokens and not their quality. I don’t think that this approach is either valuable for getting better responses or good for the environment.

In this article, I want to present a production-ready framework, Deep Context Query (DCQ), for doing exactly that.

An intent-driven context filtering system that leans on the learnings from using GitHub Copilot and is built on RegEx pattern matching. I am using it in my latest iteration on “Superbill” that I am recently dubbing “Sentinel”.

I have done some more work on the UI/UX recently using Claude Design and Stitch with Google.

In this app, for each opportunity, I on average, 30,000 tokens are used per request. The techniques I am presenting in this article achieve 40–70% token reduction while improving response quality.

The principles are applicable to other agents as well, and not only financial research agents. If your agent operates over rich contextual data across multiple turns, token optimization is not optional. It is architecture.

Subscribe now

The Attention Budget Problem

Transformers compute self-attention across every token pair in the context window, n² relationships for n tokens. As context grows, as it does with every agent loop iteration, the used token budget increases. This results in reduced recall accuracy, degraded reasoning precision, and increased hallucination.

What can be done to improve this?

Naturally, I was looking for ways to ensure that my agents perform well.

The Memory Intelligence Agent (MIA) paper found that raw context injection, dumping everything into the window, performs worse than no memory at all when accumulated trajectories are noisy. A 7B-parameter MIA model with intelligent memory management outperformed a 32B model without it by 18 points on the GAIA benchmark.

This underlines that we need to have a clearly structured memory architecture for your agents. Not clearly engineering context leads to bloated prompts.

The Economics of Bloated Prompts

Token bloat is not only a performance problem, but it is also a cost problem.

Here is a real-life example from my daily work.

A typical investment research session comprises stock data (financials, news, volume, insider trades) runs around 15,000 tokens. Add to that an 8,000-token thesis document, 12,000 tokens of tool results, 6,000 tokens of chat history, and a 3,000-token system prompt, and you’re at roughly 44,000 tokens per turn.

At $3 per million input tokens (Claude Sonnet), that’s $0.132 per turn. Over a 20-turn research session, costs reach $2.64. Scale to 100,000 queries per month and you’re looking at $264,000 per year, much of it spent transmitting context the model doesn’t need for the query at hand.

Obviously, at that scale, you need to have a clear business case to make this viable.

Deep Context Query

My approach starts before constructing the prompt, by analyzing what the user is actually asking (intent), then extracting only the data from memory needed to answer.

Something like this:

The intent analysis step is what makes the context management much more effective. Because I don’t use only the LLM to make that decision, I am using a dynamic and lightweight RegEx pattern matching approach to classify queries into semantic categories in under a millisecond, without a secondary model call, which then drives extraction from structured context subblocks.

Why RegEx, Not ML?

2 words. Speed and Determinism.

Speed: My RegEx executes in microseconds. In production, I measure a mean of 0.34ms per query (max 0.72ms) across 1,000 production queries. In comparison, LLMLingua, a learned compression method, reports ~340–400ms overhead, a 1,000× latency disadvantage for the ML approach.

Determinism: Patterns are predictable, debuggable, and version-controllable. No drift, no surprise.

Zero dependencies: No training data, no secondary model, no infrastructure.

Explicit control: Engineers can read, tune, and audit the exact logic.

The accuracy trade-off?

My DCQ solution achieves 99.2% intent classification accuracy with a 0.8% false-negative rate, competitive with trained multi-label classifiers for comparable vocabulary sizes, at a fraction of the operational cost. For structured domains like finance, legal, or medical, RegEx often matches ML before you reach for the complexity.

So, how to implement it?

Here is a simple overview of the layers

Layer 1: Intent Classification

The classification layer analyzes the user query and produces a structured intent object, with boolean flags representing distinct information needs.

In my financial use-case, I am covering three core intents: news, financials, and price:

from dataclasses import dataclass
import re

@dataclass
class Intent:
    needs_news: bool = False
    needs_financials: bool = False
    needs_price: bool = False

def analyze_intent(question: str) -> Intent:
    q_lower = question.lower()
    intent = Intent()

    intent.needs_news = bool(re.search(
        r'\b(news|article|headline|press|announcement|media|report|story|publication)\b',
        q_lower
    ))

    intent.needs_financials = bool(re.search(
        r'\b(revenue|earnings|profit|margin|eps|income|cash flow|fcf|'
        r'assets|liabilities|debt|ebitda|operating|capex|dividend|'
        r'balance sheet|income statement|financial|p/e|pe ratio)\b',
        q_lower
    ))

    intent.needs_price = bool(re.search(
        r'\b(price|quote|trading at|stock price|market cap|valuation|'
        r'52-week|high|low|chart|technical)\b',
        q_lower
    ))

    return intent

In production, you’d extend this to cover additional intents, insider trades, SEC filings, options, and portfolio context, but the pattern is identical. Each flag is independent, multiple flags can fire simultaneously, and the whole function runs in under a millisecond.

Key design choices worth noting:

1. Each pattern covers 5–15 related synonyms to maximize recall, since missing a relevant query (false negative) is worse than including slightly too much.

2. Word boundaries (\b) prevent substring false positives,“earnings” won’t trigger on “learnings”, “assets” won’t fire on “assemble”.

Layer 2: Context Extraction

Data arrives in structured, section-tagged blocks:

[Volume]
Average: 50M shares
Today: 65M shares (+30%)

[News]
- 2024-03-15: Tesla announces new factory
- 2024-03-14: Q4 earnings beat estimates

[Fundamentals]
Revenue: $100B
EPS: $5.00

DCQ extracts only the sections relevant to the detected intent, using lookahead patterns to preserve section boundaries:

def optimize_stock_context(stock_context: str, intent: Intent, max_chars: int = 3000) -> str:
    if not stock_context or len(stock_context) < 100:
        return stock_context

    sections_to_include = []

    if intent.needs_price:
        volume_match = re.search(r'(\[Volume\].*?)(?=\n\s*\[|\n\s*</stock_context>|$)',
                                  stock_context, re.DOTALL)
        if volume_match:
            sections_to_include.append(volume_match.group(1).strip())

    if intent.needs_news:
        news_match = re.search(r'(\[News\].*?)(?=\n\s*\[|\n\s*</stock_context>|$)',
                               stock_context, re.DOTALL)
        if news_match:
            sections_to_include.append(news_match.group(1).strip())
    else:
        # Include headlines only — not full articles
        news_match = re.search(r'\[News\](.*?)(?=\n\s*\[|\n\s*</stock_context>|$)',
                               stock_context, re.DOTALL)
        if news_match:
            news_lines = news_match.group(1).strip().split('\n')[:11]
            sections_to_include.append('[News]\n' + '\n'.join(news_lines))

    if intent.needs_financials:
        fundamental_match = re.search(r'(\[Fundamentals\].*?)(?=\n\s*\[|\n\s*</stock_context>|$)',
                                      stock_context, re.DOTALL)
        if fundamental_match:
            sections_to_include.append(fundamental_match.group(1).strip())

    metadata_match = re.search(r'(<stock_context>.*?\n)', stock_context, re.DOTALL)
    metadata = metadata_match.group(1) if metadata_match else '<stock_context>\n'

    if sections_to_include:
        filtered = metadata + '\n\n'.join(sections_to_include) + '\n</stock_context>'
        return filtered[:max_chars]

    return stock_context[:max_chars]

The core pattern,

r'(Volume.*?)(?=\n\s*\[|\n\s*</stock_context>|$)' ,

matches the section header, captures everything non-greedily, and stops at the next header or closing tag via lookahead without consuming those characters.

Here, the intention is to remove what’s unnecessary, preserve everything that remains intact.

Layer 3: Tool Result Deduplication

Multi-turn agents accumulate tool results across turns. Without deduplication, the same financial data gets sent to the model repeatedly:

[get_financials(TSLA)] Revenue: $100B, EPS: $5.00
[get_news(TSLA)] 15 articles fetched
[get_financials(TSLA)] Revenue: $100B, EPS: $5.00  ← Duplicate!

Research on agentic memory systems confirms that redundant tool trajectories actively degrade model performance, the model attends to duplicated noise, diluting signal from genuinely new information. In a representative 10-turn session, deduplication reduces effective API calls from 18 to 6.

def optimize_acquired_data(acquired_data: str, intent: Intent, max_results: int = 10) -> str:
    if not acquired_data or len(acquired_data) < 50:
        return acquired_data

    pattern = r'\[([\w_]+)\([^)]*\)\]\s*(.*?)(?=\n\[|\n</acquired_data>|$)'
    matches = re.findall(pattern, acquired_data, re.DOTALL)

    seen = {}
    for tool_name, result in matches:
        key = (tool_name, result[:100].strip())  # First 100 chars as signature
        if key not in seen:
            seen[key] = {
                'tool': tool_name,
                'result': result.strip(),
                'order': len(seen)
            }

    relevant_tools = []
    if intent.needs_news:
        relevant_tools.extend(['fetch_news', 'get_cached_news', 'get_news_item_url'])
    if intent.needs_financials:
        relevant_tools.extend(['get_financials', 'INCOME_STATEMENT', 'BALANCE_SHEET',
                              'CASH_FLOW', 'get_dividend_history'])
    if intent.needs_price:
        relevant_tools.extend(['get_price_history', 'get_technical_indicators'])

    if not relevant_tools:
        relevant_tools = None

    filtered_results = []
    for key, data in sorted(seen.items(), key=lambda x: x[1]['order']):
        if relevant_tools is None or data['tool'] in relevant_tools:
            filtered_results.append(f"[{data['tool']}(...)] {data['result']}")

    filtered_results = filtered_results[-max_results:]

    if filtered_results:
        return '<acquired_data>\n' + '\n\n'.join(filtered_results) + '\n</acquired_data>'

    return '<acquired_data></acquired_data>'

The deduplication key uses result[:100] as a signature, which allows same-tool, different-result calls to coexist (news fetched on different days) while eliminating true duplicates.

Layer 4: Thesis Summarization and History Windowing

Investment theses can easily grow to 10,000+ tokens over long sessions per call. For queries unrelated to reviewing or editing the thesis, DCQ replaces the full document with a section outline:

def optimize_thesis(thesis: str, intent: Intent, max_chars: int = 3000) -> str:
    if not thesis or len(thesis) < 500:
        return thesis

    if intent.needs_financials:  # Thesis likely relevant for deep financial queries
        return thesis[:max_chars]

    sections = re.findall(r'^#+\s+(.+)$', thesis, re.MULTILINE)

    if sections:
        summary = "<current_thesis>\n"
        summary += "# Thesis Outline (full version available if needed)\n\n"
        for section in sections[:8]:
            summary += f"- {section}\n"
        content_start = thesis[:300].strip()
        summary += f"\n{content_start}...\n"
        summary += "\n<!-- Full thesis omitted. Ask to review thesis for complete version. -->\n"
        summary += "</current_thesis>"
        return summary

    return thesis[:max_chars]

History windowing:

Conversation history grows linearly with turns. DCQ keeps only the 5 most recent turns, sufficient to resolve all coherent reference chains observed in production (”add that to the thesis”, “compare it to what you said about the competitor”).

Beyond that, I believe chat history is noise.

def optimize_chat_history(chat_history: str, max_turns: int = 5) -> str:
    if not chat_history or len(chat_history) < 100:
        return chat_history

    turns = re.split(r'\n(?=User:|Assistant:)', chat_history)

    if len(turns) <= max_turns:
        return chat_history

    recent_turns = turns[-max_turns:]

    summary = "<chat_history>\n"
    summary += f"<!-- Showing last {max_turns} turns of {len(turns)} total -->\n\n"
    summary += '\n'.join(recent_turns)
    summary += "\n</chat_history>"

    return summary

Agent Orchestrator

All four layers compose into a single entry point called before every prompt construction:

def optimize_context(
    stock_context: str,
    acquired_data: str,
    current_thesis: str,
    chat_history: str,
    question: str,
    max_stock_chars: int = 3000,
    max_turns: int = 5,
    max_results: int = 10
) -> tuple[str, str, str, str, Intent]:

    intent = analyze_intent(question)

    opt_stock = optimize_stock_context(stock_context, intent, max_stock_chars)
    opt_acquired = optimize_acquired_data(acquired_data, intent, max_results)
    opt_thesis = optimize_thesis(current_thesis, intent)
    opt_history = optimize_chat_history(chat_history, max_turns)

    return opt_stock, opt_acquired, opt_thesis, opt_history, intent

Results

I run this in production right now, but I also tested it across 1,000 representative queries over six weeks of pilot deployment. The results break down cleanly by query type. News-only queries achieve the largest reduction, from ~42,000 tokens down to ~12,500, a 70% drop, because the entire fundamentals block and most tool history is irrelevant and excluded. Financials-only queries drop from ~44,000 to ~16,800 tokens (62%). Technical analysis queries compress from ~38,000 to ~15,200 (60%). Even general, ambiguous questions see a 40% reduction from deduplication and history windowing alone. The average across all query types is 57.6%, from 43,000 tokens to 18,220.

In my opinion, the dominant lever is stock section extraction, which alone accounts for 46.5 percentage points of reduction by stripping full news sections and fundamentals blocks from queries that don’t require them. Thesis summarization adds the second-largest quality gain, collapsing an 8,000-token document to a 600-token outline for ~56% of queries that aren’t thesis reviews.

Every tested query achieved at least 35% reduction.

I did not observe a query type where DCQ made things worse.

My unsurprising verdict is that less is more. An optimized context improves response quality. A/B testing across 100 queries with human evaluation on a 1–5 scale (validated by the Wilcoxon signed-rank test) shows correctness moves from 4.3 to 4.4, completeness from 4.1 to 4.2, and overall quality from 4.1 to 4.4. The headline finding is relevance: +0.6 points (p < 0.001), from 3.9 to 4.5. Fewer tokens, better focus.

The model attends more precisely when the noise is removed.

False negative rate was 0.8%, 8 of 1,000 queries where a relevant section was incorrectly excluded. All cases involved ambiguous queries where intent classification failed, resolvable by expanding pattern coverage.

At 100,000 queries per month, baseline annual API costs run ~$264,000. Post-DCQ, that drops to ~$112,000, $152,000 in annual savings.

At 10,000 queries per month, the savings are ~$15,200 per year.

Lessons learned

Prioritize recall over precision. A false positive (slightly too much context included) is less harmful than a false negative (relevant section excluded). Design patterns wide, with 5–15 synonyms per category.

Use word boundaries religiously. Always wrap terms in \b...\b. “assets” should not trigger on “assemble”; “options” should not fire on “captions”.

Version-control your patterns. As domain vocabulary evolves, maintain a changelog. This creates institutional knowledge and makes regressions diagnosable.

Test against real user queries. Synthetic test cases miss natural language variation. Log actual queries and validate coverage against them quarterly.

Always have a graceful fallback. Wrap all optimization in try-except fallbacks. If RegEx fails on malformed input, return an unoptimized context rather than crashing the request.

Log everything. Emit metrics per call: original token count, optimized token count, reduction percentage, detected intent flags, and processing latency. This data drives future improvements.

Roll out progressively. Deploy to 10% of traffic, then 50%, then 100% over several weeks. Monitor error rates and user satisfaction at each step.

But this might be quite basic. What about more advanced techniques?

Advanced Optimization Techniques

Active Compaction (LLM-Assisted Summarization) is when history exceeds a volume threshold, use the model itself to compress older turns:

if len(turns) > 10:
    old_turns = '\n'.join(turns[:-5])
    summary_prompt = f"Summarize this conversation history in 200 words:\n{old_turns}"
    summary = llm.generate(summary_prompt)
    return f"<chat_history>\n{summary}\n\n{recent_5_turns}\n</chat_history>"

The cost is one additional LLM call per cycle. The benefit is that this enables much longer effective conversation windows without linear context growth.

Intent Confidence Scoring replaces binary flags with continuous confidence scores for proportional filtering:

@dataclass
class Intent:
    needs_news: float = 0.0
    needs_financials: float = 0.0
    needs_price: float = 0.0

def analyze_intent(question: str) -> Intent:
    q_lower = question.lower()
    intent = Intent()
    news_matches = len(re.findall(r'\b(news|article|headline)\b', q_lower))
    intent.needs_news = min(news_matches * 0.4, 1.0)
    return intent

If Low-confidence is detected, then I include sections defensively. In the opposite scenario, high-confidence detection, I can filter aggressively.

Multi-Turn Intent Inheritance optimizes follow-up questions as they inherit intent from prior turns using OR logic:

Turn 1: "What's the latest news on Tesla?"   → needs_news=True
Turn 2: "And what about Microsoft?"          → needs_news=True (inherited)

Here you store intent in the conversation state and merge with the current turn’s detected intent. This prevents follow-up questions from losing their implicit context.

Semantic Fallback for Unstructured Text supports the situation when context lacks section headers, and we need to use embedding-based cosine-similarity as a fallback:

def extract_relevant_paragraphs(text: str, question: str, top_k: int = 3) -> str:
    paragraphs = text.split('\n\n')
    question_emb = embed_text(question)
    para_embs = [embed_text(p) for p in paragraphs]
    scores = [cosine_similarity(question_emb, p_emb) for p_emb in para_embs]
    top_indices = argsort(scores)[-top_k:]
    return '\n\n'.join([paragraphs[i] for i in top_indices])

Use RegEx for structured data; fall back to semantic matching only for unstructured text where section boundaries don’t exist.

In Closing

Tokens are a finite resource. And I firmly believe we need to optimize them for our own sanity. They are a precious budget, and by spending it recklessly, it puts the model in an irrelevant context, actively harming the model’s ability to reason about what matters and also the environment.

Deep Context Query delivers 40–70% token reduction across diverse query types, improved response quality through higher signal-to-noise ratio, $150K+ annual savings at production scale, zero external dependencies, and deterministic behavior that’s predictable, debuggable, and version-controllable.

The implementation runs in production at DeepCQ, processing thousands of investment research queries daily. The patterns are battle-tested against real user needs, edge cases, and performance requirements.

The principle generalises. Financial analyst, coding assistant, legal researcher, customer support agent, the context window is always finite, attention always degrades, and thoughtful curation always wins.

Understand intent. Extract precisely. Optimize relentlessly.

Now that Artemis 2 is launching to the moon in just a (likely) couple of hours, I figured it might be a good idea to talk about Autonomous Agents and ISAM again.

I believe, and hope that I am right, that we are just at the cusp of a civilizational upgrade.

And in my opinion, AI in general and autonomous …

Read more

“For the things we have to learn before we can do them, we learn by doing them… we become just by doing just acts, temperate by doing temperate acts, brave by doing brave acts.”

(Nicomachean Ethics, Book II)

It was a cold, bright morning in March, and the clocks were striking seven when the first sunlights of a new dawn woke me up after a sleepless night …

Read more

What if the value of writing software has collapsed to zero?

Not the value of software or SaaS companies. Their death might be greatly exaggerated. They still remain with enormous profit margins and multi-year enterprise contracts.

But the act of writing it manually might very well be.

Som…

Read more

If you wa…

Read more

“Joe ain’t vicious, you understand. He ain’t like one of these ambitious robots you read about that make up their minds the human race is inefficient and has got to be wiped out an’ replaced by thinkin’ machines. Joe’s got ambition. If you were a machine, you’d wanna work right, wouldn’t you? That’s Joe. He want to work right. An’ he’s a logic. An’ logi…

Read more

Read more

In enterprise setti…

Read more

When …

Read more

Some of it more successfully. Some less so. Some of my work was just my own, but some was deployed across several countries in multi-hundred-billion-dollar businesses.

Over the last weeks and months, gradually but surely, code-generating agents have entered my software life cycle. And I think that once I have fully autonomous agents that act, negotiate, and collaborate through code and APIs, they, again, will bring my work to a whole new level by fundamentally transforming both the speed and quality of my Software Development Life Cycle.

But not by any means.

In this post, I aim to give you an overview of my thoughts on some hopefully interesting findings to demonstrate how Agentic AI solutions automate, optimize, and reshape each SDLC stage, and how the paradigm of “code as interface” that I wrote before will already enable highly effective agent-to-agent workflows.

Please like, share, subscribe.

Subscribe now

Introduction

If you are building software just for yourself, then obviously, a lot of the steps in the software development life cycle are done by yourself. In larger organizations, however, these steps are usually distributed across a large number of people covering a mix of in-house and external resources.

I won’t explain the different stages; I think the chart and source article do that reasonably well.

source

That said, I have formed the opinion that the complexity and dynamism of modern software delivery will benefit greatly from agentic AI. At least my workflow demands more than incremental automation; it requires a system that can reason which code segments to pick, plan how to adjust them, write test cases (unit-tests, SITs, etc), and then act with minimal human intervention, only changing the pieces that would be affected by the change and iterate on error logs.

Classical automation doesn’t cut it anymore.

Across all steps of the software development, this will influence all steps that rely on automation, like rule-based scripts and CI/CD pipelines that break down in the face of changing requirements, unforeseen dependencies, or the need for continuous adaptation.

Thus, agentic AI, by acting as a persistent collaborator (Microsoft calls it CoPilot) that communicates via code and machine-readable protocols, introduces the necessary flexibility, resilience, and intelligence for next-generation software engineering.

But how exactly will that work?

Here are some of my ideas.

Requirements engineering has traditionally posed a bottleneck for SDLC efficiency and accuracy. Business people sit together quabbling around how the software should work, but don’t understand how their ideas fit into the system. If they lack imagination or experience, they miscommunicate intentions and ambiguous specifications, which then frequently lead to costly rewrites.

But that is the past.

Requirements Engineering: Autonomous Clarification and Enrichment

Agents should be good enough to now parse natural language stakeholder input like meeting minutes, include legacy documentation (as long as it’s in digital form), and other documentation like policy requirements reliably to derive actionable software requirements. This helps to properly contextualize and refine requirements. I noticed that it helps if my agent proactively queries back to me to clarify a misunderstanding or patch a missing detail. The biggest impact I see here is that now specs are becoming even more living design documents that are constantly updated as information evolves, ensuring alignment from project inception. This means that the boundaries between living specs and agent context blur, but that also it will be incredibly difficult to control the document, and by extension the context, so it doesn’t go completely off the rails.

This problem is amplified by platforms that convert mixed human prompts, living specs, and data artifacts into structured, testable user stories and technical requirements. The solution here is that we need to maintain a clear goal document, maybe in the form of an agents.md file.

Manus’s engineering team has a great write-up.

Design and Architecture: Automated Blueprints

Design is the creative and technical nucleus of modern software, encompassing everything from system architecture to API definitions. Yet, increasingly, automated systems are generating and refining these blueprints in real time, turning design into a dynamic, adaptive process. Examples are GitHub’s Copilot or AWS Well-Architected Tool. I think this shift enables architectures that evolve alongside product requirements, rather than being locked into static early-stage decisions.

So what does agentic AI bring to the discussion?

Agentic AI can independently propose architecture patterns, leveraging patterns from large-scale code corpora, industrial best practices, and regulatory requirements. It can also generate, on demand, up-to-date UML diagrams, sequence flows, and modular blueprints from requirements. And it might suggest distributed designs optimized for scalability, security, or compliance—all grounded in domain-specific contextual understanding.

Such agents would then not only accelerate the design phase, but they also proactively identify design risks and present alternative architectures for review. In my mind, the iterative process remains, though, as feedback between design agents and human teams allows for faster convergence on robust system blueprints.

While requirements gathering and design phases will increase in importance and time spent, actual implementation time should decrease, driven by coding agents like Claude Code or OpenAI’s Codex.

Implementation: Autonomous Coding, Refactoring, and Optimization

Implementation is still the core task, though. The core of Agentic AI’s change is, in my opinion, not entirely limited to autonomous code generation and maintenance. Earlier code-generation tools that simply scaffolded boilerplate, modern development agents have been doing this for a long time already.

I think the benefit of implementation is also in turning requirements and blueprints directly into code, selecting appropriate frameworks, libraries, and architectural paradigms. Then to write features and full modules with integrated documentation and inline test suites. And finally, being able to refactor legacy code (and that is a moving target) for maintainability, migrate APIs, and address code smells without manual prompting.

In multi-agent scenarios, specialized performance agents may further scrutinize and optimize database queries, server configurations, and application bottlenecks, traditionally tasks left for more specialized engineers and time-constrained performance reviews.

Testing: Unsupervised Assurance and Remediation

I never met a person who likes bug-testing. Yet the more the upstream pipeline of the SDLC is automated, the more we rely on proper testing to ensure that the features we have implemented do not wreck the entire code base and we generate a technical debt or, even more importantly, a security hole we can’t close. What is needed here are, in my opinion, unsupervised verification (in the mathematical sense), and structured and well-defined software testing policies that can be executed well by AI Agents, leading to a more effective quality assurance.

So, instead of relying solely on manually designed tests, these agents generate comprehensive unit, integration, and vulnerability tests for both new and legacy code. They independently run simulations, load tests, and security scans to uncover hidden edge cases and vulnerabilities long before deployment. When tests fail, repair agents attempt automated bug fixes and remediation, dramatically reducing backlog churn and time-to-resolution. This closes the feedback loop: once a defect is surfaced, it is immediately analyzed, repaired, and re-tested in a continuous cycle until resolved or escalated.

That does mean we are ushering in an era of self-correcting systems.

Quite an interesting proposition, and brings up the question of what constitutes a bug.

The new software begins its life with deployment.

Deployment and Maintenance: Autonomous Orchestration

Already now, tools like Jenkins allow for efficient CI/CD. Post-development, Agentic AI systems would enable continuous and adaptive deployment and maintenance without manual oversight. Here, maybe specialized agents that provision infrastructure (platform as a service), manage cloud deployments and optimize costs, orchestrate containers, and configure environments autonomously could be value drivers. When anomalies occur, self-healing capabilities may detect, diagnose, and remediate incidents, sometimes reverting systems to stable states in seconds. Proactive monitoring agents correlate metrics and detect early warning signals, proposing long-term system improvements and optimizations.

The outcome could be near-zero downtime, leading to greater operational resilience and the ability to dynamically scale system resources in real time, aligning performance with demand. While that sounds like overkill, think what this means for robotics.

One bottleneck between the steps of the SDLC is media breaks. Interfaces from requirements to design and from design to implementation, and so forth, are not that straightforward to automate.

What if these interfaces are also code?

Code as Interface: The Architecture of Multi-Agent Collaboration

One of the most profound shifts I see when entering the agentic era is the adoption of code as the primary interface. Instead of exchanging vague instructions in natural language, AI agents communicate through well-defined code artifacts, such as standardized scripts, protocol messages, and serialized code objects. Using negotiation and delegation protocols such as Smolagents, Google’s Agent2Agent (A2A), or AG-UI, they then could dynamically coordinate responsibilities, clarify intents, and divide labor across distributed systems. In my opinion, this transforms the Software Development Life Cycle into a multi-agent ecosystem where each agent understands and builds upon the work of others. The result would pave the trail towards true autonomy → agents can evolve codebases continuously, combined with security and traceability, as every transaction is logged and auditable.

This model then also ensures seamless interoperability across human engineers, vendor systems, and in-house automation frameworks.

The key problem here is then “reduced” to orchestration.

And I plan to be writing about OpenAI’s Agentkit, Power Automate, and N8N rather sooner than later.

However, the key showstopper is still:

Compliance, Security, and Auditability

As SDLC processes increase in autonomy, oversight, and governance have to take center stage.

Every agent interaction needs to be logged, auditable, and designed for compliance with stringent industry standards. Security-focused agents continuously monitor for policy violations, anomalous behavior, or non-compliant actions, intervening and documenting their actions in real time. This embedded auditability ensures that agentic systems meet regulatory frameworks such as GDPR, SOX, and HIPAA, making them viable even in highly regulated industries. By merging intelligence with transparency, these systems bring both innovation and accountability to the core of software operations.

In Closing

The adoption of agentic AI redefines both organizational dynamics and the developer role itself. Developers evolve from coders into supervisors, validators, and collaborators—guiding digital teammates rather than micromanaging syntax. Organizations benefit from accelerated delivery cycles, shorter release cadences, and fewer manual handoffs across the SDLC. The result is higher-quality software, fewer post-deployment incidents, and systems that continuously adapt to changing requirements, regulations, and market forces. Agentic AI thus doesn’t just optimize how software is built—it transforms who builds it, turning software development into a collaborative dialogue between humans and machines.

Agentic AI has moved beyond theoretical promise to demonstrable value across every SDLC phase. By automating task orchestration, enforcing rigorous testing and remediation, and enabling code-native, protocol-driven collaboration, Agentic AI fundamentally reshapes both the productivity and resilience of software delivery.

The rise of code as interface means that agents can now negotiate, delegate, and iterate as code-aware collaborators—a leap forward that promises massive efficiency gains, greater innovation, and secure, compliant software systems that evolve as rapidly as the markets they serve.

As multi-agent protocols mature and adoption grows, Agentic AI will shift the role of human developers and architects toward higher-order strategic and creative tasks, heralding a new era of intelligent, adaptive, and scalable software development.

References:

• Zencoder.ai: Agentic AI for Full-Cycle Software Development

• AG-UI: the Agent-User Interaction Protocol

• Practical AI Agent Examples (n8n)

• Google: Announcing the Agent2Agent Protocol (A2A)

• Agent Protocol (agi-inc)

• DhiWise: Agentic AI Applications in Modern Software Development

Read more

Read more

Recently, we have seen physical AI models misbehaving when interacting with the real world.

1. Industrial Humanoid Robot (Unitree H1) Goes Berserk During Testing in China

2. Tesla’s Robotaxi Going Rogue in Austin.

3. The US Navy’s Autonomous Drone program faces setbacks.

As companies like Pony.ai expand around the world, and new entrants like Tensor Auto start building robot-taxis, the stakes are only ever increasing. The challenge these companies face, teaching artificial intelligence human-like real-world intuition, has long been a barrier in developing and especially scaling robotics and autonomous systems. Training robots has suffered from an inability to generalize for a very long time. And while there has been some progress (TRI, X-Embodiments), replicating the nuanced understanding we humans have of physical interactions, object permanence, gravity, or friction, remains unmatched.

But this might all change now. Because NVIDIA’s Cosmos Reason1-7B just topped Huggingface’s physical reasoning leaderboard. Benchmark scores confirm a strong performance: An average score of 65.7 across key robotics and autonomous vehicle tasks. Although we should consume such benchmarks with a pinch of salt.

Let’s dive into the fantastic world of Vision-Language-Action models.

Subscribe now

Table of Contents

1. Architectures of Physical AI

2. Challenges of Physical AI

3. Visual Language Action Models in general

4. NVIDIA’s Cosmos World Foundation Model (WFM)

5. NVIDIA’s Cosmos-Reason1,

6. Meta’s Vision Language World Model,

7. Wayvve’s Lingo-2, and

8. EO Robotics EmbodiedOneVision

Architectures of Physical AI

It’s nothing new that by now, large language models are really good at processing text. Some even display impressive multi-modal processing capabilities. I.e., you can upload an image into ChatGPT and it will actually “understand” it.

But that does not mean that these models are capable of translating words into physical actions. A more sophisticated architecture is needed for this.

So, before we dive into VLA models, I will outline more explicitly how the architecture for building Physical AI systems works and what challenges this brings along with it. Then I will dive into the most recent models and papers published.

We begin, similar to my traditional autonomous agents definition, with sensors.

Sensor Layer

Sensors form the foundation of this architecture as Physical AI relies on multiple sensor modalities to gather comprehensive environmental data. Multiple cameras provide visual information for object recognition and scene understanding from different angles. Although Tesla got rid of them, I still believe in Radar and LIDAR. Radar offers reliable distance and velocity measurements in various weather conditions, while LiDAR generates 3D point clouds for spatial mapping. Other sensors include GPS provides global positioning data, inertial measurement units, or ultrasonic sensors. This multi-sensor approach unlocks redundancy and delivers robustness in data collection

Perception layer

In the perception layer, the raw sensor data is processed into meaningful interpretations of the driving environment using specialized perception models and sensor fusion algorithms.

In traditional autonomous systems, we may deploy the following standard models:

Specialized perception models are task-specific algorithms where lane detection uses techniques like Hough transforms or deep learning to identify road boundaries and lane markings, traffic light detection employs convolutional neural networks to recognize signal states and timing, and traffic sign detection interprets regulatory and warning signs through trained classification models. Object detection & tracking identifies and monitors vehicles, pedestrians, and obstacles using models like YOLO or R-CNN variants, providing real-time recognition capabilities. Free space detection determines drivable areas through semantic segmentation or point cloud processing algorithms.

Sensor fusion models like Kalman filters and Extended Information filters combine a variety of sensor data with high-density maps to precisely determine the vehicle's position within the mapped environment, creating a more robust understanding than any single sensor could provide alone.

Planning layer

But autonomous driving is not only about understanding the “now”. Using the interpreted environmental data, we still need to know where we are going.

Thus,

• Route planning for determining the optimal path to the destination.

• Prediction algorithms for forecasting of the future behavior of other road users.

• Behavior planning for deciding high-level maneuvers like lane changes or turns.

  and finally

• Trajectory planning for calculating the specific path and speed profile the vehicle should follow, considering safety constraints and traffic rules.

And especially, trajectory planning is where VLAMs shine. But I am getting ahead of myself.

Control Layer

The fourth and final stage translates planned trajectories into actual vehicle motion. PID controllers manage basic control loops for steering, throttle, and braking. Model predictive control provides more sophisticated control algorithms that anticipate future states. Other control systems may include stability control, traction management, or specialized actuator control systems that physically execute the planned maneuvers through the vehicle's mechanical systems.

Challenges of Physical AI

We don’t have easily scalable and trainable autonomous vehicles yet. Just an observation. We humans are incredibly good at judging driving situations we have not encountered yet. But that is also a function of age. There is a reason why we don’t let kids drive (besides their obvious height constraints).

Problem 1: Edge cases are incredibly valuable.

Data on edge cases is incredibly difficult and expensive to acquire. The more rare the event, the more likely it is that the system might misbehave and cause physical harm. And as we can see with the fleet of Google Street Map cars, physical data is costly to collect, curate, and label.

Problem 2: The real world is a spatial and temporal environment bound by the laws of physics.

Spatial data is also a hard problem because the environment in a spatial 3D environment has objects have different shapes, sizes, and distances, while the car’s sensors might perceive them in 2D as the same. They also have to comply with basic rules of physics. Reconciling multi-camera input makes this also more complex. And then a lot of what we observe is constantly moving. Either as we pass along houses or factory floors. Or they move themselves, like pedestrians, vehicles, or drones. Therefore, decisions can’t be just about “where things are now” but “where they will be in the next second, 5 seconds, or 30 seconds.” On top, prediction errors compound quickly, making safe planning even more difficult.

Problem 3: Output modality needs to be incredibly accurate.

Even when perception and predictions are accurate in a volatile environment, the system must translate that understanding into actions. I.e., steering, braking, accelerating, grasping, or rerouting, each with different latency, precision, and safety requirements. The importance of high-quality output modality is as critical as getting the input right.

Problem 4: Trust & explainability in decision-making.

Humans can explain their decisions with natural language. But to have one human describe the actions of another human leads to no trust in the explanation. There is a reason my moderators at sports events are largely for entertainment. This is known as the back-seat driver problem: when one agent tries to explain the decisions of another, the explanation rarely inspires trust.

Btw., this also outlines the importance of agency in agentic decision making.

Visual Language Action Models

Visual Language Action Models (VLAMs) are AI systems that integrate visual perception, natural language understanding, and action planning to enable agents to interpret their environment, follow language instructions, and perform corresponding actions.

The benefit of integrating VLAMs into physical AI architectures is that it allows the system to not only perceive its environment but also interpret complex scenarios, make contextual decisions, and potentially communicate its intentions—capabilities that mirror human-like driving behavior. Cosmos can generate photorealistic, physically based, synthetic data.

Traditional autonomous systems have relied on sensor fusion and rule-based decision-making. The use-cases for VLAM are different:

1. Data curation & annotation: To reduce cost and increase quality, we need to automate high-quality dataset curation and annotation. VLAMs can interpret raw multimodal inputs, generate candidate labels, and validate them against schema constraints, reducing reliance on manual annotation.

2. Robot planning & reasoning: Planning and reasoning can guide deliberate, methodical decision-making with vision language action (VLA) models. VLAMs unify perception and instruction-following to translate natural language goals into structured action plans that remain robust under real-world uncertainty.

3. Video analytics AI agents: Extract actionable insights and perform root-cause analysis on massive video datasets. VLAMs integrate visual understanding with temporal reasoning and language-based explanation, enabling both anomaly detection and human-readable causal narratives.

The advancement here lies in the system's ability to understand nuanced real-world situations through language, with the goal of improving how vehicles handle unpredictable real-world conditions. For the transportation & logistics industry, this will be a revolution. But for us normal mortals, it means safer and more reliable autonomous vehicles.

Share

We begin with:

NVIDIA’s Cosmos World Foundation Model (WFM)

A couple of months ago, right after CES, I already touched on NVIDIA’s Cosmos World Foundation Model (WFM). I hold the belief that synthetic trajectory data is an important step towards the generalization of autonomous movement. WFM builds on this concept by establishing a pre-trained (“vanilla”) model, not unlike traditional LLMs, trained on Internet-scale datasets providing broad time-frozen knowledge across many environments and tasks. However, to perform well in a specific real-world application, such as accurately guiding a robot on a factory floor or safely navigating an autonomous vehicle through traffic, general capability is not good enough. We must fine-tune them in post-training on a smaller, targeted dataset collected from their particular environments.

This process produces a post-trained, specialized WFM: a model that retains the broad capabilities of its foundation training while being adapted to go deep into a specific Physical AI use-case. WMF in its current (Fall 2025) iteration has evolved into a suite of open, physics-aware generative AI models, including diffusion and autoregressive transformer architectures, designed to simulate and predict future states of virtual environments (via video) from inputs like text, images, or past frames.

These language models may also augment vehicle-to-infrastructure communication, enabling more efficient traffic management and coordinated mobility systems.

Within WMF, there are specialized models. Cosmos-Reason1 is one of them.

Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA's Cosmos Reason1-7B model aims to be an evolution in the domain of Physical AI through embodied reasoning. Embodied reasoning refers to the ability of an AI or robotic system to ground its reasoning and decision-making in a physical context, using sensory input and interactions with the environment. So, instead of abstract reasoning alone, it integrates perception, action, and feedback to plan and adapt in real-world settings.

Cosmos-Reason1 employs a multimodal decoder-only architecture similar to LLaVA, available in two model sizes: Cosmos-Reason1-7B and Cosmos-Reason1-56B. The architecture follows a sequential pipeline where the input video is processed through a vision encoder, then a projector that aligns visual tokens with text embeddings, before being fed into the LLM backbone. The larger 56B model incorporates a hybrid Mamba-MLP-Transformer architecture to improve efficiency in handling long sequences, combining the linear-time complexity benefits of Mamba's selective state space models with Transformer layers for comprehensive long-context modeling.

On a simple step-by-step workflow basis, Cosmos-Reason1 works like this:

1. It accepts the video input,

2. It converts video into an understandable format, and then

3. It tries to reason about it through long chain-of-thought thinking processes before generating a response in language space.

This response will be returned in natural language and includes both explanatory insights and but also embodied decisions (pose, trajectory, etc). In my opinion, what makes Cosmos Reason1 unique compared with traditional VLMs is its ability to infer and reason using physical common-sense knowledge.

So what’s the magic here?

Statistical models have no true “understanding” of the world they exist in. Humans and animals acquire a physical common sense through passive observation of the world around us. For example, infants can understand basic concepts such as object permanence and gravity already in a few months after birth. To develop this understanding computationally, models must be trained on expensive human-annotated datasets that capture object interactions, spatial relationships, and causal dynamics, enabling them to learn predictive patterns about how objects behave in three-dimensional space through labeled examples of physical phenomena.

However, unlike passive understanding, reasoning in embodied AI must be grounded in action. I.e., reasoning emerges through the coupling of perception and control layers, where the system learns by acting on the environment, observing outcomes, and adjusting its internal models accordingly.

As the real world is temporal, spatial, and bound by the laws of physics (Problem 2), Reason1’s “common sense” is grounded on an ontology of three main categories.

As shown in the image below.

This shall enable the model, and the robots it steers by extension, to not only comprehend what they currently observe but also to plan intelligently ahead.

But there is more to it.

Specifically, reasoning in physical space requires the capability to:

• Process complex sensory inputs. Unlike symbolic reasoning that relies on clean, structured data, embodied reasoning must parse noisy, partial, and ambiguous multimodal signals to extract reliable patterns for decision-making.

• Predict action effects. Every action alters the environment, so embodied reasoning requires an internal model of cause-and-effect. E.g., how objects respond to force, how a robot’s kinematics interact with terrain, or how vehicle dynamics shift under changing conditions.

• Respect physical constraints. Beyond abstract optimization, embodied reasoning must respect real-world physics, inertia, friction, and material limits while producing long-horizon plans that are both safe and efficient in execution.

• Learn from interaction. Actions generate feedback loops, and embodied reasoning depends on integrating this feedback to refine policies over time, enabling continuous improvement and dynamic adaptation to changing environments.

Here are some examples from the paper.

Cosmos-Reason1 introduces several key innovations for the advancement of physical AI reasoning capabilities. The system establishes hierarchical physical reasoning ontologies that systematically organize capabilities across space, time, and fundamental physics domains, providing a structured framework for understanding and evaluating physical common sense.

It incorporates self-supervised intuitive physics tasks such as spatial puzzles, arrow-of-time detection, and object permanence testing that are both scalable to generate and verifiable through rule-based rewards. The framework enables cross-embodiment reasoning by applying unified reasoning principles across diverse physical agents, including humans, robotic arms, humanoid robots, and autonomous vehicles.

And finally, the system features a novel asynchronous reinforcement learning infrastructure with fault tolerance and dynamic scaling capabilities, achieving approximately 160% improvement in training efficiency compared to traditional colocated frameworks while maintaining robustness through automated recovery and resource management.

Meta’s Vision Language World Model

Meta’s Vision Language World Model (VLWM) is designed to enable AI agents to plan and reason about actions using natural video inputs. “World model” facilitates this by allowing agents to internally optimize action plans, reducing the reliance on exhaustive trial-and-error in real environments.

Given a video, VLWM aims to extract a structured language representation consisting of two key components:

1. Goal: A description and interpretation of the desired outcome.

2. Procedural Plan: An action-state sequence outlining the steps to achieve the goal.

For such a video-to-text extraction task, a straightforward approach might involve providing a Vision-Language Model (VLM) with the full video and prompting it to extract these language representations. However, this approach encounters the "impossible triangle": achieving high spatial resolution for fine-grained perception, maintaining a long temporal horizon that spans many procedural steps, and utilizing a large and intelligent VLM capable of following complex instructions.

To address this challenge, VLWM employs a two-stage strategy:

1. Compression into a Dense Tree of Captions: The input video is compressed into a dense Tree of Captions, significantly reducing data volume while preserving essential semantic information.

2. Extraction of Structured Goal-Plan Representations: Structured goal-plan representations are extracted from these captions using large language models (LLMs). Since this stage operates purely on text, it enables efficient processing with large LLMs and allows for iterative quality refinement through Self-Refine techniques.

This approach enhances the efficiency of video understanding and sets a new standard in AI's capability to plan and reason based on visual data.

What makes VLWM unique is its dual-mode planning system that mirrors the psychological distinction between fast, intuitive thinking and slow, deliberative reasoning. Reminds me of HRM.

System-1 Planning operates as the model's reactive mode, generating plans through direct autoregressive text completion. When presented with visual context and a goal description, VLWM first interprets the goal by predicting both the initial world state ("Now, the kitchen is set up with necessary ingredients...") and the expected final state needed for achievement ("To achieve the goal, the eggs need to be cooked and mixed with tomatoes..."). The model then generates an interleaved sequence of actions and world state changes in a single forward pass, where each action, like "Crack eggs into a bowl and whisk them together," is immediately followed by detailed descriptions of environmental transformations. These world state descriptions function as internal reasoning chains that help track task progress, but the autoregressive nature creates a critical limitation—once an action token is generated, it becomes irreversible, potentially leading to error accumulation in complex scenarios.

System-2 Planning addresses these limitations by introducing reflective reasoning through internal trial-and-error with the learned world model. Instead of committing to a single action sequence, System-2 generates multiple candidate plans and uses VLWM to simulate their effects, predicting the resulting world states for each possibility. A separately trained critic module—a 1B parameter language model—evaluates the semantic distance between these predicted outcomes and the desired goal state, assigning costs that reflect how well each candidate plan advances toward the objective. The system then selects the action sequence with the lowest predicted cost, effectively performing reasoning by searching through the space of possible futures. This critic is trained through self-supervised learning using ranking constraints that teach it to assign lower costs to meaningful progress and higher costs to irrelevant or procedurally incorrect actions, enabling the model to optimize plans without requiring explicit reward annotations.

First, it constructs training samples by taking base partial trajectories and appending either valid next steps from coherent task continuations or distractor steps sampled from unrelated tasks, training the model to satisfy ranking constraints where valid continuations receive lower costs than irrelevant additions. Second, it generates negative samples by randomly shuffling steps in base trajectories, teaching the critic to assign higher costs to procedurally incorrect sequences and ensuring sensitivity to temporal coherence. This dual approach enables the critic to distinguish meaningful progress from both irrelevant distractors and temporally disordered actions. The architecture benefits from VLWM's video-text formulation, which allows initialization from pretrained vision-language models like PerceptionLM-8B, inheriting strong visual perception capabilities along with language understanding and commonsense knowledge from large language models, while the language-based world state representation provides computational efficiency and interpretability compared to pixel-based generative approaches.

Wayvve’s Lingo-2 - A closed-loop Vision-Language-Action-Mode

Wayve's LINGO-2 is a closed-loop Vision-Language-Action Model (VLAM) that integrates visual perception, natural language processing, and driving control to provide real-time explanations of autonomous driving decisions. Unlike its predecessor, LINGO-1, which operated as an open-loop system offering commentary without influencing vehicle behavior, LINGO-2 combines vision and language inputs to generate both driving actions and explanatory text. This integration allows the model to adapt its behavior based on natural language prompts and provide continuous commentary on its decision-making process, enhancing transparency and trust in autonomous driving systems.

The architecture of LINGO-2 comprises two primary components: a vision model and an auto-regressive language model. The vision model processes sequences of camera images into tokens, which, along with additional contextual information such as route, speed, and speed limits, are fed into the language model. The language model then predicts both a driving trajectory and corresponding commentary text. The vehicle's controller executes the predicted trajectory, enabling real-time interaction and adaptation to dynamic driving scenarios.

A distinctive feature of LINGO-2 is its ability to respond to natural language instructions and queries during operation. For instance, passengers can ask the system about traffic light statuses or request specific maneuvers, and LINGO-2 will adjust its actions accordingly while providing explanations for its decisions. This closed-loop interaction between vision, language, and action represents a significant advancement in autonomous driving technology, offering a more intuitive and interpretable interface for users.

How does LINGO-2 work?

LINGO-2 “Driving with language” integrates vision, language, and action into a unified framework. It comprises two primary modules: a vision model that processes camera images of consecutive timestamps into a sequence of tokens, and an auto-regressive language model that predicts a driving trajectory and generates commentary text. These components work in tandem to enable real-time interaction between the vehicle and its environment, facilitating both driving behavior and explanatory dialogue.

The integration of language and driving behavior in LINGO-2 introduces several capabilities that enhance human-vehicle interaction and trust in autonomous systems. Passengers may, in the future, issue natural language commands, such as "turn right" or "find a parking spot," prompting the vehicle to adapt its behavior accordingly. Although the latter example is more likely than the former. Additionally, the system can provide real-time debugging explanations of its driving decisions, answering questions like "Why did you slow down?" or "What is the current speed limit?". This is incredibly important if you need to understand how these systems make decisions.

This approach enables easier testing in synthetic environments where identical scenarios can be evaluated with different linguistic instructions, providing unprecedented insights into AI behavior and decision-making processes

EO Robotics EmbodiedOneVision

EmbodiedOneVision (EO-1), a unified embodied foundation model paired with the large-scale EO-Data1.5M dataset. Together, they represent a significant step toward building generalist embodied agents capable of understanding, reasoning, and acting across diverse environments. Unlike prior work that treats perception, reasoning, and control as loosely connected modules, EO-1 is also designed as a closed-loop system where vision, language, and action are interleaved during pretraining. This approach enables the model to predict actions in context, respond to language prompts, and reason about spatial and temporal dynamics within real-world tasks.

The system is motivated by advances in multimodal foundation models, which have shown that joint training across modalities yields superior generalization and reasoning. EO-Robotics extends these ideas into the embodied domain, emphasizing how robots can learn not only from static vision–language corpora but also from temporally grounded interaction data. With its unified model architecture, carefully curated dataset, and principled evaluation benchmark, EO-Robotics provides a foundation for the next generation of embodied intelligence research.

Architecture

At the core of EO-Robotics lies EO-1, a decoder-only transformer with approximately three billion parameters. The model integrates multiple modalities into a single shared representation space, handling images, video frames, text, and robotic actions without requiring separate specialized modules. This design departs from earlier Vision-Language-Action (VLA) model approaches that introduce action-specific heads or auxiliary training objectives.

EO-1 employs two complementary training mechanisms:

• Auto-regressive decoding for discrete modalities such as text and symbolic tokens.

• Flow-matching denoising for continuous robotic action trajectories.

Both mechanisms are unified through causal attention over the entire multimodal sequence, enabling EO-1 to capture dependencies between perception, reasoning, and action. The architecture is initialized with a pre-trained vision–language backbone, giving it broad perceptual and linguistic priors, and then adapted to the embodied domain by training on EO-Data1.5M.

To bridge action and non-action modalities, EO-1 incorporates lightweight multilayer perceptrons (MLPs) that encode and decode continuous motor actions into the tokenized sequence. This avoids retraining large action-specific modules from scratch, improving efficiency and enhancing cross-modal transfer. By aligning text, vision, and action tokens in the same autoregressive stream, EO-1 treats reasoning and acting as a single process rather than separate stages, resulting in more coherent robotic control.

Capabilities

EO-1’s abilities stem from both its architectural design and its large-scale training data. EO-Data1.5M provides over 1.5 million multimodal samples, interleaving vision, text, and robot actions. The dataset combines two key sources:

1. Web-based vision–language corpora, which provide general visual-linguistic grounding.

2. Real-world robot episodes supply action-level continuity, spatial grounding, and temporal structure.

A custom data pipeline ensures diversity, quality, and task coverage. Robot videos are filtered and clustered to reduce redundancy, split into subtasks, and captioned by both pretrained vision–language models (VLMs) and human annotators. From these subtasks, question–answer pairs are generated to probe temporal and spatial reasoning. Human annotators refine answers to ensure correctness, and the resulting interleaved sequences integrate visual tokens, linguistic reasoning, and motor actions.

Through this training regime, EO-1 acquires capabilities in several areas:

• Spatial understanding: object localization, multi-view correspondence, and trajectory reasoning.

• Task reasoning: planning sequences of actions and evaluating task progress.

• Physical commonsense: reasoning about forces, constraints, and counterfactual outcomes.

• State estimation: assessing object states such as open/closed or full/empty, and predicting the results of actions.

Performance is measured through EO-Bench, a benchmark constructed to avoid the pitfalls of conflated evaluation tasks. EO-Bench includes 648 QA pairs across four categories—spatial understanding, physical commonsense, task reasoning, and state estimation—allowing precise measurement of strengths and weaknesses. Unlike other benchmarks that mix multiple reasoning aspects in a single query, EO-Bench ensures that each question isolates a specific embodied reasoning skill, making evaluations interpretable and reliable.

Charming points

EO-Robotics distinguishes itself from prior embodied AI frameworks through three defining elements:

1. Unified Multimodal Transformer
   EO-1 processes text, vision, and actions in a single stream rather than through disjointed modules. This unified design improves alignment across modalities and enables coherent reasoning-to-action pipelines.

2. Interleaved Embodied Dataset
   EO-Data1.5M structures multimodal episodes in temporal order, linking perception, reasoning, and action. The inclusion of flexible interleaved QA pairs with robot actions enriches cross-modal grounding and provides the model with nuanced spatial–temporal reasoning skills.

3. Principled Benchmarking with EO-Bench
   EO-Bench introduces a structured evaluation suite that disentangles reasoning categories. This not only allows researchers to identify bottlenecks in embodied intelligence—such as spatial reasoning, which is emphasized in the benchmark—but also sets a standard for transparent and interpretable comparisons.

Together, these design choices create a toolchain that goes beyond conventional VLAM approaches. EO-1 does not merely predict actions from observations; it integrates reasoning, perception, and control into a single generative process. This positions EO-Robotics as a platform for generalist embodied intelligence, where robots can flexibly adapt across tasks, environments, and modalities without requiring task-specific retraining.

EO-1 is my final example of how embodied foundation models are moving beyond narrow perception or control, toward systems that unify vision, language, and action into generalist frameworks for reasoning and robotics.

In Closing

Read more

Read more

Maybe the whole concept of language-based reasoning is wrong?

China and Singapore-based Research Lab Sapie…

Read more