Multi-agent reinforcement learning / artificial life · 2026

Digital Terrarium: multi-agent RL survival world

One small neural network, shared by 30 agents, keeps 86.3% of them alive in a simulated survival world, against 62.0% for a hand-written strategy and 0% for random play.

Context: Independent project
Year: 2026
Stack: Python, NumPy, PyTorch, Double DQN, FiLM conditioning

86.3%

agents surviving with the trained network

± 9.2% across 10 world layouts

62.0%

survival with a hand-written greedy strategy

± 22.8%, same worlds

0.0%

survival with random play or the untrained network

the untrained network has the same design

The project’s 3D viewer showing a live run of a smaller 48 × 48 world with births turned on. A new agent is born partway through.

Overview

Problem

Can one small neural network, shared by every creature in a simulated world, learn to keep the population alive when food and water have to be found? And how do you separate learned behaviour from what the world's rules guarantee?

Approach

Built a 64 × 64 grid world with food, water and terrain in Python; the same seed always replays identically.
Trained one 55,343-parameter network, shared by all agents, for 65 million steps with Double DQN, a reinforcement-learning method that learns from trial and error. An action mask rules out impossible actions.
Each agent sees only its surroundings and picks one of 15 actions.
Tested on 10 world layouts with 30 agents, against a hand-written greedy strategy, random play and the same network untrained.

Results

The trained network kept 86.3% of agents alive, against 62.0% for the greedy strategy and 0% for random play and the untrained network. It forages nearby (a median 8.9 cells from food 100 steps before eating it, against 21.4 for greedy) and leaves only 28.1% of food standing, against 95.1% under random play.

The comparisons also showed what was not learned. Near-perfect eating and drinking accuracy comes from the action mask, not from training: the untrained network scores the same. Moving toward food is only slightly better than random (0.427 vs 0.394). All trained results come from one training run, and no social behaviour was seen.

What I built

Built a deterministic 64×64 survival world in Python and NumPy, where one 55,343-parameter network controls every agent.
Trained the network with reinforcement learning (Double DQN) for 65 million steps, with an action mask that rules out impossible moves.
Tested it against hand-written, random and untrained controls on 10 worlds, separating what was learned from what the world’s rules already provide.

Results in detail

One agent, one tick

Digital Terrarium — observation → policy → action → world

OBSERVATION 161 floats
needs hunger thirst energy health
time sin/cos of day
4 × nearest food (dx,dy,dist,mask)
4 × nearest water
4 × nearest agent
5×5 terrain patch, 2 channels
+ genome, emotions, social memory, affordance bits, previous action
POLICY 55,343 params
trunk 153 → 128 → 128 ReLU
head 128 → 15 Q-values
FiLM genome 8 → 32 → 512, scales both hidden layers
frozen: eval(), no grad, argmax.
One net, shared by all agents.
ACTION 1 of 15
8 compass moves
EAT DRINK REST NOOP
GIVE ATTACK MATE
HARVEST, TAKE, DROP exist in the world, not in this head
WORLD
move — water is impassable
resolve EAT, DRINK, social
physiology
death check
next observation, one tick later

MESSAGE
not implemented — agents/language.py is a stub
No signal exists in the action enum, the observation, or the world resolver.

CONSTRAINTS, WHERE THEY BITE

perception radius 8.0 cells: nothing outside it ever enters the vector
action mask: EAT, DRINK, GIVE and MATE are removed when out of reach
reward = need deltas only: 1.0·Δhunger + 1.0·Δthirst + 0.25·Δenergy + 2.0·Δhealth − 10.0 on death
no social reward: the contract forbids it

Figure 1. One agent’s turn: what it sees, the shared network, the action it picks and the world’s response. The dashed message path is not built.

Six behaviours, four conditions, ten world seeds

Each behaviour is a counting rule over world state, fixed before measuring. All four conditions run the same ten seeds (501–510). 30 agents, 3,000 ticks per run. Small marks are individual seeds; the bar is their mean. “untrained” is the SAME network and the SAME action mask at random initialisation — the training-step-0 control.

Survival

fraction of the 30 founders still alive at tick 3,000

trained

86%

scripted

62%

random

untrained

Sitting at water

fraction of sampled agent-ticks within 1.5 cells of a water cell centre

trained

21%

scripted

32%

random

untrained

33%

Food-directed movement

fraction of sampled moves that REDUCE distance to the nearest active food

trained

0.43

trained does not separate from random here

scripted

0.28

random

0.39

untrained

0.04

Standing food stock

mean fraction of the 60 food items active over the run's second half. Lower = more eaten

trained

28%

scripted

70%

random

95%

untrained

100%

Travelled to its food

median distance, in cells, from the eater to that item's position 100 ticks earlier

trained

8.9

scripted

21.4

random

4.5

untrained

22.7

Nearest-neighbour distance

mean distance, in cells, from each living agent to the closest OTHER living agent

trained

6.6

scripted

6.1

random

12.2

untrained

4.4

NOT A TRAINING CURVE

Behaviour cannot be plotted against training steps: every intermediate checkpoint was deleted on 11 September 2026. Only the episode-550 best survives.

The trained condition is ONE training seed on ten world seeds. Spread is across worlds, not across training runs.

Figure 2. Six behaviours measured under four conditions on the same ten worlds. Moving toward food is flagged: there the trained network is no better than random.

What the mask does, and what the weights do

Interaction precision is the fraction of EAT (or DRINK) presses the world resolved into an actual eat or drink event. Successes come from the world's event log, never re-derived from geometry. Pooled over ten seeds, 3,000 ticks, 30 agents.

EAT

trained policy3,399 of 3,405 presses landed

99.8%

untrained, same mask12 of 12 presses landed

100.0%

random, no mask408 of 26,842 presses landed

1.5%

DRINK

trained policy15,281 of 15,281 presses landed

100.0%

untrained, same mask2,105 of 2,105 presses landed

100.0%

random, no mask1,614 of 26,774 presses landed

6.0%

HOW TO READ THIS

The trained and untrained rows are the same to within a rounding error. Both run under cognition/masking.py, which removes EAT when no food is within 1.0 cell and DRINK when no water is within 1.5 cells. Near-perfect precision is therefore a property of the ARCHITECTURE, not something the weights learned. The random row, which ignores the mask, shows what the same presses are worth without it. Read the DRINK column for the comparison: the untrained net pressed DRINK 2,105 times, so its 100% is not a small-sample artefact. Its 12 EAT presses are, and that row proves nothing on its own.

Where the weights do show: the untrained net pressed EAT 12 times in 30 runs; the trained net pressed it 3,405 times and ate.

Figure 3. Near-perfect eating and drinking accuracy comes from the action mask, not from training: the untrained network scores the same.

Figure 4. The 3D viewer with the run’s live numbers set beside it.

In depth: the 2D simulation view

The same simulation drawn straight from the world arrays: every agent, food item and water cell, one frame per tick.

Top-down map of a cream-coloured 64 by 64 grid world with pale sage water regions, small grey food squares and about thirty dark green dots representing agents, one of them rust-coloured. — Figure 5. Thirty agents in the 64 × 64 world, drawn straight from the simulation’s data. The tracked agent is highlighted.

Figure 6. Eight seconds of the trained network running, one frame per step. The highlighted agent starves partway through and is marked with a cross.

Labelled map of a 64 by 64 grid with sage water regions and scattered food squares, annotated with the rules for water, food, perception and death. — Figure 7. The test world’s terrain, water and food, drawn from the simulation, with the rules that limit what an agent can do listed below.

Side-by-side top-down maps of the same grid world; the left has green agent dots and few food squares, the right has no agents and many food squares. — Figure 8. The same world at the same moment under two strategies: 26 of 30 trained agents alive and the food mostly eaten; the random population gone and the food untouched.

Figure 9. The same 160 steps as the 3D viewer clip, drawn in 2D. Both views were checked to show the same agents in the same places.

Figure 10. Both views of the same eight seconds, 3D on the left and 2D on the right. Their birth and death counts differ because the viewer counts only what it has watched.

Zoomed grid showing a rust dot pressed against a sage water region with a short looping trail, beside a text panel listing its final needs. — Figure 11. A documented failure: one agent kept trying to walk into water it could not cross, and starved beside the water it kept drinking from.

Technical details

The full list of measured results behind this project.

Show all 15 measurements

Measurement	Value
Survival, trained policy (10 world seeds)	86.3% ± 9.2%
Survival, scripted greedy baseline	62.0% ± 22.8%
Survival, random baseline	0.0%
Survival, untrained net, same architecture and mask	0.0%
Policy parameters	55,343
Training, environment steps	65,051,937
Training, gradient steps	3,585,106
EAT precision, trained (mask is on)	99.8% (3,399 of 3,405 presses)
DRINK precision, untrained net under the same mask	100.0% (2,105 of 2,105 presses)
Food-directed movement, trained	0.427 (random baseline 0.394; 0.5 = undirected)
Standing food stock, trained (second half of run)	28.1% of 60 items (random 95.1%)
Median distance to the food it ate, 100 ticks earlier	8.9 cells (scripted greedy 21.4)
Food eaten that did not exist 100 ticks earlier	55.9% (scripted greedy 26.8%)
Mean nearest-neighbour distance, trained	6.6 cells (random 12.2, untrained 4.4)
Default test suite	4 failed, 1953 passed, 2 skipped, 2250 deselected

Limitations (11)

Population size is 30 agents per run; the policy was trained on episodes of 16-30 agents. Nothing here speaks to larger populations.
Run length is 3,000 ticks. The project's own longer runs (30,000 ticks) reported extinctions in many world configurations; that is not measured here.
Every trained number comes from ONE training seed. The two replication training seeds were deleted on 11 September 2026, so the spread shown is across world seeds only, never across training runs. Treat single-training-seed results as provisional.
The behaviour figure is a comparison across conditions, not a curve against training steps: intermediate checkpoints no longer exist.
The untrained control is degenerate. A randomly initialised Q-net under argmax is near-deterministic (0.66 bits of action entropy versus 3.10 for the trained net), so it is a valid floor and a valid isolation of the action mask, but a poor behavioural comparator. The random policy is the better movement null.
There is no communication, language or signalling component. agents/language.py and agents/beliefs.py are docstring-only stubs, so no message analysis of any kind is possible.
GIVE was selected on 19 of 857,163 trained decisions and MATE on none. No social behaviour was observed, and none is claimed.
The clip shows one seed and one interval, chosen because a single agent's failure is legible in it. It is not a random sample of the run and it does not loop.
The agent-4 starvation case is one agent in one seed. The aggregate it illustrates - 25 starvation deaths across 300 trained agents - is the measured number; the mechanism shown is that one agent's.
The scripted greedy comparison is in this world and this config only. The project's own earlier benchmark reports the opposite ordering for an earlier checkpoint, and the two are not comparable.
The default test suite has 4 failures and deselects 2,250 of 4,209 collected tests by default; the deselected research lane fails on artifacts deleted in September.