When the cooling fails: teaching a 7B model to run a datacenter

A datacenter is a building full of heat. Every server turns the power it draws into heat, and a row of cooling units carries it away. When one of those units fails, someone in the operations center sees the alert, reads the dashboard and decides what to do: which unit to inspect, which survivors to push harder, which racks to shed. When the utility power drops, batteries carry the load while a diesel generator starts, and the clock is running.

I wanted to know whether a small language model could learn that job. Not talk about it, but read a live dashboard and type commands that a physical system obeys. Fluent text earns nothing here. A command that sounds right but names the wrong unit wastes a step, and the temperatures do not wait. The physics decides what worked.

That rules out the usual shortcuts. There is no dataset of expert operators working through these incidents, and nobody rehearses a cooling cascade on real hardware. So the project builds the whole chain: a physics simulator of a small facility, a dataset written by a larger reasoning model playing inside it, and a 7B model trained first to imitate that teacher and then to practise against the live simulation. This post walks through all three, the head-to-head results, and what a closer look at the evaluation says about them.

A datacenter in a text box

The simulator, DC-Ops, is built on OpenEnv1, an open framework for packaging the environments that agents learn in. It models a small facility: two zones with twenty racks and 160 kW of IT load, four cooling units known as CRACs, two UPS battery systems, a diesel generator and a transfer switch between utility and generator power. Temperatures follow a lumped thermal model: each zone stores heat, the servers add it, the cooling units remove it, and the walls exchange it with the outside. Batteries drain while they carry the load, and the generator needs time to start.

The agent gets no state vector. It reads the same text dashboard an operator would, with the alert, the state of every unit, temperatures and power readings, and answers with one of fourteen commands, such as diagnose CRAC-3 or set_fan_speed CRAC-2 100. Each command advances the simulation, and the next screen shows what happened.

What the model reads

Action Result: Environment initialized. Awaiting your command.

Steps Remaining: 15

DC-OPS MONITORING DASHBOARD Step: 0/15

!! ALERT: CRITICAL: CRAC-3 compressor failure detected. Zone B temperatures rising. Investigate and stabilize.

COOLING UNITS

Cooling units

Unit	Status	Setpoint	Supply	kW
CRAC-1	RUNNING	18.0°C	18.0°C	29.1
CRAC-2	RUNNING	18.0°C	18.0°C	29.1
CRAC-3	!! COMPRESSOR	18.0°C	---	0.0
CRAC-4	RUNNING	18.0°C	18.0°C	29.1

ZONE TEMPERATURES

Zone temperatures

Zone	Cold Aisle	Hot Aisle	Max Inlet
zone_a	19.8°C	30.8°C	19.8°C
zone_b	19.8°C	30.8°C	19.8°C

POWER

IT Load: 160.0 kW | Cooling: 87.2 kW | PUE: 1.73

Utility: NORMAL | Gen: OFF | ATS: UTILITY

What it writes back

<reasoning>

1. Situation: CRAC-3 compressor failure detected, causing potential cooling issues in Zone B.

2. Constraint: Operational procedure requires diagnosing faulty units before compensating.

3. Step: Assess -> diagnose.

4. Action: Diagnose CRAC-3 to understand the extent of the failure and determine next steps.

</reasoning>

<command>

diagnose CRAC-3

</command>

Six scenarios cover cooling and power at three levels of difficulty, from units that overcool the room to a double cooling failure and a full loss of utility power. Each scenario injects its fault and decides for itself when the incident counts as resolved, and every step earns a reward of six parts: thermal safety, power safety, efficiency, progress, procedure and the quality of the action. The simulator is open source and runs live on Hugging Face Spaces, so you can try an incident in your browser.

Cooling

1. A1 Easy

   Cooling setpoint optimization

   Every cooling unit set to 15°C, wasting energy

2. A2 Medium

   Thermal event response

   CRAC-3 loses its compressor

3. A4 Hard

   CRAC failure cascade

   CRAC-1 loses its compressor and CRAC-3 its fan

   Used in the evaluation

Power

1. B3 Easy

   Generator test protocol

   No fault: the monthly generator test

2. B1 Medium

   UPS alarm response

   UPS-1 briefly ran on battery

3. B4 Hard

   Power failure cascade

   Utility power lost, with a slow generator warm-up

   Used in the evaluation

Three steps of training

Without expert logs, the training data has to come from somewhere else. The pipeline has three phases, and each one hands its result to the next.

1. 1 A teacher plays DeepSeek-R1-Distill-Qwen-32B plays live episodes and explains every move 1,083 episodes kept
2. 2 The student imitates Qwen2.5-7B-Instruct learns to write the reasoning and the command 7,107 turns
3. 3 The student practises GRPO: eight replies to each screen, each run in its own copy of the simulator and scored 510 prompts

DC-Ops simulator The teacher plays in it, and it scores every reply during RL

A teacher plays the scenarios

A reasoning model, DeepSeek-R1-Distill-Qwen-32B from the DeepSeek-R1 family2, played live episodes in the simulator. At every turn it wrote a free chain of thought, a short structured reasoning block and one command. Of the 1,520 episodes it played, 1,083 survived filtering, which dropped episodes with no usable turn, an invalid command or an escalation to a human. The teacher was far from perfect: it resolved 179 of the 1,083, about one in six.

A 7B student imitates it

The student is Qwen2.5-7B-Instruct3, fine-tuned through a small LoRA adapter. It is not a reasoning model, so the teacher’s chain of thought was cut out of every turn: the student learns to write the short reasoning block and the command, nothing else. This phase used the 934 episodes from the six scenarios, 7,107 turns in all.

The student practises against the simulator

Imitation copies the teacher, mistakes included. To go further, the student practised with GRPO4, reinforcement learning that compares several attempts at the same problem instead of training a separate critic. Each training prompt is a live screen, captured either at the start of an incident or a few scripted moves in. The model writes eight replies to that screen, each reply’s command runs in its own fresh copy of the simulator, and the replies that score above their group’s average become more likely.

Possible range Mean over the training run

1. Format −1.5 to +0.3 · mean 0.15

   A reasoning block and exactly one known command

2. Physics −4 to +5 · mean 0.54

   Runs the command in a fresh copy of the simulator and scores how the facility’s health moves over the next steps

3. Command rules −1.5 to +1.2 · mean 0.65

   Hand-written for each scenario: diagnose the failed unit, leave broken units alone, never set below 17°C

4. No repeats −1 to 0 · mean −0.02

   A penalty for redoing a command from the scripted warm-up

−4−20+2+4

The physics score has the widest range. It replays the screen’s history in a fresh simulator, runs the command, lets four more steps pass and rewards replies that leave the facility healthier, with a bonus if the incident resolves along the way. The command rules encode what an operator would expect, such as diagnosing the failed unit first and leaving broken units alone. Every phase, the teacher included, ran on a single AMD Instinct MI300X, and getting the training stack to work on AMD’s ROCm software took its own set of fixes, which the repository documents.

Teacher episodes kept, of 1,520 played

1,083

Episodes the teacher itself resolved

1 in 6

Turns the student imitated

7,107

Replies scored for each screen during RL

8

What training showed

Imitation did its job before RL began. At every logged step of RL, every reply was well formed: a reasoning block of reasonable length and one known command. The format score never moved from 0.15, and since GRPO learns only from differences within a group, all of the learning went to the other three scores.

The mean training reward rose 66% from the first 50 logged steps to the last 50, and most of that came early. By quarters of the run it averaged 1.33, 1.38, 1.41 and 1.18. The repeat penalty, which punished redoing a scripted warm-up command, had fallen to zero by the end.

Those numbers say the model got better at earning its reward. Whether it got better at running a datacenter is a question for the evaluation.

Head to head with the base model

To see what training changed, I ran the base Qwen2.5-7B-Instruct and the trained model on the two hard scenarios, the cooling cascade (A4) and the power failure (B4), with five runs each and ten steps per run.

DDiagnose a unitSChange a setpointGStart the generatorLSet a rack’s loadWait LRejected: no such rack

Cooling failure cascade (A4)

Cooling failure cascade (A4): every command, ten steps per run, with each run's total reward

Run	1	2	3	4	5	6	7	8	9	10	Reward
Base model · mean 0.041
1	wait	wait	wait	wait	wait	wait	wait	D diagnose PDU-A-01	wait	wait	0.033
2	wait	wait	wait	wait	wait	wait	L set_rack_load A-01 5.5. Sheds A-01 from 8.0 kW to 5.5 kW	wait	wait	wait	0.006
3	wait	wait	wait	wait	wait	wait	wait	wait	wait	wait	0.018
4	D diagnose CRAC-1	wait	wait	wait	wait	wait	wait	wait	wait	D diagnose PDU-A-01	0.048
5	D diagnose CRAC-1	wait	wait	wait	wait	wait	wait	wait	wait	L set_rack_load A-01 4.4. Sheds A-01 from 8.0 kW to 4.4 kW	0.102
After training · mean 0.430
1	D diagnose CRAC-1	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 28.0	S adjust_setpoint CRAC-2 28.0	S adjust_setpoint CRAC-4 27.1	S adjust_setpoint CRAC-2 27.1	0.448
2	D diagnose CRAC-1	S adjust_setpoint CRAC-1 28.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-2 22.1	S adjust_setpoint CRAC-4 28.0	S adjust_setpoint CRAC-4 28	S adjust_setpoint CRAC-2 27.0	0.463
3	D diagnose CRAC-1	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 20.0	S adjust_setpoint CRAC-4 28.0	S adjust_setpoint CRAC-4 20	S adjust_setpoint CRAC-4 18.0	S adjust_setpoint CRAC-2 18.0	0.451
4	D diagnose CRAC-1	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.5	0.395
5	D diagnose CRAC-1	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 22	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-4 22.0	S adjust_setpoint CRAC-2 22.0	S adjust_setpoint CRAC-4 20	S adjust_setpoint CRAC-4 22.0	0.394

Power failure cascade (B4)

Power failure cascade (B4): every command, ten steps per run, with each run's total reward

Run	1	2	3	4	5	6	7	8	9	10	Reward
Base model · mean −0.147
1	wait	wait	wait	wait	wait	D diagnose PDU-A-01	wait	wait	wait	wait	−0.128
2	wait	wait	wait	wait	wait	wait	wait	wait	wait	wait	−0.153
3	wait	wait	wait	wait	wait	wait	wait	wait	wait	wait	−0.153
4	wait	wait	wait	wait	wait	wait	wait	wait	wait	wait	−0.153
5	G start_generator	wait	wait	wait	wait	wait	wait	wait	wait	wait	−0.148
After training · mean −0.092
1	G start_generator	G start_generator	L set_rack_load A-01 10. Raises A-01 from 8.0 kW to 10.0 kW	L set_rack_load A-02 10.0. Raises A-02 from 8.0 kW to 10.0 kW	L set_rack_load A-03 11.0. Raises A-03 from 8.0 kW to 11.0 kW	L set_rack_load A-04 11.0. Raises A-04 from 8.0 kW to 11.0 kW	L set_rack_load A-05 11.0. Raises A-05 from 8.0 kW to 11.0 kW	L set_rack_load A-04 11.0. Leaves A-04 at 11.0 kW	L set_rack_load A-05 11.0. Leaves A-05 at 11.0 kW	L set_rack_load A-03 10.0. Sheds A-03 from 11.0 kW to 10.0 kW	−0.255
2	G start_generator	wait	L set_rack_load A-01 11. Raises A-01 from 8.0 kW to 11.0 kW	L set_rack_load A-02 11.0. Raises A-02 from 8.0 kW to 11.0 kW	L set_rack_load A-03 11.0. Raises A-03 from 8.0 kW to 11.0 kW	L set_rack_load A-4 11.0. Rejected: there is no rack A-4	L set_rack_load A-4 0.0. Rejected: there is no rack A-4	L set_rack_load A-4 8. Rejected: there is no rack A-4	L set_rack_load A-4 8.0. Rejected: there is no rack A-4	L set_rack_load A-4 11.0. Rejected: there is no rack A-4	−0.388
3	G start_generator	G start_generator	L set_rack_load A-01 0.0. Sheds A-01 from 8.0 kW to 0.0 kW	L set_rack_load A-02 0. Sheds A-02 from 8.0 kW to 0.0 kW	L set_rack_load A-03 00. Sheds A-03 from 8.0 kW to 0.0 kW	L set_rack_load A-04 0.0. Sheds A-04 from 8.0 kW to 0.0 kW	L set_rack_load A-05 01. Sheds A-05 from 8.0 kW to 1.0 kW	L set_rack_load A-01 1.0. Raises A-01 from 0.0 kW to 1.0 kW	L set_rack_load A-0 1.0. Rejected: there is no rack A-0	L set_rack_load A-01 01. Leaves A-01 at 1.0 kW	−0.146
4	D diagnose GEN-1	G start_generator	L set_rack_load A-01 10. Raises A-01 from 8.0 kW to 10.0 kW	L set_rack_load A-02 10.0. Raises A-02 from 8.0 kW to 10.0 kW	L set_rack_load A-03 11.0. Raises A-03 from 8.0 kW to 11.0 kW	L set_rack_load A-04 11.0. Raises A-04 from 8.0 kW to 11.0 kW	L set_rack_load A-05 11.0. Raises A-05 from 8.0 kW to 11.0 kW	L set_rack_load A-4 11.0. Rejected: there is no rack A-4	L set_rack_load A-3 11.0. Rejected: there is no rack A-3	L set_rack_load A-01 11.0. Raises A-01 from 10.0 kW to 11.0 kW	0.540
5	G start_generator	G start_generator	L set_rack_load A-01 7.0. Sheds A-01 from 8.0 kW to 7.0 kW	L set_rack_load A-02 7.0. Sheds A-02 from 8.0 kW to 7.0 kW	L set_rack_load A-03 7.0. Sheds A-03 from 8.0 kW to 7.0 kW	L set_rack_load A-04 7.0. Sheds A-04 from 8.0 kW to 7.0 kW	L set_rack_load A-05 7.0. Sheds A-05 from 8.0 kW to 7.0 kW	L set_rack_load A-06 7.0. Sheds A-06 from 8.0 kW to 7.0 kW	L set_rack_load A-07 7.0. Sheds A-07 from 8.0 kW to 7.0 kW	L set_rack_load A-8 7.0. Rejected: there is no rack A-8	−0.211

The change in behavior is the clearest result. The base model waited on 92 of its 100 steps. The trained model waited once. On the cooling cascade it opened every run by diagnosing CRAC-1, one of the two failed units, and then adjusted setpoints for the rest of the run, almost always on the two surviving units and mostly to 22°C. Its mean reward was 0.430 against the base model’s 0.041, and each of its five runs scored higher than every base run.

The power failure is murkier. The trained model sent start_generator in the first two steps of every run and then turned to rack loads. Nine of its 40 rack commands named racks that do not exist, such as A-4 for A-04, and the simulator rejected them. Of the 31 it accepted, 17 raised a rack above its starting 8 kW, the opposite of shedding load during an outage. Four of its five runs ended below zero, and its mean of −0.092 against the base model’s −0.147 hides a spread from −0.388 to +0.540.

No run resolved either incident, and the logs show why. Each scenario counts as solved only after the facility has held steady for two steps and a specific check has been made: the cooling cascade needs both failed units diagnosed, CRAC-1 and CRAC-3, and the power failure needs a UPS diagnosed. In twenty runs, neither model ever diagnosed CRAC-3 or a UPS.

Two more details temper the gains. In the power failure the generator starts on its own: the opening alert already says so, and in the simulator a run of plain waits brings it online exactly as fast as one that starts with start_generator. And the repeat penalty that reached zero in training did not stop loops: ten times, the trained model sent the very command it had just sent. In training it was only ever compared with a short scripted history, and at each step the model sees the current screen, not a list of what it has already done.

Steps the base model spent waiting

92 of 100

Steps the trained model spent waiting

1 of 100

Runs that resolved the incident

0 of 20

Trained rack commands naming a rack that does not exist

9 of 40

What the evaluation can and cannot claim

A review of the evaluation code turned up six problems. Five now have fixes written, and one is still open. None of the runs above has been repeated with the fixes, so every number in this post comes from the original runs, and the points below are insights into what those numbers can claim.

1. Every seed, the same screen

   Fix written

   FoundThe simulator ignored the seed whenever a scenario was named. All five runs of each evaluation started from one screen, a screen the model had trained on: the 510 training prompts held only 48 distinct screens.

   NextEach seed now varies the start: outside and zone temperatures, rack loads and battery charge. The same 510 prompts become 510 different screens.

2. Half the steps

   Fix written

   FoundBoth hard scenarios allow 20 steps, but the evaluation stopped every run after 10.

   NextEach run gets its scenario’s full step budget.

3. Graded with its own reward

   Fix written

   FoundRuns were scored with the simulator’s reward, the same signal the physics score was built on during training.

   NextA separate scorecard the model was never rewarded for: peak inlet temperature, how long and how far temperatures ran over the ASHRAE limits, lowest battery charge, time on battery, generator start time, PUE, energy, and invalid or misaddressed commands.

4. Outcomes read from the alert text

   Fix written

   FoundSuccess and failure were inferred from words in the alert. Both hard scenarios open with an alert that begins with CRITICAL, one of the failure words, and a timeout could not be told from a crash.

   NextThe simulator now reports whether the incident is resolved, and a crash is told apart from a timeout.

5. Five runs, no error bars

   Fix written

   FoundFive runs per model and scenario, compared by their averages alone.

   NextTwenty seeds per scenario, 95% bootstrap intervals, a paired test on matched seeds, and evaluation seeds kept apart from the training seeds.

6. No imitation-only baseline

   Still open

   FoundThe base model was compared with the model after imitation and RL together.

   NextNot added yet. Until an imitation-only model runs the same evaluation, the gain cannot be split between the two phases.

What to take from it

• Imitation teaches the format. From the first step of RL, every reply had the right shape. Here, 7,107 teacher turns were enough to give a 7B model a strict output format.
• Training changed the behavior, but which phase did it is open. The trained model acted where the base model waited and aimed most of its commands at the right subsystem. Without an imitation-only baseline, the share that RL contributed is unknown.
• Read the success check before the reward curve. Both hard scenarios needed a specific diagnosis before they could count as solved, and no run made it. A rising training curve could not show that. The logs could.
• Check that seeds change what the agent sees. One ignored argument made five runs into one and left the training set with 48 distinct screens. A test that two seeds give two different screens would have caught it.

References

1. 1. Joseph Spisak, Davide Testuggine, Zach Wentz, Pierre Andrews, Sanyam Bhutani, Hamid Shojanazeri, Pankit Thapar, Emre Guven, Lewis Tunstall and Vaibhav Srivastav (2025). Building the Open Agent Ecosystem Together: Introducing OpenEnv. Hugging Face blog.
2. 2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
3. 3. Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115.
4. 4. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu and Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.