How ChatGPT runs my game tests

The Windows VM builds the Xbox plugin

The old Windows tools were needed for the BO1 plugin. ChatGPT sent build jobs through WinRM into a Windows 7 SP1 VM with Visual Studio 2010 SP1 and Xbox SDK 21256.3. MSBuild produced codxe.xex; the tools collected the output, recorded its identity and staged it for a game test. Opening the IDE was useful for looking at the code, but repeated builds could run without clicking through it.

↗

Project capture 1 / 1Close ×

The real BO1 build VM, shown during the September 7 walkthrough. The Windows SDK toolchain produces the Xbox plugin; Linux emulator source builds use a different path.

Open full-size capture ↗

That is different from changing an emulator or a GSC script. I wanted these distinctions visible because “ChatGPT modded the game” hides several very different jobs:

The PS4 build container used pinned Clang 19 and GCC 14’s C++ library, with persistent build and compiler-cache directories. That made small compile-and-link iterations take seconds once the initial build was done. One transferred source file kept an old timestamp and Ninja skipped it; identical parent and candidate binary hashes caught the mistake before a comparison run. A “build succeeded” line alone was not enough.

How commands, controller input and video travel

ChatGPT works through project tools in Codex, sending SSH commands to Ubuntu and reading saved output. My video viewers are a separate connection. Closing Moonlight or a video window does not stop the server-side test, so the runner needs its own deadline and cleanup.

For Xbox, Sunshine streams the existing desktop to Moonlight on my Mac or iPad over the LAN or Tailscale. For PS4, FFmpeg captures the dedicated Xorg root window and its audio, encodes H.264/AAC and sends an MPEG transport stream through a loopback listener forwarded over SSH. The PS4 viewer carries video and sound; controller commands go through the project’s input tools.

There was a practical reason for using FFmpeg on the PS4 side. The installed Sunshine build looked for RandR hardware outputs, but the headless NVIDIA screen had none. Its web interface starting did not prove it could stream that screen. FFmpeg could capture the root window and use NVENC without a connected display.

Chiaki adds one more connection: Ubuntu talks to the physical PS4 over Ethernet for Remote Play. That is how ChatGPT reached Kino der Toten on the real console. shadPS4 and Chiaki use the same PS4 session at different times; a Chiaki gameplay capture says nothing by itself about emulator compatibility.

Keeping an Xbox button press out of the PS4 test

The Xbox session uses display :44; the PS4 session has its own NVIDIA Xorg display at :45, audio server and capture route. The input split goes deeper than naming two controllers. Each emulator or check runs with a private device view containing only its selected input nodes and the GPU devices it needs.

The replay pads use Linux uinput. A helper queues validated button and axis commands through a FIFO, and the consumer sees a normal controller device. Device numbers are discovered at launch. A narrow udev rule connects each pad’s lifetime to its systemd consumers, so a disconnect stops the consumers before a replacement device can be mistaken for the old one.

emulator-session.py input ps4 tap a
emulator-session.py input ps4 stick left 0.2 0 0.5
emulator-session.py input ps4 axis rt_axis 1 0.2
emulator-session.py input ps4 center

Those are examples of the helper interface, not a recorded gameplay sequence. The button-mapping bug in Chiaki showed why valid input delivery was only the first check. ChatGPT also needed to establish what the consumer called each button and whether the game reacted. SDL2 and the pinned shadPS4 SDL3 library did not use identical defaults.

Process ownership matters for cleanup too. The app and check jobs run in session-specific systemd units. With ExitType=cgroup, Proton children remain owned even after the launcher exits. The launcher’s earlier broad Wine/Xenia kill rules were replaced with scoped cleanup. Stopping the PS4 emulator should not terminate an Xbox playtest.

The files that make a comparison worth keeping

The useful loop starts with a specific question: did the imported cabinet get its original texture, did the same hut approach still crash, or did a voice read update its byte count? ChatGPT changes the candidate or fixture, runs it and saves enough context to connect the result to that exact attempt.

1. 01

   I set the goal

   I describe the feature or report a specific failure.

2. 02

   ChatGPT edits code

   The agent changes a test copy and builds a candidate.

3. 03

   ChatGPT runs tests

   Project tools replay inputs and save screenshots and logs.

4. 04

   I review and play

   I inspect the result and playtest where needed, then report problems.

The BO1 runner’s manifest records the plugin hash, Xenia target, configuration, scenario and replay clip. Recorded controller inputs can return to a donor map or a repeatable camera view. Later PS4 runners also saved binary/configuration hashes, screenshots, logs and save-tree backups, then checked restoration afterward. These tools developed across the projects; the early WaW playtests were still largely manual.

ChatGPT was useful to me because it could operate this whole sequence and change the experiment when the evidence called for it. In BO3, it reduced minutes of menu waiting to an approximately one-second check using the game’s real voice wrapper. In BO1, my hut-crash report became a repeatable close-approach test. An experienced developer could build these tools too; my advantage was having ChatGPT write and maintain them while I decided what I wanted to play and reviewed the result.

Sometimes the screenshot needs its own test

The PS4 investigation produced sliced images that looked like broken rendering. An ordinary Vulkan cube showed similar horizontal displacement outside shadPS4. ChatGPT then briefly held its own cube process stationary, waited half a second and captured it again. The saved frames were coherent.

↗

↗

Project capture 1 / 2Close ×

Ordinary Vulkan sample, September 7 local time. The captured moving cube is split into horizontal strips. This is not a BO3 scene.

Open full-size capture ↗

That supports capture or presentation timing as an explanation for the slicing. It did not identify the exact synchronization defect, fix all BO3 graphics or resolve the GPU timeouts. The comparison mattered because an agent that only sees saved frames can otherwise start changing the renderer to “fix” a capture artifact.

The same distinction applies to logs and input. A printed success can leave an output count unchanged; a received SDL key event does not prove the guest accepted Cross; a blank software-renderer frame does not mean a level loaded. The automated loop became useful when its checks asked about the actual behavior, and my visual reviews still caught mistakes it had accepted.

The sessions still share a GPU

Separate controllers, displays and process groups do not split the RTX 5090 into independent hardware. CPU time, VRAM and the driver remain shared. The setup checks covered isolated input, audio, capture and stopping each session, plus concurrent synthetic Vulkan tests. They did not benchmark two real games running together.

Later BO3 runs hit NVIDIA Xid 109 context-switch timeouts. The cause remains unresolved, and ordinary Vulkan output was not proof that the known-failing BO3 workload was safe to repeat. Subsequent diagnostics selected Mesa lavapipe and a null GPU path to investigate CPU progress without relying on rendered gameplay. That software setup had explicit time and log limits; it still did not establish Zombies compatibility.

At the September 8 PS4 checkpoint, I asked to stop its dedicated services and free the running resources. The repositories, game copies, builds and captures were kept. The diagram describes the configured architecture, not a live status display or a claim that every component is currently running.