Testing Black Ops III with ChatGPT

Letting ChatGPT control my PS4

I wanted to see how far ChatGPT could get with BO3’s offline Zombies in shadPS4. I also had a physical PS4 available, which gave it a working version to compare against. The first job was getting reliable input and video from that console through Chiaki on Ubuntu.

ChatGPT navigated to Local Zombies, loaded Kino der Toten and sent movement, camera and firing inputs. It checked the captures afterward. This gave us a way to investigate the game without me pressing every button, although it did not make the emulator compatible with it.

Even the Options button needed debugging. Our virtual controller exposed digital trigger buttons before Back and Start in Linux’s button order. Chiaki’s default SDL2 mapping treated those positions differently, so a command named Start did not mean the PS4 received Options. ChatGPT inspected the actual events and applied a mapping only to Chiaki: Back at button 8, Start at 9, Guide at 10 and the stick clicks at 11 and 12.

↗

Project capture 1 / 1Close ×

The corrected controller mapping opens BO3’s menu on the physical PS4, September 7. The identity tab is selected; the test was whether the Options button opened this menu.

shadPS4 used a different SDL version. A small probe built against its pinned SDL3 library already passed six named buttons with the default mapping. Copying the Chiaki override into it would have mixed two different clients. That library check still did not prove that BO3 accepted every control in the emulator.

Why the opening video was taking minutes

With the owned base game and update 1.33 checked, ChatGPT tested shadPS4 revision 63cc326. The intro appeared, but advanced painfully slowly. In one controlled run, successive observations were about two, three and four seconds per frame. A deadline expiring was not enough to call that a deadlock: the video was still progressing.

ChatGPT matched the official executable to a build with debug symbols, then started it as a child of GDB. Five complete samples put the GPU command thread in the same allocation path:

Rasterizer::DispatchDirect
  → Rasterizer::BindResources
  → BufferCache::FindBuffer
  → vmaCreateBuffer
  → AllocateDedicatedMemory / AllocateVulkanMemory
  → NVIDIA driver ioctl

A separate instrumented trace recorded 512 completed allocations totaling 17.354 GB in about 14.9 seconds. That was cumulative allocation traffic, not peak VRAM usage. The cache kept replacing large buffers as small writes extended their range.

The first patch corrected which side received spare capacity in BufferCache::ResolveOverlaps. The old comparisons could reserve another 2 MiB behind a buffer whose end was growing forward. A second patch retained the buffer’s stream score after a growth step, so it did not need another series of exact-size replacements before earning spare capacity again.

↗

↗

Project capture 1 / 2Close ×

Growth-only candidate, about 40 seconds into this run. The intro showed 60 FPS at this instant; the same run later failed with an allocation error.

A focused replay used the actual overlap method with a simplified cache: a 1 GiB stream followed by 200 small appends. Keeping the stream score reduced replacements from 19 to two in that fixture. It did not execute Vulkan copying or prove a game-wide memory improvement. The combined emulator candidate then survived a three-minute run without the earlier allocation fatal and reached the incomplete menu in a follow-up.

A successful voice read that returned no useful data

The next clue was repeated calls to sceVoiceReadFromOPort. Three observed calls requested 255 bytes, returned success and left the count at 255. The stub had no declared arguments, so it could not update the output count. ChatGPT then looked at BO3’s actual x86-64 code to find out why that mattered.

The read wrapper at disassembly address 0xed3c60 starts with a request for 255 bytes. After a successful read, a nonzero count becomes count plus one; a zero count produces zero. The caller repeats while that wrapper returns a positive value. This short excerpt is copied from the saved update-1.33 disassembly; these are virtual addresses from the saved executable, not raw file offsets or addresses to reuse with another build.

d65a4b: e8 10 e2 16 00   call   0xed3c60
d65a50: 41 01 07         add    %eax,(%r15)
d65a53: 41 01 c5         add    %eax,%r13d
d65a56: 85 c0            test   %eax,%eax
d65a58: 7f b6            jg     0xd65a10

The test sets flags from the wrapper’s return in eax; jg takes the signed greater-than-zero branch back into the loop. The old stub could therefore make an empty read look like another 256-byte result. There is another helper inside the loop whose side effects were not fully analyzed, so this identified a concrete repeat condition rather than proving every path through the game.

ChatGPT also tested the console voice library’s original machine code offline with controlled port state. A valid empty output ring produced success with zero bytes; a ring containing 16 bytes copied and reported 16. Other errors preserved or cleared the count depending on the state. Those were synthetic library tests, not live measurements of BO3’s microphone or port state on my PS4.

The resulting emulator candidate declares the arguments, checks the port range and pointers, then reports an empty source. This is the actual changed function:

s32 PS4_SYSV_ABI sceVoiceReadFromOPort(u32 port_id, void* data, u32* size) {
    LOG_ERROR(Lib_Voice, "(STUBBED) called");
    if (port_id >= 64) {
        return ORBIS_VOICE_ERROR_INVALID_PORT;
    }
    if (!data || !size) {
        return ORBIS_VOICE_ERROR_INVALID_ARGUMENT;
    }
    *size = 0;
    return ORBIS_OK;
}

This remains a small candidate for a stubbed subsystem. It does not implement port allocation, voice initialization, microphone capture or encoding, and an in-range number is not proof of an allocated port.

The diagnostic plugin crashed before measuring voice

The physical-console investigation had a failure of its own. ChatGPT built a private voice probe with the OpenOrbis toolchain and GoldHEN SDK, but its first launch crashed in plugin_load before the hook or any voice measurement ran.

The crash address matched a compiler-generated movaps %xmm0,-0x40(%rbp) store. The saved base pointer put that destination eight bytes off a 16-byte boundary. The entry point had assumed stack alignment that this loading path did not provide. A correction explicitly realigned the stack at entry, using Clang’s force_align_arg_pointer attribute.

The earlier passing fixture covered the voice wrapper, not plugin startup. ChatGPT then ran the actual compiled entry code on Ubuntu with both alignment cases: the original failed under the observed misalignment, while the corrected entry passed. That supported the startup correction. It did not validate the complete plugin.

The PS4 also got stuck closing the crashed application. FTP still responding had given a misleading impression that remote recovery was available. After recovery, ordinary launch, close and reconnect checks worked; a separate minimal plugin successfully wrote a load marker. The full voice probe stayed disabled. This part of the project changed what the tests needed to cover before another hardware experiment.

Testing BO3’s own wrapper in about a second

This is the part that made ChatGPT useful to me beyond generating a patch. It could move from a slow emulator run to the smallest piece of code that answered the question, write a test around it and keep the original failure as a comparison. I would otherwise have spent much of that time rebuilding and navigating back to the same menu.

Twelve focused sanitizer cases passed against the actual voice implementation. ChatGPT also extracted the unchanged BO3 wrapper and import thunk from the verified update executable, then ran those bytes with controlled globals and guarded buffers. The comparison linked either the actual candidate or an equivalent version of the old stub:

The host driver repeated while the wrapper returned a positive value; it did not execute the caller’s other helpers. This was much stronger evidence than a test that simply repeated the patch’s intended behavior, but still stopped short of proving the full guest loop or playable Zombies. ChatGPT’s advantage here was writing, running and revising that experiment across the disassembler, C++ build and debugger. The earlier probe crash also shows why a passing automated test needs its limits spelled out.

Where I paused the PS4 work

The later normal-renderer voice-candidate run reached the title screen with substantial graphics defects, then hit an NVIDIA Xid 109 / CTX SWITCH TIMEOUT. It did not produce a verified voice-return checkpoint. The cause remains unresolved. Subsequent software-only diagnostics used lavapipe and a null GPU path; those runs could inspect CPU progress, but their blank output could not establish game compatibility.

I asked for a Git checkpoint and for the PS4 services to stop so the Ubuntu resources were free again. That checkpoint was recorded on September 8. The patches, game copies, builds and evidence remain saved; the emulator and its dedicated support services were stopped. No emulator patch had been submitted upstream.