Entry 63

eemu in Rust: piney-eemu, identical to eemu.py instruction for instruction and 20 times faster

Filescrates/piney-eemutools/test_eemu_rs.pytools/test_anim.pycrates/piney-demo/examples/demo_probe.rs

tools/eemu.py runs the game's own code for every harness that checks the port against the game. It is pure Python, a few hundred thousand instructions a second, and the world port's suite had grown to 21 minutes. The user asked for it in Rust. The rule that RE tools stay in Python was never theirs (see the corrected memory note), so a helper agent ported it.

What it is

crates/piney-eemu (about 3,100 lines) holds:

  • The interpreter. eemu's Machine: the EE core, the FPU with the EE's float rules (piney_data::field::ee, plus rsqrt.s) and the C library stand-ins.
  • The harnesses' extensions, behind switches:
    • test_anim's VU0 macro and MMI instructions;
    • test_stream_rs's VU0 integer registers and data memory;
    • test_toppage_rs's divide-by-zero rule.
  • The Python module (feature python, PyO3): eemu_rs, which mirrors eemu.Machine:
    • mem is a real 32 MB bytearray, so slicing works;
    • it has the same registers, hooks and calls;
    • it raises eemu's own Stop;
    • a harness's Python subclasses that override exec or cop2 are honoured.

Proof it is the same

tools/test_eemu_rs.py runs eemu.py and eemu_rs in lockstep. After every instruction it compares the pc, every register, the VU0 state and every byte either machine wrote.

  • Real workloads, 19.3 million steps from the world, animation, stream, top-page, morph, demo, desktop and dungeon harnesses.
  • Fuzzing. 120,000 random instructions over six machine variants, 900 random whole programs, and 1,500 calls to the string and memory functions.
  • Mutations. Five deliberately broken builds were each caught.

The agent also ran all 18 harness suites that use eemu with the Rust machine swapped in. All 150 tests passed.

I re-ran the parity test in a clean worktree: 14 tests, 140 s, OK.

Speed

Workloadeemu.pyeemu_rs
the world suite1,275 s18.5 s (69x)
all 18 suites1,908 s93 s (20x)
200 world frames40.5 s0.16 s (247x)
sinf in a loop0.66M instructions/s74M (112x)

The switch

  • The default. test_anim.machine_class() now returns eemu_rs.VuMachine when the module is built into tools/, and falls back to eemu.py (with a note on how to build it) when it is not. PINEY_EEMU=py forces the Python machine. The world, morph, stream and demo-lighting harnesses take it that way.
  • The parity test. It pins its reference to the Python machine, so it never compares Rust with Rust.
  • The other harnesses. Those that name eemu.Machine directly switch with one import line each. That waits until the agents now editing them hand back.

Also found

test_demo_rs had been broken since 54: demo_probe.rs calls piney_gs::convert::mmat, whose argument list the morph change grew, and it is built only with the trace feature, which workspace clippy does not enable. It now passes the new argument.

Checked

  • The crate. piney-eemu has 14 tests; clippy is clean with and without python.

  • The parity test. It passes in a clean worktree.

  • The switched suites. After the switch I ran them on the Rust machine in a clean worktree:

    SuiteResultTime
    test_world_rs10 tests pass40 s (over 20 minutes before)
    test_demo_rs18 tests pass70 s
    test_morph_rspasses2 s
    the stream fixtureregenerated byte-identical3.2 s (104 s before)
    the effects fixtureregenerated byte-identical0.46 s (8 s before)
Still unknown
  • What cannot be mirrored. A subclass override of load, store or set does not reach the interpreter; no harness uses one. An exec override runs near eemu.py's speed, because it is a Python call per instruction. - Differences in use. RAM cannot grow past 32 MB (eemu.py grows it), and registers are live views rather than lists. - Not switched. iopemu's machine, built without a program, stays on eemu.py.