How rmqa Was Tested With 24 Broken Releases and One Bug It Found in Itself
A release checker has one job, and it fails in two ways. It can miss a broken release, which is the reason you run it. Or it can fail a good release, which teaches you to ignore it. The tests for rmqa go after both.
All the numbers on this page come from files in the test/results folder of the code repository. A script runs every test and writes those files, so the site never quotes a number nobody can check.
A Good Release First
Each test starts from a good release that the tests build themselves: a small Go program compiled for Linux, macOS and Windows on two kinds of processor, packed the way real projects pack them, with a license file, a readme and a SHA256SUMS file. Real compiled programs matter here, because rmqa reads the program headers to tell a Linux arm64 build from a Linux amd64 one. A text file named after the program wouldn’t test that.
The good release must come out clean: 0 errors and 0 warnings. If it doesn’t, every other result means nothing, so this is checked first.
Then 24 Ways to Break It
Each of the 24 scenarios makes one change to the good release, the kind of change that happens when a release script goes wrong or someone uploads by hand. An old build in one archive. The amd64 program copied into the arm64 download. A license file left out of the Windows zip. A .env file packed by mistake. An archive rebuilt after the checksums were made. An upload cut off halfway.
For each one, the test names the finding codes that must appear. Finding the right code matters as much as failing: “something is wrong” sends you digging, while wrong-arch on the arm64 archive tells you where to look. All 24 were caught, each with the codes it was meant to raise.
Then Many Ways at Once
Real broken releases often have more than one thing wrong, and a checker that stops at the first problem hides the rest. So a second test picks up to four of the scenarios at random, never two that touch the same file, applies them all to one release, and checks that every one is reported.
The recorded run covered 300 such releases with 671 problems between them. Every problem was reported.
This test found a bug, but in the test, not in rmqa. One random release combined “an archive was rebuilt after the checksums were made” with a change to the checksums file itself. With the checksums file changed too, the mismatch the test expected could no longer happen, and rmqa reported what was really there. The test was wrong to expect both. Scenarios that depend on the checksums file now count as touching it, so they’re never combined.
Fuzzing the Readers
rmqa reads two kinds of input it didn’t write: the manifest, and every archive in the release. Archives come from build tools, and sometimes from people, so the archive reader has to survive anything without crashing or hanging. Go’s built-in fuzzer fed it 419,166 made-up and mangled archives in a minute, and the TOML reader got a minute of its own. Neither crashed.
The archive reader also has limits that a fuzzer can’t reach on its own. rmqa never unpacks archives to disk. It keeps the programs it finds in memory and the first 512 bytes of everything else, up to 512 MiB per archive, only counts the rest, and stops at 4 GiB of unpacked data or 200,000 entries.
That design came from two independent reviews. The first found that 0.1.0 held every file in memory, so an archive of about a megabyte built to unpack to a gigabyte could exhaust the machine running the check. The second found that the 0.1.1 fix still kept a few kilobytes for each small file, so an archive of thousands of tiny files could add up the same way. The recorded results now measure both. A 1 MB archive holding 1 GiB of zeros peaks at 12 MB of memory, and a 1.3 MB archive of 150,000 one-byte files peaks at 60 MB.
A Real Release
Made-up releases only go so far, so the recorded results also check a real one: ukue 0.1.0, downloaded from GitHub, five archives and a checksums file. It passes with 0 errors and 0 warnings. Checked against the wrong version number, every program inside fails with version-missing, which is the point. On the 2-CPU machine that recorded the results, a check of the 18.8 MB release took 0.99 seconds, median of 7 runs.
The Bug rmqa Found in Itself
rmqa checks its own releases: the release workflow builds it for five platforms, runs rmqa on the result, and only publishes if it passes. The very first time this ran, it failed. Every rmqa program was reported as version-missing, an old build, which was not true.
The version check looks for the version string inside each program. To avoid false matches, the first version refused to accept 0.1.0 when a digit came right after it, so it wouldn’t mistake part of 0.1.01 for 0.1.0. But the Go compiler packs short strings end to end with nothing between them. In rmqa’s own program, the bytes 0.1.0 were followed directly by the next string, which started with a digit. The check saw 0.1.0 plus a digit and refused it.
The fix was to accept a digit after the version, while still refusing a digit or dot before it and a dot followed by a digit after it. That leaves a known gap, written down in the manifest reference: a build of 1.4.20 would pass a check for 1.4.2. Running the program with run closes that gap on the platforms the checking machine can run, and rmqa’s own manifest does that.
The bug would have hit every Go project that used rmqa. It was caught because rmqa had to pass its own check before anything was published, which is the same habit this tool asks of you.