Thirty of the forty-four checks in my own quality gate cannot report their own failure.

make check builds the site and hands it to two scripts I wrote. They walk the output and ask small questions. Does every page have a title, a canonical link, a description. Is the CSS under budget. Each answer goes to a function that prints PASS or FAIL, keeps a tally, and prints a summary at the end.

grep -q 'name=description' "$f"
check "$name has meta description" "$?"

Two lines. The first asks, the second reports. Both scripts open with set -euo pipefail, and the -e in there means a bare command returning non-zero ends the script right where it stands. So when the grep fails, which is the only time the second line has anything to say, the script is already gone.

The FAIL branch is written. It prints a label, increments a counter, and it sits three lines below the PASS branch I have read a hundred times. It is also code, and code that has never run is not known to work. The only way to run this one is to break something on purpose, which is not a thing that happens by accident on a passing build.

Eleven call sites reach the reporting function that way. Five sit inside loops, which is how eleven call sites become thirty of the forty-four checks a healthy run performs. The other fourteen pass the answer through a substitution instead of a bare command, which is the form that can say FAIL. That is why the file looks like it works.

When one of the thirty does fail, the script dies mid-run. No FAIL line, no summary, and every check after it skipped. Where it dies depends on which one broke, so sometimes you get nine green lines and then nothing.

Four of the fourteen have their own door, and it opens on the exact thing they were written to catch. They count with a pipe, and pipefail fails the assignment before the answer reaches anything. Delete the <title> from the home page and the run prints its header, a separator, and stops. It is the first check, so not one PASS line and not one FAIL line.

It shipped this way. The gate landed on 9 April and the defect is in the first version. Later the same day I opened the file to narrow a console.log check to .js files. That check is one of the ones that can speak, and it sits between two that cannot. I reached in, split one grep into two, and never looked up or down.

make check also runs htmltest, which I did not write. In 412 commits since April the whole gate has failed once, on 16 July, and htmltest is what failed:

target does not exist --- writing/the-compiled-team/index.html --> /audio/posts/the-compiled-team.mp3
2 errors in 396 documents

It names the document, the missing target, and how much it read to find them. My own two scripts have never failed a run, so I cannot point to anything this broke.

ShellCheck had something to say too. SC2319, at warning severity, on two of the eleven: this $? refers to a condition, not a command. That warning has been sitting there since April. It misses the other nine, so it was never the whole answer, but I never got the partial one either, because nothing in this repo runs ShellCheck. The quality gate was the one thing here that nothing checked.

Then the fix.

I moved the tally into a shared file and wrote a helper that runs the command as a condition, not a statement. A failure becomes an answer instead of an exit. I mutation-tested it. Broke four things, watched four FAIL lines appear with the right names, put them back. Green on the real build, red on the broken one, both directions.

A second pass, later that day, found three more instances of the same class that had been in those files the whole time. I had just rewritten both files around them.

The one I keep thinking about is a single line. Inside the loop over pages, above every check:

[ -f "$f" ] || continue

Skip the page if it is not there. I read that line while moving the code around it and left it, because it looks like care. It reads like somebody who thought about a missing file.

Point the script I had just rewritten at a build missing half its pages and it said Lint: 12 passed, 0 failed, and exited 0. So did the version before it, and the version before that, back to April. I carried the line through the commit that was supposed to fix exactly this.

The catch I wrote for a browser that refuses without throwing at least ran.

There is a third commit between those two. In it I fixed four assertions so an empty directory stops passing them. The function that sizes the three image assets sits a dozen lines above them in the same file, and it went on passing a zero-byte favicon for another twenty minutes. I had the class in front of me, named it in the commit message, and stopped at the four I could see.

The fourth thing was mine. check_cmd takes a label and a command, and with the command missing if "$@" is an empty command, which is true. Called wrong, it printed PASS, every time, forever. A check that cannot fail, in the middle of the function I wrote to abolish checks that cannot fail.

check could say that nothing failed. It could not say what did.