Your Tests All Pass. I Have Bad News.
Your test suite is green. Every check passed. Merge it.
So what did green actually prove?
It proved the code, as written today, does not trip the assertions you happened to write. Whether those assertions would catch the code breaking is a different promise, and nobody measures that one.
Try this some quiet afternoon. Open a codeunit you trust, delete a line that looks important, and run the tests. If they stay green, you just learned that line has no protection at all. Put it back. Then wonder how many more of those you have.
Break the code on purpose
That afternoon experiment has a name: mutation testing. A tool does the breaking for you, one small change at a time. Flip a > to >=. Remove a SetRange. Turn Modify(true) into Modify(false). Each broken copy of your app is called a mutant, and the question is always the same: does any test notice?
A test fails: the mutant is killed. Good.
Everything stays green: the mutant survived. A survivor is the uncomfortable one: if a bug of that exact shape ever lands on that line, it ships, straight past your whole suite. Your tests just demonstrated it.
So mutation testing does not test your code. It tests your tests. Code coverage tells you a line ran. A killed mutant tells you someone would notice if it broke. Only one of those helps you sleep.
About the name
I built a tool that does this for AL. It is called LethAL, and the name is my one regret.
This is the year the X-Men finally enter the MCU. Avengers: Doomsday is in cinemas this December. And I shipped a tool whose entire job is hunting down mutants without calling it SentinAL.
In my defense: Leth is my last name. LethAL is more puns per word than SentinAL, and I’m a dad. The law is clear. 😉
Why you have never done this
Mutation testing is old news everywhere else. C# has Stryker, Java has PIT, and both are established enough to be boring. AL has had nothing, and not because nobody thought of it.
Do the math. Even a small app produces a few hundred mutants. The naive way to score one is compile, publish, run the suite, throw it away, next. Call it a couple of minutes per mutant on a good day. A few hundred mutants later, your test run is a long weekend. Nobody does that twice.
LethAL’s answer is to compile once: every mutant ships inside the same build (a big app gets a handful of builds, but you get the idea), and switching to the next one is a table write, not a recompile. The demo run below deployed 32 mutants and finished in 16 seconds, publish and baseline included. The naive loop would have spent an hour on it. How the one-build trick works is properly a post of its own. Today I care about what comes out the other end.
Before you get too excited: a real app is another story. The biggest codebase I have pointed LethAL at came out at just under 20,000 mutation sites, and no, you do not run all of those in one go. You aim it at the codeunits you care about with --only, and LethAL refuses big unscoped runs by default, so you cannot ruin your own day by accident. The demo is tiny on purpose, so you can read the whole thing.
A credit limit and a green suite
To show you what a run finds, I wrote the most ordinary extension I could think of: a credit limit check. It ships with LethAL as examples/credit-limit, so you can run everything below yourself.
Full disclosure: I wrote the tests too, and I knew where the holes were. So this is a demo, not LethAL surprising me with my own code. But the holes are the kind I have seen in plenty of real suites, and written a few of myself. On your app the surprise will be real.
procedure WouldExceedLimit(CustomerNo: Code[20]; NewOrderAmount: Decimal): Boolean
var
CreditCustomer: Record "Credit Customer";
Exposure: Decimal;
begin
CreditCustomer.Get(CustomerNo);
if CreditCustomer."Credit Limit" = 0 then
exit(false);
CreditCustomer.CalcFields(Balance);
Exposure := CreditCustomer.Balance + OutstandingOrderAmount(CustomerNo) + NewOrderAmount;
exit(Exposure > CreditCustomer."Credit Limit");
end;Balance is a FlowField over a ledger table. Open orders count toward exposure. A limit of 0 means no limit. Five tests, all green, and they would pass review: under the limit is allowed, over is blocked, a limit of 0 never blocks, open orders count, invoiced ones stop counting.
Then I pointed LethAL at it. Sixteen seconds later: 32 mutants, 17 killed, 8 never executed by any test (the posting helpers, which no test touches), 7 survived.
Three of them had no business surviving.
Nobody ordered exactly the limit
- exit(Exposure > CreditCustomer."Credit Limit");
+ exit(Exposure >= CreditCustomer."Credit Limit");All five tests passed.
The suite checks 400 against a limit of 1000 and 1200 against a limit of 1000. Never 1000 against 1000. So the boundary is unguarded: whether a customer standing exactly at their limit gets the order or an error is decided by whichever comparison happens to be in the code, and no test has an opinion about it.
The fix is one test that orders exactly up to the limit. Mutant dies.
Nobody ever owed anything
- CreditCustomer.CalcFields(Balance);
Exposure := CreditCustomer.Balance + OutstandingOrderAmount(CustomerNo) + NewOrderAmount;Remove the CalcFields and Balance keeps whatever the record variable last held, which is zero. All five tests passed anyway.
Why? No test ever touches the ledger. Even the invoiced-orders test only flips a status field; nothing posts. Every customer in the suite has a spotless payment history, so the FlowField is zero whether you calculate it or not. The test data was too polite to catch anything.
In production this is the customer who owes 900 of their 1000 limit and sails a 500 order straight through, because the check never fetched what they owe. The fix is one test that posts an invoice before it tries to order.
There was only ever one customer
- CreditOrder.SetRange("Customer No.", CustomerNo);
CreditOrder.SetRange(Status, CreditOrder.Status::Open);
CreditOrder.CalcSums(Amount);Drop the customer filter and outstanding orders are summed across every customer in the database. All five tests passed.
The suite creates one customer, C-10000, in every single test. With one customer in the table, “this customer’s orders” and “all orders” are the same set, so the filter is never load-bearing. Your test container is a much lonelier place than production.
The fix is a second customer with orders of their own. Suddenly the filter matters.
What the score is for
A run ends with a mutation score. This one said 70.8%, and to be clear about the math: that is 17 dead out of the 24 mutants a test actually covered. The eight no-coverage ones are reported next to the score, not hidden inside it. And to be fair, those eight are not really LethAL’s discovery. Plain old code coverage would have told you the same thing, the posting code is never tested. What coverage cannot tell you is the other part: a line that ran in every test and still changed nothing when it broke. That is the part only mutation testing sees.
Don’t set out to chase 100%. Some survivors are shrugs: this run turned Insert(true) into Insert(false) on a table with no OnInsert trigger, and nothing fires either way, so there is nothing to fix. Of my seven survivors, three were the sections above, two were shrugs like that one, and two pointed at a small real gap: no test ever inserts a customer without a number.
Treat the survivor list as a queue: ranked, in the code you actually care about. A survivor is a lead, not a verdict: the tool can’t read your spec, it can only tell you where your tests went quiet. Read the leads. Fix the ones that are money.
Built for agents
Who works that queue? At my desk, mostly agents. Most of my code is written by them now, and a survivor is the best-shaped task you can hand one: precise (kill this mutant), local (this line, this file), and self-grading (rerun, watch the verdict flip). No “please improve the test suite” hand-waving. A queue of small, provable jobs.
One rule though: you decide what the right behavior is before the agent writes the test. Otherwise it just writes a test that locks in whatever the code does today, right or wrong, and calls it a kill.
So LethAL was built from the ground up to be driven by something that isn’t me. The report and the progress stream have published JSON Schemas. doctor --json tells a script exactly which pre-flight check failed instead of making it parse prose. Exit codes say whether the result is usable, never whether your suite is good. And when LethAL can’t prove the server was in a sane state, it quarantines the whole run and refuses to vouch for the verdicts, because an agent will happily draw a confident conclusion from a poisoned result.
If your agent is Claude Code, there is a ready-made skill in the repo. Install it and “run mutation testing and kill the top survivor” becomes a one-line prompt.
Run it yourself
LethAL is open source and pre-alpha, in the honest sense of both words: there is no published release yet, so you clone the repo and build it yourself with Bun. You need a sandbox or dev container, never production, and the README walks you through the setup. The shape of a run:
lethal init --project path/to/your-app
lethal run --project path/to/your-app --tests path/to/your-tests \
--backend bcdev --config lethal.config.json --out report.json
lethal explain report.json --top 10Start with the credit limit demo before your own app. It is small enough to read in one sitting, every survivor in this post is in it, and its README has the exact commands.
Consider this the short introduction. A bigger one is coming when LethAL is a bit more polished, either here, at a conference, or both. We will see what comes first.
The mutants aren’t coming. They’re already in your codebase, sitting quietly behind green tests, waiting for a customer to find them first. Mine were.
Go build your Sentinels.