Testing the Rules as Executed against the Rules as Promised

Anne Schuth and Eelco Hotting. In Legal Knowledge and Information Systems: JURIX 2026, the Thirty-ninth Annual Conference, 2026.

Abstract

Where government executes the law from formalized rules, the formalization has to be tested against what the statute was meant to yield. But the expected outcomes a test needs almost always come from the formalizer, so the tests inherit the reading they were meant to check. The Dutch legislative record holds an oracle that neither the encoder nor the executing organization sets: the worked examples with which the government shows parliament what a bill should arrive at. A full-text harvest scores 5,068 passages as household-level worked examples, four in five of them genuine on a read sample, in 2,415 documents across 899 dossiers, and a recall check puts the true number about four in five higher. We transcribed seventeen of them under a rule that forbids inventing cases and ran fifteen against a public corpus of encoded statutes. The other two are for a provision the corpus does not model. Eleven reproduce the promised amount. Among the four failures is a household the government’s own amendment note had falsified before the act took effect, and auditing the expectations around it found three defects in our encodings. The same material shows what the oracle cannot do. An example states a third of the inputs an encoding needs, and rarely places one on a threshold. Of the 53 faults seeded into the allowance act, the encoders’ twelve households catch 23 and the memorandum’s two catch 12, each missing what the other finds. Examples go stale as soon as a parameter moves, and they say nothing about the open norms where the concretizations live. Worked examples belong in the record of how a statute becomes executable rules: the specification file the Council of State asked for in 2021, maintained with the law and layered with the policy rules and judgments that carry what a memorandum cannot.

Accepted for publication in the proceedings of JURIX 2026.

Links

https://github.com/NederlandseDigitaleDienst/regelrecht

Related Projects

RegelRecht — Dutch government exploration into machine-executable legislation; rulework format with the legal text alongside, BDD scenarios, Rust/WebAssembly reference engine.

Bib

@inproceedings{schuth2026testing,
  title = {Testing the Rules as Executed against the Rules as Promised},
  author = {Anne Schuth and Eelco Hotting},
  year = {2026},
  booktitle = {Legal Knowledge and Information Systems: JURIX 2026, the Thirty-ninth Annual Conference}
}