From Rule Execution to Citizen Explanation: Evaluating LLM-Generated Explanations in Rule-Based Government Systems

Floortje Bos and Anne Schuth and Jieying Chen. In Legal Knowledge and Information Systems: JURIX 2026, the Thirty-ninth Annual Conference, 2026.

Abstract

Governments increasingly use automated decision-making systems to apply legislation in domains such as social benefits, taxation, and licensing. While Rules as Code (RaC) approaches enable legally consistent and transparent decision-making through executable legislation, they do not automatically generate explanations that are understandable for citizens. Large language models (LLMs) can support natural-language explanation generation, but may hallucinate facts or exclude legally relevant information. This paper investigates how Graph-based LLM architectures can generate citizen-focused explanations from deterministic rule-engine traces. A rule engine evaluates synthetic citizen profiles against formalised Dutch legislation and produces execution traces, which are transformed into structured decision representations for explanation generation. The approach is evaluated using automated metrics and human annotation. Results show that explicitly exposing decisive legal conditions improves faithfulness, while Graph-based structuring provides further alignment with the rule-execution trace. Human evaluation shows that explanations perceived as understandable by citizens are not always considered legally correct by experts. The findings suggest that structured representations of deterministic legal reasoning can support legally faithful and citizen-focused explanations of automated government decisions. The source code is published.

Accepted for publication in the proceedings of JURIX 2026. This is the full paper based on Floortje’s MSc thesis; the thesis abstract appeared at BNAIC/BeNeLearn 2026.

Links

https://github.com/Bosfloortje/machine-law-thesis-flora

Supervised Student

Floortje Bos
MSc AI student, Vrije Universiteit Amsterdam, 2026

Related Projects

RegelRecht — Dutch government exploration into machine-executable legislation; rulework format with the legal text alongside, BDD scenarios, Rust/WebAssembly reference engine.

Bib

@inproceedings{bos2026jurix,
  title = {From Rule Execution to Citizen Explanation: Evaluating LLM-Generated Explanations in Rule-Based Government Systems},
  author = {Floortje Bos and Anne Schuth and Jieying Chen},
  year = {2026},
  booktitle = {Legal Knowledge and Information Systems: JURIX 2026, the Thirty-ninth Annual Conference}
}