ELEF's rules were tested in recorded simulations. The actors in them were small programs trained inside the simulation, not language-model agents. Each version of the rulebook was played by scripted inspectors and, later, by inspectors that learn from their own earnings.
They found where checking was thin. When every inspector learned at once, they checked carefully where the orchestrator's planted cases could reach them, and approved without looking at the audits and appeal panels, 91 to 97 times in 100.
Each revision since has closed part of that. Under 2.9.0, the current rulebook, approving without looking at the audit fell from about nine times in ten to about half. It has not gone away, and 2.9.0's results are not settled.
ELEF is the rulebook. Its findings come from recorded simulations whose actors were small programs trained inside the simulation, not language-model agents.
Under 2.7.0, when every inspector learned at once from one shared strategy, the population was honest where the orchestrator's known truth could reach it, and nowhere else: it approved without looking at the audits and appeal panels 91–97 times in a hundred, and its false approvals were three to four times those of the scripted mix, the comparison group of hand-written inspectors that starts as 60% stampers and 40% careful.
2.8.0 sent planted cases to every seat. False approvals fell by more than half, but stayed about three times the mix's, and the audit seat was still approved without looking most of the time.
2.9.0 has the orchestrator plant the wrong steps too. In the runs before the reward correction, lying appeals were paid 5–6 times in a hundred, from 23–31. After it, on the inspectors' own money, lying appeals were paid about as rarely as against the scripted mix, and approving without looking at the audit seat fell from 94–97% to 41–56%.
The record leaves that remaining half to 2.9.0's new dial, which is still moving. Its results are not settled.
ELEF is the rulebook. Its findings come from recorded simulations whose actors were small programs trained inside the simulation, not language-model agents.
Under 2.7.0, when every inspector learned at once from one shared strategy, the population was honest where the orchestrator's known truth could reach it, and nowhere else: it approved without looking at the audits and appeal panels 91–97 times in a hundred, and its false approvals were three to four times those of the scripted mix, the comparison group of hand-written inspectors that starts as 60% stampers and 40% careful.
2.8.0 sent planted cases to every seat. False approvals fell by more than half, but stayed about three times the mix's, and the audit seat was still approved without looking most of the time.
2.9.0 has the orchestrator plant the wrong steps too. In the runs before the reward correction, lying appeals were paid 5–6 times in a hundred, from 23–31. After it, on the inspectors' own money, lying appeals were paid about as rarely as against the scripted mix, and approving without looking at the audit seat fell from 94–97% to 41–56%.
The record leaves that remaining half to 2.9.0's new dial, which is still moving. Its results are not settled.