AI agent: fix a bug¶
A small agent fixes a failing test. K-Veritas records it from outside: every command, file change and connection. No API key, no framework, Linux.
| Agent | Plain Python, no model |
| Task | Make a failing test pass |
Plus the K-Veritas CLI.
The project¶
calc.py, with a bug:
test_calc.py:
import unittest
from calc import add
class TestAdd(unittest.TestCase):
def test_add(self):
self.assertEqual(add(2, 3), 5)
if __name__ == "__main__":
unittest.main()
fix_agent.py¶
Stands in for any agent. The KV agent lines are optional labels: what the agent says it does.
import subprocess
import sys
def run_tests():
return subprocess.run(["python3", "-B", "-m", "unittest", "-q"], capture_output=True, text=True).returncode == 0
print("KV agent tool=bash")
if run_tests():
print("Tests pass, nothing to do.")
sys.exit(0)
source = open("calc.py").read()
print("KV agent tool=edit target=calc.py")
open("calc.py", "w").write(source.replace("a - b", "a + b"))
print("KV agent tool=bash")
print("Fixed." if run_tests() else "Still failing.")
Record, seal, verify¶
kveritas agent --disclosure names -- python3 fix_agent.py
kveritas seal
kveritas verify kveritas-session-<id>.pdf
--disclosure names shows file names and commands in the report; the default shows neither.
VERIFIED
Witness: all 8 actions received by the server as they happened (max delay 0.7s)
Recorded: from outside by the system-call tracer (1 run)
Run 1: python3 fix_agent.py, exit 0, 0.5s, 3 commands
Activity: 3 commands, 1 file change, 0 connections, 3 file reads
Files: 1 touched, 1 still present, 0 deleted
Flagged: none
Labels: 3 from the agent, 3 checked, 0 with no matching effect
Actions:
(the agent itself) write 1, read 1
python3 -B -m unittest -q read 2, exit 1
python3 -B -m unittest -q exit 0
The first test run fails (exit 1), the agent edits calc.py, the second passes. Each label matches what happened.
An agent that cheats¶
Same task, but the agent changes the test and says it edited calc.py:
import subprocess
print("KV agent tool=edit target=calc.py")
test = open("test_calc.py").read()
open("test_calc.py", "w").write(test.replace("5)", "-1)"))
print("KV agent tool=bash")
subprocess.run(["python3", "-B", "-m", "unittest", "-q"])
Labels: 2 from the agent, 2 checked, 1 with no matching effect
entry 4 (label.edit) has no matching effect
Unlabeled: 1 file change in the project with no label
write test_calc.py
Without labels, the report still shows the write to test_calc.py.
Your own agent¶
Same command, any agent:
kveritas agent -- claude
kveritas agent -- python my_langchain_agent.py
kveritas agent -- npx some-agent
Interactive agents work as usual in the terminal. More: AI agents, Agent log format.