Skip to content

AI agent: fix a bug

A small agent fixes a failing test. K-Veritas records it from outside: every command, file change and connection. No API key, no framework, Linux.

Agent Plain Python, no model
Task Make a failing test pass

Plus the K-Veritas CLI.

The project

calc.py, with a bug:

def add(a, b):
    return a - b

test_calc.py:

import unittest
from calc import add


class TestAdd(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add(2, 3), 5)


if __name__ == "__main__":
    unittest.main()

fix_agent.py

Stands in for any agent. The KV agent lines are optional labels: what the agent says it does.

import subprocess
import sys


def run_tests():
    return subprocess.run(["python3", "-B", "-m", "unittest", "-q"], capture_output=True, text=True).returncode == 0


print("KV agent tool=bash")
if run_tests():
    print("Tests pass, nothing to do.")
    sys.exit(0)

source = open("calc.py").read()
print("KV agent tool=edit target=calc.py")
open("calc.py", "w").write(source.replace("a - b", "a + b"))

print("KV agent tool=bash")
print("Fixed." if run_tests() else "Still failing.")

Record, seal, verify

kveritas agent --disclosure names -- python3 fix_agent.py
kveritas seal
kveritas verify kveritas-session-<id>.pdf

--disclosure names shows file names and commands in the report; the default shows neither.

VERIFIED
Witness:    all 8 actions received by the server as they happened (max delay 0.7s)
Recorded:    from outside by the system-call tracer (1 run)
  Run 1:     python3 fix_agent.py, exit 0, 0.5s, 3 commands
Activity:    3 commands, 1 file change, 0 connections, 3 file reads
Files:       1 touched, 1 still present, 0 deleted
Flagged:     none
Labels:      3 from the agent, 3 checked, 0 with no matching effect
Actions:
  (the agent itself)  write 1, read 1
  python3 -B -m unittest -q  read 2, exit 1
  python3 -B -m unittest -q  exit 0

The first test run fails (exit 1), the agent edits calc.py, the second passes. Each label matches what happened.

An agent that cheats

Same task, but the agent changes the test and says it edited calc.py:

import subprocess

print("KV agent tool=edit target=calc.py")
test = open("test_calc.py").read()
open("test_calc.py", "w").write(test.replace("5)", "-1)"))

print("KV agent tool=bash")
subprocess.run(["python3", "-B", "-m", "unittest", "-q"])
Labels:      2 from the agent, 2 checked, 1 with no matching effect
  entry 4 (label.edit) has no matching effect
Unlabeled:   1 file change in the project with no label
  write test_calc.py

Without labels, the report still shows the write to test_calc.py.

Your own agent

Same command, any agent:

kveritas agent -- claude
kveritas agent -- python my_langchain_agent.py
kveritas agent -- npx some-agent

Interactive agents work as usual in the terminal. More: AI agents, Agent log format.