Skip to content

Checkpoints and branches

Save a point, try things, go back. Like git, except nothing can be erased: every step is sent to the server as it happens.

kveritas checkpoint -m "epoch 50 stable" --files model_ep50.pt
kveritas status                  # what changed since the checkpoint
kveritas restore 3c8e1a2b        # put the files back
kveritas branch quick-test       # try something on the side
kveritas set-aside 7d97432a -m "bad config"
kveritas log                     # see it all as a tree

Commands

Command What it does
checkpoint -m "..." Saves your code and config files. Add big files like a model with --files.
status Shows CLEAN, or the files that changed since your checkpoint.
restore <id> Puts the files back as they were. Unsaved changes are kept in a new checkpoint first.
branch <name> Labels the next runs, so side experiments stay apart. branch alone lists them.
set-aside <run> Moves a run out of the main results. --undo brings it back.
log The whole session as a tree. -v adds every result and changed file.

In your browser

kveritas web

Opens the same history as a page on your computer:

  • the tree of runs, checkpoints and branches,
  • CLEAN, or the files changed since your checkpoint,
  • buttons to save a checkpoint, switch branch, restore, or set a run aside.

Click a row to see its results. Right-click for actions. Drag the "you are here" row onto a checkpoint to restore it.

Only you can open it: it runs on your machine, and the link carries a secret key. On a remote machine, forward the port first:

ssh -L 7701:127.0.0.1:7701 <host>

Pick up where you left off

Your program has to save its own progress, like a model file. Then:

  1. Checkpoint after epoch 50, with --files model_ep50.pt.
  2. Something breaks at epoch 63.
  3. kveritas restore the checkpoint. Your code and model_ep50.pt come back.
  4. Run again from epoch 50. Only epochs 50 to 63 are redone.

The report shows the whole path: the run that broke, the restore, and the run that finished.

What you can and can't do

You can You can't
Save as many checkpoints and branches as you want Delete a run, failed ones included
Go back to any checkpoint Change a run after it ends
Set runs aside Hide that you restored, branched or set something aside
Continue later, even offline Remove set-aside runs from the report

In the report

  • Your current branch leads the results.
  • Set-aside runs and other branches get their own section, still signed.
  • A History table lists every step, in order.

Privacy

Saved files stay on your machine. The server only gets a fingerprint of each step.