Checkpointing, Resuming, and Output Structure

Why Checkpointing Matters

A batch of 1,000 prompts might take several hours to complete. During that time, many things can go wrong: your laptop goes to sleep, your network connection drops, your API provider returns a 429 rate-limit error that crashes the process, or you simply need to stop and restart the machine. Without checkpointing, you would need to start over from prompt one every time.

The Hermes batch runner checkpoints progress continuously. Every time a prompt completes, its result is immediately written to disk and the checkpoint file is updated. If the process stops for any reason, you can resume it and it will pick up exactly where it left off -- skipping every prompt that already completed and only processing the ones that did not.

This is not just a convenience feature. For large datasets, checkpointing is what makes batch processing practically viable. A job that would be risky to run overnight becomes safe when you know a crash at hour three does not erase the previous three hours of work.

The Output Directory Structure

Every batch run writes to data/<run_name>/. Inside that directory, you will find four types of files:

trajectories.jsonl is the merged output file containing every completed trajectory from the run. This is the file you use downstream -- for fine-tuning, for evaluation scoring, for any analysis of the results. When the run finishes, this file is the deliverable.

batch_N.jsonl files are intermediate batch files, one per batch of prompts. If you specified --batch_size=10 and ran 100 prompts, you will have batch_1.jsonl through batch_10.jsonl. These are merged into trajectories.jsonl at the end. During a run, you can inspect these files to see results as they come in without waiting for everything to complete.

checkpoint.json tracks which prompts have been completed, identified by their content (not by index). This file is what enables safe resume. Do not delete or modify it while a run is in progress.

statistics.json contains aggregate metrics for the completed run: total prompts, prompts filtered, average tool calls per session, total reasoning tokens used, and other summary statistics. After the run completes, this file gives you a quick health check on the dataset quality.

Resuming an Interrupted Run

Resuming is straightforward. Use exactly the same command you used to start the run, but add --resume:

python batch_runner.py --dataset_file=data/prompts.jsonl --batch_size=10 --run_name=my_first_run --resume

The runner reads the checkpoint file, identifies which prompts have already completed, skips them, and processes only the remaining ones. The output files are updated in place -- new trajectories are appended to the existing batch files, and the checkpoint is updated as each new prompt completes.

You do not need to pass any additional flags or modify any configuration. The runner remembers everything about the original run from the checkpoint file and the existing output directory. Just add --resume and it picks up where it left off.

Content-Based Matching: Why It Is More Robust Than Index-Based

Most checkpointing systems track progress by index -- they remember that prompts 0 through 347 are done and resume from 348. This is fragile. If you need to add new prompts to your dataset, reorder existing prompts, or remove some entries before resuming, the index-based checkpoint becomes wrong. It might re-run prompts that already completed or skip prompts that have not been processed yet.

The Hermes batch runner uses content-based matching instead. It identifies completed prompts by their actual text content, not by their position in the file. This means you can safely reorder your dataset, insert new prompts at any position, or remove entries -- the runner will correctly identify which prompts have completed regardless of where they now appear in the file.

Content-based matching does have one implication: if you modify the text of a prompt that has already completed, the runner treats it as a new, unprocessed prompt. This is the correct behavior -- a changed prompt should be re-run -- but it means you should not modify prompts midway through a run unless you actually want them re-processed.

Inspecting Results Before the Run Finishes

Because intermediate results are written to the batch files in real time, you can inspect trajectories before the full run completes. Open any batch_N.jsonl file and read the entries. Each line is a complete trajectory in ShareGPT format that you can examine immediately.

This is especially valuable for long runs. If you start a 10-hour batch and want to know whether the outputs are looking good at hour two, just open one of the completed batch files. You do not have to wait until the end to catch problems.

If you inspect early results and find a systematic issue -- the agent is misinterpreting a common prompt pattern, or a tool is returning unexpected errors -- you can stop the run, fix the underlying problem, and resume. The work you did before stopping is preserved.

Reading the Statistics File

After a run completes, statistics.json gives you a quick health assessment. The most important metrics to check:

• Filter rate. What percentage of prompts were filtered by quality checks? A filter rate above 20% suggests your prompts are systematically producing low-quality trajectories -- either they are too simple (not triggering reasoning) or the model is struggling with them.
• Average tool calls. This tells you how complex the agent's behavior was. Very low average tool calls (close to 0) means the agent is answering most prompts without using tools, which may indicate prompts that do not require tool use at all.
• Average reasoning tokens. Zero reasoning tokens means the model is not using its reasoning capability. For training data intended to produce a model that reasons, this is a problem.

These signals should inform whether you run the next batch with adjusted prompts or configuration before scaling up further.