Running Your First Batch

The Basic Command

Once you have a JSONL dataset ready, running a batch is a single command:

python batch_runner.py --dataset_file=data/prompts.jsonl --batch_size=10 --run_name=my_first_run

That is the minimum viable invocation. It reads your dataset, processes prompts in groups of 10, and writes results to data/my_first_run/. The default configuration uses 4 parallel workers and the model configured in your Hermes settings.

The batch runner lives inside the Hermes agent repository at ~/.hermes/hermes-agent/. Navigate there before running it, or provide the full path. Everything else -- API keys, model selection, tool configuration -- comes from your existing Hermes configuration, so there is nothing extra to set up if Hermes is already working interactively.

Required Flags

Two flags are required every time you run the batch runner:

--dataset_file points to your input JSONL file. Provide a path relative to the current directory or an absolute path.

--run_name names this batch run. The runner creates a directory at data/<run_name>/ and writes all outputs there. Choose a name that will be meaningful when you look at it later -- something that encodes the date, the dataset, and the model is a good pattern.

There is no --batch_size default -- it is required in practice even though the documentation lists it as configurable. A batch size of 10 is a reasonable starting point for most jobs.

Key Optional Flags

The optional flags are where you tune the batch for your specific use case.

--num_workers controls how many agent sessions run in parallel. The default is 4. Increasing this speeds up throughput but increases your API usage rate. If you hit rate limits from your provider, lowering this is the first thing to try. A good rule of thumb: set it to the number of parallel requests your API tier allows, not higher.

--model overrides the model you use for this batch run. This is useful for running the same dataset against multiple models for comparison, or for using a cheaper model for a first-pass run before committing to an expensive one for the final dataset.

--max_turns limits how many tool-calling iterations the agent can make per prompt. The default is 10. For simple prompts that should require only 1 or 2 tool calls, lowering this prevents runaway sessions from consuming credits when the agent gets confused. For complex multi-step tasks, you may need to raise it.

--reasoning_effort controls how much reasoning the model is allowed to do per turn. Higher reasoning effort produces better outputs but costs more tokens and takes longer. For training data generation where output quality matters, you might want full reasoning effort. For evaluation runs where you want consistent, comparable behavior, a fixed effort level keeps results comparable across runs.

Listing Available Toolsets

Before you run a batch, it is worth checking which toolsets are available in your Hermes setup:

python batch_runner.py --list_distributions

This prints the available toolset configurations -- combinations of tools that can be activated for a batch run. Different toolsets are appropriate for different task types: a coding evaluation needs code execution tools, a web research task needs browser tools, a file processing job needs file system tools.

If your prompts require specific tools, confirm the right toolset is available before starting a large batch. Discovering halfway through a 1,000-prompt run that the tool your prompts rely on is not active is an expensive mistake.

Reading Live Output

The batch runner prints progress to stdout as it runs. You will see lines like:

• Worker started for prompt index 42
• Prompt 42 completed in 8.3s -- 3 tool calls, 2,400 reasoning tokens
• Batch 5 of 50 complete -- 100 prompts done, 900 remaining
• Quality filter: prompt 67 removed -- no reasoning tokens found

The completion lines tell you throughput per prompt and whether the session used reasoning. The quality filter lines tell you which prompts were excluded and why. Pay attention to the filter lines -- if a large fraction of your prompts are being filtered, your prompts may be too simple (not triggering reasoning) or the model may be struggling with them.

If you see error lines, read them carefully before letting the run continue. A systematic error -- a missing tool, a malformed prompt template, a rate limit being hit -- will repeat on every subsequent prompt. It is better to stop, fix the issue, and resume than to let 900 prompts fail with the same error.

Adjusting Throughput vs. Cost

The throughput/cost tradeoff is the central tuning decision in batch processing. Here is how the key flags interact:

The most expensive mistake in batch processing is running a full dataset at high reasoning effort only to discover that half the prompts are poorly designed and produce unusable trajectories. Always run a small test batch first -- 20 to 50 prompts -- and inspect the trajectories before committing to the full run.

What Success Looks Like

A successful batch run ends with the runner printing a summary: total prompts processed, prompts filtered by quality checks, average tool calls per session, and average reasoning token usage. The data/<run_name>/ directory contains the trajectory files, a checkpoint file, and a statistics JSON file with the aggregate metrics.

Open a few trajectory files and read them. Are the tool calls sensible? Does the reasoning match what you expected? Are the outputs formatted correctly? Spot-checking trajectories after a run is essential -- the quality metrics give you aggregate signals, but only reading actual trajectories tells you whether the agent is behaving the way you intended.