What Is Batch Processing and When to Use It

The Problem With Running 500 Prompts One at a Time

Imagine you need to generate 500 structured reports: one per customer, each requiring Hermes to read a file, call a tool, and produce a formatted JSON output. If you run them interactively, you are looking at hours of babysitting a terminal, manually kicking off the next prompt every time one finishes. You also have to worry about context accumulation -- each new prompt runs in a session that has already seen 50 previous ones, and that prior context can subtly warp the outputs in ways that are hard to detect.

Batch processing exists to solve exactly this. Instead of running prompts one after another in a shared session, Hermes's batch mode spins up hundreds of isolated agent environments in parallel, processes each prompt independently, and writes structured results to disk -- all without you watching.

What Batch Processing Actually Is

In Hermes, batch processing means running the agent across a dataset of prompts using a dedicated script called batch_runner.py. You provide a JSONL file where each line is a JSON object containing a prompt field. The runner distributes those prompts across a configurable pool of worker processes, each of which launches a full Hermes agent session, runs the prompt to completion, and writes the trajectory to disk.

The key word here is isolated. Each prompt gets its own clean environment. The agent starts with no conversation history, no leftover tool state, and no memory of the prompts that ran before it. This isolation is what makes batch outputs reproducible and comparable -- you are not measuring the cumulative effect of prior context, you are measuring how the agent handles each prompt on its own merits.

This is fundamentally different from running a loop of hermes chat commands. A loop inside an interactive session accumulates context. Batch mode does not.

Inside batch_runner.py: The Architecture

The batch runner operates as an orchestrator on top of the Hermes agent. Here is what happens when you invoke it:

• It reads your JSONL dataset and builds a queue of prompts.
• It spins up N worker processes (default: 4) via Python multiprocessing.
• Each worker claims a prompt from the queue, launches a Hermes agent session, and runs the prompt to completion (up to --max_turns tool-calling iterations).
• When the agent finishes, the worker serializes the full session as a trajectory and writes it to the output directory.
• The worker claims the next prompt and repeats.
• A checkpoint file tracks which prompts have completed, enabling safe resume after interruption.

The runner is not a thin wrapper -- it manages concurrency, checkpointing, output organization, and quality filtering. Understanding this architecture helps you configure it correctly and diagnose problems when they arise.

What Is a Trajectory?

A trajectory is the complete record of a single agent session. It includes the original prompt, every message exchanged between the user and the model, every tool call and its result, and the final response. It also captures statistics: how many tool calls were made, how many reasoning tokens were used, and whether the run passed quality filters.

Trajectories are stored in ShareGPT format, which is a standard structure used widely in the fine-tuning community. This means the output of a batch run is immediately usable for training or evaluation without any format conversion.

Think of a trajectory as a session recording. If you ran a prompt interactively and took notes on every step the agent took, that would be your trajectory. Batch mode generates those recordings automatically at scale.

The Three Primary Use Cases

Batch processing shines in three scenarios:

Training data generation. If you want to fine-tune a model on tool-use behavior, you need thousands of high-quality trajectory examples showing the model reasoning, calling tools, and producing correct outputs. Batch mode lets you generate those trajectories at scale across a diverse prompt set, with automatic quality filtering to remove low-quality samples.

Model evaluation. Evaluating an agent across a standardized benchmark requires running each benchmark prompt under identical conditions. Batch mode's isolation guarantee means your evaluation results reflect only the prompt, not accumulated session state. You can also run the same benchmark against multiple model configurations in parallel.

Structured output generation at scale. Sometimes you just need to process a large dataset: generate a summary for each document, extract entities from each record, or produce a structured JSON output for each input. Batch mode handles this efficiently and writes the results in a structured format you can process downstream.

When to Stick With Interactive Mode

Batch mode is not always the right tool. There are situations where interactive mode -- the normal hermes chat experience -- is clearly better:

• Exploratory work. When you do not yet know what prompt structure will work, you want to iterate quickly in a conversation, not batch 500 variations.
• Multi-turn problem solving. If your task requires genuine back-and-forth -- asking clarifying questions, refining outputs based on feedback -- batch mode cannot help. It runs each prompt to completion once.
• Human-in-the-loop tasks. Anything that requires a human to review intermediate results and decide what to do next belongs in interactive mode.
• Small scale. If you have 10 prompts, the overhead of configuring and running the batch system is not worth it. Just run them interactively.

A good mental model: use batch mode when you are sure about what you want and you need it done at a scale that would be impractical manually. Use interactive mode when you are still figuring out what you want.

Prerequisites Before You Begin

Batch processing requires a few things to be in place before you run your first job. You need a working Hermes installation with the agent configured for at least one provider. You need Python's uv package manager available, which Hermes installs automatically during setup. The batch_runner.py script lives inside the Hermes agent repository at ~/.hermes/hermes-agent/.

The next two lessons cover building your input dataset and running the runner with the right flags. Once those are in place, the rest of the track covers how to configure providers, manage credentials, and optimize costs -- all of which apply equally to batch runs and interactive sessions.