Designing Your Input Dataset

JSONL: One Prompt Per Line

The batch runner reads input from a JSONL file -- JSON Lines format, where every line is a complete, self-contained JSON object. There is no outer array, no wrapper structure. Just one JSON object per line, each representing one prompt to run.

The minimum valid entry looks like this:

{"prompt": "Summarize the key risks in this earnings report. [file: report.txt]"}

That single field -- prompt -- is the only requirement. Everything else is optional. This simplicity is intentional: the format should be easy to generate programmatically, whether you are writing a Python script, exporting from a database, or building prompts from a template.

A dataset file might contain 10 lines or 10,000 lines. The batch runner processes them all the same way regardless of size, checkpointing progress so you can resume safely if the run is interrupted.

The Required Field: prompt

The prompt field is what the agent receives as its first user message. Write it exactly as you would type it into a Hermes chat session. Everything that applies to interactive prompts applies here: you can reference files, include context, specify output formats, and use any Hermes-compatible syntax.

The quality of your trajectories depends heavily on prompt quality. Vague prompts produce vague trajectories that are noisy as training data. Specific, well-structured prompts produce clear tool-use patterns that make better training examples. Spend time on your prompts -- they are the foundation of everything else.

For training data generation specifically, you want diversity. A dataset of 500 prompts that all ask the agent to summarize documents will produce trajectories that only show document-summarization behavior. If you want a model trained on these trajectories to generalize, your prompts need to cover a wide range of tasks, difficulty levels, tool-use patterns, and domains.

Optional Fields: Image and Docker Overrides

Two optional fields let you control the execution environment on a per-prompt basis.

The docker_image field specifies a Docker image to use for that prompt's agent session. This is useful when you have prompts that need specific dependencies, language runtimes, or toolsets that should not pollute a shared environment. Different prompts in the same dataset can use different images.

The image field is the shorthand alias for the same thing. Use whichever reads more naturally in your context -- they do the same thing.

The cwd field sets the working directory for the agent session. This is valuable when your prompts reference files at specific paths. Instead of hardcoding absolute paths in every prompt, you can set the working directory to where the relevant files live and use relative paths in the prompt text itself.

A more complete entry using these fields:

{"prompt": "Run the test suite and report which tests fail.", "docker_image": "python:3.12-slim", "cwd": "/workspace/my-project"}

Designing for Training Data Quality

If your goal is training data, the structure of your dataset directly determines the quality of what you get out. Here are the key design principles:

Cover multiple task types. Include prompts that require file reading, code execution, web search, multi-step reasoning, and output formatting. A model trained on trajectories that only show one task type will struggle to generalize.

Balance difficulty levels. Mix easy prompts (single tool call, clear answer) with hard ones (multiple tools, ambiguous requirements, error recovery). The hard prompts are often more valuable for training because they show the agent handling complexity.

Include negative examples. Prompts that are ambiguous, contradictory, or incomplete force the agent to handle edge cases gracefully. Those trajectories teach the model how to ask for clarification or handle failure modes.

Vary the domain. Technical prompts, creative prompts, analytical prompts, coding prompts -- spread them across domains so the resulting model is not narrowly specialized.

Keep prompts self-contained. Each prompt runs in isolation, so it cannot rely on context established in a previous prompt. Every prompt must provide all the context the agent needs to complete the task.

Designing for Evaluation

Evaluation datasets have different requirements than training datasets. For evaluation, you want consistency and comparability above all.

Each prompt should have a clear, verifiable correct answer or a well-defined rubric for scoring. Open-ended prompts that could be answered many different ways make evaluation results hard to interpret.

Standardize the output format across your benchmark. If some prompts ask for JSON and others ask for markdown tables, comparing results across prompts becomes complicated. Pick a format and use it consistently.

Version your evaluation dataset. If you update prompts between evaluation runs, you cannot directly compare results from different runs. Treat the evaluation dataset as immutable once you start collecting results.

Generating Datasets Programmatically

For large datasets, hand-writing prompts does not scale. Most practitioners generate JSONL datasets programmatically from a template or a source data collection.

A simple Python script can read a CSV of customer records, fill a prompt template with each record's fields, and write one JSONL line per record. The result is a dataset that is consistent, reproducible, and easy to modify by changing the template.

You can also use Hermes interactively to generate prompts for batch processing -- ask it to produce 50 diverse coding prompts, format them as JSONL, and write the file. Then run that file through the batch runner. This self-referential workflow is surprisingly effective for building training datasets quickly.

Whatever method you use to generate the dataset, validate the JSONL file before running the batch. Every line must be valid JSON, every line must have a prompt field, and the file must not have a trailing newline that produces an empty last entry. A quick Python one-liner -- for line in open('file.jsonl'): json.loads(line) -- will catch malformed lines before they cause a batch failure midway through.

File Organization

Keep your dataset files organized alongside your batch run outputs. A simple convention: store your input JSONL files in a data/ directory with descriptive names that include the date and purpose -- for example, data/2026-07-coding-evals.jsonl or data/2026-07-customer-reports.jsonl.

The batch runner writes outputs to data/<run_name>/, so your inputs and outputs stay co-located. When you come back to a batch run three months later, you want to be able to find both the input that produced the results and the results themselves without searching through a disorganized directory tree.