Learn › ComfyUI Advanced: Training, Kontext, and Automation › LoRA Training, Part 1: Building Your Dataset

LoRA Training, Part 1: Building Your Dataset

Advanced 🕐 15 min Lesson 6 of 12
What you'll learn
  • Identify the VRAM requirements for FLUX LoRA training and describe the Vast.ai GPU rental process
  • Prepare a character training dataset: image count, angle variety, resolution, and background guidance
  • Caption training images correctly: trigger word placement, what to describe, what not to describe, and which auto-captioning tool to use for FLUX
  • Set up the correct Kohya folder structure and calculate the right repeat count to hit a 2000-3000 step target

What LoRA Training Actually Does

You know from ComfyUI Image Generation how to load and apply a LoRA — the small weight file that pushes a checkpoint toward a specific style or character when you include its trigger word. Training a LoRA means creating that file from scratch, using your own images.

The training process adjusts a small set of additional weights layered on top of the base model. The base model itself does not change. Instead, the LoRA learns to associate your trigger word with the visual patterns in your training images. When you call the trigger word at generation time, those weights activate and steer the output toward your subject or style.

The quality of the LoRA depends almost entirely on two things: the quality of your images, and the accuracy of your captions. Poor images produce poor LoRAs regardless of how many training steps you run. Inaccurate captions produce LoRAs that either fail to activate reliably or activate the wrong attributes. This lesson is about getting both right before training starts.

Hardware: What FLUX LoRA Training Actually Requires

FLUX LoRA training is demanding. At standard precision, training FLUX.1 [dev] at 1024px requires:

  • 24 GB VRAM minimum — enough to run training at 512px resolution or with aggressive gradient checkpointing
  • 48 GB VRAM for 1024px training at comfortable speed (approximately 90 minutes for a 2,000-step session)

If your GPU has less than 24 GB VRAM, you have two options:

  • SimpleTuner with NF4 quantization: runs FLUX LoRA training in approximately 9 GB VRAM by heavily quantizing the model during training. Quality is slightly lower than full-precision training but the results are usable. This requires setting up SimpleTuner (a Python training toolkit) rather than Kohya.
  • Rent a GPU: the most practical approach for a single training session. A 24 GB GPU on Vast.ai typically costs $0.30–$0.60 per hour. A full training session runs 1–2 hours. Total cost: under $1.50.

For SDXL LoRA training (a simpler model), 12 GB VRAM is sufficient and the process runs well on a local GPU if you have one.

Renting a GPU on Vast.ai

Vast.ai is a GPU marketplace where independent providers rent out their hardware. Prices are generally lower than dedicated AI cloud providers, and the selection includes RTX 4090, RTX 3090, A10G, and A100 instances.

Setup (one-time):

  1. Create an account at vast.ai and deposit at least $5 to credit your balance.
  2. Generate an SSH key pair locally: run ssh-keygen -t ed25519 -C "vastai" in your terminal. This creates a private key (keep it secure) and a public key (.pub file).
  3. In the Vast.ai console, go to Account → SSH Keys and paste the contents of your public key. This must be done before you create an instance — the key is applied at container creation and cannot be added afterward.

Finding and renting an instance:

  1. Go to Search in the console. Filter by GPU type (search for "RTX 4090" or "RTX 3090"), minimum RAM, and price. For LoRA training, you need 24 GB GPU RAM and at least 16 GB system RAM.
  2. For the template, select a PyTorch image (e.g., pytorch/pytorch:2.x-cuda12.x-cudnn9-runtime) or a Kohya-preconfigured template if one is available. A preconfigured Kohya template saves several hours of setup.
  3. Set Direct SSH Access to enabled — this is required to connect from your terminal.
  4. Click Rent. The instance starts within a few minutes.

Connecting and transferring files:

  • Once the instance is running, find it under Instances. Copy the SSH command shown (format: ssh -p PORT root@HOSTNAME).
  • Connect from your terminal using that command and your private key: ssh -i ~/.ssh/id_ed25519 -p PORT root@HOSTNAME.
  • Upload your training images using scp or rsync: scp -P PORT -r ./my-dataset/ root@HOSTNAME:/workspace/
  • Download your finished LoRA the same way: scp -P PORT root@HOSTNAME:/workspace/output/lora.safetensors ./

Billing: GPU charges accrue while the instance is running. Click Stop in the console to pause billing when not actively training — the instance persists but the GPU is released. Storage still costs while stopped, so destroy the instance once you have downloaded your trained LoRA.

RunPod as an Alternative

RunPod is a similar GPU cloud service with a more polished interface and less price volatility than Vast.ai, at a slightly higher cost. It offers pre-built pod templates specifically for Kohya and Stable Diffusion training. If the Vast.ai marketplace-style instance selection feels complex, RunPod's templates provide a simpler starting point. The training workflow is identical once you are connected — only the rental interface differs.

Image Selection: Character Datasets

A character LoRA needs enough images to teach the model what the character looks like from multiple angles and in different contexts, without overfitting to a single look.

Quantity: 15–30 images is the practical range. Fewer than 15 often produces an unstable LoRA; more than 40–50 images with a character that has consistent appearance provides diminishing returns at the cost of longer training time.

Angle and expression variety: Include a mix of front-facing, three-quarter view, and side views. Include different expressions if they are part of the character's identity. A dataset of 25 identical front-facing images will produce a LoRA that struggles when you ask for the character from the side.

Resolution: 1024×1024 is the target for FLUX training. Images smaller than 512px in any dimension will be stretched or padded, which degrades the training signal. Crop images to square or near-square if they have significant empty space around the subject.

Background variety: A character consistently photographed on a white studio background may cause the LoRA to associate that background with the trigger word. Include images with varied backgrounds, or use a background removal tool to isolate the subject and place them on different solid colours across the dataset.

Consistency in key features: The features that define the character — hairstyle, eye colour, distinctive clothing — should be consistent across all images. Images where the character is in costume, wearing unusual lighting, or is partially obscured should be excluded or treated carefully.

Image Selection: Style Datasets

Style LoRA training has different requirements from character training. You are teaching the model a visual aesthetic, not an identity.

Fewer images often work: 10–20 high-quality images representing the style clearly often produces better results than 50 varied images that cover the style inconsistently. Quality and consistency of the style expression matters more than quantity.

Avoid describing the style in captions: This is the single most important rule for style training. If you caption your oil painting images with "oil painting, thick brushstrokes, impasto technique," the model learns that these text tokens produce that style — and the style bleeds into unrelated prompts that happen to include similar words. For style training, describe only the subject content ("a woman in a garden"), not the visual style. The style should be learned from the images, not the captions.

Avoid mixed styles: A dataset containing both loose watercolour and tight technical illustration will produce a confused LoRA that does neither cleanly. Pick one coherent aesthetic and stick to it across the whole dataset.

Captioning Your Dataset

Every training image needs a text caption file (a .txt file with the same name as the image). The caption tells the model what to credit to its trigger word versus what to associate with the general content. Poor captions are the most common cause of LoRA failure.

Auto-captioning tools: Generating captions manually for 25 images is feasible but slow. Two tools produce good results automatically:

  • Florence-2: Available as a ComfyUI custom node (comfyui-florence2 via Manager). Load your images in batch and run them through the Florence-2 detailed captioning node. It produces natural language descriptions. Edit the outputs to add your trigger word and remove any misidentified content.
  • WD14 Tagger: Available as a ComfyUI custom node (comfyui-wd14-tagger). Generates comma-separated tag lists rather than sentences. Better for SD 1.5 and SDXL training; Florence-2 produces better results for FLUX which handles natural language well.

The trigger word: Choose a word that is rare or absent in the base model's training data. Common words ("woman," "man," "painting") will activate broadly and produce a weak trigger signal. Invented words, uncommon names, or deliberate misspellings work best: "zrn1char," "jksmith2024," "mxstyle01." The trigger word should be placed at the very start of every caption.

Caption format:

  • For character training: Trigger word first, then clothing and accessories, then setting and lighting. Do not describe facial features — the model should learn faces from the images, not the captions. Example: jksmith2024, wearing a blue denim jacket and white sneakers, standing on a city street, natural daylight
  • For style training: Trigger word first, then subject content only. Example: mxstyle01, a woman reading in a garden

Review every caption manually. Auto-captioning tools make mistakes — misidentifying clothing colour, describing reflections as additional subjects, missing key attributes. Each wrong caption adds noise to the training. Spend time reviewing and correcting the auto-generated captions before training starts.

Folder Structure for Kohya

Kohya expects a specific folder structure. The training images and their caption .txt files go inside a folder whose name encodes two pieces of information: the number of times each image is repeated per epoch, and the training concept.

The folder naming convention is: {repeat_count}_{trigger_word}

Example: a folder named 10_jksmith2024 tells Kohya to use each image in that folder 10 times per epoch. The full training steps are calculated as: images × repeats × epochs.

For a 20-image character dataset targeting 2,000 steps at 5 epochs:

  • 20 images × 20 repeats × 5 epochs = 2,000 steps
  • Folder name: 20_jksmith2024

Targeting 2,000–3,000 total steps is a good starting point for FLUX LoRA training. Under 1,000 steps usually produces an under-trained LoRA that activates weakly; over 5,000 steps can produce overfitting where the trigger word overrides other prompt elements.

The complete folder structure Kohya expects:

dataset/
└── img/
    └── 20_jksmith2024/
        ├── image01.jpg
        ├── image01.txt
        ├── image02.jpg
        ├── image02.txt
        └── ...

Lesson 7 covers the Kohya training interface, setting the training parameters, and running the training session on your rented GPU. Lesson 8 covers evaluating the result and diagnosing common failure modes.

Key takeaways
  • FLUX LoRA training requires 24 GB VRAM minimum; renting an RTX 4090 on Vast.ai at roughly $0.40/hour means a complete training session costs under $1.50
  • Upload your SSH public key in Vast.ai before creating an instance — it is applied at container creation and cannot be added afterward
  • For character training: 15-30 images, varied angles and backgrounds, 1024px target resolution; do not describe facial features in captions — let the model learn them from the images
  • The trigger word must be rare or invented — common words produce weak LoRAs because they already carry strong associations in the base model
  • Total training steps = images × repeats × epochs; target 2000-3000 steps for FLUX as a starting point, with folder name {repeats}_{trigger_word} encoding the repeat count for Kohya
Learn more every week Get 5 hand-picked prompts + new AI tools every Monday. Free.