ControlNet: Structure and Composition Control

How ControlNet Controls Your Image

ControlNet is an additional neural network that runs alongside your checkpoint and forces the generated image to follow a specific structure. You provide a control image — an edge map, a depth map, a skeleton — and ComfyUI uses it to constrain the composition while your text prompt handles the visual style. The result is generated images that respect a predetermined layout with whatever artistic treatment your prompt describes.

This makes ControlNet essential for any work where you need consistent compositions: product photography with controlled angles, character poses that match a reference, architectural renders with correct depth, or illustrations built on a sketch.

Installation and Setup

Standard ComfyUI does not include ControlNet preprocessors by default. You need two things:

• Preprocessor nodes: Install the comfyui_controlnet_aux custom node package via ComfyUI-Manager. Search for "controlnet aux" and install it. This adds preprocessor nodes like CannyEdgePreprocessor, MidasDepthMapPreprocessor, and DWPreprocessor to your node menu.
• ControlNet models: Download .safetensors files for each ControlNet type you want to use. These go in your ComfyUI/models/controlnet/ folder. Common sources are Hugging Face (search "lllyasviel/ControlNet-v1-1") and CivitAI. Choose models that match your base checkpoint architecture — SD 1.5 ControlNets do not work with SDXL checkpoints.

After installing the custom nodes, restart ComfyUI and the preprocessor nodes will appear in the node menu under the ControlNet category.

The ControlNet Node Chain

A ControlNet workflow adds four nodes to your standard text-to-image graph:

1. Load Image: loads your reference photo or sketch
2. Preprocessor node (e.g., CannyEdgePreprocessor): converts your image into a control map that the ControlNet model understands
3. Load ControlNet Model: loads the .safetensors file from models/controlnet/
4. Apply ControlNet: takes your positive conditioning, the control image, and the loaded model as inputs, and outputs modified conditioning that carries the structural guidance

The Apply ControlNet node's output (conditioning) plugs into the positive conditioning input of your KSampler, replacing or supplementing your normal CLIP Text Encode output. The rest of the workflow — negative conditioning, latent, VAE Decode — stays the same.

Preprocessor Types and When to Use Each

Each preprocessor extracts different structural information from your reference image:

• Canny Edge (CannyEdgePreprocessor): detects all edges in the image as a black-and-white line drawing. Best for: preserving exact shapes, copying architectural outlines, or using a clean sketch as a reference. Parameters: low_threshold and high_threshold control edge sensitivity.
• Depth Map (MidasDepthMapPreprocessor): generates a grayscale depth map showing near-to-far distance. Best for: preserving 3D spatial relationships and scene depth without locking down exact shapes. Works well with any photo.
• OpenPose / DWPose (DWPreprocessor): detects human body skeleton keypoints (joints, face landmarks). Best for: matching a specific human pose exactly. The output is a colored skeleton overlay, not a photo.
• Scribble (ScribblePreprocessor): converts an image to a rough line sketch that gives loose structural guidance. Best for: when you want rough composition control without strict edge adherence.
• Normal Map (MeshGraphormerDepthMapPreprocessor): detects surface normal directions. Best for: precise 3D surface lighting and material rendering.

Add a Preview Image node (right-click the preprocessor output) to see what structure the ControlNet will actually receive before you run the full generation. This catches problems early.

Tuning ControlNet Strength

The Apply ControlNet node has a strength parameter (0.0 to 2.0, default 1.0). This controls how strictly the generated image must follow the control map:

• 1.0: full adherence — the output closely follows the control structure
• 0.5–0.8: loose guidance — the composition follows roughly but allows creative variation
• 0.3: very light touch — barely influences composition; mostly just nudges it
• Above 1.2: can cause artifacts; rarely needed

Start at 0.8 for most workflows. Lower if the ControlNet is fighting your prompt too aggressively. The start_percent and end_percent parameters (visible in advanced mode) let you apply ControlNet only during a specific portion of the sampling process — applying it from 0.0 to 0.5 means it influences early structure without constraining final detail.

Stacking Multiple ControlNets

You can chain multiple Apply ControlNet nodes. For example: OpenPose (pose) → Apply ControlNet → Canny (edges) → Apply ControlNet → KSampler. Each conditioning output feeds into the next Apply ControlNet as input. Lower the strength on each one when stacking to avoid over-constraining the generation. A common stack for character work is OpenPose at 0.7 and Depth at 0.5.