AnimateDiff: Text-to-Video Basics

How AnimateDiff Works

AnimateDiff adds motion to Stable Diffusion 1.5 by injecting a temporal attention layer — a motion module — into the existing image generation process. Instead of generating one image, the model generates a sequence of frames that flow smoothly from each other. Because it runs on top of SD 1.5, every technique you know from image generation carries over: any SD 1.5 checkpoint, any LoRA, any ControlNet. You get animated versions of your existing image workflows.

The result is short animated clips: typically 8 to 32 frames at around 8 FPS, creating 1–4 second loops. For longer, higher-quality video, Wan 2.2 and HunyuanVideo (covered in later lessons) are better choices. AnimateDiff's strength is creative flexibility — the entire SD 1.5 ecosystem applies to animation.

Installation

AnimateDiff requires a custom node package and a motion module model file:

Step 1: Install the custom node

1. Open ComfyUI-Manager → Install Custom Nodes
2. Search for "AnimateDiff Evolved" and install "ComfyUI-AnimateDiff-Evolved" (by Kosinkadink)
3. Restart ComfyUI

Step 2: Download the motion module

Download mm_sd_v15_v3.safetensors from Hugging Face (search "guozinan/AnimateDiff" or "comfyanonymous/AnimateDiff"). Place the file in:

• ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models/

This is the correct location for AnimateDiff-Evolved — not the standard models/ directory. The motion module is separate from your SD 1.5 checkpoint and specifically handles temporal animation.

Building the Workflow

An AnimateDiff workflow is an SD 1.5 text-to-image workflow with three additions:

1. AnimateDiff Loader node: loads the motion module. Connect its output to the model flow between Load Checkpoint and KSampler. Set motion_model to mm_sd_v15_v3.safetensors and beta_schedule to "sqrt_linear" (required for SD 1.5 compatibility).
2. Context Options node: controls how many frames to generate and how the temporal attention spans across the clip. Set context_length to 16 (standard for 16-frame clips) and context_stride to 1. Connect this to the AnimateDiff Loader's context_options input.
3. Video Combine node: from the Video Helper Suite package (install separately via Manager — search "VideoHelperSuite"). Connects to the IMAGE batch output and saves frames as a GIF or MP4. Set frame_rate to 8 and format to "video/h264-mp4".

The Empty Latent Image node controls the video resolution. Use 512×512 for SD 1.5 AnimateDiff. The batch_size parameter in Empty Latent Image should be set to match the number of frames you want to generate (e.g., 16 for a 16-frame clip).

Key AnimateDiff Settings

• Frame count (batch_size in Empty Latent Image): 8–32 frames. More frames use more VRAM and time. 16 frames at 8 FPS = 2 seconds.
• beta_schedule: sqrt_linear is required for SD 1.5. Using the wrong schedule produces noisy, unstable animation.
• KSampler steps: use 20 steps minimum for AnimateDiff; fewer steps produce flickery output because temporal consistency depends on thorough denoising.
• Seed: fixed seed produces loopable animations; random seed varies the motion each run.

Getting the Motion Example Workflows

The ComfyUI-AnimateDiff-Evolved GitHub repository includes example workflow JSON files for standard text-to-video, LoRA animation, and ControlNet animation. Download the relevant workflow JSON and drag it directly onto the ComfyUI canvas to load it fully configured. This is faster than building from scratch and ensures the node wiring is correct. The example workflows are in the repository's examples/ folder on GitHub.

Prompting for AnimateDiff

Prompt AnimateDiff the same way you prompt SD 1.5 for images — tag-based with quality tokens. Add motion-oriented descriptors to influence the type of movement: "slow camera pan," "gentle breeze," "flowing hair," "camera dolly forward," "time-lapse clouds." The model responds to these terms through the temporal attention layer. More specific motion language produces more predictable motion types; abstract prompts produce more organic but less controlled movement.