Video-to-Video: Applying Styles to Existing Footage

Taking Control of Existing Footage

Video-to-video (V2V) means using an existing video clip as the starting point for generation rather than creating from scratch. You might convert a webcam recording into an illustrated animation, apply a painted art style to a live-action clip, or use reference footage as a structural guide while replacing the visual style entirely.

In ComfyUI, video-to-video works by treating the video as a sequence of images: extract the frames, process each frame through a standard image generation pipeline (img2img, ControlNet, or both), and reassemble the frames into a new video. The primary tool for loading and saving video in ComfyUI is the Video Helper Suite.

Installing Video Helper Suite

Video Helper Suite (VHS) is a custom node package that handles all video file operations in ComfyUI:

1. Open ComfyUI-Manager → Install Custom Nodes
2. Search for "VideoHelperSuite" and install "ComfyUI-VideoHelperSuite"
3. Restart ComfyUI

VHS adds several key nodes: Load Video, Video Combine, Merge Image Batches, and frame selection utilities. It also handles audio extraction and reinsertion, so you can process video without losing the original audio track.

The Video-to-Video Workflow Structure

The core V2V workflow has four stages:

1. Load: VHS Load Video → extracts frames from an MP4, GIF, or other video file as an IMAGE batch. Set frame_load_cap to limit how many frames you process (start small — 24 frames = 1 second at 24 FPS).
2. Preprocess (optional): run each frame through a ControlNet preprocessor if you want structural guidance. Connect the IMAGE batch to a CannyEdgePreprocessor or MidasDepthMapPreprocessor to extract structure maps from every frame simultaneously.
3. Generate: run the IMAGE batch through your generation pipeline. For img2img style transfer, use VAE Encode → KSampler (denoise 0.6–0.75) → VAE Decode. For ControlNet-guided V2V, apply your ControlNet conditioning to the full frame batch.
4. Reassemble: Video Combine takes the processed IMAGE batch and saves it as a new video file at the same FPS as the original.

Controlling Temporal Coherence

Processing frames independently produces flickering — each frame looks good on its own but neighboring frames vary in subtle ways that become obvious in motion. Several approaches reduce this:

• Fixed seed: use the same seed for all frames. This makes the generation more consistent across frames because the random noise starting point is the same. The trade-off is slightly less variety in the generated content.
• Low denoise (0.5–0.65): keeping the denoise lower means more of the original frame's structure is preserved across each frame, reducing drift between neighboring frames.
• AnimateDiff on video frames: pipe the frames through AnimateDiff instead of a standard KSampler. AnimateDiff's temporal attention layer specifically addresses frame-to-frame coherence.
• ControlNet from original video: extract ControlNet structure maps (canny or depth) from the original video frames and use them to guide the styled output. This keeps the composition coherent while transforming the visual style.

Practical V2V Tip: Start Short

Processing 240 frames (10 seconds at 24 FPS) through a V2V pipeline is slow. Start with 24 frames (1 second) to verify your settings produce good results before committing to a longer clip. Once you have parameters that work on the short test, scale up to the full clip. Use the frame_load_cap parameter in VHS Load Video to limit the frame count during testing.