AI & Creativity

Vid2Vid with ComfyUI: AnimateDiff, ControlNet, and FaceID

Otama
3 min read
Modified
Progress 7 / 7
Table of Contents

In this article, I explain detail of Vid2Vid workflow using ComfyUI and usage method. Finally I also post generation result, so please look.

I wrote title and overview of article looking like that for now, but reason I am writing article is just “Because Vid2Vid went relatively well so I want you to see it!”.

Honestly just posting generated video was fine, but since I created workflow of ComfyUI, I will introduce its usage method simply.

Availability of Workflow

Since there might be people saying “Don’t need explanation”, I put workflow first.

Explanation of Workflow

For model and post-processing, I mainly incorporate below.

  • Model: AnimateDiff(V3) + ControlNet + IPAdapter(FaceID)
  • Post-processing: FaceDetailer + Upscale(ESRGAN) + Frame Interpolation

Regarding explanation of each method and usage method in ComfyUI etc., I think they are summarized roughly below, so please read if you like.

Usage

1. Installation of ComfyUI

Introduce ComfyUI by either method below. Those already introduced please update ComfyUI to latest version too.

Method 1: Introduce ComfyUI directly

I think this is good for beginners.

【Stable Diffusion】ComfyUIを使って画像生成AIで遊んでみよう【導入編】

>-

blog.otama-playground.com

Method 2: Install via StabilityMatrix (Integrated Environment)

StabilityMatrixの導入方法：Stable Diffusion関連ツールを効率的に管理

>-

blog.otama-playground.com

2. Import of Workflow

Please download from link destination below and import.

animatediff_controlnet_ipadapter.json
drive.google.com

3. Installation of Extension

Execute Install Missing Custom Nodes via ComfyUI-Manager, or install following extensions manually.

Required

Optional (Convenient ones)

4. Manual Installation of Library

Since python library necessary for FaceID is not installed by error, install manually.

Download wheel of insightface from link below, then hit install command with pip.

Assets/Insightface at main · Gourieff/Assets

Contribute to Gourieff/Assets development by creating an account on GitHub.

github.com
Terminal window
# In case of standalone version, replace python command with path to execution binary of python
python -m pip install (path to downloaded whl) onnxruntime onnxruntime-gpu

Referred video:

5. Download of Necessary Models

Exclusive models are necessary download for IPAdapter, ControlNet, AnimateDiff respectively. Download necessary ones from repository etc.

6. Create Video to become Input

Since AnimateDiff does not support high fps video (blurs), please convert video to 8fps. Using ffmpeg etc. is easy.

7. Generate

After that, tweak various parameters and generate.

Since I think error comes out if settings or models are missing, please try executing while fixing appropriately.

Tips

  • Can generate quickly if changing to form using AnimateDiff-Lightning or LCM-LoRA etc.
  • FaceID is not needed if just generating video
  • Please switch ControlNet according to original video or video you want to generate
    • Lineart, SoftEdge, and Canny etc. if line drawing/anime style
    • Depth if depth is important
    • OpenPose if movement is important
    • etc.

Generation Result

Since I couldn’t paste to Hatena Blog due to relationship of file size/format, Youtube.

Generated nicely isn’t it!!! Although there are points concerned slightly like face not moving and creepy, or arm disappearing occasionally, I end parameter adjustment here due to time constraint.

Since I was doing trial and error for about a week, I can sleep feeling good today. Thank you for reading until end.

Related Posts