AI & Creativity

Image Generation AI with ComfyUI: ControlNet Edition

Otama
2 min read
Modified
Progress 4 / 15
Table of Contents

Today I will try to specify pose of appearing person using method called ControlNet. By adding person’s posture (feature points, segmentation, image etc.) as input in addition to prompt, it becomes possible to control appearing person’s posture. Please calculate following article for specific method.

Stable Diffusionモデルで姿勢を学習・制御するControlNetの基礎

>-

blog.otama-playground.com

In this article, I will actually generate images using ControlNet with ComfyUI.

Work Flow

1. Installation of ComfyUI

Please refer to this article.

【Stable Diffusion】ComfyUIを使って画像生成AIで遊んでみよう【導入編】

>-

blog.otama-playground.com

2. Download ControlNet Model

Download from favorite place and place in ComfyUI/models/controlnet. I brought from ↓.

lllyasviel/ControlNet-v1-1 at main

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

Looking at filenames, there assume unfamiliar strings like canny, openpose, lineart, these are names of preprocessors for ControlNet input image generation when learning. Since it gets long I won’t touch in article, but as input to controlnet, corresponding ones must be used.

sae-chan.net
sae-chan.net

3. Creation of Workflow

Create by myself from default workflow.

  1. Load Default to return workflow to default
  2. Create Load ControlNet Model(diff), Load Image, Apply ControlNet nodes.
  3. Connect like figure below (Yellow are added nodes)
Workflow
Workflow

Added Load Image node represents input to ControlNet, include posture info here (openpose or lineart etc.)

4. Generate Image

  1. Select model downloaded earlier in Load ControlNet Model(diff) node
  2. Select image representing posture (one for preprocessor corresponding to model) in Load Image node
  3. Tweak other parameters as you like, press Queue Prompt

Generation Result

Posture Data (derived from: https://civitai.com/models/195165/openpose-praying)
Posture Data (derived from: https://civitai.com/models/195165/openpose-praying)
Generated Image
Generated Image

I think it is easy to understand if you compare generation result and used posture data, but you can see it is generated almost exactly as specified.

Conclusion

In this article, I introduced method to perform image generation with ComfyUI using ControlNet. Watching image being generated exactly as specified posture, weren’t you able to feel powerful control function of ControlNet? With this, higher degree of freedom image generation becomes possible.

If you want to try other techniques related to image generation, please utilize the link collection below.

Stable Diffusionガイド：画像生成に役立つリンク集

>-

blog.otama-playground.com

Related Posts