How do you Create Smooth Transitions from a Single Image using ControlNet v2v?

[AndyHTu]

I'm using a reference photo to create a GIF, but when I employ a ControlNet model, I encounter challenges in understanding and adjusting the ControlNet weight, Start, and End sweep. I'm facing two main issues:

1. Either I struggle to achieve significant motion in my animation while maintaining the desired pose from the source image.
2. Or, when I do manage to introduce motion, the animation no longer retains the original pose from the source image.

I've experimented with the control sliders extensively, but I can't seem to find the right timing to capture the initial source pose and smoothly transition out of it. Could someone please clarify how the ControlNet weight, Start, and End sweep parameters work together? And is it possible to use them to transition from a source image to a different animation seamlessly?

00288-3784361451.mp4

[continue-revolution]

1 picture is an extremely strong signal and will be broadcasted to ALL frames with the same weight (advanced weight will be in a later update)

CN V2V is mainly for video to video conversion, meaning that each frame has a corresponding control frame, instead of each frame having a common single control.

[AndyHTu]

Oh, it seems that achieving this with just one image isn't currently feasible. I've spent the entire day experimenting, only to discover that it's not possible yet. LOL

I'm also curious about experimenting with ControlNet Batch. I wonder if it's possible to upload three different reference images to ControlNet, each with a different end sweep, to create a flexible animation. Is this a viable option?

P.S.: I was interested in joining your Patreon to learn more about how to use this, but it seems you haven't started it yet. :)

[continue-revolution]

Yes. This is exactly what I want to support for CN V2V.

I was interested in joining your Patreon to learn more about how to use this, but it seems you haven't started it yet.

I believe that I've already setup sponsorship (if not, please let me know). However, I am not planning to establish any group chat or something because I don't have time to manage that. Feel free to ask more questions and I will try my best to answer.

If you would like to sponsor, I would prefer you to use ko-fi, thanks.

[AndyHTu]

So does the Starting Control Step and Ending Control Step work well with a Single Image? It seems to only work as "On" or "Off" no matter how I adjust the slider. I have my frames set at 8FPS and 32 frames, so its a 4 second clip. Does the duration affect the Start and End Control Steps?

Here's my test using Control Weight, but I can't really do much with the Start/End Steps.
https://youtu.be/tGwVhnq-6t8

[continue-revolution]

In theory, I don’t think I break anything for start / end. However, when I get up, I will investigate to make sure. I've made sure that there is no problem with that.

ControlNet is a very strong signal. How is start / end affect single image generation?

[AndyHTu]

I'm not too sure I understand your question, but I at default Control Step of 0 and End Step at 1, you get a very static generation. By adding Control Step to maybe .5 I get a completely different animation than whats in the source image. So just to make sure:

The Starting Control Step is when it starts to use the Control Weight? If you increase it, it should delay the ControlWeight and create random animation until it reaches that point?

Is the Start/End step based on percentage of frame numbers ?