Sparse Control and Frame Interpolation in General

[NergiZed]

I had recently made a issue that was resolved which fixed Sparse Control functionality in AnimateDiff.

It now 'works' again, but I really gotta ask. Is this how it's supposed to work?

In this, I'm interpolating 5 keyframes, generated in an SDXL model (ControlNet guided), over in AnimateDiff using a SD 1.5 model, and this is the result. I get these pulses for whenever the keyframes appear in the animation. The SDXL model is Autism, the SD 1.5 model is Aux.

I'm trying to do frame interpolation with AnimateDiff uses frames generated in SDXL, but really, the source is irrelevant, could be from a live action movie or an animation, the logic is the same.

Here's the exact same logic for Toonify, a model with both a good SD 1.5 and SDXL model. Although here, I just yanked 5 frames from another animatedDiff animation.
Just to clarify, this animateDifff vid is just 5 frames interpolated, with no ControlNets and basically no prompt, and the original keyframes and the interpolated frames are generated using the same model.

For comparison, this is the source of those 5 frames above. (Ignore the weird initial frame, that's irrelevant)
This is animateDiff guided by depth, openpose and canny ControlNets as well as a prompt. Checkpoint is Toonify.

So, I'm really curious to know if any of you have used Sparse Control to do frame interpolation, and what were the results. Because there's very little online about it, even less specifically pertaining to using Sparse Control within the AnimateDiff webui extension.

I'm using either the official V3 motion model or the AnimateDiff LCM motion model found here: https://civitai.com/models/326698/animatediff-lcm-motion-model

Is there any way for me to improve my results? My goal is to interpolate multiple keyframes, not just between two frames, this is something the original paper claims to be able to do, but I haven't found any examples of sparse control use cases of frame interpolation of more than two frames.

Also, is this pulsating keyframe from Sparse Control a bug? It feels like a bug almost, because the vid is massively improved when the pulsating original keyframes are removed. Here I've removed the original keyframes from the animation and then interpolated more frames in flow frames.

Even then, the keyframes can't just be removed because it affects the overall pacing of the shot. This can't be the final workflow.

I'm aware of and indeed use software like flowframes (https://nmkd.itch.io/flowframes), but that's not really keyframe interpolation, that's basically just for increasing framerates and smoothing out vids.

tldr; how make sparse control wurk gud? halp?

[continue-revolution]

You can’t really use sdxl with SparseCtrl. I don’t really know how you made it without error. It can only be applied to sd1.5. Also, sparsectrl does not really have a good performance.

Unfortunately, individual researchers get insufficient data and compute, so it’s hard for most of them to compete their for-profit counterpart.

[NergiZed]

You can’t really use sdxl with SparseCtrl. I don’t really know how you made it without error. It can only be applied to sd1.5. Also, sparsectrl does not really have a good performance.

Unfortunately, individual researchers get insufficient data and compute, so it’s hard for most of them to compete their for-profit counterpart.

I didn't use SDXL directly with Sparse Control. I used images generated with SDXL as a base, then did img2img with a SD 1.5 checkpoint with Sparse Control. The workflow is complicated, so I understand the confusion.

I get the feeling that this is the extent of the power of the model, which is fine; can't do anything about that.

My main question was that the difference in quality between the keyframes provided and interpolated frames are huge. The vid is much more consistent when they're not included. Can they be removed from the output during the generation process?

Shift-attention has a similar option called 'drop original frames'. It's a useful little option.