Introducing TextureFlow: regaining control

From the day I learned how to code, I've been interested in leveraging technology to create art and to turn our computers into advanced, steerable paintbrushes. It always felt to me like the function of technology is to uncover new ways of creating and experiencing that were simply unaccessible before.

At the end of this post you'll be capable of levering this new workflow to mix and mash textures and shapes at will, enabling tons of creative outputs like logo animations, video mapping content, animated QR-codes and super sick VJ loops.

In terms of new capabilities for creating, it feels very likely that AI will introduce a more dramatical shift than cinema, photography or even the printing press. But, what AI yields in terms of flexibility and ease of use, it often lacks in terms of steerability. When you prompt an image generator, you typically have something very specific in mind but the AI model wont always produce exactly what you wanted, sometimes for the better (pleasant surprises), often for the worse.

Luckily, the toolstack around generative AI is evolving rapidly and there's a lot of amazing lego blocks to play with (shoutout to ComfyUI, Banadoco and the many brilliant open source contributors that are part of this AI renaissance).

In this post I want to talk about a new Eden tool we're releasing called TextureFlow. The main idea behind TextureFlow is to make it possible to seamlessly combine shapes and textures to produce steerable animations with AI.

The TextureFlow tool is available under our /video endpoints on the new Eden website!

TextureFlow was built entirely in ComfyUI and it's a rather complicated beast with lots and lots of tweakable parameters and different modes, allowing a whole range of possible creations. In this blogpost we're going to try to cover all the variations so you can get the most out of this amazing tool.

TLDR: basic tips for using TextureFlow on Eden

For those who dont wanna read through the full post, here are a bunch of quick tips on how to get the best results out of this new tool:

• Run fast experiments using low n_frames, sampling steps and resolutions until you find a good combo of style_images / control_image / settings. Once you've hit the sweet spot, drive up the settings to generate your final animation. You can optionally also activate the upscale for really good output resolution.

• Think about how the AI model might apply your style_image(s) to the controlnet image, do the style images contain features that can be easily repurposed for drawing the control shape?

• style_images are always centre-square-cropped and resized to 224x224 before usage in IP-adapter. So the style_images DO NOT need to be high-quality at all! Also, it can help to crop them to a square before uploading so you're sure the right section of your image gets used to drive the content!

Playing with Texture

Lets start with the basics. The main driver behind the TextureFlow tool is an animation model called AnimateDiff, built on top of StableDiffusionv1.5

The most basic flow involves picking a few "style images" and using them to drive the content of the animation using IP-adapter:

Notice that the style_images are not exactly reproduced in the animation. This is a deliberate choice to allow for different ways to drive the desired motion later on. Technically speaking, the style_images are fed into an IP-adapter, but no controlnets are active on them. If you want a tool that interpolates between given images, check out the awesome Dough from Banadoco!

Basic settings

Lets start by covering some of the basic settings in TextureFlow. The first one is the "motion_mode" which controls how the style_images get injected into the final animation, from left to right, these are:

"concentric_circles", "rotating_segments", "progressive_rotating_segmentes", "horizontal_stripes":

Controlnet

Alright, basics covered, lets start making things interesting! The first obvious thing to improve the control over the generated animations is to add a simple controlnet into the mix. TextureFlow allows up to two different controlnets to be active at the same time and picking the right mix can be crucial to get good results.

Below is a nice example where I animated the new Eden logo using TextureFlow. Make sure to not overdo it with the controlnet strength(s) as this will result in less creative freedom for the AI model.

If you want something more subtle, you can tweak the controlnet settings (increasing the strength and eg adding a luminance model) and the style_images (using images with less information dense textures) to achieve something like this:

Video Mapping with TextureFlow

Now, for all the VJ's out there, when you say controlnet, a logical next step is to think videomapping. And yes, TextureFlow is absolutely awesome when it comes to producing animated pixels that are perfect for specific projection surfaces.

A few weeks ago, we had a little party over here:

I ran this image through TextureFlow with some style_images I hunted on Pinterest (highly recommend for texture hunting!!) using only a single "depth" controlnet model at strength 0.45:

Then you just get your beamer, aim, add some firespinning tricks and you got yourself a show 🔥 ¯\_(ツ)_/¯

QR-code animations

A fun little byproduct is that you can easily drop-in a QR-code (preferably one with high-error correction built in) and make a super slick, animated version of it. For the best results, you want to select the "QR-code" controlnet with a strength around 0.5 Try to scan this one with your camera:

Advanced animation techniques

Now, using a static image as controlnet input is great, but wouldn't it be cool if we could use a driving motion video instead? Well, TextureFlow also supports this and this is where things can get really awesome and creative!

Lets take the following simple, looping animation:

Next, Lets pick a few texture images like these:

Running these inputs through TextureFlow produces this:

If we simply change the style images like so:

We get the following animation:

Awesome, right?

TextureFlow makes it trivial to swap out the texture images, eg stretch out the driving motion video and add some subtle audio-reactivity: