Audio Reactive AI Animation Masterclass ft. Cerspense
TLDRIn this masterclass, Tyler hosts Spence, a Runway ML researcher, to explore the intersection of audio reactivity and AI animation. Spence shares his expertise in creating visuals for musical performances, discussing his journey with AI and visual technologies like Notch and TouchDesigner. He demonstrates how to build an audio-reactive animation system using these tools, incorporating AI-generated images and real-time 3D modeling. The session includes a sound check, an overview of Spence's work, and a step-by-step guide to achieve audio-reactive visuals. Spence also provides resources, including a TouchDesigner file and Notch workflows, to help participants dive into the creative process. The class concludes with a Q&A, offering insights into Spence's creative workflow and his thoughts on the future of real-time motion graphics.
Takeaways
- 🎉 Spence, a guest on the civii Friday guest Creator stream, has a background in creating visuals for musical performances and now works with Runway ML focusing on audio reactive projects.
- 🚀 Spence has created a presentation and shared workflows on the Twitch chat, including comfy UI workflows and a touch designer file, to help viewers understand his process.
- 🔍 Spence introduced Notch, a real-time visual effects and graphics software used for major tours, and demonstrated how to create 3D models and animations that can be used in other programs.
- 🎨 He discussed the use of touch designer, a node-based program for real-time visuals, and how it can interact with APIs and run Python to generate custom content.
- 🌟 Spence mentioned his experience with AI, including training style Gan models and using gpt3, and how he integrates image generation workflows to automate and customize his creative process.
- 📈 He provided advice for those new to node-based workflows, emphasizing the importance of understanding how nodes connect and suggesting to start by tweaking existing workflows.
- 🎥 Spence showcased how to render loops in Notch and then use them in Comfy UI for further development, including the use of depth maps and image references.
- 🔬 He demonstrated the use of control nets in Comfy UI to refine the AI image generation process, including using depth maps to enhance the quality and detail of the output.
- ⚙️ Spence explained the process of using audio reactive techniques in touch designer to synchronize visuals with music, creating an immersive audio-visual experience.
- 🧩 He also talked about the importance of finding the right balance between the technical and creative aspects when building complex systems for artistic expression.
- 🌐 Spence encouraged viewers to explore various tools and platforms, such as Unreal Engine's Avalanche features, Blender, and Max MSP, to expand their creative possibilities.
Q & A
What is the main topic of the masterclass?
-The main topic of the masterclass is audio reactive AI animation, featuring a presentation by Spence, who discusses his workflow and techniques in creating visuals for musical performances using various software like Notch, TouchDesigner, and AI models.
What is Spence's professional background?
-Spence has been creating visuals for about a decade, primarily for musical performances. He has worked on concert tour visuals and virtual production visuals for major tours with Silent Partner studio. He also developed custom systems and technical integrations for creative possibilities in shows. Spence got into AI during 2022 and now works as a researcher at Runway ML.
What is the role of music in Spence's creative process?
-Music is a significant source of inspiration for Spence. It drives him to explore technology and use it to create visuals that express his feelings when he listens to music and to create things that truly represent the music itself.
What are some of the software and tools mentioned in the masterclass?
-The masterclass mentions several tools and software, including Notch, TouchDesigner, Cinema 4D, AI models like style GAN and stable diffusion, GPT-3, and programs like Comfy UI, FlowFrame, and various nodes and systems developed by Simulate.
How does Spence use AI in his work?
-Spence uses AI to generate images and integrate them into his video workflows. He has trained models like style GAN and stable diffusion and has used GPT-3. He also fine-tuned a model called ZeroScope, which contributed to his role at Runway ML.
What is the purpose of the workflow page that Spence uploaded?
-The workflow page uploaded by Spence provides a comprehensive guide for participants to follow along with the masterclass. It includes two different Comfy UI workflows, a TouchDesigner file, and a variety of masks for creative exploration in the animation process.
What are some of the key features of Notch that Spence discusses?
-Notch is a real-time visual effects and graphics software used for big tours and events. It allows for quick and intuitive creation of graphics. It's known for its expressiveness and fun interface, which is more enjoyable than game engines for creating audio-visual graphics.
How does Spence approach learning new software or tools?
-Spence suggests starting by looking at existing workflows and tweaking them. The next step is to build a simple workflow from scratch to understand the core concepts. Troubleshooting and figuring out issues helps build confidence and understanding of the software.
What is the significance of the audio reactive aspect in Spence's animations?
-The audio reactive aspect allows the animations to change and adapt in real-time based on the audio input. This creates a dynamic and immersive experience where the visuals are synchronized with the music or sound, enhancing the overall performance.
How does Spence manage the complexity of node-based workflows?
-Spence emphasizes that it's not necessary to understand every single node. Instead, the key is to learn how the nodes connect and interact with each other. He also mentions that node-based interfaces are more intuitive and easier to manipulate compared to traditional programming.
What advice does Spence give for someone starting with node-based software like TouchDesigner?
-Spence advises not to be scared of the complexity. He suggests starting with simple rendering and gradually moving to more complex tasks. He also recommends using a second computer to render more videos simultaneously, which can speed up content creation.
Outlines
🎥 Introduction to the Guest Creator Stream
Tyler introduces the guest creator stream, mentioning that they will cover a lot of ground with the guest, Spence. He encourages viewers to submit questions through chat and provides a link to download two different comfy workflows and a touch designer file. Tyler has been working with Spence for personal projects and appreciates his help. Spence works for Runway ML and specializes in audio reactive visuals, and Tyler is excited to see what he will demonstrate.
🎨 Spence's Background and Workflow Overview
Spence introduces himself, discussing his decade-long experience in creating visuals for musical performances. He explains his process of starting with Cinema 4D, moving to professional concert visuals, and then exploring AI in 2022. Spence details his use of various technologies like Disco Diffusion, Style GAN models, and GPT-3. He also talks about his work with touch designer, a node-based program for real-time visuals, and his creation of a fine-tuned model called zeroscope.
🚀 Starting the Creative Process in Notch
Spence begins the creative process using Notch, a real-time visual effects software. He demonstrates how to create a loopable animation by cloning objects and manipulating them with a noise texture. He also discusses Notch's capabilities, its use in major tours, and the existence of a learning version. Spence provides alternatives for creating loops, such as using Blender or downloading pre-made animations.
🌟 Crafting Loops and Exploring Notch's Features
Spence continues to work on creating loops in Notch, experimenting with different settings to achieve interesting visual effects. He discusses the ability to add 2D effects to 3D objects and emphasizes the speed of rendering in Notch. Spence also shares his personal experience with rendering in Cinema 4D and how Notch's real-time rendering capabilities were a significant improvement for him.
🔍 Navigating the Learning Curve of Node-Based Workflows
Spence shares advice for overcoming the initial overwhelm of learning node-based workflows. He suggests starting with existing workflows, tweaking them, and then building from scratch to build confidence. He reassures that node-based interfaces are intuitive and useful for artists, allowing for expressive control and automation of complex tasks.
🎚️ Exploring Notch's Expressive Capabilities
Spence demonstrates Notch's rendering capabilities and its use in creating expressive and complex visuals. He talks about the program's advantages over game engines like Unreal Engine, especially in terms of creative director feedback and ease of use. Spence also discusses the process of rendering animations and the use of different effects to enhance the visuals.
🖥️ Transitioning to Comfy UI and Setting Up the Workflow
Spence transitions from Notch to Comfy UI, starting with video loading and processing. He discusses the use of different nodes for efficient loading, image processing, and rendering. Spence also mentions his custom node pack for iterating through directories and selecting specific videos or images for processing.
🔄 Automating Content Creation with Comfy UI
Spence sets up a system to automatically generate content using Comfy UI. He uses his custom nodes to iterate through a directory of videos and images, applying random masks and creating loops. The system is designed to run indefinitely, producing a continuous stream of content that can be reviewed later.
🎛️ Diving into TouchDesigner for Audio Reactive Visuals
Spence opens TouchDesigner and begins setting up an audio reactive system. He discusses the program's capabilities, its node-based interface, and its real-time processing power. Spence also provides a link to the TouchDesigner file he's using, allowing viewers to follow along and experiment with audio reactive visuals.
🎧 Beat Detection and Speed Modulation in TouchDesigner
Spence demonstrates how to use audio analysis in TouchDesigner to detect beats and modulate the speed of video playback. He uses the kick and snare channels to control the speed, creating a dynamic audio reactive effect. Spence also discusses the use of a program called FlowFrame for interpolation and slow-motion effects.
🎇 Creating Dynamic Audio Reactive Visuals
Spence shows how to composite different video elements in TouchDesigner based on audio cues. He uses a combination of nodes to create a flashing outline effect that responds to snare beats. Spence also discusses the use of a threshold to mask the snare animation and the importance of adjusting parameters to achieve the desired visual effect.
📽️ Rendering and Exporting the Final Video
Spence talks about rendering the final video in TouchDesigner, mentioning the limitations of the free version regarding export formats. He demonstrates recording the video with a specific audio track and previews the result. Spence also invites questions from the audience and discusses his full system for creating visuals, including his use of presets and MIDI controllers.
🌐 Sharing Resources and Final Thoughts
Spence shares his social media handles and encourages the audience to follow his work. He discusses the tools he's excited about, such as Unreal Engine's Avalanche and Blender, and provides advice for aspiring creatives. Spence emphasizes the importance of passion and continuous learning, and he thanks the audience for joining the session.
📁 Providing Workflow Files and Upcoming Streams
The host provides information about the contents of the zip file available on the workflow page, which includes masks for Notch and an audio reactive time setup in TouchDesigner. The host also previews upcoming streams, mentioning guests like Noah Miller and Dot Simulate, and encourages the audience to follow the platform for more informative sessions.
Mindmap
Keywords
💡Audio Reactive
💡Notch
💡Touch Designer
💡Cinema 4D
💡Stable Diffusion
💡GPT-3
💡Runway ML
💡AI Art
💡Workflow
💡Real-Time Visuals
💡Node-Based Program
Highlights
Spence, a guest creator at the civii Friday guest Creator stream, shares his expertise in audio reactive visuals and AI integration.
Spence works for Runway ML and has extensive experience in creating visuals for musical performances, concert tours, and virtual production.
He has developed custom systems and technical integrations to enhance creative possibilities in various shows.
Spence explores AI, training style Gan models, and using GPT3 for video generation and automation of image generation workflows.
He demonstrates the use of Notch, a real-time visual effects software, for creating 3D models and animations for audio reactive visuals.
The presentation includes a workflow page with downloadable resources to help attendees create their own AI and audio reactive projects.
Spence discusses the use of Touch Designer, a node-based program, for real-time visuals and its interaction with APIs and Python for content generation.
He provides an overview of his creative process, from generating visuals in Notch to integrating them with audio reactivity in Touch Designer.
The tutorial covers the use of multiple programs and workflows, including Comfy UI, to create a seamless audio-visual experience.
Spence shares his insights on overcoming the initial fear of diving into node-based workflows and offers advice for budding creators.
He emphasizes the importance of passion and continuous learning in the field of graphics and real-time visuals.
The stream features a live demonstration of creating loops and animations in Notch and rendering them for further processing in Comfy UI.
Spence explains the use of audio reactive techniques to synchronize visuals with the beat of music, enhancing the immersive experience.
He showcases the capabilities of Touch Designer for real-time graphics generation and its potential for creative expression.
The session includes interactive Q&A, allowing attendees to gain a deeper understanding of the tools and techniques discussed.
Spence provides a comprehensive guide to setting up an audio reactive project from scratch, including tips for optimizing performance.
The tutorial concludes with a discussion on the future of real-time motion graphics and the potential of integrating AI with existing tools.