How to Use Generative Audio | Runway Academy
TLDRIn this Runway Academy tutorial, we explore the generative audio tool, which allows text-to-speech conversion, custom voice model training with clean audio, and creating lip-sync videos. The process includes generating audio from text, saving it in the assets folder, and customizing voice models for unique outputs. Additionally, we learn to sync audio with images or videos, with tips to enhance the reversing effect for longer audio clips, providing a comprehensive guide to leveraging Runway's audio capabilities.
Takeaways
- 🎙️ Use the generative audio tool in Runway to convert text into spoken audio.
- 🔍 Preview and select from a list of default voices to generate audio.
- ⏱ Generation time varies based on script length but is generally quick.
- 📂 Audio files are automatically saved in the generative audio folder within assets.
- 📁 Custom save locations can be chosen via a drop-down menu.
- 🎧 Train a custom voice model with a few minutes of clean audio.
- 📝 Ensure the audio for custom voice models is as clear as possible.
- 🖼️ Create lip-sync videos using an image or video with a full viewable face.
- 🔄 Lip-sync can accommodate generated, recorded, or uploaded audio.
- 🎥 Convert images to video using Gen 2 for video-based lip-sync.
- 🔁 If audio is longer than video, the video will loop to match audio duration.
- 🎨 Use motion brush for subject motion to minimize the reversing effect in videos.
- 💡 Join the Runway community on Discord for more resources and assistance.
Q & A
What is the main topic of the Runway Academy video?
-The main topic of the video is generative audio, which includes text to speech, custom voice models, and creating lip sync videos in Runway.
How do you access the generative audio tool in Runway?
-You access the generative audio tool by clicking on it from the Runway dashboard at the top.
What is the first step after typing in the text for the generative audio tool?
-The first step is to preview the text and choose a voice from the default voice list.
What is the default name of the voice in the provided example?
-The default voice provided in the example is named James.
How long does it usually take for the audio generation to complete?
-The generation times depend on the total script length, but they usually go pretty quickly.
Where are the audio generations saved by default in Runway?
-By default, audio generations are saved to the generative audio folder inside the main assets folder in Runway.
What is required to train a custom voice model in Runway?
-To train a custom voice model, you need a few minutes of clean audio which can be imported or recorded within the generative audio tool.
What should be ensured while recording the audio for a custom voice model?
-The audio should be as clean as possible to ensure the best results for the custom voice model.
What is needed to create a lip sync video in Runway?
-To create a lip sync video, you need an image or video of a person with their full face viewable within the frame.
How can you add new text to speech for a lip sync video?
-You can add new text to speech by typing the text, choosing your voice, and clicking on the generate button.
What happens if the audio is longer than the video in a lip sync video?
-If the audio is longer than the video, once the video reaches its end, it will reverse and go back to the beginning for the duration of the audio.
What is a pro tip for using the video workflow in Runway to avoid a noticeable reversing effect?
-A pro tip is to avoid using camera motion parameters and just add subject motion with the motion brush to make the reversing effect less noticeable.
How can viewers find more helpful resources and join the community for Runway?
-Viewers can join the community on Discord for more information and experimentation using Runway, or find specific answers using the button on the dashboard at any time.
Outlines
🎙️ Introduction to Generative Audio
This paragraph introduces the topic of the video, which is generative audio in Runway Academy. It covers text-to-speech, custom voice models, and creating lip-sync videos. The speaker explains how to access the generative audio tool from the dashboard, input text, and select a voice from the default list. The process of generating audio from the script is described, including the automatic saving of audio files to the assets folder and the option to save them elsewhere. The paragraph also mentions the possibility of training a custom voice model using a few minutes of clean audio.
🔊 Custom Voice Model Training
The second paragraph delves into the process of training a custom voice model within the generative audio tool. It details the requirement of having a few minutes of clean audio, which can be imported or recorded directly in Runway. The speaker suggests reading from the provided script or using one's own, emphasizing the importance of audio clarity. Once the audio is ready, the user is instructed to name the voice model, and it will be quickly ready for use with text-to-speech functionality.
🎥 Creating Lip-sync Videos
This paragraph explains how to create lip-sync videos using the generative audio tool. It requires an image or video of a person with a full face visible within the frame. The user can upload their own media or choose from preset characters. The paragraph outlines the process of adding generated audio from text-to-speech, recorded audio, or uploaded audio, and then selecting a voice to generate the lip-sync effect. It also provides a tip on how to handle videos longer than the audio by using Gen 2 to create a video from an image and then adding lip-sync in the generative audio tool, with a note on avoiding camera motion parameters for a smoother reversing effect.
📚 Conclusion and Additional Resources
The final paragraph wraps up the video with a conclusion, thanking viewers for their time and encouraging them to engage with the community on Discord for more information and experimentation with Runway. It also mentions the availability of a button on the dashboard for finding specific answers to questions. The speaker reiterates the invitation to get started with the work at hand and ends the video with a warm appreciation for the viewers' attention.
Mindmap
Keywords
💡Generative Audio
💡Text to Speech
💡Custom Voice Models
💡Lip Sync
💡Runway Dashboard
💡Generative Audio Tool
💡Audio Generation
💡Clean Audio
💡Preset Characters
💡Gen 2
💡Motion Brush
💡Discord Community
Highlights
Introduction to generative audio in Runway Academy.
Generative audio includes text to speech, custom voice models, and creating lip sync videos.
Accessing the generative audio tool from the Runway dashboard.
Type in text to convert it into a spoken audio file.
Preview and select a voice from the default voice list.
Generation times vary based on script length but are usually quick.
Audio Generations are automatically saved to the generative audio folder.
Option to save audio to a different location through the drop-down menu.
Training a custom voice model with clean audio.
Recording or importing audio for custom voice model training.
Naming the voice model and its readiness for use with text to speech.
Creating a lip sync video with an image or video of a person.
Ensuring the full face is viewable within the frame for lip sync.
Uploading custom media or using preset characters for lip sync.
Using lip sync with generated, recorded, or uploaded audio.
Adding text to speech and generating audio for lip sync.
Turning an image into a video using Gen 2 for adding lip sync.
Handling audio longer than video duration with a reversing effect.
Pro tip for avoiding camera motion parameters in the video workflow.
Using motion brush for subject motion to reduce reversing effect visibility.
Invitation to join the Runway community on Discord for more resources.
Using the dashboard button for finding specific answers to questions.
