Runway Just Changed Video Forever...Again.

Theoretically Media
25 Nov 202411:35

TLDRRunway ML has revolutionized the AI image and video space with two groundbreaking updates: 'Expand Video' and their proprietary 'Frames' image model. 'Expand Video' impressively transforms 9x6 videos into 16x9 and vice versa, filling in missing frame areas through inpainting, similar to Adobe's Gen Fill but for video. It handles stock footage with remarkable accuracy and speed, taking only 36 seconds for video generation. The feature also allows for creative text prompts to generate unique scenes. While not perfect, it offers exciting possibilities for video editing and creation. Runway's 'Frames' promises precise design control for any world, with examples including photorealistic portraits and vintage album art. These updates are set to change video and image editing forever.

Takeaways

  • 🚀 Runway ML has introduced two groundbreaking updates in the AI image and video space.
  • 🎥 The first feature is 'Expand Video', which can transform 9x6 videos into 16x9 or vice versa, filling in missing areas using inpainting technology.
  • 🖼️ The second feature is Runway's own image model, which is distinct from other generators like Flux.
  • 📹 'Expand Video' can handle stock videos with minimal inconsistencies and no weird morphing effects.
  • ⏱️ The generation time for video expansion is impressively fast, taking only 36 seconds.
  • 🌿 The model adds slight lens distortion, mimicking a wide-angle look and enhancing the overall video quality.
  • 📱 It can convert 16x9 videos into vertical videos without the need for cropping, which is a significant advancement.
  • 🤖 Combining video expansion with text prompts can lead to creative and sometimes weird results, like a 'UFO flying in the sky' prompt resulting in a comet-like object.
  • 🎨 The model can also adjust color grading and sharpness, offering a different visual style not seen in other generations.
  • 📺 When used on old TV show footage, the model might infer the content and output in a 4:3 aspect ratio, suggesting it identifies and adapts to the source material.
  • 🎬 There are fun and interesting creative possibilities with 'Expand Video', as seen with meme extensions and unique reinterpretations of iconic movie scenes.

Q & A

  • What are the two new features unveiled by Runway ML?

    -Runway ML has unveiled two new features: 'Expand Video' and their own image model called 'Frames'.

  • What does the 'Expand Video' feature do?

    -The 'Expand Video' feature can take any 9x6 video and turn it into a 16x9 video or vice versa, filling out the missing areas of the frame via inpainting, similar to Adobe's Content-Aware Fill but for video.

  • How does the 'Expand Video' feature handle the generation of video content?

    -The 'Expand Video' feature generates content by infilling missing frame areas and maintaining consistency without causing weird morphs or distortions to the moving subjects within the video.

  • How long does it take for 'Expand Video' to generate a new video frame?

    -The time to generate a new video frame is 36 seconds, which is considered fast for the capabilities of the Gen 3 Alpha model.

  • Can 'Expand Video' be used to create vertical videos from 16x9 videos?

    -Yes, 'Expand Video' can be used to convert 16x9 videos into vertical videos without having to punch in, which is a significant feature.

  • What is the potential issue with combining text prompts with 'Expand Video'?

    -Combining text prompts with 'Expand Video' can sometimes lead to unexpected results, such as the appearance of a 'Comet missile' due to an incorrect text prompt.

  • How does 'Expand Video' handle the transformation of a vertical video to a 16x9 format?

    -When transforming a vertical video to a 16x9 format, 'Expand Video' does a great job, accurately representing the original footage and even mimicking slight lens distortions.

  • What is the 'Frames' image model and what is its purpose?

    -The 'Frames' image model is Runway ML's own image generator, designed to allow users to design with precision, creating the look, feel, and atmosphere of any world they want to bring to life.

  • What kind of images can be generated with 'Frames'?

    -With 'Frames', users can generate photorealistic portraits, 1970s album art, fantasy images like orcs, and old school disposable camera looks.

  • How does 'Expand Video' perform when given user-shot footage?

    -The performance of 'Expand Video' with user-shot footage can vary, sometimes resulting in inaccuracies or odd results, such as the appearance of a blue puffy jacket on a character.

  • Can 'Expand Video' be used to extend scenes from classic films?

    -Yes, 'Expand Video' can be used to extend scenes from classic films, such as 'The Twilight Zone' and 'The Shining', though the results may not always be accurate or as expected.

Outlines

00:00

🚀 Runway ML's AI Updates: Expand Video & Image Model

Runway ML has introduced two groundbreaking features in the AI image and video space: 'Expand Video' and their proprietary 'Frames' image model. 'Expand Video' is capable of converting 9x6 videos to 16x9 or vice versa, filling in missing frame areas through inpainting, similar to Adobe's Gen Fill but for video. The feature is demonstrated with stock footage, showcasing its ability to handle water inconsistencies and avoid morphing artifacts. It can also transform 16x9 videos into vertical formats. Experiments with text prompts reveal the feature's potential for creative applications, such as adding a 'destroyed city' background to a video. While not perfect, the results are impressive, especially considering the speed of generation, which is a significant advantage of Gen 3 Alpha.

05:01

🎬 Creative Experiments with Expand Video

The script delves into creative experiments with Runway ML's 'Expand Video' feature, testing its capabilities with various video clips. It explores the transformation of a Twilight Zone episode, applying surreal effects and color grading, and speculates on the AI's ability to recognize and replicate the style of old TV shows. The feature is also used to extend scenes from The Shining and The Lord of the Rings, demonstrating the AI's capacity to generate new visual content based on existing footage. Challenges such as maintaining aspect ratios and avoiding odd artifacts are discussed, along with solutions like providing a reference frame to guide the AI's inpainting process. The summary highlights the potential for creative storytelling and the AI's ability to generate content that is both consistent and imaginative.

10:03

🖼️ Introducing Runway's Frames: A New Image Generator

Runway ML has announced 'Frames,' their new image generator, which is distinct from other models like Flux. The company claims that Frames will allow for precise design of any desired world, offering a range of styles from photorealistic portraits to 1970s album art and fantasy imagery. The script expresses excitement about the potential integration of Frames with the Gen 3 video model and the opportunities for creative exploration. The speaker looks forward to testing the new image model and encourages viewers who have had the chance to use it to share their experiences. The summary concludes with anticipation for the new capabilities that Frames will bring to the AI image and video space.

Mindmap

Keywords

💡Runway ML

Runway ML is a company that specializes in AI-driven image and video processing. In the context of the video, Runway ML has made significant updates in the AI, image, and video space, indicating that they are at the forefront of technological advancements in these areas. The video discusses two new features introduced by Runway ML that are game-changing for video editing and creation.

💡Expand Video

Expand Video is a new feature unveiled by Runway ML that allows users to transform videos between different aspect ratios, such as from 9x6 to 16x9 or vice versa. It fills in the missing areas of the frame using a technique called inpainting, similar to Adobe's Content-Aware Fill but for video. This feature is showcased in the video by taking stock footage and expanding it without any prompts, demonstrating the capability to generate new content that fits the desired aspect ratio.

💡Inpainting

Inpainting is a technique used in image and video editing to fill in missing or damaged parts of a picture. In the video, it is used in the context of the Expand Video feature to generate new content for the areas of the frame that are not originally filled. This allows for the creation of content that appears seamless and consistent, even when the aspect ratio of the video is changed.

💡Aspect Ratio

Aspect ratio refers to the proportional relationship between the width and the height of an image or video. In the video, the aspect ratio is a key focus as the Expand Video feature allows for the conversion between different aspect ratios, such as from 9x6 to 16x9, which is crucial for适配不同的显示设备和平台.

💡Gen 3

Gen 3, or Generation 3, is the third iteration of Runway ML's AI model. The video highlights the speed and efficiency of Gen 3, particularly in the context of video generation and editing. It is used to demonstrate the capabilities of the Expand Video feature, as well as the new image model that Runway ML is introducing.

💡Text Prompting

Text prompting is a method used in AI where the user provides a text description or command to guide the AI in generating specific content. In the video, text prompting is used in combination with the Expand Video feature to create customized video content, such as adding a 'destroyed city in the background' or creating a 'UFO flying in the sky'.

💡Vertical Video

Vertical video refers to video content that is shot or formatted in a portrait orientation, with a taller height than width. The video discusses how Runway ML's Expand Video feature can convert horizontal 16x9 videos into vertical 9x6 videos, which is significant for content creators who want to适配 mobile platforms where vertical video is more commonly viewed.

💡Frames

Frames is the new image generator introduced by Runway ML, which is distinct from other image models like Flux. The video mentions that Frames allows for precise design of any world, look, feel, or atmosphere, indicating a high level of customization and control over image creation. This feature is expected to complement Runway ML's video capabilities.

💡Photorealistic Portraits

Photorealistic portraits are a type of image that is created to look extremely lifelike, as if they were taken by a high-quality camera. In the video, Runway ML's Frames image generator is praised for its ability to create photorealistic portraits, showcasing the advanced capabilities of the AI in mimicking real-world visuals.

💡Content-Aware Fill

Content-Aware Fill is a feature in Adobe software that automatically fills in selected areas of an image with content that matches the surrounding area. The video compares Runway ML's inpainting technique in the Expand Video feature to Adobe's Content-Aware Fill, highlighting the advanced capabilities of AI in video editing to fill in missing areas in a seamless manner.

💡Pillar Boxing

Pillar boxing refers to the addition of vertical bars on the sides of a video to change its aspect ratio, often used to convert a wide-screen shot into a more square or vertical format. The video discusses how Runway ML's Expand Video feature can add pillar boxing to maintain the original aspect ratio while expanding the video to fit different screen dimensions.

Highlights

Runway ML has introduced two game-changing updates in the AI image and video space.

The first feature unveiled is 'Expand Video', which can transform 9x6 videos into 16x9 or vice versa.

Expand Video uses inpainting to fill out missing areas of the frame.

The process is similar to Adobe's Content-Aware Fill but for video.

Gen 3 can handle stock videos with impressive results in just 36 seconds.

Expand Video can create a wide-angle look with slight lens distortion.

Vertical videos can be created from 16x9 videos without additional input.

Text prompts can be combined with video expansion for creative outcomes.

The model can generate parts of a scene with no prior information, maintaining a consistent look.

Expand Video can create surreal and unique outcomes, such as a mix of The Twilight Zone and Salvador Dali.

The model sometimes infers aspects of old TV shows, like giving a 4:3 output for an old episode.

Expand Video can be used to create Ultra Widescreen effects from films shot in 2.35:1.

The model can extend scenes with impressive results, such as the opening of The Dark Knight.

Runway's new image generator, Frames, allows for precise design of any world.

Frames offers photorealistic portraits, 1970s album art, fantasy characters, and disposable camera looks.

Frames is an entirely new image model built by Runway, separate from other generators.

Create Unique Logos for Your Business

Generate professional logos and brand kits in minutes.

Create Unique Logos for Your BusinessGenerate Now