SDXL ComfyUI Stability Workflow - What I use internally at Stability for my AI Art
TLDRScott Weather from Stability demonstrates a core workflow for AI Art using ComfyUI and SDXL. He guides viewers through setting up a basic graph for quality assurance, then introduces advanced techniques like using a refiner and samplers for detailed image enhancement. The tutorial also covers creative steps, such as conditioning latent noise for unique results, encouraging experimentation for those looking to master AI Art creation.
Takeaways
- 📊 This video demonstrates how to use SDXL inside ComfyUI, which is an internal workflow at Stability for AI art.
- 🔧 The workflow shown in the video serves as a core structure that can be customized and expanded upon.
- 🧩 Starting with the basic setup, the video progresses to more complex configurations, including elements not yet available in Automatic 1111.
- 📦 The initial steps involve loading the SDXL checkpoint and conditioning the CLIP model, which includes setting positive and negative prompts.
- 💡 The importance of organizing the graph for clarity and sanity, such as changing node colors, is highlighted.
- 📐 Setting dimensions and converting frequently used inputs to node inputs for efficiency is recommended.
- 🛠 The workflow incorporates advanced samplers, empty latents, and VAE decoders to produce the AI-generated images.
- 🎨 Refining steps are demonstrated, showcasing how to add more detail and refinement to the initial images.
- 🔄 The concept of conditioning latent noise before serious sampling is introduced as a new experimental approach.
- 💾 Emphasis is placed on saving and previewing images correctly, managing file prefixes, and avoiding sharing metadata inadvertently.
- 🛡 Tips on protecting proprietary workflows while still sharing useful information with the community are provided.
Q & A
What is the main topic of the video presented by Scott Weather?
-The main topic of the video is to demonstrate the process of using SDXL within Comfy UI for AI Art, which is part of the workflow at Stability.
What does Scott Weather suggest as the starting point for using SDXL in Comfy UI?
-Scott Weather suggests starting with a basic Comfy graph for SDXL and then getting creative from there, as he believes this is a good starting point before complicating the process.
What is the purpose of using a checkpoint in the workflow described by Scott Weather?
-The checkpoint serves as a starting point for loading the base SDXL model into the workflow for further processing and refinement.
Why does Scott Weather recommend setting the width and height to 4096 in the workflow?
-Scott Weather recommends setting the width and height to 4096 to ensure high-quality output, as smaller sizes might lead to quality issues.
What is the role of the 'positive prompt' in the AI Art creation process as described in the script?
-The 'positive prompt' serves as a guide for the AI, providing it with a description of the desired outcome, such as 'a robot shopping at Walgreens'.
What is the significance of the 'negative prompt' in the workflow, according to Scott Weather?
-The 'negative prompt' is used to define what should not appear in the AI-generated image, and it helps to refine the image by excluding unwanted elements.
Why does Scott Weather suggest converting the clip G and clip element to node inputs?
-Converting the clip G and clip element to node inputs allows for easier modification and reuse of the prompts throughout the workflow without having to repeatedly type them.
What is the purpose of the 'Advanced Sampler' in the workflow shown by Scott Weather?
-The 'Advanced Sampler' is used to sample from the conditioned clip, and it offers more capabilities for fine-tuning the AI Art generation process.
How does Scott Weather handle the refiner in the workflow?
-Scott Weather introduces the refiner by loading it like another checkpoint and then uses a different clip conditioner and sampler to work with it, aiming to refine the image further.
What is the innovative step that Scott Weather introduces towards the end of the video?
-The innovative step introduced by Scott Weather is adding an additional refining step before the base step, which conditions the latent noise, offering a new way to influence the initial state of the AI Art generation.
Why does Scott Weather recommend overlapping the steps of the base sampler and the refiner?
-Overlapping the steps allows for a smoother transition between the base sampling and the refining process, ensuring a more detailed and refined final image.
Outlines
📚 Introduction to SDXL Workflow in Comfy GUI
Scott Weather introduces a tutorial on setting up the Stable Diffusion XL (SDXL) workflow within Comfy, a GUI for AI art generation. He emphasizes the importance of this workflow as a starting point for quality assurance and stability in AI-generated images. Scott plans to demonstrate the basic setup first and then explore more advanced techniques that are not yet possible in the automated version of the software. He also mentions the use of checkpoints, loaders, and conditioners in the process.
🎨 Building the Core SDXL Graph in Comfy
The speaker walks through the process of creating a basic SDXL graph in Comfy, starting with loading a checkpoint and setting up a refiner. He discusses the importance of clip conditioning, using SDXL's own conditioners, and setting target width and height for the image. Scott also explains how to use node inputs for positive prompts and suggests using a primitive for text input. He highlights the use of a basic positive prompt 'a robot shopping at Walgreens' and a negative prompt 'rocks', which represents the opposite of the desired scene. The paragraph concludes with setting up an advanced sampler for the initial image generation.
🔍 Refining the Image with Additional Samplers
Scott demonstrates how to enhance the basic SDXL graph by introducing a refiner, which is another checkpoint loaded similarly to the base one. He explains the process of setting up a second sampler to work with the refiner, including the use of different clip conditioners for the refiner version. Scott also discusses the importance of aesthetic scores in the conditioning process and suggests using specific values as a starting point. The paragraph details the steps of connecting the refiner to the base sampler, emphasizing the need for both positive and negative prompts for the refiner to function correctly.
🛠️ Advanced Techniques and Experimentation
The tutorial continues with advanced techniques, such as adding an extra refining step before the base sampling begins, which Scott refers to as an initializer or latent noise conditioner. He explains how to set up this additional refiner with specific steps and start values, suggesting that this method can lead to different results by conditioning the latent noise from the outset. Scott encourages experimentation with these steps and values, hinting at the potential for discovering unique and creative outcomes.
🔄 Conclusion and Continuing the Workflow
In the final paragraph, Scott wraps up the tutorial by discussing how to save and reload the graph for further work, mentioning the importance of removing metadata if the image is to be shared publicly. He invites viewers to share their thoughts and workflows in the comments and offers assistance for any issues they might encounter. Scott concludes by thanking the audience for watching and promises more videos on the subject.
Mindmap
Keywords
💡SDXL
💡ComfyUI
💡Workflow
💡Checkpoint
💡Refiner
💡Conditioning
💡Sampler
💡Latent Noise
💡VAE Decoder
💡Preview
💡Metadata
Highlights
Introduction to the core workflow for AI Art using SDXL in ComfyUI by Scott Weather.
Explanation of the importance of starting with the basic workflow before getting creative.
Demonstration of loading a checkpoint in ComfyUI for quality assurance.
Use of the refiner in the workflow and its role in enhancing image quality.
Technique of conditioning the clip with positive and negative prompts for AI Art generation.
Conversion of prompt inputs into node inputs for efficiency in the workflow.
Setting up the sampler with specific steps and the significance of start and end steps.
The choice of DPM++ SDE GPU sampler for advanced AI Art generation.
Importance of using a fixed seed for initial configuration and stability in results.
How to preview and save the AI-generated image within ComfyUI.
Introduction of the refiner's role in adding detail and fixing elements like faces in AI Art.
The process of combining base sampling with refiner sampling for enhanced results.
Concept of using a refiner as an initializer for latent noise conditioning.
Experimentation with different steps and start points for unique AI Art outcomes.
The idea of conditioning latent noise before starting the main sampling process.
Scott Weather's personal exploration and experimentation with the workflow.
Invitation for viewers to share their own workflow ideas and suggestions.
Closing remarks and an offer of further assistance for those facing difficulties with the workflow.