How to transform any 360-degree video into 3D (using photogrammetry)
TLDRThis tutorial demonstrates two methods for converting 360-degree videos into 3D models using photogrammetry. It covers converting 360 footage into an equirectangular image, subsampling for better photogrammetry results, and using software like Insta360, DaVinci Resolve, Metashape, and Meshroom. The video compares outcomes from direct 360 photo use and splitting images into perspective projections, highlighting the ease of capturing scenes with a 360 camera versus traditional methods.
Takeaways
- 📹 Convert 360-degree video footage into an equirectangular format using the Insta360 app to prepare for photogrammetry.
- 🔍 Subsample the equirectangular video to reduce the frame rate and increase the separation between frames for better photogrammetry results.
- 🎨 Use video editing software like DaVinci Resolve to adjust the speed and resolution of the 360-degree footage.
- 🌐 Import the subsampled equirectangular images into Agisoft Metashape for photogrammetry, selecting 'spherical' for camera calibration.
- 🔄 Align images in Metashape using the 'sequence' setting to take advantage of the video's natural order.
- 📐 Split equirectangular photos into multiple perspective projections using Meshroom for compatibility with other photogrammetry software.
- 🛠 Meshroom's 'split 360 images' feature prepares the photos for use in standard photogrammetry processes.
- 🏞️ Compare the results of using direct 360-degree photos and split perspective projections in Metashape and Reality Capture.
- 📈 Adjust settings in Metashape like accuracy, key point limit, and mesh quality to achieve the desired level of detail in the 3D model.
- 📊 Use Metashape's confidence color map to evaluate the accuracy of the reconstructed 3D model, identifying areas of high and low confidence.
- 🚀 The process simplifies 3D scene capture, allowing for quick walkthroughs with a 360-degree camera instead of meticulous photo alignment.
Q & A
What is photogrammetry and how is it used in the context of 360-degree videos?
-Photogrammetry is a technique used to create 3D models from photographs. In the context of 360-degree videos, it is used to transform the video into a 3D model with textures. This involves converting the native files from the 360-degree camera into an equirectangular image, which is a typical representation of 360 photos and videos, and then using photogrammetry software to generate a 3D model.
What is the equirectangular image format and why is it used for 360-degree videos?
-The equirectangular image format is a projection method used to represent 360-degree photos and videos. It maps the spherical images captured by a 360-degree camera onto a 2D plane, making it look stretchy at the poles but somewhat readable in the middle. This format is used because it allows for a comprehensive view of the entire 360-degree scene in a single image, which is ideal for photogrammetry.
How can the Insta360 app be used to convert 360-degree footage into an equirectangular image?
-The Insta360 app can be used to convert 360-degree footage by loading the video file and then clicking export. You select the tab on the right to output the raw 360 footage, which gives you a video file with an aspect ratio of 2x1, meaning it is twice as wide as it is tall. This equirectangular video can then be used for further processing in photogrammetry.
Why is it necessary to subsample the 360-degree video footage before using it for photogrammetry?
-Subsampling the 360-degree video footage is necessary because photogrammetry requires enough separation between different poses in the images. At high frame rates, such as 24 frames per second, there is not enough separation between successive frames for the photogrammetry software to work effectively. Subsampling reduces the number of frames, providing better separation and improving the photogrammetry process.
What is the recommended frame rate for capturing 360-degree footage suitable for photogrammetry?
-The ideal frame rate for capturing 360-degree footage suitable for photogrammetry is not explicitly stated in the script, but it implies that a slower frame rate is preferable to allow for more separation between frames. This is typically achieved by subsampling the video to reduce the number of frames per second.
How can you subsample a 360-degree video using DaVinci Resolve?
-To subsample a 360-degree video in DaVinci Resolve, you load the video with a 2x1 aspect ratio, ensuring the project settings match the resolution of the 360 footage. Then, you change the clip speed by right-clicking on the clip and entering a value such as 1000% or 2000%, depending on the speed of movement during capture. This process effectively reduces the frame rate, creating a subsampled video.
What is the recommended export format for the subsampled 360-degree video in DaVinci Resolve?
-The recommended export format for the subsampled 360-degree video in DaVinci Resolve is JPEG, as it provides a good balance between quality and file size. However, if lossless compression is preferred, formats like TIFF or PNG can be chosen instead.
How does Agisoft Metashape handle equirectangular photos for photogrammetry?
-Agisoft Metashape can handle equirectangular photos directly for photogrammetry. You import the equirectangular photos and under the Tools menu, select Camera Calibration and choose 'spherical' as the option. This applies the setting for all imported images, and then you can proceed with the workflow to align images and generate a 3D model.
What is Meshroom and how can it be used to prepare equirectangular photos for photogrammetry?
-Meshroom is a free, open-source photogrammetry software. While it can be used for the photogrammetry process itself, in this context, it is used to prepare equirectangular photos by splitting them into multiple perspective projections. This preprocessing step transforms the special 360-degree photos into a more standard format that can be used with various photogrammetry software that do not natively support equirectangular photos.
How does splitting equirectangular photos into multiple perspective projections help with photogrammetry?
-Splitting equirectangular photos into multiple perspective projections helps with photogrammetry by transforming the special 360-degree photos into a format that is more standard and recognizable by a wider range of photogrammetry software. This allows for greater flexibility in choosing software and ensures that the software can align and use the photos to reconstruct the scene in 3D.
What are the advantages of using a 360-degree camera for capturing scenes compared to traditional photography?
-The main advantage of using a 360-degree camera for capturing scenes is the ease and speed of data capture. With a 360-degree camera, you can quickly walk around a scene once and capture a comprehensive view without needing to ensure overlap between photos or capture from multiple angles to avoid missing details.
Outlines
📹 Introduction to 360 Video Photogrammetry
This paragraph introduces a tutorial on converting 360-degree video footage into a 3D model with textures. The video will cover two methods for achieving this using a 1x3 camera. The first step involves converting the heavily distorted images from the dual lenses into an equirectangular format, which is the standard representation for 360 photos and videos. The Insta360 app is suggested for this conversion. The tutorial then discusses the need to subsample the video to ensure there is enough separation between frames for photogrammetry software to process effectively. The recommended approach is to use a nonlinear video editor to adjust the frame rate and export the video in a 2x1 aspect ratio, which is suitable for photogrammetry.
🔍 Processing 360° Footage for Photogrammetry
The second paragraph delves into the technical process of preparing 360-degree footage for photogrammetry. It explains how to use DaVinci Resolve to downsample the equirectangular video, adjusting the clip speed to ensure there is adequate separation between frames for better reconstruction. The paragraph also covers the exportation of the video in a format like JPEG, which is suitable for photogrammetry. The user is then guided through the use of Agisoft Metashape, a photogrammetry software, to import equirectangular photos and align them using the software's camera calibration tools. The paragraph also introduces an alternative method involving the use of Meshroom to split equirectangular photos into multiple perspective projections, making them compatible with various photogrammetry software that do not natively support 360-degree inputs.
🏆 Comparing Photogrammetry Results from Different Methods
The final paragraph presents a comparison of the results obtained from using Metashape with both direct 360-degree photos and the split perspective projections. It details the settings used for camera alignment, mesh generation, and texture generation in Metashape, highlighting the high resolution and accuracy of the resulting 3D mesh. The paragraph also discusses the use of a confidence color map to assess the accuracy of the reconstruction. Additionally, it compares the results with those from Reality Capture, another photogrammetry software, using the same set of split images. The conclusion emphasizes the ease and efficiency of capturing scenes with a 360-degree camera and the high-quality results that can be achieved using the described methods.
Mindmap
Keywords
💡Photogrammetry
💡360-degree video
💡Equi-rectangular image
💡Insta360 App
💡Subsampling
💡DaVinci Resolve
💡Agisoft Metashape
💡Meshroom
💡Perspective projections
💡Reality Capture
Highlights
Learn how to transform a 360-degree video into a 3D model using photogrammetry.
Two methods are shown for converting 360-degree footage into an equirectangular image.
Using Insta360 app to convert native files of 360-degree footage into equirectangular format.
Importance of having enough separation between different poses for photogrammetry to work effectively.
Subsampling the 360-degree video to reduce frame rate for better photogrammetry results.
Using DaVinci Resolve to subsample and adjust the video for photogrammetry.
Exporting the subsampled video in a lossless format like JPEG, TIFF, or PNG.
Using Agisoft Metashape to perform photogrammetry directly on equirectangular photos.
Setting camera calibration to spherical in Metashape for equirectangular images.
The option to split equirectangular photos into multiple perspective projections for standard photogrammetry software.
Using Meshroom to preprocess equirectangular photos for compatibility with various photogrammetry tools.
Meshroom's feature to split 360 images into four perspective projections for standard reconstruction.
Importing split photos into Metashape and setting reference pre-selection to estimated.
Comparing the results of photogrammetry using direct 360-degree photos versus split images.
The high-resolution mesh generated from the 360-degree photos with Metashape.
Observing the confidence color map in Metashape to assess the accuracy of the reconstruction.
Comparing the quality of the mesh generated from split images versus direct 360-degree photos.
Using Reality Capture to produce a mesh from the same set of split images for comparison.
The ease of capturing a scene quickly with a 360-degree camera versus traditional photogrammetry methods.