I Built a CoPilot+ AI PC (without Windows)

Jeff Geerling
4 Jun 202412:49

TLDRIn this video, the creator unveils a custom CoPilot AI PC, a Raspberry Pi-based system, contrasting it with Microsoft's Copilot. The PC integrates a 13 TOPS Hailo NPU, offering impressive machine learning capabilities for real-time video tasks, such as object detection and pose estimation, with less power consumption than traditional GPUs. Despite the AI hype, the video emphasizes practical applications like monitoring production lines and robotics. The creator also explores the potential of combining multiple AI accelerators to increase computational power, highlighting the limitations and possibilities of such a setup.

Takeaways

  • 🤖 The CoPilot PC is a custom Raspberry Pi AI PC, not to be confused with Microsoft's Copilot.
  • 🔢 The CoPilot PC aims to exceed Microsoft's suggested 40 TOPS of neural compute, with a potential of 51 TOPS using additional hardware.
  • 💰 The Raspberry Pi AI kit, priced at $70, includes an M.2 HAT and a 13 TOPS Hailo NPU for machine learning tasks.
  • 🚀 The Hailo-8L NPU is a high-efficiency, modern device that outperforms the Coral TPU in terms of TOPS per watt.
  • 📈 The script discusses the AI hype and its application in real-world problems like machine vision and robotics.
  • 📹 Raspberry Pi is focusing on real-time video applications, which can save resources by only processing when needed.
  • 🌟 The video demonstrates the significant performance and efficiency improvements of using the Hailo NPU over the Pi's CPU for AI tasks.
  • 🔧 The script explores the idea of adding more neural compute power to the CoPilot PC using various PCI Express devices.
  • 🚫 The experiment with multiple NPU boards shows the limitations in power supply and compatibility when pushing the system to its limits.
  • 🛠️ The video suggests that while the AI kit is powerful, it may be too niche for most users but will be valuable for specific use cases like machine vision.
  • 🔮 The future of AI on Raspberry Pi might include more integrated solutions and easier access to documentation and models.

Q & A

  • What is the main difference between the CoPilot PC and Microsoft's Copilot?

    -The CoPilot PC is a custom Raspberry Pi AI PC, not related to Microsoft's Copilot. The creator built it because they didn't like the way Microsoft marketed their product.

  • What is the minimum neural compute required for Microsoft's Copilot according to their specifications?

    -Microsoft states that at least 40 TOPS of neural compute is required for Copilot.

  • What is the neural compute performance of the Raspberry Pi AI kit with the Hailo NPU?

    -The Raspberry Pi AI kit with the Hailo NPU has 13 TOPS of neural compute performance.

  • How does the efficiency of the Hailo-8L NPU compare to the Coral TPU?

    -The Hailo-8L NPU operates at 13 TOPS with an efficiency of 8 TOPS per watt, which is more efficient than the Coral TPU, which offers 2 TOPS at around 2 TOPS per watt.

  • What is the main purpose of the Raspberry Pi AI kit?

    -The Raspberry Pi AI kit is designed to accelerate machine learning tasks such as object detection, pose estimation, and image segmentation, particularly for applications involving cameras.

  • Why did the creator not include an NPU in the main chip of the Raspberry Pi like Rockchip did with the RK3588?

    -Raspberry Pi does not build their own main chips; that responsibility lies with Broadcom. The creator opted for an add-on AI approach for flexibility.

  • What are some practical applications of the Raspberry Pi AI kit in real-world scenarios?

    -Practical applications include real-time video analysis for security, traffic planning, factory production line monitoring, and agricultural spoilage detection.

  • Why might GPUs not be the best choice for the Raspberry Pi AI kit despite their capabilities in AI?

    -GPUs typically have higher power requirements and can be challenging to work with, especially for low-power edge devices like the Raspberry Pi.

  • What limitations does the Raspberry Pi AI kit have in terms of model training and execution?

    -The AI kit has limited RAM, which restricts the complexity of the models that can be run on it. Additionally, training models on the Pi is very slow compared to modern GPUs.

  • How does the performance of the Raspberry Pi AI kit compare when running an object identification model on its CPU versus the Hailo NPU?

    -Running the model on the Hailo NPU is significantly faster and more efficient, with real-time object detection and near-idle CPU usage, compared to the slow and power-intensive performance on the Pi's CPU.

  • What is the final neural compute performance achieved by the creator when attempting to combine multiple AI accelerators with the Raspberry Pi?

    -The creator managed to connect hardware that provides up to 47 TOPS (actually 51) of neural compute, although not all of it was usable due to potential power and configuration issues.

Outlines

00:00

🤖 Custom CoPilot PC with Raspberry Pi AI Kit

The video script introduces a custom CoPilot PC, which is a Raspberry Pi AI PC, distinct from Microsoft's Copilot. The creator expresses reservations about Microsoft's marketing approach and details the Raspberry Pi's AI capabilities. The CoPilot PC is equipped with a $70 Raspberry Pi AI kit that includes an M.2 HAT and a 13 TOPS Hailo NPU, enhancing its machine learning performance. The script compares the performance of the Raspberry Pi with other tech giants and highlights the practical applications of AI in real-world scenarios such as object detection, pose estimation, and image segmentation. It also discusses the limitations and potential of the Raspberry Pi AI kit, emphasizing its suitability for machine vision and robotics.

05:04

🔌 Exploring AI Kit's Capabilities and Limitations

This paragraph delves into the practical applications and limitations of the Raspberry Pi AI kit. It discusses the use of pose estimation for gesture-based applications and behavior prediction, as well as image segmentation for effects similar to iPhone's portrait mode. The script also touches on the AI kit's ability to handle multiple feeds for monitoring and the potential for model tuning with custom data. Hailo's model zoo and Raspberry Pi's developing documentation are highlighted as valuable resources. However, it also points out that the AI kit is not suitable for tasks like image generation or hosting large language models due to its limited RAM and processing power. The creator then embarks on an experiment to surpass Microsoft's neural compute requirements by connecting multiple AI accelerators to the Raspberry Pi, resulting in a complex setup that pushes the limits of the device's capabilities.

10:09

🛠 Overcoming Power and Configuration Challenges

The final paragraph recounts the challenges faced when attempting to utilize the full neural compute capacity of the connected AI accelerators. The script describes issues with power requirements and initialization problems, suggesting that the Raspberry Pi's PCI Express port cannot supply sufficient power for all the devices. Despite recognizing 47 TOPS (51 actually) of neural compute, the system fails to operate optimally due to potential brownouts. The creator contemplates alternative solutions, such as using an external power source or a more robust NPU for higher performance needs. The paragraph concludes with thoughts on the future of AI and the Raspberry Pi AI kit, considering its niche appeal and the potential for further development in the field of AI and machine vision.

Mindmap

Keywords

💡CoPilot PC

The term 'CoPilot PC' refers to a custom-built personal computer that is designed to function alongside a user's primary computer, enhancing its capabilities. In the video, the creator distinguishes their CoPilot PC from Microsoft's Copilot, expressing a preference for their own version built on a Raspberry Pi platform. This custom PC is intended to serve specific use cases, particularly in the realm of artificial intelligence (AI) and machine learning, showcasing the creator's dissatisfaction with Microsoft's marketing approach and their innovative spirit.

💡TOPS

TOPS stands for 'tera operations per second' and is a unit of measurement for the computational performance of AI processors. In the context of the video, the creator discusses the computational power of their CoPilot PC, noting that it initially has 13 TOPS and later aims to increase this to 47 TOPS (or 51), which would surpass the capabilities of Apple's M4 and Snapdragon X processors. TOPS is crucial for understanding the processing speed and efficiency of AI models, especially in real-time applications.

💡Raspberry Pi

Raspberry Pi is a series of small, low-cost, single-board computers that have been widely used for various DIY projects and educational purposes. The video script mentions a 'Raspberry Pi AI kit,' which includes an M.2 HAT and a 13 TOPS Hailo NPU, effectively turning the Raspberry Pi into a capable AI machine. The Raspberry Pi's role in the video is central to demonstrating how affordable and accessible hardware can be leveraged to create powerful AI solutions.

💡Hailo NPU

An NPU, or Neural Processing Unit, is a type of computer hardware designed to accelerate machine learning tasks. The 'Hailo NPU' specifically refers to the Hailo-8L chip mentioned in the video, which provides 13 TOPS of computational power with an efficiency of 8 TOPS per watt. The Hailo NPU is depicted as a key component of the CoPilot PC, emphasizing its importance in enabling high-performance AI capabilities on a Raspberry Pi platform.

💡Machine Learning

Machine learning is a subset of artificial intelligence that involves the development of algorithms and statistical models that enable computers to learn from and make predictions or decisions based on data. In the video, machine learning is synonymous with AI and is the core functionality that the CoPilot PC is designed to support. The script discusses various machine learning applications, such as object detection and pose estimation, which are made possible through the CoPilot PC's hardware.

💡Coral TPU

The Coral TPU, or Tensor Processing Unit, is a piece of hardware designed by Google to accelerate machine learning tasks. In the video, the creator references using a Coral TPU with their Frigate server for tasks like car, person, and bird detection. The Coral TPU is highlighted as an alternative to the Hailo NPU, offering 2 TOPS of computational power at a lower price point.

💡AI Hype Train

The 'AI Hype Train' is a metaphorical term used in the video to describe the current trend of intense interest and investment in artificial intelligence technologies by major tech companies. The creator uses this term to express their skepticism about the excessive hype surrounding AI while acknowledging the genuine utility of AI in certain applications, such as the CoPilot PC project.

💡Machine Vision

Machine vision refers to the ability of computers to interpret and understand visual information from the world, often using AI and machine learning algorithms. In the context of the video, machine vision is a key application for the CoPilot PC, with potential uses in areas such as factory inspection, quality assurance, and robotics. The script emphasizes the practicality of machine vision in solving real-world problems.

💡Pose Estimation

Pose estimation is a computer vision technique used to determine the position and orientation of objects within an image or video, often applied to human bodies. The video script mentions pose estimation as a capability of the CoPilot PC, suggesting its potential for creating gesture-based applications or for enhancing safety features in vehicles, such as predicting a pedestrian's behavior.

💡Image Segmentation

Image segmentation is the process of partitioning an image into multiple segments or regions to simplify and/or change the representation of an image into something more meaningful and easier to analyze. In the video, image segmentation is discussed in the context of applications like iPhone's portrait mode, where the technology is used to isolate a person from the background for aesthetic effects or to apply unique lighting.

💡HatBrick! Commander

The HatBrick! Commander is a PCI Express board mentioned in the video that adapts one port into two, allowing for the connection of additional hardware to the Raspberry Pi. The creator uses this device in an attempt to increase the neural compute power of their CoPilot PC by connecting multiple AI accelerators, demonstrating the experimental and innovative nature of their project.

Highlights

Introduction of a custom CoPilot PC built without Windows, different from Microsoft's Copilot.

The CoPilot PC is a Raspberry Pi AI kit with a 13 TOPS Hailo NPU, offering high efficiency in machine learning tasks.

Comparison with Microsoft's requirement of 40 TOPS for Copilot, and the goal to achieve 47 TOPS [actually 51] with the CoPilot setup.

The $70 Raspberry Pi AI kit includes an M.2 HAT and a high-performance Hailo NPU for machine learning acceleration.

The Coral TPU's efficiency and its use in the Frigate server for detecting objects.

The Hailo-8L's superior performance with 13 TOPS and 8 TOPS per watt efficiency compared to the Coral TPU.

The disclaimer on the hype around AI and its practical applications in machine vision with the Raspberry Pi AI kit.

Raspberry Pi's focus on real-time video applications such as saving clips with detected people instead of continuous recording.

The potential of machine vision in various industries including traffic planning, factory monitoring, and farming.

The question of why Raspberry Pi doesn't integrate an NPU into their main chip and the role of Broadcom in this decision.

Different approaches to AI hardware, comparing custom NPUs, edge devices, and AI extensions on the CPU.

The limitations of using a GPU for AI tasks in low-power devices like the Raspberry Pi.

Demonstration of the performance difference between running an AI model on the Pi's CPU and the Hailo NPU.

The capabilities of the Hailo NPU for real-time object detection, pose estimation, and image segmentation.

The potential of the AI kit for machine vision and robotics, and its limitations in terms of RAM and model training.

Hailo's model zoo and Raspberry Pi's documentation efforts to support and integrate the AI kit into projects.

The unsuccessful attempt to exceed 13 TOPS by connecting multiple AI accelerators to the Raspberry Pi, due to power limitations.

Recommendations for using external power or a more powerful NPU for those requiring higher performance.

The future potential of the AI kit and its role as an alternative to the Coral, especially for machine vision and robotics enthusiasts.

Closing thoughts on the niche appeal of the AI kit and its value for specific use cases.