iOS 18 - Apple's NEW AI Model.. Beyond Siri!

Brandon Butch
13 Apr 202416:21

TLDRApple's upcoming iOS 18 update is set to introduce significant AI advancements, including a new multimodal large language model called Faret UI, which surpasses GPT-4V in UI tasks. This model的理解能力超越了传统的文本基础的LLMs，能够理解图像、视频和音频等多模态元素。iOS 18还将可能引入AI驱动的Safari浏览器助手和加密视觉搜索功能。此外，iOS 17.5 beta版带来了新的游戏和改进的查找我的网络功能。苹果公司还计划在Mac产品线中全面采用M4芯片，增强AI处理能力，并可能推出带有内置摄像头的新Apple TV。

Takeaways

  • 🚀 iOS 18 is introducing a new AI system called Feret UI, a multimodal large language model (MLLM) that goes beyond text to understand elements like images, video, and audio.
  • 📈 Feret UI has been benchmarked against OpenAI's MLLM and reportedly surpasses it on all elementary UI tasks, showing its capability to recognize and interact with a user's home screen elements.
  • 🔍 iOS 18 may include a new AI-powered browsing assistant for Safari and encrypted visual search, potentially enhancing the existing visual lookup feature.
  • 🎮 Game emulators may become available on the App Store due to EU regulations, but developers are responsible for the software offered, meaning no pirated games or ROMs.
  • 🆕 iOS 17.5 beta introduces new features such as a connection to Game Center for puzzles in the News app, and a new puzzle game called 'Cor Tiles'.
  • 💰 Apple now displays the total cost of combined subscriptions when adding a new one, providing users with a clearer view of their monthly expenses.
  • 🔔 With iOS 17.5, third-party item trackers like Google's and Chipolo's will be integrated into Apple's Find My network, alerting users to unknown trackers.
  • 📱 Performance and battery life for iOS 17.5 beta 1 are reportedly on par with previous versions, with some users experiencing battery drain issues.
  • 💻 The future of Mac is closely tied to the upcoming M4 chip, which will focus on AI processing capabilities and may come in multiple variants like M4, M4 Pro, and M4 Ultra.
  • 📺 A new Apple TV might feature a built-in camera for FaceTime and gesture-based controls, and there are considerations for a lightweight smart display as a mobile solution around the home.
  • 🔧 Apple will start allowing used parts for repair in the fall, with activation lock extended to these parts to deter theft and illegal disassembly of iPhones.

Q & A

  • What is the main focus of Apple's new AI system, Feret UI?

    -Apple's new AI system, Feret UI, is a multimodal large language model (MLLM) designed to understand not only text but also multimodal elements such as images, video, and audio. It is trained to recognize different elements of a user's home screen, such as app icons and small text, and can propose goal-oriented actions in interaction conversation and deduce the overall function of the screen via function interface.

  • How does Feret UI compare to GPT-4V in terms of performance?

    -According to a research paper published by Apple, Feret UI excels beyond most open-source UI MLMs and surpasses GPT-4V on all of the elementary UI tasks. This indicates that Feret UI has superior performance in understanding and interacting with user interface elements compared to GPT-4V.

  • What new AI-powered feature might be added to iOS 18 for Safari browsing?

    -iOS 18 might add a new AI-powered browsing assistant for Safari, which could enhance the user's browsing experience by providing more intelligent and personalized assistance.

  • What is the potential change in the App Store regarding game emulators?

    -Due to pressure from the EU, Apple may allow game emulators on the App Store globally. However, developers are responsible for the software offered in the emulator application, and pirated games or ROMs will not be permitted.

  • What new features are coming to iOS 17.5?

    -iOS 17.5 is expected to introduce a new game, 'Cor Tiles', in the News application, connected to Game Center for tracking stats. Additionally, there will be an alert for unknown third-party item trackers following the user, thanks to a partnership between Apple and Google.

  • How is Apple addressing the issue of battery drain in iOS 17.5?

    -While there have been reports of battery drain on iOS 17.5 beta 1 and 17.4.5, the performance on beta 1 has remained stable. Apple is likely to address any battery issues in subsequent beta releases leading up to the final release of iOS 17.5.

  • What can we expect from the future of the Mac with the introduction of the M4 chip?

    -The M4 chip, set to equip every Mac in Apple's lineup, will focus on AI processing capabilities. There will be at least three variants of the M4 chip, and the highest-end desktop Mac might increase its RAM ceiling to 512 GB. The updated computers, including new iMacs, 14- and 16-inch MacBook Pros, and Mac minis with M4 chips, are expected to be released from late 2024 into early 2025.

  • What is the significance of the new Apple TV rumor regarding built-in camera and gesture-based controls?

    -The rumored new Apple TV with a built-in camera and gesture-based controls could enhance the user experience for video calling applications and potentially work seamlessly with the iPhone and Vision Pro. This suggests a move towards more integrated and interactive home entertainment systems.

  • How is Apple addressing the issue of iPhone theft and part resale?

    -Apple will begin allowing the use of genuine used Apple parts for repair, with on-device calibration after the part is installed. Activation lock will also be extended to iPhone parts, deterring thieves from stealing phones and selling their parts.

  • What is the irony in the story of thieves stealing Apple AirTags?

    -The irony is that the thieves stole AirTags, which have location tracking built into the product. This allowed the police to use the Find My application to locate and arrest the thieves, demonstrating the effectiveness of Apple's security features against theft.

Outlines

00:00

📱 Apple's iOS 18 and AI Advancements

This paragraph discusses the latest updates on Apple's iOS 18 and introduces a new AI system from Apple. The focus is on 'Ferit UI', a multimodal large language model (MLLM) that differs from traditional LLMs by understanding not just text, but also multimodal elements like images, video, and audio. The research paper compares Ferit UI to OpenAI's MLLM and GPT-4V, highlighting its superior performance on UI tasks. Although the paper is sophisticated and vague on implementation specifics, it's clear that Ferit UI's ability to recognize on-screen elements opens up new possibilities for the iPhone. The paragraph also mentions potential features in iOS 18, such as an AI-powered browsing assistant for Safari and encrypted visual search, indicating Apple's increasing focus on AI integration.

05:00

🎮 Game Emulators and iOS 17.5 Updates

This paragraph covers the EU's influence on Apple allowing game emulators on the App Store and the potential for these to launch globally. It emphasizes that developers are responsible for the software in emulator applications, suggesting that pirated games or ROMs will not be permissible. The discussion then shifts to iOS 17.5, currently in beta, which is adding new features such as a connection to Game Center for puzzle games in the News app, and a new puzzle game called 'Cor Tiles'. The paragraph also notes an increase in the number of Geekbench scores for iOS 17.5 beta 1 and mentions ongoing issues with battery drain in the beta versions. The performance and battery life experiences are personal, and the paragraph concludes with expectations for the release schedule of iOS 17.5.

10:01

💻 Future of Mac with M4 Chip and Apple TV Rumors

The future of Mac is discussed with a focus on the upcoming M4 chip, which is expected to enhance AI processing capabilities. Apple plans to equip its entire Mac lineup with the M4 chip, which will have at least three variants. The report suggests a potential increase in RAM for the highest-end Mac Pro to 512GB, but no changes in base storage or RAM are expected. The release of updated Mac models, including low-end MacBook Pro, high-end MacBook Pros, and Mac minis with M4 chips is anticipated from late 2023 to early 2024. Additionally, there are rumors about a new Apple TV that might include a built-in camera for video calling and gesture-based controls, as well as a lightweight smart display for home use. The paragraph concludes with information about Apple's decision to allow used parts for repair and the extension of activation lock to iPhone parts.

15:02

📱 AirTag Heist and iPhone 16 Rumors

This paragraph narrates an ironic story of thieves in California who stole Apple AirTags but were caught using the Find My application. The thieves faced multiple charges, and more stolen AirTags were found in their possession. The paragraph then transitions to discuss early rumors about the iPhone 16, suggesting it may come in seven different color options. While such rumors are speculative, there is a desire for more color choices than the previous iPhone models. The summary of the video's content concludes with a reminder to subscribe for updates on Apple news.

Mindmap

Keywords

💡iOS 18

iOS 18 is the latest operating system update from Apple for its mobile devices, which is expected to introduce new features and improvements over its predecessor. In the context of the video, iOS 18 is particularly notable for its integration of a new AI model, indicating a significant step forward in terms of智能化 and user experience on Apple devices.

💡AI Model

An AI Model, or Artificial Intelligence Model in this context, refers to the sophisticated algorithms and systems that Apple is developing to perform specific tasks, such as understanding and processing language, images, and user interactions. The video highlights the introduction of a new multimodal large language model (MLLM) called Feret UI, which is a significant advancement over previous models as it can understand not just text, but also visual elements like images and videos.

💡Multimodal Large Language Model (MLLM)

A Multimodal Large Language Model (MLLM) is a type of AI system that is designed to process and understand multiple forms of input, such as text, images, and audio. Unlike traditional LLMs that focus primarily on text, MLLMs like Feret UI can interpret and respond to a broader range of data, providing a more comprehensive and dynamic interaction with users. This capability is crucial for creating more immersive and intelligent user interfaces.

💡Safari Browsing Assistant

A Safari Browsing Assistant refers to an AI-powered feature that is expected to be included in iOS 18, designed to enhance the user's browsing experience on Apple's Safari web browser. This assistant would likely utilize machine learning algorithms to provide personalized recommendations, streamline navigation, and improve the overall browsing efficiency.

💡Encrypted Visual Search

Encrypted Visual Search is a feature that allows users to search for and identify visual content, such as images or objects, in a secure manner. This type of search is enhanced by encryption, which ensures that the data processed during the search is protected and maintains the user's privacy. The video implies that this feature could be an upgraded version of Apple's existing visual lookup feature, potentially offering more advanced search capabilities.

💡Game Emulators

Game Emulators are software applications that replicate the functionality of older gaming consoles, allowing users to play classic games on modern devices. The video discusses the possibility of game emulators being available on the App Store due to regulatory changes, which would be a significant shift from previous restrictions.

💡iOS 17.5

iOS 17.5 is a beta version of Apple's operating system for mobile devices, which includes new features and improvements over the previous versions. The video provides an update on the current beta stage of iOS 17.5, mentioning additional features such as a new puzzle game and enhanced integration with Game Center.

💡Find My Network

Find My Network is a feature provided by Apple that allows users to locate and secure their lost or misplaced Apple devices using a network of Bluetooth-enabled devices. The video discusses an upcoming enhancement to this feature in iOS 17.5, which will alert users if an unknown third-party item tracker is following them, providing an additional layer of security and privacy.

💡M4 Chip

The M4 Chip is the anticipated successor to the M3 Chip, which is a series of Apple-designed ARM-based processors used in Mac computers. The M4 Chip is reported to focus on enhancing AI processing capabilities, suggesting that future Macs will have improved performance and智能化 features. The video indicates that the M4 Chip will be included in every Mac in Apple's lineup, signifying a significant upgrade in computing power and智能化 for Apple devices.

💡Apple TV

Apple TV is a digital media player developed by Apple that allows users to stream content from the internet to their televisions. The video mentions a new Apple TV that might include a built-in camera for FaceTime and other video calling applications, as well as support for gesture-based controls, indicating a potential shift towards more interactive and智能化 home entertainment systems.

💡iPhone 16 Rumors

iPhone 16 Rumors refer to the speculative information and predictions about the potential features, design, and release date of Apple's future smartphone model. The video briefly touches on a rumor from a social media post suggesting that the iPhone 16 might come in seven different color options, indicating a possible trend towards greater customization and variety in Apple's product design.

💡AirTag

AirTag is a small, wireless device developed by Apple that helps users locate their personal belongings using the Find My app. The video shares a story about thieves who stole AirTags and were subsequently located and arrested by the police using the Find My application, highlighting the tracking capabilities of AirTags and their role in enhancing security and aiding in the recovery of lost or stolen items.

Highlights

iOS 18 introduces Apple's new AI system, a multimodal large language model (MLLM) called Feret UI.

Feret UI differs from traditional LLMs like GPT as it understands not only text but also multimodal elements such as images, video, and audio.

Apple's research paper compares Feret UI to OpenAI's MLLM and finds it surpasses GPT-4V on all UI tasks.

Feret UI can recognize elements of a user's home screen, such as app icons and small text, without text or voice inputs.

The AI can discuss visual elements in detail, propose goal-oriented actions, and deduce the overall function of the screen.

iOS 18 might add an AI-powered browsing assistant for Safari and encrypted visual search features.

The EU has forced Apple to allow game emulators on the App Store globally, with developers being responsible for the software offered.

iOS 17.5 beta introduces new features such as a connection to Game Center for puzzles and a new daily puzzle game, Cor Tiles.

Apple now shows the total amount a user will pay per month for combined subscriptions when adding a new service.

With iOS 17.5, third-party item trackers like Google's and Chipolo will be able to send alerts if an unknown tracker is following you.

Apple is expected to release iOS 17.5 in early May, alongside new iPad Pro and iPad Air models.

Apple is nearing production of M4-equipped Macs, with a focus on AI processing capabilities and at least three variants of the chip.

The highest-end desktop Mac may increase its RAM ceiling to 512 GB, but there's no word on increasing base storage or RAM.

Apple plans to update every Mac in the lineup with the M4 chip, starting late this year and extending into early next year.

A new Apple TV might have a built-in camera for FaceTime and other video calling applications, and support gesture-based controls.

Apple is considering a lightweight smart display for use around the house, as an alternative to a low-end iPad.

Apple will begin allowing used parts for repair in the fall, with activation lock extended to iPhone parts as well.

Rumor has it that the iPhone 16 could come in seven different color options, though this should be taken with a grain of salt.

Thieves in California were caught after stealing Apple AirTags, which were tracked using the Find My application.