AI Frontiers: Jesper Hvirring Henriksen (OpenAI DevDay)
TLDRJesper Hvirring Henriksen from Be My Eyes introduces a new feature, Be My AI on GPT-4V, in partnership with OpenAI. This AI assistant aims to provide blind and low-vision users with 24/7 visual assistance, enhancing their independence. With the ability to describe images and navigate digital content, GPT-4 Vision has opened up a world of accessibility, receiving overwhelmingly positive feedback from users. The technology's impact on assistive technologies is projected to be profound, as it continues to improve accessibility for all.
Takeaways
- 🌟 Be My Eyes, in partnership with OpenAI, has launched a new feature called Be My AI on GPT-4V.
- 👥 The app aims to provide visual assistance to blind and low-vision individuals through a community of over 7 million volunteers.
- 📱 Users can request help through video calls, allowing volunteers to lend their eyes to assist with tasks and navigation.
- 🚀 The introduction of Be My AI offers users an alternative to human assistance, providing an AI assistant available 24/7.
- 💡 GPT-4 vision enables computers to 'see' and describe the physical and digital world, enhancing accessibility for visually impaired users.
- 🌐 The AI can provide detailed descriptions of images, including those in group chats, websites, and other online platforms.
- 📈 An example showcased was the AI's ability to describe a graph from a US government website, which was previously inaccessible to screen reader users.
- 🎥 Beta testers like Caroline and Lucy have provided positive feedback, significantly increasing their independence and quality of life.
- 📊 Since the beta launch, the service has provided around a million image descriptions per month, with a 95% satisfaction rate.
- 🗣️ The AI supports over 36 languages, responding in the user's language, and has been integrated into enterprise customer support products like Microsoft's Disability Answer Desks.
- 🌈 The development and integration of AI models that can see and understand human speech are expected to greatly improve future accessibility and assistive technologies.
Q & A
What is Be My Eyes and what does it aim to provide for blind and low-vision individuals?
-Be My Eyes is an application that provides visual assistance to blind and low-vision individuals through a community of volunteers. It allows these individuals to request help via video calls, giving them access to sighted volunteers who can assist them in real-time.
How many users and volunteers does Be My Eyes have currently?
-Be My Eyes currently has over half a million blind and low-vision users supported by more than 7 million volunteers.
What was the motivation behind developing the Be My AI feature on GPT-4V?
-The Be My AI feature was developed to provide users with an alternative to human assistance. It aims to offer blind and low-vision individuals the ability to maintain their independence and receive assistance from an AI assistant that is available 24/7.
What are some of the real-world applications of GPT-4 vision?
-GPT-4 vision has a wide range of real-world applications. It can be used for utility-based tasks, such as describing objects or assisting with navigation. It can also enhance digital accessibility by providing meaningful alt text for images on websites and apps, making media and photos understandable to those who can't see them.
How does Be My AI assist users with images they encounter online or in apps?
-Be My AI provides thorough descriptions of any image users encounter online or in apps. This includes photos in group chats, images on websites, and other digital media that typically lack accessible alt text.
What was the reaction of one of the beta testers, Caroline, to the Be My AI feature?
-Caroline, a long-time user of Be My Eyes, became a Be My AI beta tester and has since used the feature to describe more than 700 images. This indicates a significant increase in her engagement with the service and suggests a positive reception of the feature.
How does Lucy Edwards incorporate Be My AI into her daily life?
-Lucy Edwards, another beta tester, uses Be My AI for various daily tasks such as checking eggs for shells, reading expiry dates on products, sorting laundry, identifying food on her plate, and even describing images on social media. This shows how Be My AI can assist with both practical tasks and enhance social inclusion.
What was the outcome of deploying Be My AI to a small Beta System group and later to all iOS users?
-After deploying Be My AI to a small Beta System group and later to all iOS users, the service has seen about a million image descriptions per month. The satisfaction ratings from users are extremely high, with a 95% approval rate when excluding downtime and system errors.
How many languages does Be My AI currently support and how was this achieved?
-Be My AI now supports up to 36 languages. This was achieved by instructing the model to respond in the same language that the user is using, demonstrating the flexibility and adaptability of the technology.
How has Be My AI been integrated into enterprise customer support?
-Be My AI has been deployed into the enterprise customer support product, with an example being Microsoft's Disability Answer Desks. Users can now choose to start with a chatbot instead of a call, and in most cases, the chatbot is able to resolve their issues without the need for a call.
What is the long-term vision for AI models that can see and understand human language?
-The long-term vision is that AI models capable of seeing and understanding human language will profoundly improve accessibility and assistive technologies in the future. They will not only enhance the lives of those with visual impairments but also contribute to making digital interactions more human-like and intuitive for everyone.
Outlines
🌟 Introduction to Be My AI and Its Impact
This paragraph introduces Jesper from Be My Eyes, a platform that connects blind and low-vision individuals with sighted volunteers for visual assistance through video calls. The introduction of Be My AI on GPT-4V, a partnership with OpenAI, marks a significant advancement. With over half a million users and seven million volunteers, the service aims to provide independence and reduce the burden on others. The new AI feature offers a 24/7 visual assistant, addressing various user feedback and scenarios where human interaction might not be desired or possible. The GPT-4 vision capability allows computers to 'see' and has a wide range of applications, from practical tasks to digital accessibility, transforming the way blind and low-vision users interact with the world around them.
📸 Real-Life Applications and User Experiences with Be My AI
This paragraph delves into the practical applications of Be My AI, showcasing its utility in everyday life. Users can now receive detailed descriptions of images they encounter online or in apps, improving their understanding and participation in social interactions. The narrative follows Lucy Edwards, a beta tester, as she demonstrates the use of Be My AI in her daily routine, from cooking and laundry to reading product labels and managing her social media. The feature's success is evident in the overwhelmingly positive feedback and high satisfaction ratings. The integration of GPT-4 Vision into Be My Eyes has also expanded language support and been implemented in enterprise customer support products, significantly enhancing accessibility and user experience.
Mindmap
Keywords
💡Be My Eyes
💡GPT-4V
💡Independence
💡Visual Assistance
💡Alt Text
💡Navigation
💡Digital Accessibility
💡Beta Tester
💡Enterprise Customer Support
💡Language Support
💡Assistive Technologies
Highlights
Jesper Hvirring Henriksen introduces Be My Eyes and its new feature, Be My AI, in partnership with OpenAI.
Be My Eyes is an app that provides visual assistance to blind and low-vision individuals through a community of volunteers.
The app has over half a million blind and low-vision users supported by more than 7 million volunteers.
Volunteers assist users through video calls, but the new AI feature offers an alternative for users seeking independence.
GPT-4V enables computers to 'see' and assist with tasks that were previously challenging for visually impaired users.
The AI can describe images, including those on websites and in apps, making content more accessible.
GPT-V provides detailed and accurate descriptions, enhancing the user experience for visually impaired individuals.
The AI's responses are not only accurate but also exhibit a level of wit and human-like interaction.
Beta tester Caroline increased her usage from two calls a year to over 700 image descriptions with Be My AI.
Lucy Edwards, another beta tester, demonstrates how Be My AI integrates into daily life in a video.
Be My AI helps users with tasks like checking eggs for shells, reading expiry dates, and identifying laundry colors.
The AI can also assist with meal identification, whether dining in or at a buffet.
Users can receive descriptions of perfume scents from images, enhancing their understanding of products.
Social media photo descriptions are provided by Be My Eyes, improving engagement for visually impaired users.
Feedback from beta testers has been overwhelmingly positive, with high satisfaction ratings reported.
The feature was rolled out to a small Beta group in March and later to all iOS users, resulting in a million image descriptions monthly.
Language support has been expanded to 36 languages by instructing the model to respond in the user's language.
Be My AI has been integrated into enterprise customer support products, such as Microsoft's Disability Answer Desks.
9 out of 10 users who start with a chatbot do not escalate to a call, showcasing the effectiveness of the AI in resolving issues.
The development and integration of AI models that can see and hear are expected to greatly improve accessibility and assistive technologies in the future.
