Automated Content Quality Assurance for Crowdsourcing Educational Platforms - Brainly x deepsense.ai

The script begins with a technical issue where the presenter is testing the audio, specifically the music stream for the audience. After resolving the audio issue, the presenter, John Maros Pagania, introduces the Data Science Milan community, highlighting its foundation in February 2016 and its growth to over 1,700 members. The community meets monthly on YouTube and is part of the Italian Association for Machine Learning. John also mentions the staff and volunteers behind the events and invites new volunteers, directing interested individuals to their website and Slack workspace. The community offers a newsletter, blog summaries, and recordings of past events.

John introduces himself and his role at Greenlee as the Director of Artificial Intelligence. He also introduces Artur Zagado, a Senior Data Scientist at Deep Sense AI, who will be presenting alongside him. The presentation focuses on the automated content quality assurance system developed for educational platforms, specifically for Brainly, a Q&A platform for students. The system aims to personalize the learning experience and provide predictive interventions to assist students in their educational journey. The script discusses the importance of content quality and the strategy to achieve it through investments in data, user attributes, visual search, and curriculum analysis.

The script explores traditional content moderation strategies such as community moderation, external moderation services, and shadow banning. It highlights the limitations of these methods, including bias, delays, and scalability issues. The introduction of AI into the moderation process is discussed, moving from a binary classification approach to a multi-label classification problem. The AI system categorizes content into high risk, low risk, and 'request to fix' groups, improving the moderation process by providing immediate action on harmful content and suggestions for improvement on lower-quality content.

A live demonstration of the AI content moderation system is provided, showcasing its capabilities in detecting various types of content issues such as spam, non-English content, and mathematical expressions that may be misinterpreted by natural language processing models. The system uses an ensemble of models and an assembly function to filter and prevent low-quality questions. It also maintains a blacklist feed for client applications to filter content. The demonstration highlights the system's ability to detect and suggest improvements on content based on pre-calculated predictions.

Artur Zagado from Deep Sense AI shares lessons learned from developing the Automated Content Quality Assurance (AQUA) system. The first lesson emphasizes the importance of manually annotating a data sample to understand the problem better. The team used a multi-label scheme for content quality taxonomy, which helped in setting up guidelines for future annotations and improving the quality of labels over time.

The script discusses the importance of analyzing data patterns to identify potential issues and find quick solutions. It mentions the use of regular expressions to improve spam filtering and the challenges of mathematical expressions for NLP models. The presentation also highlights the use of state-of-the-art NLP solutions such as Detoxify, Perspective API, Gibberish Detector, and Microsoft Presidio for various content detection tasks.

Artur explains how complex problems like incomplete question detection can be broken down into simpler sub-problems. The team identified four major sub-problems: missing context, missing questions, missing choices, and truncated contents. A baseline model combining heuristics and NLP is proposed for detecting incomplete questions, which involves checking for specific tokens, multiple-choice formats, and imperative verbs to identify missing elements in questions.

The presentation concludes with future directions for the AQUA system, including improving baseline models with more Brainly-related data, researching mathematical formula detection, subject classification, and using interpretability methods and weak supervision for smarter data annotation. The team also plans to expand the solution to other markets and languages. The script ends with an invitation for questions and contact information for those interested in career opportunities with the companies involved.

The final part of the script addresses a question about experiment tracking, highlighting the use of neptune.ai for managing experiments efficiently. The presenter also mentions an upcoming case study on using neptune for tracking computer vision models and encourages viewers to stay tuned for its release. The session concludes with holiday wishes and an invitation for further questions.