Meta Announces Llama 3 at Weights & Biases’ conference

The speaker, Joe Speac from Meta, welcomes the audience and introduces himself as someone deeply involved in the AI space for over a decade. He talks about his work with open source, PyTorch, and his role in building teams at Meta. Joe discusses the creation of an AI image of a llama and segues into the history of Llama, starting from its inception in February 2023. He mentions the collaboration of various teams across Meta and their collective effort in advancing AI technology. Joe also highlights his involvement in advising and investing in companies where he has close relationships with the founders.

The speaker delves into the timeline and development of Llama, starting from its release in July, its commercial availability, and the subsequent release of Code Llama for code-specific models. He discusses the impressive download statistics and the creation of derivative models, emphasizing the wide adoption and impact of Llama 2. Joe also introduces Purple Llama, an open trust and safety project, and its importance in the generative AI era. He outlines the updates and improvements made in Llama 3, including training on more data, a larger vocabulary, and a new tokenizer, and shares the excitement around the model's performance and reception.

Joe explains the four key areas of focus in developing Llama 3: model architecture, training data, training infrastructure, and post-training. He details the use of a dense Auto-regressive Transformer and the introduction of a new tokenizer. The speaker emphasizes the significant scaling up of training data and the custom-built infrastructure used for training. He also discusses the importance of post-training, which includes human annotations and techniques like rejection sampling and DPO. Joe stresses the balance between maximizing model helpfulness and ensuring safety, as well as the concept of red teaming to evaluate and mitigate potential risks.

The speaker discusses the importance of safety in AI models, particularly in the context of harmful use cases. He talks about evaluating models for their potential to assist in generating harmful content, such as bioweapons, and the need for dedicated teams to address these risks. Joe also covers the licensing aspects of Llama 3, its commercial and research usage, and the addition of branding guidelines. He highlights the ecosystem around Llama, including hardware vendors, enterprise platforms, and the open-source community, and the role of Purple Llama in managing safety through evaluation and mitigation strategies.

Joe presents data on Llama 3's performance, focusing on the refusal rate versus violation rate and its resistance to prompt injection attacks. He compares Llama 3's performance to other models and discusses the mitigation of overly cautious behavior in Code Llama 70b. The speaker also introduces Llama Guard 2, an open-source model based on Llama 3, and Code Shield, a tool for filtering insecure code produced by LLMs. He mentions TorchTune, a PyTorch fine-tuning library, and teases a larger model in development with impressive metrics, indicating future advancements in AI technology.

The speaker concludes by emphasizing the importance of multilingual and multimodal capabilities in AI, reflecting the global reach of Meta's family of apps. He discusses the company's commitment to safety and open-sourcing safety measures, as well as the intention to build a community and standardization around safety. Joe invites the audience to try out Llama 3 through Meta's platforms, where they can experiment with text generation and image creation, showcasing the practical applications of the technology.