An LLM journey speed run: Going from Hugging Face to Vertex AI

The video begins with Skander Hannachi and Rajesh Thallam introducing themselves as part of Google Cloud's Vertex AI applied engineering team. They express excitement about discussing the augmentation of a Gemini-based generative AI solution with open source models from Hugging Face. The focus is on integrating a time series forecasting model into a Gemini-based large language model using Google's open source time series foundation model, TimesFM, recently launched on Hugging Face. The discussion aims to guide viewers on using the Vertex AI platform to build generative AI applications, with a specific use case of enhancing retail and e-commerce product catalogs with demand forecasting capabilities.

Skander and Rajesh delve into the thought process and options available for constructing generative AI applications using the Vertex AI platform. They highlight the platform's foundational model, which is a multi-task model suitable for various use cases. They emphasize that there's no single model that fits all scenarios, advocating for a 'garden' approach where task-specific or smaller models are integrated as needed. The speakers also touch on the differences between Google AI Studio and Vertex AI Studio, with the latter offering an end-to-end machine learning platform. They outline the three components of the Vertex AI platform: the Vertex Model Garden for model repositories, the Vertex Model Builder for custom use case development, and the Vertex Agent Builder for configuring agent-based use cases.

The conversation shifts to a concrete industry example, focusing on integrating an open source model with Vertex AI to add demand forecasting capabilities to an e-commerce product catalog solution. Skander discusses the challenges retailers face in managing product catalogs and how the Vertex AI team's solution automates the generation and enhancement of product content using Gemini and Imagine. The solution involves a user journey that starts with a merchant uploading a product description and image, followed by category detection, attribute filtering, and the generation of detailed product descriptions and image edits. The system design is likened to a single path state machine, with potential for expansion into agentic systems that assist with tasks like demand forecasting.

Skander addresses the complexities of demand forecasting, traditionally managed by either cumbersome ERP systems or custom ML pipelines. He introduces TimesFM, Google's newly released open source time series foundational model, which is trained on synthetic data and capable of zero-shot learning for various time series forecasting tasks. TimesFM's strength lies in its ability to generate forecasts on the fly without extensive training, making it accessible for merchants who lack the operational capacity for more traditional forecasting tools. The model's performance is benchmarked against dedicated models, showcasing its accuracy and efficiency.

Rajesh demonstrates the practical steps for integrating the TimesFM model from Hugging Face into Vertex AI. He outlines the user journey for reviewing item performance by building an agent using foundational models from Vertex AI and Hugging Face. Rajesh explains the concept of a generative AI agent, which goes beyond simple text generation to perform complex tasks using tools and orchestration. He details the four components necessary for building agents: models, tools, orchestration, and runtime. The process involves deploying the TimesFM model, defining Python functions as tools, creating a LangChain agent, and deploying the agent on Vertex AI's Reasoning Engine for scalability and security.

The video proceeds with a live demonstration of deploying the TimesFM model from Hugging Face onto Vertex AI's prediction endpoint. Rajesh guides viewers through data preparation, including downloading a dataset from Kaggle, uploading it to BigQuery, and transforming it for time series forecasting. He then demonstrates deploying the model using Vertex AI SDK, creating a custom container, and testing it locally before pushing it to the model registry and deploying it to an endpoint. The endpoint's response, which includes point and quantile forecasts, is used to illustrate how the model can be utilized for demand forecasting.

Rajesh defines functions as tools that will interact with BigQuery and the TimesFM model on Vertex AI. He creates Python functions to fetch data from BigQuery and invoke the TimesFM model's forecasting capabilities. The video shows testing these functions locally and plotting the forecasts against historical sales data. The process of defining a LangChain agent and binding the model with these tools is detailed, including setting up the agent with a prompt prefix to ensure the model uses the correct dataset. The agent is tested locally, and its ability to query BigQuery and generate forecasts is demonstrated.

The final steps of deploying the agent on Vertex AI's Reasoning Engine are covered. Rajesh defines the agent, including its dependencies on Google BigQuery and Vertex AI SDK, and deploys it to Cloud Run. He tests the remote agent's functionality, showing how it can query BigQuery and generate forecasts when prompted. The video concludes with Skander and Rajesh summarizing the key takeaways: the importance of integrating various models for different tasks, the ease of deploying Hugging Face models on Vertex AI, and the potential of combining generative AI with predictive models for powerful solutions. They also highlight the flexibility of deploying scalable generative AI agents using function calling and Reasoning Engine. The presenters invite viewers to discuss further if they have additional questions.