World’s Fastest Talking AI: Deepgram + Groq

In this paragraph, Greg introduces the concept of combining a high-speed language model (LLM) with a fast text-to-speech model to create a rapid AI conversational system. He collaborates with Deepgram to test their new text-to-speech model and decides to use the Grock API for its impressive token processing speed. The process involves three main components: speech-to-text (STT), language model (LLM), and text-to-speech (TTS). Greg outlines the need for a transcription model to convert audio into text, a language model to process the text and generate a response, and a TTS model to convert the response back into audio. He emphasizes the importance of latency and the role of endpointing in conversational AI, where the system identifies natural breaks in speech to improve responsiveness. The chosen STT model is Deep Nova 2, which is noted for its speed and accuracy, and supports various scenarios and streaming capabilities.

Greg continues by demonstrating the use of Grock, a new model provider specializing in serving models quickly using custom chips called LPUs. Grock's proficiency lies in inference speed, which is showcased as Greg tests the model's ability to handle a long poem about trees, achieving an impressive token processing speed. The API is explored in both batch and streaming modes, with a focus on the latter for its real-time capabilities. The paragraph then shifts to the text-to-speech component, introducing Deepgram Aura, a new model that utilizes Deepgram's extensive audio data to create high-quality speech from text. The streaming feature is highlighted, which allows for data to be processed and returned in chunks, enabling almost real-time audio playback. Greg measures the time to first byte (the initial data chunk) and emphasizes the efficiency of the streaming process, which outperforms traditional batch processing in terms of latency.

In the final paragraph, Greg discusses the implementation of a conversational AI loop that incorporates memory via Lang chain to maintain context during interactions. He explains the use of an exit word, 'goodbye', to terminate the conversation. The process flow includes receiving transcription, passing it to the LLM for processing, obtaining a response, and converting it back to speech using the TTS model. The conversation manager class is introduced to handle the sequence of operations. Greg also addresses the trade-offs between latency and user experience, mentioning common practices like using filler words to mask delays. He touches on the complexity of implementing interruptions in a conversational AI system and leaves the audience with food for thought by suggesting a technique where the LLM could predict the remainder of a user's sentence based on the initial part, allowing for a response to be generated even before the user has finished speaking. The paragraph concludes with Greg inviting others to share their speech models on Twitter and reminding them of the available code in the description.