Writing Better Code with Ollama

Matt Williams
26 Jan 202404:43

TLDRThe video discusses the recent release of an official Node.js library by the Ollama team, which the speaker plans to use alongside its Python equivalent. They demonstrate using Ollama by calling the chat endpoint with a system prompt and question, and explore the difference between the default non-streaming response and the streaming response, which returns an async generator. The speaker highlights the use of Llama Coder, an AI-assisted coding tool that suggests code completions and can write code based on comments. They also mention Continue.dev, a platform that allows users to ask coding questions and receive answers. Both tools are praised for being free and functional offline, which is beneficial for the speaker living on an island. The video provides a brief guide on setting up Ollama, Llama Coder, and Continue, and encourages viewers to share their local coding tool setups or configurations.

Takeaways

  • 🚀 The Ollama team has released an official Node.js library, which can be found on GitHub.
  • 🤖 The author plans to start building with the Node.js library and its Python equivalent soon.
  • 💡 The chat endpoint can be called with a system prompt and initial question, and the response is a JSON blob with output and performance metrics.
  • 🔁 Setting the stream to true changes the function to return an async generator, which requires a different handling method.
  • 📝 Llama Coder, an alternative to Copilot, can auto-complete code based on typed prompts and even write code from comments.
  • 📜 To get the tokens without newlines, the author suggests using `process.stdout.write` instead of `console.log`.
  • 🏝 The author lives on an island near Seattle and appreciates tools that work offline, such as Llama Coder and Continue.
  • 🛠️ Llama Coder and Continue are free VS Code extensions that provide a good alternative to Copilot, even without an internet connection.
  • ⚙️ The author recommends configuring the model in Llama Coder for optimal performance and trying different models to see which works best.
  • 🔍 Continue.dev allows users to ask questions about code and provides answers, with the ability to select and use different models.
  • 📄 The JSON config file can be updated to add any model available with Ollama to Continue.
  • 📚 The author encourages viewers to review the documentation for Continue and other VS Code extensions for more features and customization options.

Q & A

  • What is the name of the official Node.js library recently released by the Ollama team?

    -The official Node.js library released by the Ollama team is simply referred to as 'Ollama' in the transcript.

  • Where can one find the Node.js library for Ollama?

    -The Node.js library for Ollama can be found on GitHub.

  • What is the default behavior of the chat endpoint in the Ollama library?

    -The chat endpoint in the Ollama library defaults to not streaming, which means the response is a JSON blob with the output and some performance metrics.

  • What happens when you set the stream to true in the Ollama library?

    -Setting the stream to true changes the function to return an async generator instead of a JSON blob, which allows you to start seeing content sooner.

  • What is the name of the AI coding assistant mentioned in the transcript?

    -The AI coding assistant mentioned is called 'Llama Coder'.

  • How does Llama Coder assist in writing code?

    -Llama Coder assists by autocompleting code as you type and even generating code based on comments describing what you want the code to do.

  • What is Continue.dev and how does it help in coding?

    -Continue.dev is a VS Code extension that allows users to ask questions about their code and get suggestions or solutions. It can remind users about alternative methods or functions that they might not be aware of.

  • Why are Llama Coder and Continue considered good alternatives to Copilot?

    -Llama Coder and Continue are considered good alternatives to Copilot because they are completely free, work offline, and provide a similar level of assistance in coding.

  • What is the advantage of using process.stdout.write over console.log?

    -process.stdout.write is an alternative to console.log that doesn't automatically add a newline character, allowing for more control over the output formatting.

  • How can one configure the Llama Coder extension in VS Code?

    -To configure the Llama Coder extension, one needs to select the appropriate model that best suits their needs in terms of speed and accuracy. The model 'deepseek coder 1.3b q4' is mentioned as a fast and effective option in the transcript.

  • What is the purpose of the JSON config file in Continue?

    -The JSON config file in Continue allows users to add or update the model used by the extension. This enables customization and the use of any model available with Ollama.

  • How can one disable telemetry in Continue to avoid internet usage?

    -The transcript mentions that there is a setting in the Continue documentation to disable telemetry, which prevents the extension from using the internet for any purpose.

Outlines

00:00

🚀 Introduction to Ollama's Node.js Library

The video begins with the presenter sharing an update about the Ollama team's recent release of an official Node.js library, which is available on GitHub. The presenter expresses their intention to start experimenting with this library alongside its Python counterpart. They demonstrate the process of importing the library, instantiating it, and using the chat endpoint with a system prompt and initial question. The default setting for the endpoint is non-streaming, resulting in a JSON response containing the output and performance metrics. However, the presenter highlights the perceived faster response when streaming is enabled, as content becomes visible sooner. They encounter an error when attempting to enable streaming, which is clarified as a result of the function returning an async generator instead of a JSON blob. The presenter also mentions the use of Llama Coder, an alternative to Copilot, which assists in code generation and accepts user-provided comments to generate corresponding code snippets. The video emphasizes the benefits of using these tools, especially in offline scenarios, and provides a brief guide on how to set up and configure the necessary extensions for optimal use.

Mindmap

Keywords

💡Ollama

Ollama is a software tool or platform mentioned in the video that has recently released an official Node.js library. It is used for building applications and seems to have features that assist in coding, such as auto-generating code snippets based on user prompts. In the context of the video, Ollama is central to the discussion as the presenter uses it to demonstrate how it can facilitate the coding process.

💡Node.js library

A Node.js library refers to a collection of JavaScript code that provides specific functionality which can be reused in programs running under Node.js runtime. In the video, the presenter discusses an official Node.js library released by the Ollama team, which they intend to use for building applications, indicating its importance in the development workflow.

💡GitHub

GitHub is a web-based platform used for version control and collaboration that allows developers to work on projects from anywhere. The video mentions GitHub as the location where one can find the newly released official Node.js library for Ollama. It is a key platform for developers to share and access code, and in this context, it serves as the source for the library the presenter is discussing.

💡Chat endpoint

In the context of web development, a chat endpoint refers to a specific location in a program that handles chat-related functionalities. The video script describes using the chat endpoint with a system prompt and an initial question, suggesting that it is a part of the Ollama tool used for interacting with the AI or automating responses.

💡JSON blob

A JSON blob is a term used to describe a large, often hard-to-read, piece of JavaScript Object Notation (JSON) data. In the video, the presenter mentions that the chat endpoint's default behavior is to return a JSON blob containing the output and some metrics, which implies that it is a way to receive structured data from the endpoint.

💡Streaming

Streaming in the context of this video refers to the process of sending or receiving data in a continuous, sequential flow, rather than all at once. The presenter discusses setting the stream to true, which changes the function to return an async generator instead of a JSON blob, allowing the user to see content sooner as it is generated.

💡Async generator

An async generator is a type of iterator in JavaScript that can asynchronously yield values. When the presenter sets the stream to true, the function returns an async generator, which is a different way of handling the output compared to a JSON blob. This allows for a more dynamic and potentially more efficient way of processing the data stream.

💡Llama Coder

Llama Coder is an extension for Visual Studio Code (VS Code) mentioned in the video as an alternative to Copilot. It assists in coding by auto-completing code snippets and writing code based on user comments. The presenter uses Llama Coder to enhance their coding experience, particularly when they are offline or have limited internet connectivity.

💡Continue.dev

Continue.dev is another VS Code extension highlighted in the video. It allows users to ask questions about their code and receive suggestions or solutions. The presenter uses it to inquire about alternatives to console.log that don't add a newline, demonstrating its utility in solving coding problems and improving the code writing process.

💡Process.stdout.write

Process.stdout.write is a method in Node.js used for writing output to the standard output stream without adding a newline character. The video script mentions it as an alternative to console.log when the presenter wants to print content without newlines, which is useful for formatting the output in a specific way.

💡Telemetry

Telemetry is the process of collecting and transmitting measurements at a distance, often used in the context of software to gather usage statistics. The video mentions disabling telemetry in the Continue extension to prevent it from using the internet, which is important for the presenter who works offline and wants to ensure their tools work without an internet connection.

Highlights

Ollama team released an official Node.js library for Ollama.

The speaker plans to start building with the Node.js library and its Python equivalent.

Demonstration of importing Ollama and instantiating it in the code.

Usage of the chat endpoint with a system prompt and initial question.

Default behavior of the endpoint to return a JSON blob with output and performance metrics.

The perceived speed improvement when using streaming to receive responses.

Error encountered when setting stream to true, which changes the function's return type.

Autocompletion feature of Llama Coder for typing console.log and stream.

Llama Coder's ability to write code based on described comments.

The need to refine code to handle JSON blobs and tokens without newlines.

Use of Continue.dev for expert code advice and alternative solutions.

Alternative to console.log suggested: process.stdout.write.

Comparison of Llama Coder and Continue with other VS Code extensions like Copilot.

Advantages of using these extensions offline, especially in areas with no internet connection.

Instructions on setting up Ollama, Llama Coder, and Continue with model configurations.

Recommendation to review Continue's documentation for additional features and settings.

Mention of other VS Code extensions like Code GPT and Ollama Autocoder.

The speaker's interest in trying out Cody from Sourcegraph and discussing local alternatives to Copilot.

Invitation for viewers to share their setups or configurations in the comments.