Build Knowledge Search Apps with Generative AI and Document Understanding

Table of Contents

• Introduction to Generative AI for Customized Search
• Demo: AI-Powered Search App for Oil and Gas Documents
• Key Components for Building Your Own AI Search Application
• Conclusion and Next Steps for Getting Started

Introduction to Generative AI for Customized Search

Generative AI models like Claude can be very powerful for answering questions and generating text. However, they are typically trained on publicly available data, which means they may not have detailed knowledge of domain-specific topics like oil and gas or company-specific information.

In this blog post, we'll explore an approach to combine the power of large language models like Claude with focused knowledge bases, allowing us to create customized and intelligent search applications.

Problem: Ingesting Domain-Specific Documents into AI

Many companies have decades of internal documents - PDFs, PowerPoints, Word files, memos, and more. These documents contain a wealth of company knowledge, but it can be difficult to leverage that information with today's AI systems. We need a way to ingest these domain-specific documents so that the AI can consult and understand this information when answering questions or generating text.

Solution: Combining Large Models with Focused Knowledge Bases

The solution is to index these internal documents in a search engine optimized for enterprise data. Amazon Kendra is a great choice here - it can crawl documents from various sources including S3 and on-prem filesystems. When a user asks a question, we first search this focused Kendra knowledge base to find the most relevant text passages. We pass this context to a large language model like Claude, which is then able to format a rich, detailed response by expanding on the key information found in the knowledge base.

Demo: AI-Powered Search App for Oil and Gas Documents

To demonstrate this approach, we built a sample web application using Streamlit that is powered by Claude and Kendra behind the scenes. We indexed public oil and gas industry articles into Kendra to serve as our domain-specific knowledge base.

When asking questions about topics like the Permian Basin geology or the OSDU data standard vision, our AI search engine provides detailed answers with inline citations back to source documents. This shows the possibilities for leveraging legacy company content to build customized and intelligent applications.

Key Components for Building Your Own AI Search Application

The key components leveraged to create this demo provide a blueprint for how you can build your own AI search application tailored to your company's documents and knowledge.

Leverage Large Language Models like Anthropic Claude

Services like Amazon Comprehend or tools like Hugging Face provide easy access to state-of-the-art models for natural language processing. Anthropic's Claude takes this a step further for dialog applications. By handling the prompting, Claude allows us to focus on the search and indexing logic while benefiting from powerful text generation capabilities.

Ingest Documents into Amazon Kendra for Search

Amazon Kendra handles the complexity of natural language search, leveraging machine learning for semantic indexing and retrieval. We can point Kendra to our various knowledge sources, and it automatically indexes and processes the content. When users ask questions, Kendra finds the most relevant content within the ingested documents to feed context into downstream NLP models like Claude.

Conclusion and Next Steps for Getting Started

This demo has shown an effective template leveraging Claude, Kendra, Streamlit and Linkchain to unlock legacy company knowledge for next-generation applications. The same principles apply across any industry or domain.

We invite you to explore these technologies and consider how AI search could transform use cases within your organization. Please reach out if you would like help building a proof-of-concept prototype tailored to your business content and objectives.

FAQ

Q: What is a large language model in AI?
A: Large language models like Anthropic and GPT-3 are AI systems trained on massive text data to generate human-like text and power applications like search and chatbots.

Q: How does Amazon Kendra work?
A: Amazon Kendra is an intelligent enterprise search service that ingests documents and data sources using ML and NLP to create a powerful search index.

Q: What is prompt engineering for AI?
A: Prompt engineering is the practice of carefully crafting the inputs or 'prompts' to AI systems to get better, more focused outputs.