* This blog post is a summary of this video.

Can Chatbots Like Bard & ChatGPT Actually Read The Internet?

Table of Contents

Chatbots Struggle to Summarize Specific Web Pages

As AI chatbots like Google Bard, Anthropic Cloud, and Bing Chat become more popular, many people expect them to be able to read and summarize content from the internet. However, as we discovered in testing, these chatbots still face significant limitations when trying to summarize specific web pages.

Bard & Cloud Generate Inaccurate Summaries

When asked to summarize an article from the Everyday AI website, both Google Bard and Anthropic Cloud generated summaries that sounded reasonable but contained almost no factual information from the actual page. They appeared to fabrication information based on keywords rather than truly reading and comprehending the page's content.

Bing Chat Pulls Irrelevant Info From Other Sites

Similarly, the Bing chatbot produced a summary containing information about AI in healthcare, which were relevant topics but not actually discussed in the specific article I asked it to summarize. Further inspection revealed that Bing Chat had simply searched those keywords and pulled sentences from other websites.

How Chatbots Search The Web vs Read Pages

These experiences reveal key differences between how large language models like those behind Bard, Cloud, and Bing Chat search the internet versus how humans actually read and comprehend content.

Large Models Query Using Keywords

The foundation of these chatbots is a technique called predictive language modeling. Essentially, they have been trained on a huge dataset of text to predict the next word in a sequence. This allows them to generate remarkably human-like text when prompted with keywords. However, it does not mean they truly understand the content they are generating or summarizing. So when presented with a specific webpage, these models seem to simply extract keywords and then use those to query their internal knowledge, drawing text from their training data rather than reading and comprehending the actual page.

Plugins Needed for ChatGPT to Read Pages

In contrast, ChatGPT was the only chatbot that produced an accurate summary of the page when used with the appropriate plugins. This is because those plugins allow a human to provide additional instructions to the model, telling it to actually load and read the specific page rather than just searching keywords. So while the base capabilities of ChatGPT aren't so different from Bard or Cloud, the ability to provide more granular human feedback and correction is critical for completing more complex information processing tasks like summarizing a specific document.

Tips to Get Accurate Summaries from Chatbots

Given the limitations discussed above, here are some key tips to follow if you want to successfully leverage today's AI chatbots to summarize specific webpages:

Use Correct Plugins With ChatGPT

As mentioned, you need more specialized plugins and instructions when using ChatGPT for web page comprehension tasks. Make sure to install and properly configure capabilities like SearchAI and WebGPT3 to allow the model to actually load and read specific pages.

Check If Summaries Include Relevant Links

When summarizing a specific page, any good summary should include key facts, figures, quotes, or insights referenced or linked from the source page. Be skeptical of any summary, even if well-written, that does not link back to or accurately reflect information on the site. Checking the supporting links and facts is crucial to determining if a chatbot has truly read and understood a page versus just queried it for keywords and fabricated a response based on those.

The Future of Teaching Bots to Read Web Content

While today's chatbots clearly still struggle with tasks requiring true comprehension like summarizing web pages, rapid progress is being made. Google, Anthropic, and others are pouring resources into improving web-scale knowledge extraction and semantic understanding capabilities.

Key Takeaways on Chatbots Reading Web Pages

In summary, key takeaways include:

  • Chatbots like Bard and Cloud can generate reasonable-sounding summaries but often lack factual accuracy

  • These models currently query using keywords rather than actually reading and comprehending content

  • Instructions and plugins are needed for more accurate page summarization in ChatGPT

  • Checking that summaries link back to original sources is important

FAQ

Q: Can chatbots read entire web pages?
A: No, most chatbots like Bard and Cloud cannot accurately read and summarize specific web pages. They tend to search keywords and provide general info.

Q: Why did only ChatGPT summarize the page correctly?
A: ChatGPT was used with plugins that forced it to only reference the given webpage, instead of searching keywords like other chatbots.

Q: Do chatbot page summaries always include relevant links?
A: Not necessarily. You should check if chatbot summaries include links to verify the info is from the referenced page, not other sites.

PreNext