acco.io
about

URLs make ChatGPT stupid

2024-02-26|2 min

Ran into something funny today. I asked ChatGPT (v4) to read a particular web page and extract the data into a simple markdown structure:

Me asking ChatGPT to summarize specific content on HubSpot into a Markdown format

Its response was uncharacteristically oblivious to my request:

ChatGPT blowing past my request and providing me with a generic summary

Notably, it clearly did not access the website – there was no moment that it switched into a “Browsing...” state.

My theory was that I had triggered some sort of cache. I linked it to a URL it's probably seen before. I’m assuming web browsing is expensive and therefore something it tries to avoid.

What’s funny is that it clearly didn’t cache the page, but instead cached its response to the page (way cheaper!)

Dropping a URL in does not automatically shut its faculties off:

It answers my question about chicken/pollo, then notices the URL I dropped in - but does not summarize it

It seems like the kill switch is when I ask it to perform an operation on the URL!

I ask it to check the page for the word 'spanish' and it blows past my request, providing me with a generic summary

It really doesn’t want to access the page. When I tell it to check the page again, it clearly doesn’t access the page (the word “checkbox” does indeed appear on the page):

I ask it to check the page for the word 'spanish' and it blows past my request, providing me with a generic summary

However, it did cheekily name our conversation “Crear Objectos Personalizados”, which is just delightful:

The name of our conversation

To confirm my theory, I tried something else: I saved the webpage as a PDF, then uploaded the PDF to ChatGPT. That worked just fine:

I ask it to check the page for the word 'spanish' and it blows past my request, providing me with a generic summary