I am puzzled by the difference in behavior from ChatGPT when used from Drafts as opposed to being used from its own web site - with the same API key.
I created a prompt that I use in Drafts asking ChatGPT to collect information about books from Amazon and to return that information in a specified format - including transforming the Amazon link to a book using a tiny url. That works fine, including with recently published books. By “recent” I mean well after the nominal 2021 cutoff. The returned information is unmistakably coming from the Amazon site.
When exactly the same prompt is used from the OpenAI web site with either ChatGPT 3.5 or 4, the AI says it has no ability to retrieve information from the web or to create a tiny url.
It is great that this kind of prompt works from Drafts but I wonder if someone can explain the difference in behavior.
I’m going to go ahead and state upfront that I can’t explain it with any authority.
It is my understanding, and similar experience, that a VLLM like ChatGPT has a corpus of data it works with, much like a search engine, to return results. It does not do things like go out and make network requests on your behalf to generate new data in its results.
If it is returning a tiny URL for you in some cases, that might be because someone else in the past has also generated a tiny URL for the same page and it knows about it? Maybe?
LIke if you ask it to summarize a URL for you, it will, if that URL is in the corpus, but it will be based on some past version of that page when it was snapshot last.
As for why the API and web-versions differ? They are probably addressing different revisions of the corpus? Maybe.
Further investigation… A prompt that received a “no can do” response from the Open AI web site, resulted in invented information being returned by the same prompt used in Drafts. The “fiction” was plausible - sounded good until compared to the actual information on the Amazon site.
it would be interesting to try it 20 x on each. I have a feeling it’s a random thing. Even on the website or on other GPT API-accessing apps, sometimes it makes something up and sometimes it says it can’t access the web (although I never use their website directly so I can’t say if it’s more one than the other now).
Either way, it’s always been a liar when summarizing anything based on URL. It’s best to download the contents of the page yourself and feed that to it to summarize.