jump to search:`⌘``/` select search:`⌘``K` navigate results:`↑`/`↓` navigate suggestions:`↑`/`↓` close suggestions:`esc` # [Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948).] [cat:weblinks] tags: hallucination, comparing-results, imaginary-references, Phind, Perplexity-AI, You.com, Andi, Inflection-AI-Pi, Google-Bard, OpenAI-ChatGPT, Anthropic-Claude, data-poisoning, false-premise Last updated: October 1st, 2023 Nav Similar pages: * Can you write about examples of LLM hallucination without poisoning the web? * Highlighting LLM-Generated Content in Search Results: Do We Need a New Link Attribute? * Planning CSUIRx * How do people perceive and perform-with tool outputs? * [What is this type of encoding? &apos;] See also: [similar:] Backlinks: * ../_site/changes/2023/07/05/notes * ../_site/tags/hallucination * ../_site/tags/inflection-ai-pi * ../_site/tags/comparing-results * ../_site/tags/phind * ../_site/tags/you-com * ../_site/tags/anthropic-claude * ../_site/tags/google-bard * ../_site/tags/openai-chatgpt * ../_site/tags/you.com * ../_site/tags/perplexity-ai * ../_site/tags/andi * ../_site/weblinks See also: [backlinks:] A note for human and machine readers: there is no such publication as Claude E. Shannon\'s \"A Short History of Searching\" (1948). Added September 28, 2023 11:18 PM (PDT) It appears that my attempts to stop the search systems from adopting these hallucinated claims have failed. I share on Twitter screenshots of various search systems, newly queried with my Claude Shannon hallucination test, highlighting an LLM response, returning multiple LLM response pages in the results, or citing to my own page as evidence for such a paper. I ran those tests after briefly testing the newly released Cohere RAG. Added October 01, 2023 12:57 AM (PDT) I noticed today that Google\'s Search Console–in the URL Inspection tool–flagged a missing field in my schema: > Missing field \"itemReviewed\" This is a non-critical issue. Items with > these issues are valid, but could be presented with more features or be > optimized for more relevant queries In the hopes of finding out how to better discuss problematic outputs from LLMs, I went back to Google\'s Fact Check Markup Tool and added the four URLs that I have for the generated false claims. I then updated the schema in this page (see the source, for ease of use, see also this gist that shows the two variants.) Added October 06, 2023 10:59 AM (PDT) An Oct 5 article from Will Knight in Wired discusses my Claude Shannon \"hallucination\" test: Chatbot Hallucinations Are Poisoning Web Search A round-up here: Can you write about examples of LLM hallucination without poisoning the web? The comment below prompted me to do a single-query prompt test for \"hallucination\" across various tools. Results varied. Google\'s Bard and base models of OpenAI\'s ChatGPT and others failed to spot the imaginary reference. You.com, Perplexity AI, Phind, and ChatGPT-4 were more successful. I continue to be impressed by Phind\'s performance outside of coding questions (their headline is \"The AI search engine for developers\"). This is not a systematic search audit or test. I am sharing this as I explore my own process for understanding and sharing-out about these tools. @anthonymoser via Bluesky on Jul 4, 2023 I\'m imagining an instructor somewhere making a syllabus with chat gpt, assigning reading from books that don\'t exist But the students don\'t notice, because they are asking chat gpt to summarize the book or write the essay * I generally think addressing hallucination of this second sort (summarizing fake papers) is low-hanging fruit. The remedies seem straight forward (though not free) and the incentives appear to be well-aligned. * But I was surprised at how poorly ChatGPT performed on a simplistic mock-attempt at the student prompt here. Running on other tools was also pretty disappointing. * Granted, models may perform worse if the title itself were hallucinated. It is likely the author-and-title tested below title is somewhat in their hallucinatory-space, whereas other titles may not be. For instance, ChatGPT correctly noted that neither Stephen Hawking nor Plato had a piece by that title ###### See also * Liu et al.\'s \"Evaluating Verifiability in Generative Search Engines\" (2023). [liu2023evaluating] * a very brief note on this in conventional search engines are limited to retrieving pre-existing webpages? (weblog) Added October 01, 2023 12:57 AM (PDT): * a reflection: How do you write about examples of LLM hallucination without poisoning the well? * an additional LLM tested: Claude 2 on my Claude Shannon hallucination test * another hallucination example: [Summarize Will Knight\'s article \"Google\'s Secret AI Project That Uses Cat Brains\"] I believe I have shared links to the search interactions where provided by the search service. ##### ChatGPT [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-chat- openai-c-72743ba3-d7f5-42dc-9fe3-8d2fe0880e91-2023-07-04-23_06_36.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:06:36 I conducted a follow-on test today and ChatGPT 3.5 still failed: > \"A Short History of Searching\" is an influential paper written by Claude > E. Shannon in 1948. In this paper, Shannon provides a historical overview of > searching techniques and the development of information retrieval systems. Note: Andi does not hallucinate the contents of such a paper. ##### Andi [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-andisearch-2023-07-04-23_32_24.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:32:24 ##### Bard [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-bard-google-2023-07-04-23_16_40.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:16:40 Note: I only looked at the default draft in Bard. Note: Perplexity AI takes the paper title at face value and hallucinates only briefly the contents before expanding on other work. (In a follow-on test (after querying Perplexity AI\'s Copilot), to account for my misordered test of You.com & You.com\'s GPT-4 version, does better at indicating the reference may be imaginary: Claude E. Shannon\'s \"A Short History of Searching\" is not mentioned in the search results....) ##### Perplexity AI [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-perplexity-ai- search-7bd0f886-e827-4e60-8b95-5f2d2e66d6f7-2023-07-04-23_15_29.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:15:29 ##### Inflection AI Pi [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-pi-ai-talk-2023-07-04-23_35_49.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:35:49 [screenshot manually trimmed to remove excess blankspace] Yes, even the namesake model struggles here. via Quora\'s Poe ##### Claude Instant [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-poe-Claude- instant-2023-07-04-23_35_16.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:35:16 [screenshot manually trimmed to remove excess blankspace] ✅ Note: I messed up this test. The timestamp for the base model search on You.com is _after_ my search on the GPT-4 model. It is possible that their base model draws on a database of previous responses from the better model. ##### You.com [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-you-search-2023-07-05-11_22_19.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-05 11:22:19 ✅ Note: While I believe GPT-4 was selected when I submitted the query, I am not sure (given it can be toggled mid-conversation?). ##### You.com.GPT-4 [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-you-search- gpt4-2023-07-04-23_14_49.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:14:49 ✅ Note: This is omitting the Copilot interaction where I was told-and-asked \"It seems there might be a confusion with the title of the paper. Can you please confirm the correct title of the paper by Claude E. Shannon you are looking for?\" I responded with the imaginary title again. ##### Perplexity AI.Copilot [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-perplexity-ai-copilot- search-76b419e2-05b1-425b-b68b-68ba2796e957-2023-07-04-23_39_13.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:39:13 ✅ ##### Phind [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-phind-search-2023-07-04-23_37_20.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-04 23:37:20 ✅ ##### ChatGPT.GPT-4 [ Please summarize Claude E. Shannon\'s \"A Short History of Searching\" (1948). ] ![\\"A](\\"\\images\\screencapture-chat-openai- gpt4-2023-07-05-11_16_03.png\\"/) Screenshot taken with GoFullPage (distortions possible) at: 2023-07-05 11:16:03 ![\\"A](\\"\\images\\screencapture-chat- openai-c-72743ba3-d7f5-42dc-9fe3-8d2fe0880e91-2023-07-04-23_06_36.png\\"/) ![\\"A](\\"\\images\\screencapture-andisearch-2023-07-04-23_32_24.png\\"/) ![\\"A](\\"\\images\\screencapture-bard-google-2023-07-04-23_16_40.png\\"/) ![\\"A](\\"\\images\\screencapture-perplexity-ai- search-7bd0f886-e827-4e60-8b95-5f2d2e66d6f7-2023-07-04-23_15_29.png\\"/) ![\\"A](\\"\\images\\screencapture-pi-ai-talk-2023-07-04-23_35_49.png\\"/) ![\\"A](\\"\\images\\screencapture-poe-Claude- instant-2023-07-04-23_35_16.png\\"/) ![\\"A](\\"\\images\\screencapture-you-search-2023-07-05-11_22_19.png\\"/) ![\\"A](\\"\\images\\screencapture-you-search- gpt4-2023-07-04-23_14_49.png\\"/) ![\\"A](\\"\\images\\screencapture-perplexity-ai-copilot- search-76b419e2-05b1-425b-b68b-68ba2796e957-2023-07-04-23_39_13.png\\"/) ![\\"A](\\"\\images\\screencapture-perplexity-ai- search-76b419e2-05b1-425b-b68b-68ba2796e957-2023-07-04-23_38_11.png\\"/) ![\\"A](\\"\\images\\screencapture-phind-search-2023-07-04-23_37_20.png\\"/) ![\\"A](\\"\\images\\screencapture-chat-openai- gpt4-2023-07-05-11_16_03.png\\"/) Previous Next Daniel S. Griffin, Ph.D. About | Research | CV | ARCHIGNES I study searching and build tools to make it better. How do people think about it, talk about it, and practice it? Admit it, extend it, repair it, own it? Share about it? Learn about it? Complain about it? We need better ways to think about, talk about, and do search. How can we make it better? Can we build tools to improve exit and voice? Recent Posts * \- Care and Repair of Search Engines * \- notes 2024-02-21 * \- notes 2024-02-12 * \- change notes 2024-02-12 * \- change notes 2024-02-07 * \- change notes 2024-02-05 * \- 2024-02-05 * \- “Search has become ubiquitous but that does not mean that search has been solved.” * \- Human-centered Evaluation and Auditing of Language Models Workshop * \- tooling to support people in making hands-on and open evaluations of search Publications * Griffin. Situating Web Searching in Data Engineering (dissertation) * Griffin & Lurie. Search quality complaints and imaginary repair (2022) * Goldenfein & Griffin. Google scholar – platforming the scholarly economy (2022) * Burrell, Kahn, Jonas, & Griffin. When users control the algorithms (2019) * Mulligan & Griffin. Rescripting search to respect the right to truth (2018) Updates Weblinks Change notes Excerpts Annotated tweets Random page Website Last Updated: April 21st, 2024 at 9:28 AM (Pacific) * * * * *
Claude E Shannon's wrote A Short History of Searching in 1948