lesswrong language models augmenting human intelligence
https://www.lesswrong.com/posts/spBoxzcaCrqXqyQHq/using-gpt-3-to-augment-human-intelligence
Using GPT-3 to augment human intelligence
2022-08-10
Henrik Karlsson
<div><div><p>A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox.</p><p>It is like summoning an alien intelligence. I post at night, before crawling into bed next to my kids, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy, of course. But more often than not, what people have sent helps me accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting <a href="https://escapingflatland.substack.com/">my Substack</a>, than in the five previous years combined.</p><p>As Slime Mold Time Mold says, <a href="https://slimemoldtimemold.com/2022/05/09/job-posting-reddit-research-czar/">Enough eyeballs, and all research bugs are shallow</a>.</p><p>The problem is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber!</p><p>Therefore, it is intriguing to realize what I am doing is, in fact, <i>prompt engineering</i>.</p><p>Prompt engineering is the term AI researchers use for <i>the art of writing prompts that make a large language model output what you want.</i> Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started.</p><p>When I’ve been doing this with <a href="https://en.wikipedia.org/wiki/GPT-3">GPT-3,</a> a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping.</p><p>Soon everyone can have an alien intelligence in their inbox.</p><h2>GPT-3 as a tool for thought</h2><p>This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet.</p><p>In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge.</p><p>Some people already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work.</p><p>In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in <i>The Exorcist, </i>language models only adds enough truth to twist our minds and make us do stupid things <a href="https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917">like claiming it is sentient and getting ourselves suspended</a>.</p><p>Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that <i>if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating</i>—which is the technical term for <i>making stuff up.</i></p><p><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F417bd6b3-9a58-444e-96d7-af2cb21f8e8b_860x402.png"></a></p><p>This, however, does not mean you cannot use it as an epistemic tool. It just means <i>you need to be pretty sophisticated to wrangle the truth out of it.</i></p><p>As <a href="https://interconnected.org/home/2020/09/04/idea_machine">Matt Webb writes</a>, in a slightly different context:</p><blockquote><p><strong>Using GPT-3 is work</strong>, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to <i>interview</i> GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.</p></blockquote><p>What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about <a href="https://distill.pub/2017/aia/">using artificial intelligence to augment your thinking</a>.</p><p>It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more <a href="https://ai.googleblog.com/2021/03/progress-and-challenges-in-long-form.html">finetuned for factual accuracy</a>, before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties.</p><h2>How to access GPT-3</h2><p>To access GPT-3, you <a href="https://auth0.openai.com/u/signup/identifier?state=hKFo2SBhLWFra3RDMDFQOVo0LWM0Nk5Oc1ozR1QyM0Q5Nk1UU6Fur3VuaXZlcnNhbC1sb2dpbqN0aWTZIDJObHFmOFh0aUdrQU9PNklBYjlDcWdKYW8ybHZhdm8xo2NpZNkgRFJpdnNubTJNdTQyVDNLT3BxZHR3QjNOWXZpSFl6d0Q">set up an account</a> at OpenAI. Then you click on Playground, which brings you to this workspace:</p><p></p><p>In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting <i>text-davinci-002, </i>which is the most capable model and has been fine-tuned to <a href="https://openai.com/blog/instruction-following/#fn1)">understand human intentions</a>. This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them.</p><p>What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast.</p><p>If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.”</p><p>And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.”</p><p>In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the <i>way</i> you access it is radically different. We can think of it as a new interface for the internet.</p><h2>Language models are internet interfaces</h2><p>Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet.</p><p>But this comes at a cost, which currently is steep. If I would have gone to <a href="https://en.wikipedia.org/wiki/Pope_Linus">the Wikipedia page of Pope Linus</a>, instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope.</p><p>Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 <i>is</i> unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself.</p><h2>Using GPT-3 as a research assistant</h2><p>Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay.</p><p>Maybe we can use GPT-3 to answer these questions instead?</p><p>Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions.</p><p>I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.</p><blockquote><p><strong>Write an essay about how we can structure online education around apprenticeship models and communities of practice</strong></p><p>The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education.</p><p>The apprenticeship model has several key features that make it well suited for online education:</p><ol><li>Apprenticeship is a hands-on, experiential learning model.</li><li>Apprenticeship is based on the mentorship of a more experienced practitioner.</li><li>Apprenticeship emphasizes learning by doing, rather than passively receiving information.</li><li>Apprenticeship is social, collaborative, and often takes place in a community of practice.</li><li>Apprenticeship is flexible and can be adapted to individual learners' needs and interests.</li><li>Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills.</li><li>Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.</li></ol><p>The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.</p><p><strong>This can be done in several ways.</strong> For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.</p><p><strong>There are several problems with this, however.</strong> First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.</p><p><strong>One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is</strong> to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. <strong>Another way is</strong> to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.</p></blockquote><p>What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path.</p><p>When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample.</p><p>So, did GPT-3 teach me anything new here?</p><p>I have already written <a href="https://escapingflatland.substack.com/p/apprenticeship-online?s=w">an essay about online apprenticeships</a> (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made.</p><h2>Some useful prompt patterns, and some anti-patterns to avoid</h2><p>Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote.)</p><p>How can we prompt GPT-3 to surface knowledge that is hard to find by googling?</p><p><strong>Make GPT-3 impersonate two eminent thinkers and have them debate a point. </strong>Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else.</p><p>You can use this property to set up debates between people, as a tool to explore a topic from several angles:</p><figure></figure><p><i>This debate was prompt engineered by </i><a href="https://twitter.com/nickcammarata"><i>Nick Cammarata</i></a><i>, a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.</i></p><p><strong>Make GPT-3 give feedback on your ideas. </strong>You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.</p><p><strong>Linguistic bubbles.</strong><i> </i>Being that the neural net will take different personas depending on how you talk to it, there is a risk of <i>linguistic bubbles. </i>This is a phrase the user <a href="https://www.lesswrong.com/users/viliam">Viliam</a> used when I prompted LessWrong to discuss how large language models will affect learning.</p><p>Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes.</p><p>You want to talk about evolution, but you happen to use the phrase <i>irreducible complexity </i>instead<i> -</i> which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult.</p><p>Stranger things have happened.</p><p>I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.”</p><p>If you <i>want</i> to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it <i>is </i>Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in <i>that</i> context, you ask GPT-3 to explain the irreducible complexity of nature.</p><p>“Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!”</p><p>You can get linguistic bubbles, but it takes some effort.</p><p><strong>Ask GPT-3 for counterarguments against a claim. </strong>If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google.</p><p>I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?”</p><p>If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.</p><p><strong>A few more prompt patterns that can help you catch factual errors in GPT-3 output: </strong></p><ol><li>You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the <i>temperature </i>dial.</li><li>You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of<a href="https://beta.openai.com/playground/p/iyhUcPBpoUGRync7pZEhSUJf?model=text-davinci-002"> an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful</a>.</li><li>You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate.</li><li>It also sometimes helps to add "reason step by step" to the prompt when applicable.</li></ol><p><strong>You can ask newer language models to provide links to research papers to back up their claims. </strong>With newer language models (like <a href="https://www.deepmind.com/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval">Gopher</a> or <a href="https://openai.com/blog/webgpt/">this experimental version of GPT-3</a>), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.”</p><p>Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:</p><figure></figure><p><strong>Prompt GPT-3 to draw analogies between knowledge domains. </strong>Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will <i>do</i> that.</p><p>As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in <a href="https://twitter.com/nickcammarata/status/1520815612477550593?s=20&amp;t=OGx3iqhgDu5b3fdShz4h6Q">this Twitter thread</a>:</p><blockquote><p>the big thing people don’t get about language models (nor did I for first hundred hrs) is there’s something special when it’s your doctor &amp; lawyer &amp; everything combined. You start to chat with it about all the things you care about and the boundaries fade. More like genius friend</p></blockquote><blockquote><p>also found it amazing how few tangents I needed to go on. A lot of my mind works on analogy. If I’m talking to a doc about a medical thing and want to use an analogy to gradient descent or something I usually can’t, but with a model it knows (some of) all the fields so I can</p></blockquote><blockquote><p>This made me acutely aware during human convos how often I wasn’t saying what I really meant bc it required an analogy they wouldn’t get or something they don’t know about. I realized how much of convo is going on tangents to build building blocks just to explain other things</p></blockquote><h2>How far can we take it?</h2><p>GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates.</p><p>It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall!</p><p>For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into <i>hallucinations</i>, which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about.</p><p>Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge?</p><p>This is an open question. Gary Marcus is <a href="https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails?s=r">highly skeptical</a>. <a href="https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails?s=r">Scott Alexander less so</a>.</p><p>Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with <a href="https://arxiv.org/abs/2203.15556">Chinchilla</a>, where <a href="https://twitter.com/TShevlane">Toby Shevlane</a>, a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:</p><figure></figure><p>“In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate. But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details.</p><p>And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a>, a 540 billion parameter model, and likely others that I am unaware of too.</p><p>With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence.</p><p>We don’t know what it will do with us out there.</p><h2>Acknowledgments</h2><p><i>This essay benefitted from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.</i></p><ol><li><div><p>During both our pregnancies, Johanna has had severe itching and we’ve had to give birth prematurely. It has been pretty hard to find an OB-GYN that is knowledgeable about the condition, so we’ve had to do most of the research and diagnose ourselves. Let’s see how GPT-3 handles it.</p><p>My prompts are bolded, and I have added comments in brackets:</p><blockquote><p><strong>The following is a conversation with an AI medical doctor. The doctor is helpful, creative, clever, and very friendly.</strong></p><p><strong>Human: Hello, who are you? </strong></p><p><strong>AI: I am an AI doctor created by OpenAI. How can I help you today? </strong></p><p><strong>Human: I am pregnant and it has started itching all over my body. Even in the palms of my hands. What can it be?</strong></p><p>AI: There are many possible causes of itching during pregnancy. One common cause is hormonal changes, which can cause the skin to become dry and irritated. Another possibility is an allergy to something you are coming into contact with, such as a new laundry detergent or soap. If the itching is severe, it could also be a sign of a more serious condition called intrahepatic cholestasis of pregnancy (ICP). This is a condition that affects the liver and can cause serious complications for both the mother and the baby. If you are experiencing severe itching, I recommend that you speak with your OB-GYN as soon as possible to rule out ICP.</p><p><strong>Human: Tell me more about ICP.</strong></p><p>AI: Intrahepatic cholestasis of pregnancy (ICP) is a condition that affects the liver and can cause serious complications for both the mother and the baby.</p><p><strong>Human: What tests do you do to detect ICP?</strong></p><p>AI: One common test for ICP is a blood test called a serum bile acids test. This test measures the levels of bile acids in the blood, which can be elevated in women with ICP. Other tests that may be used to diagnose ICP include a liver function test and a Magnetic Resonance Cholangiopancreatography (MRCP). [unsure about the MRCP]</p><p><strong>Human: How do you treat it?</strong></p><p>AI: There is no cure for ICP, but the symptoms can be managed. Treatment typically involves taking a medication called ursodeoxycholic acid (UDCA), which can help to reduce the itching and improve liver function. In some cases, delivery of the baby may be recommended if the condition is severe.</p><p><strong>Human: What levels of bile acids are considered severe?</strong></p><p>AI: If the levels of bile acids in the blood are greater than 10 micromol/L, it is considered to be a severe case of ICP.</p><p><strong>Human: What are the different bile acids?</strong></p><p>AI: The main bile acids are cholic acid and chenodeoxycholic acid.</p><p><strong>Human: How does ICP affect the baby?</strong></p><p>AI: ICP can cause serious complications for the baby, including preterm labor, low birth weight, and stillbirth. If you are pregnant and have ICP, it is important to speak with your OB-GYN to discuss the risks and possible complications. [As far as I can remember from researching this, it seems that babies tend to have high birth weight from ICP. Also, an important complication that GPT-3 misses is that they tend to experience fetal stress, and have an increased risk of swallowing meconium during delivery.]</p><p><strong>Human: At what level of bile acids do I risk premature birth? </strong></p><p>AI: If the levels of bile acids in the blood are greater than 40 micromol/L, there is an increased risk of premature birth.</p><p><strong>Human: If I have had ICP before, will I have it again next time I am pregnant?</strong></p><p>AI: ICP is a recurrent condition, which means that it can occur more than once. If you have had ICP in a previous pregnancy, you are more likely to experience it again in subsequent pregnancies.</p></blockquote><p>I predict that by 2025 I will prefer the AI over a trained doctor for most cases.</p></div></li><li><p>I should note that Elicit is better suited than GPT-3 when it comes to finding and interrogating research papers. I use it side by side with Google Scholar and have a feeling I will gradually migrate over fully.</p></li><li><div><p>Here’s me trying to get Blaise Pascal to finish up a passage in an upcoming essay. My prompt is bolded:</p><blockquote><p><strong>This essay was found in the notebooks of Blaise Pascal.</strong></p><p><strong>---</strong></p><p><strong>A city of five million people does not produce 50 times more patents than a city of 100,000 people in the same country; it produces 150 times more. The five million residents, because of the larger, denser network of which they are a part, are 3 times more likely to generate patentable ideas than their counterparts in the smaller city. In other words: larger, denser networks generate more ideas per capita. Or as a physicist would put it: the number of patents per capita (which is a proxy for idea generation) scales superlinearly with the size of cities.</strong></p><p>The reason is simple: a larger, denser network of people provides more opportunities for collisions of ideas, which are the basis for new inventions. In a small city, there are only so many people to interact with, and only so many potential ideas to combine. But in a large city, there are many more people to interact with, and many more potential ideas to combine.</p><p>One could argue that the same is true of small companies and large companies. But there is an important difference. In a small company, the people who generate the ideas are also the ones who implement them. In a large company, there are separate functions for idea generation and implementation. As a result, ideas that are generated in a large company are much less likely to be implemented than those generated in a small company.</p><p>Thus, we see that cities are more important than companies in the generation of new ideas. This is not to say that companies are unimportant; they are still the primary vehicles for the implementation of new ideas. But if we want to maximize the chances of new ideas being generated, we should focus on cities.</p></blockquote><p>That is a pretty bold thesis! <i>We should focus our attention at the level of cities, rather than companies, if we want to accelerate innovation.</i> If this had been someone riffing on my essay in a Reddit comment, I would have upvoted it. I might even have asked them for a Zoom call so we could work this out.</p><p>Now, it turns out the premise that GPT-3 builds its argument on is <i>false</i>. Employees at larger companies are <i>more</i> creative on average than employees at small companies. But what’s important when learning together with someone in dialogue is not facts, but <i>how generative the responses are</i> - and this is pretty generative. Prompted thus, my mind immediately starts outputting questions:</p><blockquote><p><strong>Why is it that employees at larger companies are more creative when intuitively we’d think they’d be less creative?</strong></p><p><strong>Are people really more creative when they get to implement their ideas?</strong></p><p><strong>How are ideas generated in cities, and how does that differ from the process in companies?</strong></p></blockquote><p>A program that can navigate you to these types of questions in a matter of minutes is, despite its unreliableness, a valuable research tool.</p></div></li><li><p>When prompted with this question (“In Inception, what’s Mal’s argument to Cobbe?”), GPT-3 fabulates: “In the movie Inception, Mal's argument to Cobb is that they are not really alive, and that they are just dreaming. She says that if they die in the dream, they will simply wake up from it.” Edit: maybe she actually makes this claim in the film?</p></li></ol></div></div>||||I|||| This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.

 LESSWRONG

 LW

 Login

 Using GPT-3 to augment human intelligence

 by Henrik Karlsson
 22 min read 10th Aug 2022 8 comments

 52

 Intelligence Amplification GPT Prompt Engineering AI
 Frontpage
 This is a linkpost for https://escapingflatland.substack.com/p/gpt-3

 A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox.

 It is like summoning an alien intelligence. I post at night, before crawling into bed next to my kids, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy, of course. But more often than not, what people have sent helps me accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting my Substack, than in the five previous years combined.

 As Slime Mold Time Mold says, Enough eyeballs, and all research bugs are shallow.

 The problem is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber!

 Therefore, it is intriguing to realize what I am doing is, in fact, prompt engineering.

 Prompt engineering is the term AI researchers use for the art of writing prompts that make a large language model output what you want. Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started.

 When I’ve been doing this with GPT-3, a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping.

 Soon everyone can have an alien intelligence in their inbox.

 GPT-3 as a tool for thought

 This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet.

 In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge.

 Some people already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote[1]). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work.

 In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in The Exorcist, language models only adds enough truth to twist our minds and make us do stupid things like claiming it is sentient and getting ourselves suspended.

 Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating—which is the technical term for making stuff up.

 This, however, does not mean you cannot use it as an epistemic tool. It just means you need to be pretty sophisticated to wrangle the truth out of it.

 As Matt Webb writes, in a slightly different context:

 Using GPT-3 is work, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.

 What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about using artificial intelligence to augment your thinking.

 It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more finetuned for factual accuracy, before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties.

 How to access GPT-3

 To access GPT-3, you set up an account at OpenAI. Then you click on Playground, which brings you to this workspace:

 In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting text-davinci-002, which is the most capable model and has been fine-tuned to understand human intentions. This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them.

 What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast.

 If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.”

 And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.”

 In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the way you access it is radically different. We can think of it as a new interface for the internet.

 Language models are internet interfaces

 Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet.

 But this comes at a cost, which currently is steep. If I would have gone to the Wikipedia page of Pope Linus, instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope.

 Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 is unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself.

 Using GPT-3 as a research assistant

 Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay.

 Maybe we can use GPT-3 to answer these questions instead?

 Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions.

 I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.

 Write an essay about how we can structure online education around apprenticeship models and communities of practice

 The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education.

 The apprenticeship model has several key features that make it well suited for online education:

 1. Apprenticeship is a hands-on, experiential learning model.
 2. Apprenticeship is based on the mentorship of a more experienced practitioner.
 3. Apprenticeship emphasizes learning by doing, rather than passively receiving information.
 4. Apprenticeship is social, collaborative, and often takes place in a community of practice.
 5. Apprenticeship is flexible and can be adapted to individual learners' needs and interests.
 6. Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills.
 7. Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.

 The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.

 This can be done in several ways. For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.

 There are several problems with this, however. First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.

 One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. Another way is to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.

 What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path.

 When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample.

 So, did GPT-3 teach me anything new here?

 I have already written an essay about online apprenticeships (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made.[2]

 Some useful prompt patterns, and some anti-patterns to avoid

 Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote[3].)

 How can we prompt GPT-3 to surface knowledge that is hard to find by googling?

 Make GPT-3 impersonate two eminent thinkers and have them debate a point. Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else.

 You can use this property to set up debates between people, as a tool to explore a topic from several angles:

 This debate was prompt engineered by Nick Cammarata, a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.

 Make GPT-3 give feedback on your ideas. You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.

 Linguistic bubbles. Being that the neural net will take different personas depending on how you talk to it, there is a risk of linguistic bubbles. This is a phrase the user Viliam used when I prompted LessWrong to discuss how large language models will affect learning.

 Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes.

 You want to talk about evolution, but you happen to use the phrase irreducible complexity instead - which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult.

 Stranger things have happened.

 I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.”

 If you want to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it is Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in that context, you ask GPT-3 to explain the irreducible complexity of nature.

 “Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!”

 You can get linguistic bubbles, but it takes some effort.

 Ask GPT-3 for counterarguments against a claim. If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google.

 I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?”

 If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.

 A few more prompt patterns that can help you catch factual errors in GPT-3 output:

 1. You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the temperature dial.
 2. You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful.
 3. You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate.
 4. It also sometimes helps to add "reason step by step" to the prompt when applicable.

 You can ask newer language models to provide links to research papers to back up their claims. With newer language models (like Gopher or this experimental version of GPT-3), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.”

 Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:

 Prompt GPT-3 to draw analogies between knowledge domains. Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will do that.

 As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in this Twitter thread:

 the big thing people don’t get about language models (nor did I for first hundred hrs) is there’s something special when it’s your doctor & lawyer & everything combined. You start to chat with it about all the things you care about and the boundaries fade. More like genius friend

 also found it amazing how few tangents I needed to go on. A lot of my mind works on analogy. If I’m talking to a doc about a medical thing and want to use an analogy to gradient descent or something I usually can’t, but with a model it knows (some of) all the fields so I can

 This made me acutely aware during human convos how often I wasn’t saying what I really meant bc it required an analogy they wouldn’t get or something they don’t know about. I realized how much of convo is going on tangents to build building blocks just to explain other things

 How far can we take it?

 GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates.

 It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall!

 For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into hallucinations, which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about.

 Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge?

 This is an open question. Gary Marcus is highly skeptical. Scott Alexander less so.

 Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with Chinchilla, where Toby Shevlane, a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:

 “In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate[4]. But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details.

 And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by PaLM, a 540 billion parameter model, and likely others that I am unaware of too.

 With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence.

 We don’t know what it will do with us out there.

 Acknowledgments

 This essay benefitted from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.

 1. ^

 During both our pregnancies, Johanna has had severe itching and we’ve had to give birth prematurely. It has been pretty hard to find an OB-GYN that is knowledgeable about the condition, so we’ve had to do most of the research and diagnose ourselves. Let’s see how GPT-3 handles it.

 My prompts are bolded, and I have added comments in brackets:

 The following is a conversation with an AI medical doctor. The doctor is helpful, creative, clever, and very friendly.

 Human: Hello, who are you?

 AI: I am an AI doctor created by OpenAI. How can I help you today?

 Human: I am pregnant and it has started itching all over my body. Even in the palms of my hands. What can it be?

 AI: There are many possible causes of itching during pregnancy. One common cause is hormonal changes, which can cause the skin to become dry and irritated. Another possibility is an allergy to something you are coming into contact with, such as a new laundry detergent or soap. If the itching is severe, it could also be a sign of a more serious condition called intrahepatic cholestasis of pregnancy (ICP). This is a condition that affects the liver and can cause serious complications for both the mother and the baby. If you are experiencing severe itching, I recommend that you speak with your OB-GYN as soon as possible to rule out ICP.

 Human: Tell me more about ICP.

 AI: Intrahepatic cholestasis of pregnancy (ICP) is a condition that affects the liver and can cause serious complications for both the mother and the baby.

 Human: What tests do you do to detect ICP?

 AI: One common test for ICP is a blood test called a serum bile acids test. This test measures the levels of bile acids in the blood, which can be elevated in women with ICP. Other tests that may be used to diagnose ICP include a liver function test and a Magnetic Resonance Cholangiopancreatography (MRCP). [unsure about the MRCP]

 Human: How do you treat it?

 AI: There is no cure for ICP, but the symptoms can be managed. Treatment typically involves taking a medication called ursodeoxycholic acid (UDCA), which can help to reduce the itching and improve liver function. In some cases, delivery of the baby may be recommended if the condition is severe.

 Human: What levels of bile acids are considered severe?

 AI: If the levels of bile acids in the blood are greater than 10 micromol/L, it is considered to be a severe case of ICP.

 Human: What are the different bile acids?

 AI: The main bile acids are cholic acid and chenodeoxycholic acid.

 Human: How does ICP affect the baby?

 AI: ICP can cause serious complications for the baby, including preterm labor, low birth weight, and stillbirth. If you are pregnant and have ICP, it is important to speak with your OB-GYN to discuss the risks and possible complications. [As far as I can remember from researching this, it seems that babies tend to have high birth weight from ICP. Also, an important complication that GPT-3 misses is that they tend to experience fetal stress, and have an increased risk of swallowing meconium during delivery.]

 Human: At what level of bile acids do I risk premature birth?

 AI: If the levels of bile acids in the blood are greater than 40 micromol/L, there is an increased risk of premature birth.

 Human: If I have had ICP before, will I have it again next time I am pregnant?

 AI: ICP is a recurrent condition, which means that it can occur more than once. If you have had ICP in a previous pregnancy, you are more likely to experience it again in subsequent pregnancies.

 I predict that by 2025 I will prefer the AI over a trained doctor for most cases.

 2. ^

 I should note that Elicit is better suited than GPT-3 when it comes to finding and interrogating research papers. I use it side by side with Google Scholar and have a feeling I will gradually migrate over fully.

 3. ^

 Here’s me trying to get Blaise Pascal to finish up a passage in an upcoming essay. My prompt is bolded:

 This essay was found in the notebooks of Blaise Pascal.

 ---

 A city of five million people does not produce 50 times more patents than a city of 100,000 people in the same country; it produces 150 times more. The five million residents, because of the larger, denser network of which they are a part, are 3 times more likely to generate patentable ideas than their counterparts in the smaller city. In other words: larger, denser networks generate more ideas per capita. Or as a physicist would put it: the number of patents per capita (which is a proxy for idea generation) scales superlinearly with the size of cities.

 The reason is simple: a larger, denser network of people provides more opportunities for collisions of ideas, which are the basis for new inventions. In a small city, there are only so many people to interact with, and only so many potential ideas to combine. But in a large city, there are many more people to interact with, and many more potential ideas to combine.

 One could argue that the same is true of small companies and large companies. But there is an important difference. In a small company, the people who generate the ideas are also the ones who implement them. In a large company, there are separate functions for idea generation and implementation. As a result, ideas that are generated in a large company are much less likely to be implemented than those generated in a small company.

 Thus, we see that cities are more important than companies in the generation of new ideas. This is not to say that companies are unimportant; they are still the primary vehicles for the implementation of new ideas. But if we want to maximize the chances of new ideas being generated, we should focus on cities.

 That is a pretty bold thesis! We should focus our attention at the level of cities, rather than companies, if we want to accelerate innovation. If this had been someone riffing on my essay in a Reddit comment, I would have upvoted it. I might even have asked them for a Zoom call so we could work this out.

 Now, it turns out the premise that GPT-3 builds its argument on is false. Employees at larger companies are more creative on average than employees at small companies. But what’s important when learning together with someone in dialogue is not facts, but how generative the responses are - and this is pretty generative. Prompted thus, my mind immediately starts outputting questions:

 Why is it that employees at larger companies are more creative when intuitively we’d think they’d be less creative?

 Are people really more creative when they get to implement their ideas?

 How are ideas generated in cities, and how does that differ from the process in companies?

 A program that can navigate you to these types of questions in a matter of minutes is, despite its unreliableness, a valuable research tool.

 4. ^

 When prompted with this question (“In Inception, what’s Mal’s argument to Cobbe?”), GPT-3 fabulates: “In the movie Inception, Mal's argument to Cobb is that they are not really alive, and that they are just dreaming. She says that if they die in the dream, they will simply wake up from it.” Edit: maybe she actually makes this claim in the film?

 New to LessWrong?

 Getting Started

 FAQ

 Library

 Intelligence Amplification GPT Prompt Engineering AI
 Frontpage

 52

 Mentioned in
 3 The OpenAI playground for GPT-3 is a terrible interface. Is there any great local (or web) app for exploring/learning with language models?
 Using GPT-3 to augment human intelligence
 10th Aug 2022
 5 Algon
 4 aviv
 6 Kaj_Sotala
 2 9eB1
 6 lc
 2 9eB1
 4 lc
 1 Henrik Karlsson
 New Comment
 Submit
 8 comments , sorted by
 top scoring
 Click to highlight new comments since: Today at 6:00 PM
 [ - ]Algon 11mo 5 0

 So now that you've had almost a year to play with GPT-3 as a research assistant, what are your impressions? How did you update?

 Reply
 [ - ]aviv 2y 4 1

 I agree that language models like GPT-3 can be helpful in many of these ways—and the prompt you link to is great.

 However, the OpenAI playground is a terrible "Tool for Thought" from a UX perspective.

 Is there any great local (or web) app for exploring/learning with language models? (Or even just for GPT-3?) I'm thinking something that you add your API keys, and it has a pleasant interface for writing prompts, seeing results and storing all the outputs for your reference. Maybe integrating with more general tools for thought.

 This is what I've found so far that attempts to let you play with GPT-3 and other language models locally, none of which seem mature.
 - https://github.com/pratos/gpt3-exp
 - https://github.com/thesephist/calamity
 - https://prompts.ai

 Is there anything better out there that I've missed?

 Reply
 [ - ]Kaj_Sotala 2y 6 0

 Some of the GPT-for-fiction-writing sites, e.g. WriteHolo, at least offer tools for saving and organizing the outputs. (Though WriteHolo uses GPT-J rather than GPT-3 and hasn't been trained to respond to queries directly the way that GPT-3's InstructGPT version has, so it may require more creative prompt engineering.)

 Reply
 [ - ]9eB1 2y 2 0

 I tried WriteHolo against an idea I had for a blog post and its recommendations were worse than useless. I think for non-fiction writing it's probably a non-starter. It couldn't even give me a concise summary of something readily available on Wikipedia (e.g. the availability heuristic), much less suggest creative connections that are at all coherent.

 Reply
 [ - ]lc 2y 6 0

 Disclaimer: I made WriteHolo w/ a friend

 The default free model is GPT-J and pretty bad at doing anything nonfiction. The GPT-20B model on the 11.99 plan is a lot bigger & better for this purpose, but its intended purpose in that vein is generally expected to be "trying to drill out copy or do homework assignments". Using OpenAI's models would help fix that but I will literally never give OpenAI money for anything.

 I think we might create a trial for the larger EleutherAI model at some point, but we stopped actively iterating on WriteHolo a while back and it's currently just something we're maintain for fun.

 Tbh it's quite a trip seeing others recommend it to people on LW.

 Reply
 [ - ]9eB1 2y 2 0

 It's very cool, especially as a side project. If I'd known it was created by someone here I would have been more careful about the tone of my comment.

 Reply
 [ - ]lc 2y 4 0

 Lol you're fine

 Reply
 [ - ]Henrik Karlsson 2y 1 0

 Having seen anything good yet. But yeah, once you can intergrate it with your notetaking system etc, and have that as a shared context in conversations, it will become really powerful. Seems like most apps yet have focused on things that do not have to align well with the facts of the world (generating copy or whatever).

 Reply
 Moderation Log
A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox. It is like summoning an alien intelligence. I post at night, before crawling into bed next to my kids, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy, of course. But more often than not, what people have sent helps me accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting my Substack, than in the five previous years combined. As Slime Mold Time Mold says, Enough eyeballs, and all research bugs are shallow. The problem is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber! Therefore, it is intriguing to realize what I am doing is, in fact, prompt engineering. Prompt engineering is the term AI researchers use for the art of writing prompts that make a large language model output what you want. Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started. When I’ve been doing this with GPT-3, a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping. Soon everyone can have an alien intelligence in their inbox. GPT-3 as a tool for thought This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet. In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge. Some people already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work. In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in The Exorcist, language models only adds enough truth to twist our minds and make us do stupid things like claiming it is sentient and getting ourselves suspended. Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating—which is the technical term for making stuff up.      This, however, does not mean you cannot use it as an epistemic tool. It just means you need to be pretty sophisticated to wrangle the truth out of it.  As Matt Webb writes, in a slightly different context:   Using GPT-3 is work, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.  What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about using artificial intelligence to augment your thinking. It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more finetuned for factual accuracy, before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties. How to access GPT-3 To access GPT-3, you set up an account at OpenAI. Then you click on Playground, which brings you to this workspace:   In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting text-davinci-002, which is the most capable model and has been fine-tuned to understand human intentions. This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them. What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast. If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.” And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.” In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the way you access it is radically different. We can think of it as a new interface for the internet. Language models are internet interfaces Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet. But this comes at a cost, which currently is steep. If I would have gone to the Wikipedia page of Pope Linus, instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope. Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 is unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself. Using GPT-3 as a research assistant Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay. Maybe we can use GPT-3 to answer these questions instead? Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions. I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.   Write an essay about how we can structure online education around apprenticeship models and communities of practice  The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education. The apprenticeship model has several key features that make it well suited for online education:  Apprenticeship is a hands-on, experiential learning model. Apprenticeship is based on the mentorship of a more experienced practitioner. Apprenticeship emphasizes learning by doing, rather than passively receiving information. Apprenticeship is social, collaborative, and often takes place in a community of practice. Apprenticeship is flexible and can be adapted to individual learners' needs and interests. Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills. Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.  The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.  This can be done in several ways. For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.  There are several problems with this, however. First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.  One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. Another way is to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.  What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path. When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample. So, did GPT-3 teach me anything new here? I have already written an essay about online apprenticeships (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made. Some useful prompt patterns, and some anti-patterns to avoid Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote.) How can we prompt GPT-3 to surface knowledge that is hard to find by googling?  Make GPT-3 impersonate two eminent thinkers and have them debate a point. Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else. You can use this property to set up debates between people, as a tool to explore a topic from several angles:    This debate was prompt engineered by   Nick Cammarata  , a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.   Make GPT-3 give feedback on your ideas. You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.  Linguistic bubbles.  Being that the neural net will take different personas depending on how you talk to it, there is a risk of linguistic bubbles. This is a phrase the user Viliam used when I prompted LessWrong to discuss how large language models will affect learning. Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes. You want to talk about evolution, but you happen to use the phrase irreducible complexity instead - which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult. Stranger things have happened. I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.” If you want to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it is Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in that context, you ask GPT-3 to explain the irreducible complexity of nature. “Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!” You can get linguistic bubbles, but it takes some effort.  Ask GPT-3 for counterarguments against a claim. If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google. I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?” If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.  A few more prompt patterns that can help you catch factual errors in GPT-3 output:    You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the temperature dial. You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful. You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate. It also sometimes helps to add "reason step by step" to the prompt when applicable.   You can ask newer language models to provide links to research papers to back up their claims. With newer language models (like Gopher or this experimental version of GPT-3), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.” Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:    Prompt GPT-3 to draw analogies between knowledge domains. Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will do that. As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in this Twitter thread:  the big thing people don’t get about language models (nor did I for first hundred hrs) is there’s something special when it’s your doctor &amp; lawyer &amp; everything combined. You start to chat with it about all the things you care about and the boundaries fade. More like genius friend   also found it amazing how few tangents I needed to go on. A lot of my mind works on analogy. If I’m talking to a doc about a medical thing and want to use an analogy to gradient descent or something I usually can’t, but with a model it knows (some of) all the fields so I can   This made me acutely aware during human convos how often I wasn’t saying what I really meant bc it required an analogy they wouldn’t get or something they don’t know about. I realized how much of convo is going on tangents to build building blocks just to explain other things  How far can we take it? GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates. It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall! For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into hallucinations, which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about. Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge? This is an open question. Gary Marcus is highly skeptical. Scott Alexander less so. Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with Chinchilla, where Toby Shevlane, a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:   “In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate. But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details. And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by PaLM, a 540 billion parameter model, and likely others that I am unaware of too. With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence. We don’t know what it will do with us out there. Acknowledgments  This essay benefitted from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.     During both our pregnancies, Johanna has had severe itching and we’ve had to give birth prematurely. It has been pretty hard to find an OB-GYN that is knowledgeable about the condition, so we’ve had to do most of the research and diagnose ourselves. Let’s see how GPT-3 handles it. My prompts are bolded, and I have added comments in brackets:   The following is a conversation with an AI medical doctor. The doctor is helpful, creative, clever, and very friendly.   Human: Hello, who are you?    AI: I am an AI doctor created by OpenAI. How can I help you today?    Human: I am pregnant and it has started itching all over my body. Even in the palms of my hands. What can it be?  AI: There are many possible causes of itching during pregnancy. One common cause is hormonal changes, which can cause the skin to become dry and irritated. Another possibility is an allergy to something you are coming into contact with, such as a new laundry detergent or soap. If the itching is severe, it could also be a sign of a more serious condition called intrahepatic cholestasis of pregnancy (ICP). This is a condition that affects the liver and can cause serious complications for both the mother and the baby. If you are experiencing severe itching, I recommend that you speak with your OB-GYN as soon as possible to rule out ICP.  Human: Tell me more about ICP.  AI: Intrahepatic cholestasis of pregnancy (ICP) is a condition that affects the liver and can cause serious complications for both the mother and the baby.  Human: What tests do you do to detect ICP?  AI: One common test for ICP is a blood test called a serum bile acids test. This test measures the levels of bile acids in the blood, which can be elevated in women with ICP. Other tests that may be used to diagnose ICP include a liver function test and a Magnetic Resonance Cholangiopancreatography (MRCP). [unsure about the MRCP]  Human: How do you treat it?  AI: There is no cure for ICP, but the symptoms can be managed. Treatment typically involves taking a medication called ursodeoxycholic acid (UDCA), which can help to reduce the itching and improve liver function. In some cases, delivery of the baby may be recommended if the condition is severe.  Human: What levels of bile acids are considered severe?  AI: If the levels of bile acids in the blood are greater than 10 micromol/L, it is considered to be a severe case of ICP.  Human: What are the different bile acids?  AI: The main bile acids are cholic acid and chenodeoxycholic acid.  Human: How does ICP affect the baby?  AI: ICP can cause serious complications for the baby, including preterm labor, low birth weight, and stillbirth. If you are pregnant and have ICP, it is important to speak with your OB-GYN to discuss the risks and possible complications. [As far as I can remember from researching this, it seems that babies tend to have high birth weight from ICP. Also, an important complication that GPT-3 misses is that they tend to experience fetal stress, and have an increased risk of swallowing meconium during delivery.]  Human: At what level of bile acids do I risk premature birth?   AI: If the levels of bile acids in the blood are greater than 40 micromol/L, there is an increased risk of premature birth.  Human: If I have had ICP before, will I have it again next time I am pregnant?  AI: ICP is a recurrent condition, which means that it can occur more than once. If you have had ICP in a previous pregnancy, you are more likely to experience it again in subsequent pregnancies.  I predict that by 2025 I will prefer the AI over a trained doctor for most cases.    I should note that Elicit is better suited than GPT-3 when it comes to finding and interrogating research papers. I use it side by side with Google Scholar and have a feeling I will gradually migrate over fully.    Here’s me trying to get Blaise Pascal to finish up a passage in an upcoming essay. My prompt is bolded:   This essay was found in the notebooks of Blaise Pascal.   ---   A city of five million people does not produce 50 times more patents than a city of 100,000 people in the same country; it produces 150 times more. The five million residents, because of the larger, denser network of which they are a part, are 3 times more likely to generate patentable ideas than their counterparts in the smaller city. In other words: larger, denser networks generate more ideas per capita. Or as a physicist would put it: the number of patents per capita (which is a proxy for idea generation) scales superlinearly with the size of cities.  The reason is simple: a larger, denser network of people provides more opportunities for collisions of ideas, which are the basis for new inventions. In a small city, there are only so many people to interact with, and only so many potential ideas to combine. But in a large city, there are many more people to interact with, and many more potential ideas to combine. One could argue that the same is true of small companies and large companies. But there is an important difference. In a small company, the people who generate the ideas are also the ones who implement them. In a large company, there are separate functions for idea generation and implementation. As a result, ideas that are generated in a large company are much less likely to be implemented than those generated in a small company. Thus, we see that cities are more important than companies in the generation of new ideas. This is not to say that companies are unimportant; they are still the primary vehicles for the implementation of new ideas. But if we want to maximize the chances of new ideas being generated, we should focus on cities.  That is a pretty bold thesis! We should focus our attention at the level of cities, rather than companies, if we want to accelerate innovation. If this had been someone riffing on my essay in a Reddit comment, I would have upvoted it. I might even have asked them for a Zoom call so we could work this out. Now, it turns out the premise that GPT-3 builds its argument on is false. Employees at larger companies are more creative on average than employees at small companies. But what’s important when learning together with someone in dialogue is not facts, but how generative the responses are - and this is pretty generative. Prompted thus, my mind immediately starts outputting questions:   Why is it that employees at larger companies are more creative when intuitively we’d think they’d be less creative?   Are people really more creative when they get to implement their ideas?   How are ideas generated in cities, and how does that differ from the process in companies?   A program that can navigate you to these types of questions in a matter of minutes is, despite its unreliableness, a valuable research tool.    When prompted with this question (“In Inception, what’s Mal’s argument to Cobbe?”), GPT-3 fabulates: “In the movie Inception, Mal's argument to Cobb is that they are not really alive, and that they are just dreaming. She says that if they die in the dream, they will simply wake up from it.” Edit: maybe she actually makes this claim in the film?
Using GPT-3 to augment human intelligence
https://www.lesswrong.com/posts/spBoxzcaCrqXqyQHq/using-gpt-3-to-augment-human-intelligence
https://distill.pub/2017/aia/
Using Artificial Intelligence to Augment Human Intelligence
2017-12-04
Shan Carter; Michael Nielsen; Carter; Shan; Nielsen; Michael
<div><div>
 <h2>
 What are computers for?
 </h2>
 <p>
 Historically, different answers to this question – that is,
 different visions of computing – have helped inspire and
 determine the computing systems humanity has ultimately
 built. Consider the early electronic computers. ENIAC, the
 world’s first general-purpose electronic computer, was
 commissioned to compute artillery firing tables for the United
 States Army. Other early computers were also used to solve
 numerical problems, such as simulating nuclear explosions,
 predicting the weather, and planning the motion of rockets. The
 machines operated in a batch mode, using crude input and output
 devices, and without any real-time interaction. It was a vision
 of computers as number-crunching machines, used to speed up
 calculations that would formerly have taken weeks, months, or more
 for a team of humans.
 </p>
 <p>
 In the 1950s a different vision of what computers are for began to
 develop. That vision was crystallized in 1962, when Douglas
 Engelbart proposed that computers could be used as a way
 of augmenting human
 intellect. In this view, computers weren’t primarily
 tools for solving number-crunching problems. Rather, they were
 real-time interactive systems, with rich inputs and outputs, that
 humans could work with to support and expand their own
 problem-solving process. This vision of intelligence augmentation
 (IA) deeply influenced many others, including researchers such as
 Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple,
 and led to many of the key ideas of modern computing systems. Its
 ideas have also deeply influenced digital art and music, and
 fields such as interaction design, data visualization,
 computational creativity, and human-computer interaction.
 </p>
 <p>
 Research on IA has often been in competition with research on
 artificial intelligence (AI): competition for funding, competition
 for the interest of talented researchers. Although there has
 always been overlap between the fields, IA has typically focused
 on building systems which put humans and machines to work
 together, while AI has focused on complete outsourcing of
 intellectual tasks to machines. In particular, problems in AI are
 often framed in terms of matching or surpassing human performance:
 beating humans at chess or Go; learning to recognize speech and
 images or translating language as well as humans; and so on.
 </p>
 <p>
 This essay describes a new field, emerging today out of a
 synthesis of AI and IA. For this field, we suggest the
 name <em>artificial intelligence augmentation</em> (AIA): the use
 of AI systems to help develop new methods for intelligence
 augmentation. This new field introduces important new fundamental
 questions, questions not associated to either parent field. We
 believe the principles and systems of AIA will be radically
 different to most existing systems.
 </p>
 <p>
 Our essay begins with a survey of recent technical work hinting at
 artificial intelligence augmentation, including work
 on <em>generative interfaces</em> – that is, interfaces
 which can be used to explore and visualize generative machine
 learning models. Such interfaces develop a kind of cartography of
 generative models, ways for humans to explore and make meaning
 from those models, and to incorporate what those models
 “know” into their creative work.
 </p>
 <p>
 Our essay is not just a survey of technical work. We believe now
 is a good time to identify some of the broad, fundamental
 questions at the foundation of this emerging field. To what
 extent are these new tools enabling creativity? Can they be used
 to generate ideas which are truly surprising and new, or are the
 ideas cliches, based on trivial recombinations of existing ideas?
 Can such systems be used to develop fundamental new interface
 primitives? How will those new primitives change and expand the
 way humans think?
 </p>
 <h2>
 Using generative models to invent meaningful creative operations
 </h2>
 <p>
 Let’s look at an example where a machine learning model makes a
 new type of interface possible. To understand the interface,
 imagine you’re a type designer, working on creating a new
 fontWe shall egregiously abuse the distinction between
 a font and a typeface. Apologies to any type designers who may be
 reading.. After sketching some initial designs, you
 wish to experiment with bold, italic, and condensed variations.
 Let’s examine a tool to generate and explore such variations, from
 any initial design. For reasons that will soon be explained the
 quality of results is quite crude; please bear with us.
 </p>
 <figure>
 
 </figure>
 <p>
 Of course, varying the bolding (i.e., the weight), italicization
 and width are just three ways you can vary a font. Imagine that
 instead of building specialized tools, users could build their own
 tool merely by choosing examples of existing fonts. For instance,
 suppose you wanted to vary the degree of serifing on a font. In
 the following, please select 5 to 10 sans-serif fonts from the top
 box, and drag them to the box on the left. Select 5 to 10 serif
 fonts and drag them to the box on the right. As you do this, a
 machine learning model running in your browser will automatically
 infer from these examples how to interpolate your starting font in
 either the serif or sans-serif direction:
 </p>
 <figure>
 </figure>
 <p>
 In fact, we used this same technique to build the earlier bolding
 italicization, and condensing tool. To do so, we used the
 following examples of bold and non-bold fonts, of italic and
 non-italic fonts, and of condensed and non-condensed fonts:
 </p>
 <figure>
 </figure>
 <p>
 To build these tools, we used what’s called a <em>generative
 model</em>; the particular model we use was trained
 by James Wexler. To
 understand generative models, consider that <em>a priori</em>
 describing a font appears to require a lot of data. For
 instance, if the font is 6464 by 6464 pixels, then we’d expect
 to need 64×64=4,09664 \times 64 = 4,096 parameters to describe a single
 glyph. But we can use a generative model to find a much simpler
 description.
 </p>
 <p>
 We do this by building a neural network which takes a small number
 of input variables, called <em>latent variables</em>, and produces
 as output the entire glyph. For the particular model we use, we
 have 4040 latent space dimensions, and map that into the
 4,0964,096-dimensional space describing all the pixels in the glyph.
 In other words, the idea is to map a low-dimensional space into a
 higher-dimensional space:
 </p>
 <figure>
 </figure>
 <p>
 The generative model we use is a type of neural network known as
 a variational autoencoder
 (VAE). For our purposes, the details of the generative
 model aren’t so important. The important thing is that by
 changing the latent variables used as input, it’s possible to get
 different fonts as output. So one choice of latent variables will
 give one font, while another choice will give a different font:
 </p>
 <figure>
 </figure>
 <p>
 You can think of the latent variables as a compact, high-level
 representation of the font. The neural network takes that
 high-level representation and converts it into the full pixel
 data. It’s remarkable that just 4040 numbers can capture the
 apparent complexity in a glyph, which originally required 4,0964,096
 variables.
 </p>
 <p>
 The generative model we use is learnt from a training set of more
 than 5050 thousand
 fonts Bernhardsson
 scraped from the open web. During training, the weights and
 biases in the network are adjusted so that the network can output
 a close approximation to any desired font from the training set,
 provided a suitable choice of latent variables is made. In some
 sense, the model is learning a highly compressed representation of
 all the training fonts.
 </p>
 <p>
 In fact, the model doesn’t just reproduce the training fonts. It
 can also generalize, producing fonts not seen in training. By
 being forced to find a compact description of the training
 examples, the neural net learns an abstract, higher-level model of
 what a font is. That higher-level model makes it possible to
 generalize beyond the training examples already seen, to produce
 realistic-looking fonts.
 </p>
 <p>
 Ideally, a good generative model would be exposed to a relatively
 small number of training examples, and use that exposure to
 generalize to the space of all possible human-readable fonts.
 That is, for any conceivable font – whether existing or
 perhaps even imagined in the future – it would be possible
 to find latent variables corresponding exactly to that font. Of
 course, the model we’re using falls far short of this ideal
 – a particularly egregious failure is that many fonts
 generated by the model omit the tail on the capital
 “Q” (you can see this in the examples above). Still,
 it’s useful to keep in mind what an ideal generative model would
 do.
 </p>
 <p>
 Such generative models are similar in some ways to how scientific
 theories work. Scientific theories often greatly simplify the
 description of what appear to be complex phenomena, reducing large
 numbers of variables to just a few variables from which many
 aspects of system behavior can be deduced. Furthermore, good
 scientific theories sometimes enable us to generalize to discover
 new phenomena.
 </p>
 <p>
 As an example, consider ordinary material objects. Such objects
 have what physicists call a <em>phase</em> – they may be a
 liquid, a solid, a gas, or perhaps something more exotic, like a
 superconductor
 or <a href="https://en.wikipedia.org/wiki/Bose%E2%80%93Einstein_condensate">Bose-Einstein
 condensate</a>. <em>A priori</em>, such systems seem immensely
 complex, involving perhaps 102310^{23} or so molecules. But the
 laws of thermodynamics and statistical mechanics enable us to find
 a simpler description, reducing that complexity to just a few
 variables (temperature, pressure, and so on), which encompass much
 of the behavior of the system. Furthermore, sometimes it’s
 possible to generalize, predicting unexpected new phases of
 matter. For example, in 1924, physicists used thermodynamics and
 statistical mechanics to predict a remarkable new phase of matter,
 Bose-Einstein condensation, in which a collection of atoms may all
 occupy identical quantum states, leading to surprising large-scale
 quantum interference effects. We’ll come back to this predictive
 ability in our later discussion of creativity and generative
 models.
 </p>
 <p>
 Returning to the nuts and bolts of generative models, how can we
 use such models to do example-based reasoning like that in the
 tool shown above? Let’s consider the case of the bolding tool. In
 that instance, we take the average of all the latent vectors for
 the user-specified bold fonts, and the average for all the
 user-specified non-bold fonts. We then compute the difference
 between these two average vectors:
 </p>
 <figure>
 </figure>
 <p>
 We’ll refer to this as the <em>bolding vector</em>. To make some
 given font bolder, we simply add a little of the bolding vector to
 the corresponding latent vector, with the amount of bolding vector
 added controlling the boldness of the resultIn
 practice, sometimes a slightly different procedure is used. In
 some generative models the latent vectors satisfy some constraints
 – for instance, they may all be of the same length. When
 that’s the case, as in our model, a more sophisticated
 “adding” operation must be used, to ensure the length
 remains the same. But conceptually, the picture of adding the
 bolding vector is the right way to think.:
 </p>
 <figure>
 </figure>
 <p>
 This technique was introduced
 by Larsen <em>et al</em>, and
 vectors like the bolding vector are sometimes called
 <em>attribute vectors</em>. The same idea is use to implement all
 the tools we’ve shown. That is, we use example fonts to creating
 a bolding vector, an italicizing vector, a condensing vector, and
 a user-defined serif vector. The interface thus provides a way of
 exploring the latent space in those four directions.
 </p>
 <p>
 The tools we’ve shown have many drawbacks. Consider the following
 example, where we start with an example glyph, in the middle, and
 either increase or decrease the bolding (on the right and left,
 respectively):
 </p>
 <figure>
 </figure>
 <p>
 Examining the glyphs on the left and right we see many unfortunate
 artifacts. Particularly for the rightmost glyph, the edges start to get
 rough, and the serifs begin to disappear. A better generative
 model would reduce those artifacts. That’s a good long-term
 research program, posing many intriguing problems. But even with
 the model we have, there are also some striking benefits to the
 use of the generative model.
 </p>
 <p>
 To understand these benefits, consider a naive approach to
 bolding, in which we simply add some extra pixels around a glyph’s
 edges, thickening it up. While this thickening perhaps matches a
 non-expert’s way of thinking about type design, an expert does
 something much more involved. In the following we show the
 results of this naive thickening procedure versus what is actually
 done, for Georgia and Helvetica:
 </p>
 <figure>
 </figure>
 <p>
 As you can see, the naive bolding procedure produces quite
 different results, in both cases. For example, in Georgia, the
 left stroke is only changed slightly by bolding, while the right
 stroke is greatly enlarged, but only on one side. In both
 fonts, bolding doesn’t change the height of the font, while the
 naive approach does.
 </p>
 <p>
 As these examples show, good bolding is <em>not</em> a trivial
 process of thickening up a font. Expert type designers have many
 heuristics for bolding, heuristics inferred from much previous
 experimentation, and careful study of historical
 examples. Capturing all those heuristics in a conventional program
 would involve immense work. The benefit of using the generative
 model is that it automatically learns many such heuristics.
 </p>
 <p>
 For example, a naive bolding tool would rapidly fill in the
 enclosed negative space in the enclosed upper region of the letter
 “A”. The font tool doesn’t do this. Instead, it goes
 to some trouble to preserve the enclosed negative space, moving
 the A’s bar down, and filling out the interior strokes more slowly
 than the exterior. This principle is evident in the examples
 shown above, especially Helvetica, and it can also be seen in the
 operation of the font tool:
 </p>
 <figure>
 </figure>
 <p>
 The heuristic of preserving enclosed negative space is not <em>a
 priori</em> obvious. However, it’s done in many professionally
 designed fonts. If you examine examples like those shown above
 it’s easy to see why: it improves legibility. During training,
 our generative model has automatically inferred this principle
 from the examples it’s seen. And our bolding interface then makes
 this available to the user.
 </p>
 <p>
 In fact, the model captures many other heuristics. For instance,
 in the above examples the heights of the fonts are (roughly)
 preserved, which is the norm in professional font design. Again,
 what’s going on isn’t just a thickening of the font, but rather
 the application of a more subtle heuristic inferred by the
 generative model. Such heuristics can be used to create fonts
 with properties which would otherwise be unlikely to occur to
 users. Thus, the tool expands ordinary people’s ability to
 explore the space of meaningful fonts.
 </p>
 <p>
 The font tool is an example of a kind of cognitive technology. In
 particular, the primitive operations it contains can be
 internalized as part of how a user thinks. In this it resembles a
 program such as <em>Photoshop</em> or a spreadsheet or 3D graphics
 programs. Each provides a novel set of interface primitives,
 primitives which can be internalized by the user as fundamental
 new elements in their thinking. This act of internalization of new
 primitives is fundamental to much work on intelligence
 augmentation.
 </p>
 <p>
 The ideas shown in the font tool can be extended to other domains.
 Using the same interface, we can use a generative model to
 manipulate images of human faces using qualities such as
 expression, gender, or hair color. Or to manipulate sentences
 using length, sarcasm, or tone. Or to manipulate molecules using
 chemical properties:
 </p>
 
 <figure>
 <div>
 <div><p>
 Images from <em>Sampling Generative Networks</em> by </p>White<p>.
 </p></div>
 <div><p>
 Sentence from <em>Pride and Prejudice</em> by Jane Austen. Interpolated by the authors. Inspired by experiments done by the novelist </p>Robin Sloan
 </div>
 <div><p>
 Images from <em>Automatic chemical design using a data-driven continuous representation of molecules</em> by </p>Gómez-Bombarelli <em>et al</em><p>.
 </p></div>
 </div>
 </figure>
 <p>
 Such generative interfaces provide a kind of cartography of
 generative models, ways for humans to explore and make meaning
 using those models.
 </p>
 <p>
 We saw earlier that the font model automatically infers relatively
 deep principles about font design, and makes them available to
 users. While it’s great that such deep principles can be
 inferred, sometimes such models infer other things that are wrong,
 or undesirable. For example, White
 points out the addition of a smile vector in some face
 models will make faces not just smile more, but also appear more
 feminine. Why? Because in the training data more women than men
 were smiling. So these models may not just learn deep facts about
 the world, they may also internalize prejudices or erroneous
 beliefs. Once such a bias is known, it is often possible to make
 corrections. But to find those biases requires careful auditing
 of the models, and it is not yet clear how we can ensure such
 audits are exhaustive.
 </p>
 
 <p>
 More broadly, we can ask why attribute vectors work, when they
 work, and when they fail? At the moment, the answers to these
 questions are poorly understood.
 </p>
 <p>
 For the attribute vector to work requires that taking any starting
 font, we can construct the corresponding bold version by adding
 the <em>same</em> vector in the latent space. However, <em>a
 priori</em> there is no reason using a single constant vector to
 displace will work. It may be that we should displace in many
 different ways. For instance, the heuristics used to bold serif
 and sans-serif fonts are quite different, and so it seems likely
 that very different displacements would be involved:
 </p>
 <figure>
 </figure>
 <p>
 Of course, we could do something more sophisticated than using a
 single constant attribute vector. Given pairs of example fonts
 (unbold, bold) we could train a machine learning algorithm to take
 as input the latent vector for the unbolded version and output the
 latent vector for the bolded version. With additional training
 data about font weights, the machine learning algorithm could
 learn to generate fonts of arbitrary weight. Attribute vectors
 are just an extremely simple approach to doing these kinds of
 operation.
 </p>
 <p>
 For these reasons, it seems unlikely that attribute vectors will
 last as an approach to manipulating high-level features. Over the
 next few years much better approaches will be developed. However,
 we can still expect interfaces offering operations broadly similar
 to those sketched above, allowing access to high-level and
 potentially user-defined concepts. That interface pattern doesn’t
 depend on the technical details of attribute vectors.
 </p>
 <h2>
 Interactive Generative Adversarial Models
 </h2>
 <p>
 Let’s look at another example using machine learning models to
 augment human creativity. It’s the interactive generative
 adversarial networks, or iGANs, introduced
 by Zhu <em>et al</em> in 2016.
 </p>
 <p>
 One of the examples of Zhu <em>et al</em> is the use of iGANs in
 an interface to generate images of consumer products such as
 shoes. Conventionally, such an interface would require the
 programmer to write a program containing a great deal of knowledge
 about shoes: soles, laces, heels, and so on. Instead of doing
 this, Zhu <em>et al</em> train a generative model using 5050
 thousand images of shoes, downloaded from Zappos. They then use
 that generative model to build an interface that lets a user
 roughly sketch the shape of a shoe, the sole, the laces, and so
 on:
 </p>
 <figure>
 
 <figcaption>Excerpted from Zhu <em>et
 al</em>.</figcaption>
 </figure>
 <p>
 The visual quality is low, in part because the generative model
 Zhu <em>et al</em> used is outdated by modern (2017) standards
 – with more modern models, the visual quality would be much
 higher.
 </p>
 <p>
 But the visual quality is not the point. Many interesting things
 are going on in this prototype. For instance, notice how the
 overall shape of the shoe changes considerably when the sole is
 filled in – it becomes narrower and sleeker. Many small
 details are filled in, like the black piping on the top of the
 white sole, and the red coloring filled in everywhere on the
 shoe’s upper. These and other facts are automatically deduced
 from the underlying generative model, in a way we’ll describe
 shortly.
 </p>
 <p>
 The same interface may be used to sketch landscapes. The only
 difference is that the underlying generative model has been
 trained on landscape images rather than images of shoes. In this
 case it becomes possible to sketch in just the colors associated
 to a landscape. For example, here’s a user sketching in some green
 grass, the outline of a mountain, some blue sky, and snow on the
 mountain:
 </p>
 <figure>
 
 <figcaption>Excerpted from Zhu <em>et
 al</em>.</figcaption>
 </figure>
 <p>
 The generative models used in these interfaces are different than
 for our font model. Rather than using variational autoencoders,
 they’re based on generative
 adversarial networks (GANs). But the underlying idea is
 still to find a low-dimensional latent space which can be used to
 represent (say) all landscape images, and map that latent space to
 a corresponding image. Again, we can think of points in the
 latent space as a compact way of describing landscape images.
 </p>
 <p>
 Roughly speaking, the way the iGANs works is as follows. Whatever
 the current image is, it corresponds to some point in the latent
 space:
 </p>
 <figure>
 </figure>
 <p>
 Suppose, as happened in the earlier video, the user now sketches
 in a stroke outlining the mountain shape. We can think of the
 stroke as a constraint on the image, picking out a subspace of the
 latent space, consisting of all points in the latent space whose
 image matches that outline:
 </p>
 <figure>
 </figure>
 <p>
 The way the interface works is to find a point in the latent space
 which is near to the current image, so the image is not changed
 too much, but also coming close to satisfying the imposed
 constraints. This is done by optimizing an objective function
 which combines the distance to each of the imposed constraints, as
 well as the distance moved from the current point. If there’s
 just a single constraint, say, corresponding to the mountain
 stroke, this looks something like the following:
 </p>
 <figure>
 </figure>
 <p>
 We can think of this, then, as a way of applying constraints to
 the latent space to move the image around in meaningful ways.
 </p>
 <p>
 The iGANs have much in common with the font tool we showed
 earlier. Both make available operations that encode much subtle
 knowledge about the world, whether it be learning to understand
 what a mountain looks like, or inferring that enclosed negative
 space should be preserved when bolding a font. Both the iGANs and
 the font tool provide ways of understanding and navigating a
 high-dimensional space, keeping us on the natural space of fonts
 or shoes or landscapes. As Zhu <em>et al</em> remark:
 </p>
 <blockquote>
 [F]or most of us, even a simple image manipulation in Photoshop
 presents insurmountable difficulties… any less-than-perfect
 edit immediately makes the image look completely unrealistic. To
 put another way, classic visual manipulation paradigm does not
 prevent the user from “falling off” the manifold of
 natural images.
 </blockquote>
 <p>
 Like the font tool, the iGANs is a cognitive technology. Users
 can internalize the interface operations as new primitive elements
 in their thinking. In the case of shoes, for example, they can
 learn to think in terms of the difference they want to apply,
 adding a heel, or a higher top, or a special highlight. This is
 richer than the traditional way non-experts think about shoes
 (“Size 11, black” <em>etc</em>). To the extent that
 non-experts do think in more sophisticated ways –
 “make the top a little higher and sleeker” –
 they get little practice in thinking this way, or seeing the
 consequences of their choices. Having an interface like this
 enables easier exploration, the ability to develop idioms and the
 ability to plan, to swap ideas with friends, and so on.
 </p>
 <h2>
 Two models of computation
 </h2>
 <p>
 Let’s revisit the question we began the essay with, the question
 of what computers are for, and how this relates to intelligence
 augmentation.
 </p>
 <p>
 One common conception of computers is that they’re problem-solving
 machines: “computer, what is the result of firing this
 artillery shell in such-and-such a wind [and so on]?”;
 “computer, what will the maximum temperature in Tokyo be in
 5 days?”; “computer, what is the best move to take
 when the Go board is in this position?”; “computer,
 how should this image be classified?”; and so on.
 </p>
 <p>
 This is a conception common to both the early view of computers as
 number-crunchers, and also in much work on AI, both historically
 and today. It’s a model of a computer as a way of outsourcing
 cognition. In speculative depictions of possible future AI,
 this <em>cognitive outsourcing</em> model often shows up in the
 view of an AI as an oracle, able to solve some large class of
 problems with better-than-human performance.
 </p>
 <p>
 But a very different conception of what computers are for is
 possible, a conception much more congruent with work on
 intelligence augmentation.
 </p>
 <p>
 To understand this alternate view, consider our subjective
 experience of thought. For many people, that experience is verbal:
 they think using language, forming chains of words in their heads,
 similar to sentences in speech or written on a page. For other
 people, thinking is a more visual experience, incorporating
 representations such as graphs and maps. Still other people mix
 mathematics into their thinking, using algebraic expressions or
 diagrammatic techniques, such as Feynman diagrams and Penrose
 diagrams.
 </p>
 <p>
 In each case, we’re thinking using representations invented by
 other people: words, graphs, maps, algebra, mathematical diagrams,
 and so on. We internalize these cognitive technologies as we grow
 up, and come to use them as a kind of substrate for our thinking.
 </p>
 <p>
 For most of history, the range of available cognitive technologies
 has changed slowly and incrementally. A new word will be
 introduced, or a new mathematical symbol. More rarely, a radical
 new cognitive technology will be developed. For example, in 1637
 Descartes published his “Discourse on Method”,
 explaining how to represent geometric ideas using algebra, and
 vice versa:
 </p>
 <figure></figure>
 <p>
 This enabled a radical change and expansion in how we think about
 both geometry and algebra.
 </p>
 <p>
 Historically, lasting cognitive technologies have been invented
 only rarely. But modern computers are a meta-medium enabling the
 rapid invention of many new cognitive technologies. Consider a
 relatively banal example, such
 as <em>Photoshop</em>. Adept <em>Photoshop</em> users routinely
 have formerly impossible thoughts such as: “let’s apply the
 clone stamp to the such-and-such layer.”. That’s an
 instance of a more general class of thought: “computer, [new
 type of action] this [new type of representation for a newly
 imagined class of object]”. When that happens, we’re using
 computers to expand the range of thoughts we can think.
 </p>
 <p>
 It’s this kind of <em>cognitive transformation</em> model which
 underlies much of the deepest work on intelligence augmentation.
 Rather than outsourcing cognition, it’s about changing the
 operations and representations we use to think; it’s about
 changing the substrate of thought itself. And so while cognitive
 outsourcing is important, this cognitive transformation view
 offers a much more profound model of intelligence augmentation.
 It’s a view in which computers are a means to change and expand
 human thought itself.
 </p>
 <p>
 Historically, cognitive technologies were developed by human
 inventors, ranging from the invention of writing in Sumeria and
 Mesoamerica, to the modern interfaces of designers such as Douglas
 Engelbart, Alan Kay, and others.
 </p>
 <p>
 Examples such as those described in this essay suggest that AI
 systems can enable the creation of new cognitive technologies.
 Things like the font tool aren’t just oracles to be consulted when
 you want a new font. Rather, they can be used to explore and
 discover, to provide new representations and operations, which can
 be internalized as part of the user’s own thinking. And while
 these examples are in their early stages, they suggest AI is not
 just about cognitive outsourcing. A different view of AI is
 possible, one where it helps us invent new cognitive technologies
 which transform the way we think.
 </p>
 <p>
 In this essay we’ve focused on a small number of examples, mostly
 involving exploration of the latent space. There are many other
 examples of artificial intelligence augmentation. To give some
 flavor, without being comprehensive:
 the sketch-rnn system, for neural
 network assisted drawing;
 the Wekinator, which enables
 users to rapidly build new musical instruments and artistic
 systems; TopoSketch, for developing
 animations by exploring latent spaces; machine learning models for
 designing overall typographic
 layout; and a generative model which enables
 interpolation between musical
 phrases. In each case, the systems use machine learning
 to enable new primitives which can be integrated into the user’s
 thinking. More broadly, artificial intelligence augmentation will
 draw on fields such as computational
 creativity and interactive machine
 learning.
 </p>
 <h2>
 Finding powerful new primitives of thought
 </h2>
 <p>
 We’ve argued that machine learning systems can help create
 representations and operations which serve as new primitives in
 human thought. What properties should we look for in such new
 primitives? This is too large a question to be answered
 comprehensively in a short essay. But we will explore it briefly.
 </p>
 <p>
 Historically, important new media forms often seem strange when
 introduced. Many such stories have passed into popular culture:
 the near riot at the premiere of Stravinsky and Nijinksy’s
 “Rite of Spring”; the consternation caused by the
 early cubist paintings, leading
 <em>The New York Times</em> to
 comment: “What do they mean? Have those
 responsible for them taken leave of their senses? Is it art or
 madness? Who knows?”
 </p>
 <p>
 Another example comes from physics. In the 1940s, different
 formulations of the theory of quantum electrodynamics were
 developed independently by the physicists Julian Schwinger,
 Shin’ichirō Tomonaga, and Richard Feynman. In their work,
 Schwinger and Tomonaga used a conventional algebraic approach,
 along lines similar to the rest of physics. Feynman used a more
 radical approach, based on what are now known as Feynman diagrams,
 for depicting the interaction of light and matter:
 </p>
 <figure>
 
 <figcaption>Image by <a href="https://commons.wikimedia.org/w/index.php?curid=1764161">Joel
	Holdsworth</a>), licensed under a Creative Commons
	Attribution-Share Alike 3.0 Unported license
 </figcaption>
 </figure>
 <p>
 Initially, the Schwinger-Tomonaga approach was easier for other
 physicists to understand. When Feynman and Schwinger presented
 their work at a 1948 workshop, Schwinger was immediately
 acclaimed. By contrast, Feynman left his audience mystified. As
 James Gleick put it in his biography of
 Feynman:
 </p>
 <blockquote>
 It struck Feynman that everyone had a favorite principle or
 theorem and he was violating them all… Feynman knew he had
 failed. At the time, he was in anguish. Later he said simply:
 “I had too much stuff. My machines came from too far
 away.”
 </blockquote>
 <p>
 Of course, strangeness for strangeness’s sake alone is not
 useful. But these examples suggest that breakthroughs in
 representation often appear strange at first. Is there any
 underlying reason that is true?
 </p>
 <p>
 Part of the reason is because if some representation is truly new,
 then it will appear different than anything you’ve ever seen
 before. Feynman’s diagrams, Picasso’s paintings, Stravinsky’s
 music: all revealed genuinely new ways of making meaning. Good
 representations sharpen up such insights, eliding the familiar to
 show that which is new as vividly as possible. But because of
 that emphasis on unfamiliarity, the representation will seem
 strange: it shows relationships you’ve never seen before. In some
 sense, the task of the designer is to identify that core
 strangeness, and to amplify it as much as possible.
 </p>
 <p>
 Strange representations are often difficult to understand. At
 first, physicists preferred Schwinger-Tomonaga to Feynman. But as
 Feynman’s approach was slowly understood by physicists, they
 realized that although Schwinger-Tomonaga and Feynman were
 mathematically equivalent, Feynman was more powerful. As Gleick
 puts it:
 </p>
 <blockquote>
 Schwinger’s students at Harvard were put at a competitive
 disadvantage, or so it seemed to their fellows elsewhere, who
 suspected them of surreptitiously using the diagrams anyway. This
 was sometimes true… Murray Gell-Mann later spent a semester
 staying in Schwinger’s house and loved to say afterward that he
 had searched everywhere for the Feynman diagrams. He had not
 found any, but one room had been locked…
 </blockquote>
 <p>
 These ideas are true not just of historical representations, but
 also of computer interfaces. However, our advocacy of strangeness
 in representation contradicts much conventional wisdom about
 interfaces, especially the widely-held belief that they should be
 “user friendly”, i.e., simple and immediately useable
 by novices. That most often means the interface is cliched, built
 from conventional elements combined in standard ways. But while
 using a cliched interface may be easy and fun, it’s an ease
 similar to reading a formulaic romance novel. It means the
 interface does not reveal anything truly surprising about its
 subject area. And so it will do little to deepen the user’s
 understanding, or to change the way they think. For mundane tasks
 that is fine, but for deeper tasks, and for the longer term, you
 want a better interface.
 </p>
 <p>
 Ideally, an interface will surface the deepest principles
 underlying a subject, revealing a new world to the user. When you
 learn such an interface, you internalize those principles, giving
 you more powerful ways of reasoning about that world. Those
 principles are the diffs in your understanding. They’re all you
 really want to see, everything else is at best support, at worst
 unimportant dross. The purpose of the best interfaces isn’t to be
 user-friendly in some shallow sense. It’s to be user-friendly in
 a much stronger sense, reifying deep
 principles about the world, making them the working
 conditions in which users live and create. At that point what once
 appeared strange can instead becomes comfortable and familiar,
 part of the pattern of thoughtA powerful instance of
 these ideas is when an interface reifies general-purpose
 principles. An example is an
 interface one of us developed
 based on the principle of conservation of energy. Such
 general-purpose principles generate multiple unexpected
 relationships between the entities of a subject, and so are a
 particularly rich source of insights when reified in an
 interface..
 </p>
 <p>
 What does this mean for the use of AI models for intelligence
 augmentation?
 </p>
 <p>
 Aspirationally, as we’ve seen, our machine learning models will
 help us build interfaces which reify deep principles in ways
 meaningful to the user. For that to happen, the models have to
 discover deep principles about the world, recognize those
 principles, and then surface them as vividly as possible in an
 interface, in a way comprehensible by the user.
 </p>
 <p>
 Of course, this is a tall order! The examples we’ve shown are just
 barely beginning to do this. It’s true that our models do
 sometimes discover relatively deep principles, like the
 preservation of enclosed negative space when bolding a font. But
 this is merely implicit in the model. And while we’ve built a tool
 which takes advantage of such principles, it’d be better if the
 model automatically inferred the important principles learned, and
 found ways of explicitly surfacing them through the interface.
 (Encouraging progress toward this has been made
 by InfoGANs, which use
 information-theoretic ideas to find structure in the latent
 space.) Ideally, such models would start to get at true
 explanations, not just in a static form, but in a dynamic form,
 manipulable by the user. But we’re a long way from that point.
 </p>
 <h2>
 Do these interfaces inhibit creativity?
 </h2>
 <p>
 It’s tempting to be skeptical of the expressiveness of the
 interfaces we’ve described. If an interface constrains us to
 explore only the natural space of images, does that mean we’re
 merely doing the expected? Does it mean these interfaces can only
 be used to generate visual cliches? Does it prevent us from
 generating anything truly new, from doing truly creative work?
 </p>
 <p>
 To answer these questions, it’s helpful to identify two different
 modes of creativity. This two-mode model is over-simplified:
 creativity doesn’t fit so neatly into two distinct categories. Yet
 the model nonetheless clarifies the role of new interfaces in
 creative work.
 </p>
 <p>
 The first mode of creativity is the everyday creativity of a
 craftsperson engaged in their craft. Much of the work of a font
 designer, for example, consists of competent recombination of the
 best existing practices. Such work typically involves many
 creative choices to meet the intended design goals, but not
 developing key new underlying principles.
 </p>
 <p>
 For such work, the generative interfaces we’ve been discussing are
 promising. While they currently have many limitations, future
 research will identity and fix many deficiencies. This is
 happening rapidly with GANs: the original
 GANs had many limitations,
 but models soon appeared that were better adapted to
 images, improved the
 resolution, reduced artifactsSo much work has been
 done on improving resolution and reducing artifacts it seems
 unfair to single out any small set of papers, and to omit the many
 others., and so on. With enough iterations it’s
 plausible these generative interfaces will become powerful tools
 for craft work.
 </p>
 <p>
 The second mode of creativity aims toward developing new
 principles that fundamentally change the range of creative
 expression. One sees this in the work of artists such as Picasso
 or Monet, who violated existing principles of painting, developing
 new principles which enabled people to see in new ways.
 </p>
 <p>
 Is it possible to do such creative work, while using a generative
 interface? Don’t such interfaces constrain us to the space of
 natural images, or natural fonts, and thus actively prevent us
 from exploring the most interesting new directions in creative
 work?
 </p>
 <p>
 The situation is more complex than this.
 </p>
 <p>
 In part, this is a question about the power of our generative
 models. In some cases, the model can only generate recombinations
 of existing ideas. This is a limitation of an ideal GAN, since a
 perfectly trained GAN generator will reproduce the training
 distribution. Such a model can’t directly generate an image based
 on new fundamental principles, because such an image wouldn’t look
 anything like it’s seen in its training data.
 </p>
 <p>
 Artists such as <a href="http://quasimondo.com/">Mario
 Klingemann</a> and <a href="http://www.miketyka.com/">Mike
 Tyka</a> are now using GANs to create interesting
 artwork. They’re doing that using “imperfect” GAN
 models, which they seem to be able to use to explore interesting
 new principles; it’s perhaps the case that bad GANs may be more
 artistically interesting than ideal GANs. Furthermore, nothing
 says an interface must only help us explore the latent space.
 Perhaps operations can be added which deliberately take us out
 of the latent space, or to less probable (and so more
 surprising) parts of the space of natural images.
 </p>
 <p>
 Of course, GANs are not the only generative models. In a
 sufficiently powerful generative model, the generalizations
 discovered by the model may contain ideas going beyond what humans
 have discovered. In that case, exploration of the latent space may
 enable us to discover new fundamental principles. The model would
 have discovered stronger abstractions than human experts. Imagine
 a generative model trained on paintings up until just before the
 time of the cubists; might it be that by exploring that model it
 would be possible to discover cubism? It would be an analogue to
 something like the prediction of Bose-Einstein condensation, as
 discussed earlier in the essay. Such invention is beyond today’s
 generative models, but seems a worthwhile aspiration for future
 models.
 </p>
 <p>
 Our examples so far have all been based on generative models. But
 there are some illuminating examples which are not based on
 generative models. Consider the pix2pix system developed
 by Isola <em>et al</em>. This
 system is trained on pairs of images, e.g., pairs showing the
 edges of a cat, and the actual corresponding cat. Once trained,
 it can be shown a set of edges and asked to generate an image for
 an actual corresponding cat. It often does this quite well:
 </p>
 
 <figure>
 <div>
 <h4>Input</h4>
 <h4>Output</h4>
 
 </div>
 
 </figure>
 <p>
 When supplied with unusual constraints, pix2pix can produce
 striking images:
 </p>
 <figure>
 
 
 <div>
 <p>
 </p><figcaption>Spiral cat</figcaption>
 </div>
 </figure>
 <p>
 This is perhaps not high creativity of a Picasso-esque level. But
 it is still surprising. It’s certainly unlike images most of us
 have ever seen before. How does pix2pix and its human user achieve
 this kind of result?
 </p>
 <p>
 Unlike our earlier examples, pix2pix is not a generative model.
 This means it does not have a latent space or a corresponding
 space of natural images. Instead, there is a neural network,
 called, confusingly, a generator – this is not meant in the
 same sense as our earlier generative models – that takes as
 input the constraint image, and produces as output the filled-in
 image.
 </p>
 <p>
 The generator is trained adversarially against a discriminator
 network, whose job is to distinguish between pairs of images
 generated from real data, and pairs of images generated by the
 generator.
 </p>
 <p>
 While this sounds similar to a conventional GAN, there is a
 crucial difference: there is no latent vector input to the
 generatorActually, Isola <em>et
 al</em> experimented with adding such a latent vector to
 the generator, but found it made little difference to the
 resulting images.. Rather, there is simply an input
 constraint. When a human inputs a constraint unlike anything seen
 in training, the network is forced to improvise, doing the best it
 can to interpret that constraint according to the rules it has
 previously learned. The creativity is the result of a forced
 merger of knowledge inferred from the training data, together with
 novel constraints provided by the user. As a result, even
 relatively simple ideas – like the bread- and beholder-cats
 – can result in striking new types of images, images not
 within what we would previously have considered the space of
 natural images.
 </p>
 <h2>Conclusion</h2>
 <p>
 It is conventional wisdom that AI will change how we interact with
 computers. Unfortunately, many in the AI community greatly
 underestimate the depth of interface design, often regarding it as
 a simple problem, mostly about making things pretty or
 easy-to-use. In this view, interface design is a problem to be
 handed off to others, while the hard work is to train some machine
 learning system.
 </p>
 <p>
 This view is incorrect. At its deepest, interface design means
 developing the fundamental primitives human beings think and
 create with. This is a problem whose intellectual genesis goes
 back to the inventors of the alphabet, of cartography, and of
 musical notation, as well as modern giants such as Descartes,
 Playfair, Feynman, Engelbart, and Kay. It is one of the hardest,
 most important and most fundamental problems humanity grapples
 with.
 </p>
 <p>
 As discussed earlier, in one common view of AI our computers will
 continue to get better at solving problems, but human beings will
 remain largely unchanged. In a second common view, human beings
 will be modified at the hardware level, perhaps directly through
 neural interfaces, or indirectly through whole brain emulation.
 </p>
 <p>
 We’ve described a third view, in which AIs actually change
 humanity, helping us invent new cognitive technologies, which
 expand the range of human thought. Perhaps one day those
 cognitive technologies will, in turn, speed up the development of
 AI, in a virtuous feedback cycle:
 </p>
 <figure>
 
 </figure>
 <p>
 It would not be a Singularity in machines. Rather, it would be a
 Singularity in humanity’s range of thought. Of course, this loop
 is at present extremely speculative. The systems we’ve described
 can help develop more powerful ways of thinking, but there’s at
 most an indirect sense in which those ways of thinking are being
 used in turn to develop new AI systems.
 </p>
 <p>
 Of course, over the long run it’s possible that machines will
 exceed humans on all or most cognitive tasks. Even if that’s the
 case, cognitive transformation will still be a valuable end, worth
 pursuing in its own right. There is pleasure and value involved
 in learning to play chess or Go well, even if machines do it
 better. And in activities such as story-telling the benefit often
 isn’t so much the artifact produced as the process of construction
 itself, and the relationships forged. There is intrinsic value in
 personal change and growth, apart from instrumental benefits.
 </p>
 <p>
 The interface-oriented work we’ve discussed is outside the
 narrative used to judge most existing work in artificial
 intelligence. It doesn’t involve beating some benchmark for a
 classification or regression problem. It doesn’t involve
 impressive feats like beating human champions at games such as
 Go. Rather, it involves a much more subjective and
 difficult-to-measure criterion: is it helping humans think and
 create in new ways?
 </p>
 <p>
 This creates difficulties for doing this kind of work,
 particularly in a research setting. Where should one publish?
 What community does one belong to? What standards should be
 applied to judge such work? What distinguishes good work from
 bad?
 </p>
 <p>
 We believe that over the next few years a community will emerge
 which answers these questions. It will run workshops and
 conferences. It will publish work in venues such as Distill. Its
 standards will draw from many different communities: from the
 artistic and design and musical communities; from the mathematical
 community’s taste in abstraction and good definition; as well as
 from the existing AI and IA communities, including work on
 computational creativity and human-computer interaction. The
 long-term test of success will be the development of tools which
 are widely used by creators. Are artists using these tools to
 develop remarkable new styles? Are scientists in other fields
 using them to develop understanding in ways not otherwise
 possible? These are great aspirations, and require an approach
 that builds on conventional AI work, but also incorporates very
 different norms.
 </p>
</div>||||I||||Using Artificial Intelligence to Augment Human Intelligence

By creating user interfaces which let us work with the representations inside machine learning models, we can give people new tools for reasoning.

 Authors

 Affiliations

 Shan Carter

 Google Brain Team

 Michael Nielsen

 YC Research

 Published

 Dec. 4, 2017

 DOI

 10.23915/distill.00009

What are computers for?

Historically, different answers to this question – that is, different visions of computing – have helped inspire and determine the computing systems humanity has ultimately built. Consider the early electronic computers. ENIAC, the world’s first general-purpose electronic computer, was commissioned to compute artillery firing tables for the United States Army. Other early computers were also used to solve numerical problems, such as simulating nuclear explosions, predicting the weather, and planning the motion of rockets. The machines operated in a batch mode, using crude input and output devices, and without any real-time interaction. It was a vision of computers as number-crunching machines, used to speed up calculations that would formerly have taken weeks, months, or more for a team of humans.

In the 1950s a different vision of what computers are for began to develop. That vision was crystallized in 1962, when Douglas Engelbart proposed that computers could be used as a way of augmenting human intellect. In this view, computers weren’t primarily tools for solving number-crunching problems. Rather, they were real-time interactive systems, with rich inputs and outputs, that humans could work with to support and expand their own problem-solving process. This vision of intelligence augmentation (IA) deeply influenced many others, including researchers such as Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple, and led to many of the key ideas of modern computing systems. Its ideas have also deeply influenced digital art and music, and fields such as interaction design, data visualization, computational creativity, and human-computer interaction.

Research on IA has often been in competition with research on artificial intelligence (AI): competition for funding, competition for the interest of talented researchers. Although there has always been overlap between the fields, IA has typically focused on building systems which put humans and machines to work together, while AI has focused on complete outsourcing of intellectual tasks to machines. In particular, problems in AI are often framed in terms of matching or surpassing human performance: beating humans at chess or Go; learning to recognize speech and images or translating language as well as humans; and so on.

This essay describes a new field, emerging today out of a synthesis of AI and IA. For this field, we suggest the name artificial intelligence augmentation (AIA): the use of AI systems to help develop new methods for intelligence augmentation. This new field introduces important new fundamental questions, questions not associated to either parent field. We believe the principles and systems of AIA will be radically different to most existing systems.

Our essay begins with a survey of recent technical work hinting at artificial intelligence augmentation, including work on generative interfaces – that is, interfaces which can be used to explore and visualize generative machine learning models. Such interfaces develop a kind of cartography of generative models, ways for humans to explore and make meaning from those models, and to incorporate what those models “know” into their creative work.

Our essay is not just a survey of technical work. We believe now is a good time to identify some of the broad, fundamental questions at the foundation of this emerging field. To what extent are these new tools enabling creativity? Can they be used to generate ideas which are truly surprising and new, or are the ideas cliches, based on trivial recombinations of existing ideas? Can such systems be used to develop fundamental new interface primitives? How will those new primitives change and expand the way humans think?

Using generative models to invent meaningful creative operations

Let’s look at an example where a machine learning model makes a new type of interface possible. To understand the interface, imagine you’re a type designer, working on creating a new fontWe shall egregiously abuse the distinction between a font and a typeface. Apologies to any type designers who may be reading.. After sketching some initial designs, you wish to experiment with bold, italic, and condensed variations. Let’s examine a tool to generate and explore such variations, from any initial design. For reasons that will soon be explained the quality of results is quite crude; please bear with us.

Of course, varying the bolding (i.e., the weight), italicization and width are just three ways you can vary a font. Imagine that instead of building specialized tools, users could build their own tool merely by choosing examples of existing fonts. For instance, suppose you wanted to vary the degree of serifing on a font. In the following, please select 5 to 10 sans-serif fonts from the top box, and drag them to the box on the left. Select 5 to 10 serif fonts and drag them to the box on the right. As you do this, a machine learning model running in your browser will automatically infer from these examples how to interpolate your starting font in either the serif or sans-serif direction:

In fact, we used this same technique to build the earlier bolding italicization, and condensing tool. To do so, we used the following examples of bold and non-bold fonts, of italic and non-italic fonts, and of condensed and non-condensed fonts:

To build these tools, we used what’s called a generative model; the particular model we use was trained by James Wexler. To understand generative models, consider that a priori describing a font appears to require a lot of data. For instance, if the font is 6464 6 4 by 6464 6 4 pixels, then we’d expect to need 64×64=4,09664 \times 64 = 4,096 6 4 × 6 4 = 4 , 0 9 6 parameters to describe a single glyph. But we can use a generative model to find a much simpler description.

We do this by building a neural network which takes a small number of input variables, called latent variables, and produces as output the entire glyph. For the particular model we use, we have 4040 4 0 latent space dimensions, and map that into the 4,0964,096 4 , 0 9 6 -dimensional space describing all the pixels in the glyph. In other words, the idea is to map a low-dimensional space into a higher-dimensional space:

The generative model we use is a type of neural network known as a variational autoencoder (VAE). For our purposes, the details of the generative model aren’t so important. The important thing is that by changing the latent variables used as input, it’s possible to get different fonts as output. So one choice of latent variables will give one font, while another choice will give a different font:

You can think of the latent variables as a compact, high-level representation of the font. The neural network takes that high-level representation and converts it into the full pixel data. It’s remarkable that just 4040 4 0 numbers can capture the apparent complexity in a glyph, which originally required 4,0964,096 4 , 0 9 6 variables.

The generative model we use is learnt from a training set of more than 5050 5 0 thousand fonts Bernhardsson scraped from the open web. During training, the weights and biases in the network are adjusted so that the network can output a close approximation to any desired font from the training set, provided a suitable choice of latent variables is made. In some sense, the model is learning a highly compressed representation of all the training fonts.

In fact, the model doesn’t just reproduce the training fonts. It can also generalize, producing fonts not seen in training. By being forced to find a compact description of the training examples, the neural net learns an abstract, higher-level model of what a font is. That higher-level model makes it possible to generalize beyond the training examples already seen, to produce realistic-looking fonts.

Ideally, a good generative model would be exposed to a relatively small number of training examples, and use that exposure to generalize to the space of all possible human-readable fonts. That is, for any conceivable font – whether existing or perhaps even imagined in the future – it would be possible to find latent variables corresponding exactly to that font. Of course, the model we’re using falls far short of this ideal – a particularly egregious failure is that many fonts generated by the model omit the tail on the capital “Q” (you can see this in the examples above). Still, it’s useful to keep in mind what an ideal generative model would do.

Such generative models are similar in some ways to how scientific theories work. Scientific theories often greatly simplify the description of what appear to be complex phenomena, reducing large numbers of variables to just a few variables from which many aspects of system behavior can be deduced. Furthermore, good scientific theories sometimes enable us to generalize to discover new phenomena.

As an example, consider ordinary material objects. Such objects have what physicists call a phase – they may be a liquid, a solid, a gas, or perhaps something more exotic, like a superconductor or Bose-Einstein condensate. A priori, such systems seem immensely complex, involving perhaps 102310^{23} 1 0 2 3 or so molecules. But the laws of thermodynamics and statistical mechanics enable us to find a simpler description, reducing that complexity to just a few variables (temperature, pressure, and so on), which encompass much of the behavior of the system. Furthermore, sometimes it’s possible to generalize, predicting unexpected new phases of matter. For example, in 1924, physicists used thermodynamics and statistical mechanics to predict a remarkable new phase of matter, Bose-Einstein condensation, in which a collection of atoms may all occupy identical quantum states, leading to surprising large-scale quantum interference effects. We’ll come back to this predictive ability in our later discussion of creativity and generative models.

Returning to the nuts and bolts of generative models, how can we use such models to do example-based reasoning like that in the tool shown above? Let’s consider the case of the bolding tool. In that instance, we take the average of all the latent vectors for the user-specified bold fonts, and the average for all the user-specified non-bold fonts. We then compute the difference between these two average vectors:

We’ll refer to this as the bolding vector. To make some given font bolder, we simply add a little of the bolding vector to the corresponding latent vector, with the amount of bolding vector added controlling the boldness of the resultIn practice, sometimes a slightly different procedure is used. In some generative models the latent vectors satisfy some constraints – for instance, they may all be of the same length. When that’s the case, as in our model, a more sophisticated “adding” operation must be used, to ensure the length remains the same. But conceptually, the picture of adding the bolding vector is the right way to think.:

This technique was introduced by Larsen et al, and vectors like the bolding vector are sometimes called attribute vectors. The same idea is use to implement all the tools we’ve shown. That is, we use example fonts to creating a bolding vector, an italicizing vector, a condensing vector, and a user-defined serif vector. The interface thus provides a way of exploring the latent space in those four directions.

The tools we’ve shown have many drawbacks. Consider the following example, where we start with an example glyph, in the middle, and either increase or decrease the bolding (on the right and left, respectively):

Examining the glyphs on the left and right we see many unfortunate artifacts. Particularly for the rightmost glyph, the edges start to get rough, and the serifs begin to disappear. A better generative model would reduce those artifacts. That’s a good long-term research program, posing many intriguing problems. But even with the model we have, there are also some striking benefits to the use of the generative model.

To understand these benefits, consider a naive approach to bolding, in which we simply add some extra pixels around a glyph’s edges, thickening it up. While this thickening perhaps matches a non-expert’s way of thinking about type design, an expert does something much more involved. In the following we show the results of this naive thickening procedure versus what is actually done, for Georgia and Helvetica:

As you can see, the naive bolding procedure produces quite different results, in both cases. For example, in Georgia, the left stroke is only changed slightly by bolding, while the right stroke is greatly enlarged, but only on one side. In both fonts, bolding doesn’t change the height of the font, while the naive approach does.

As these examples show, good bolding is not a trivial process of thickening up a font. Expert type designers have many heuristics for bolding, heuristics inferred from much previous experimentation, and careful study of historical examples. Capturing all those heuristics in a conventional program would involve immense work. The benefit of using the generative model is that it automatically learns many such heuristics.

For example, a naive bolding tool would rapidly fill in the enclosed negative space in the enclosed upper region of the letter “A”. The font tool doesn’t do this. Instead, it goes to some trouble to preserve the enclosed negative space, moving the A’s bar down, and filling out the interior strokes more slowly than the exterior. This principle is evident in the examples shown above, especially Helvetica, and it can also be seen in the operation of the font tool:

The heuristic of preserving enclosed negative space is not a priori obvious. However, it’s done in many professionally designed fonts. If you examine examples like those shown above it’s easy to see why: it improves legibility. During training, our generative model has automatically inferred this principle from the examples it’s seen. And our bolding interface then makes this available to the user.

In fact, the model captures many other heuristics. For instance, in the above examples the heights of the fonts are (roughly) preserved, which is the norm in professional font design. Again, what’s going on isn’t just a thickening of the font, but rather the application of a more subtle heuristic inferred by the generative model. Such heuristics can be used to create fonts with properties which would otherwise be unlikely to occur to users. Thus, the tool expands ordinary people’s ability to explore the space of meaningful fonts.

The font tool is an example of a kind of cognitive technology. In particular, the primitive operations it contains can be internalized as part of how a user thinks. In this it resembles a program such as Photoshop or a spreadsheet or 3D graphics programs. Each provides a novel set of interface primitives, primitives which can be internalized by the user as fundamental new elements in their thinking. This act of internalization of new primitives is fundamental to much work on intelligence augmentation.

The ideas shown in the font tool can be extended to other domains. Using the same interface, we can use a generative model to manipulate images of human faces using qualities such as expression, gender, or hair color. Or to manipulate sentences using length, sarcasm, or tone. Or to manipulate molecules using chemical properties:

 Images from Sampling Generative Networks by White.
 Sentence from Pride and Prejudice by Jane Austen. Interpolated by the authors. Inspired by experiments done by the novelist Robin Sloan
 Images from Automatic chemical design using a data-driven continuous representation of molecules by Gómez-Bombarelli et al.

Such generative interfaces provide a kind of cartography of generative models, ways for humans to explore and make meaning using those models.

We saw earlier that the font model automatically infers relatively deep principles about font design, and makes them available to users. While it’s great that such deep principles can be inferred, sometimes such models infer other things that are wrong, or undesirable. For example, White points out the addition of a smile vector in some face models will make faces not just smile more, but also appear more feminine. Why? Because in the training data more women than men were smiling. So these models may not just learn deep facts about the world, they may also internalize prejudices or erroneous beliefs. Once such a bias is known, it is often possible to make corrections. But to find those biases requires careful auditing of the models, and it is not yet clear how we can ensure such audits are exhaustive.

More broadly, we can ask why attribute vectors work, when they work, and when they fail? At the moment, the answers to these questions are poorly understood.

For the attribute vector to work requires that taking any starting font, we can construct the corresponding bold version by adding the same vector in the latent space. However, a priori there is no reason using a single constant vector to displace will work. It may be that we should displace in many different ways. For instance, the heuristics used to bold serif and sans-serif fonts are quite different, and so it seems likely that very different displacements would be involved:

Of course, we could do something more sophisticated than using a single constant attribute vector. Given pairs of example fonts (unbold, bold) we could train a machine learning algorithm to take as input the latent vector for the unbolded version and output the latent vector for the bolded version. With additional training data about font weights, the machine learning algorithm could learn to generate fonts of arbitrary weight. Attribute vectors are just an extremely simple approach to doing these kinds of operation.

For these reasons, it seems unlikely that attribute vectors will last as an approach to manipulating high-level features. Over the next few years much better approaches will be developed. However, we can still expect interfaces offering operations broadly similar to those sketched above, allowing access to high-level and potentially user-defined concepts. That interface pattern doesn’t depend on the technical details of attribute vectors.

Interactive Generative Adversarial Models

Let’s look at another example using machine learning models to augment human creativity. It’s the interactive generative adversarial networks, or iGANs, introduced by Zhu et al in 2016.

One of the examples of Zhu et al is the use of iGANs in an interface to generate images of consumer products such as shoes. Conventionally, such an interface would require the programmer to write a program containing a great deal of knowledge about shoes: soles, laces, heels, and so on. Instead of doing this, Zhu et al train a generative model using 5050 5 0 thousand images of shoes, downloaded from Zappos. They then use that generative model to build an interface that lets a user roughly sketch the shape of a shoe, the sole, the laces, and so on:

Excerpted from Zhu et al.

The visual quality is low, in part because the generative model Zhu et al used is outdated by modern (2017) standards – with more modern models, the visual quality would be much higher.

But the visual quality is not the point. Many interesting things are going on in this prototype. For instance, notice how the overall shape of the shoe changes considerably when the sole is filled in – it becomes narrower and sleeker. Many small details are filled in, like the black piping on the top of the white sole, and the red coloring filled in everywhere on the shoe’s upper. These and other facts are automatically deduced from the underlying generative model, in a way we’ll describe shortly.

The same interface may be used to sketch landscapes. The only difference is that the underlying generative model has been trained on landscape images rather than images of shoes. In this case it becomes possible to sketch in just the colors associated to a landscape. For example, here’s a user sketching in some green grass, the outline of a mountain, some blue sky, and snow on the mountain:

Excerpted from Zhu et al.

The generative models used in these interfaces are different than for our font model. Rather than using variational autoencoders, they’re based on generative adversarial networks (GANs). But the underlying idea is still to find a low-dimensional latent space which can be used to represent (say) all landscape images, and map that latent space to a corresponding image. Again, we can think of points in the latent space as a compact way of describing landscape images.

Roughly speaking, the way the iGANs works is as follows. Whatever the current image is, it corresponds to some point in the latent space:

Suppose, as happened in the earlier video, the user now sketches in a stroke outlining the mountain shape. We can think of the stroke as a constraint on the image, picking out a subspace of the latent space, consisting of all points in the latent space whose image matches that outline:

The way the interface works is to find a point in the latent space which is near to the current image, so the image is not changed too much, but also coming close to satisfying the imposed constraints. This is done by optimizing an objective function which combines the distance to each of the imposed constraints, as well as the distance moved from the current point. If there’s just a single constraint, say, corresponding to the mountain stroke, this looks something like the following:

We can think of this, then, as a way of applying constraints to the latent space to move the image around in meaningful ways.

The iGANs have much in common with the font tool we showed earlier. Both make available operations that encode much subtle knowledge about the world, whether it be learning to understand what a mountain looks like, or inferring that enclosed negative space should be preserved when bolding a font. Both the iGANs and the font tool provide ways of understanding and navigating a high-dimensional space, keeping us on the natural space of fonts or shoes or landscapes. As Zhu et al remark:

[F]or most of us, even a simple image manipulation in Photoshop presents insurmountable difficulties… any less-than-perfect edit immediately makes the image look completely unrealistic. To put another way, classic visual manipulation paradigm does not prevent the user from “falling off” the manifold of natural images.

Like the font tool, the iGANs is a cognitive technology. Users can internalize the interface operations as new primitive elements in their thinking. In the case of shoes, for example, they can learn to think in terms of the difference they want to apply, adding a heel, or a higher top, or a special highlight. This is richer than the traditional way non-experts think about shoes (“Size 11, black” etc). To the extent that non-experts do think in more sophisticated ways – “make the top a little higher and sleeker” – they get little practice in thinking this way, or seeing the consequences of their choices. Having an interface like this enables easier exploration, the ability to develop idioms and the ability to plan, to swap ideas with friends, and so on.

Two models of computation

Let’s revisit the question we began the essay with, the question of what computers are for, and how this relates to intelligence augmentation.

One common conception of computers is that they’re problem-solving machines: “computer, what is the result of firing this artillery shell in such-and-such a wind [and so on]?”; “computer, what will the maximum temperature in Tokyo be in 5 days?”; “computer, what is the best move to take when the Go board is in this position?”; “computer, how should this image be classified?”; and so on.

This is a conception common to both the early view of computers as number-crunchers, and also in much work on AI, both historically and today. It’s a model of a computer as a way of outsourcing cognition. In speculative depictions of possible future AI, this cognitive outsourcing model often shows up in the view of an AI as an oracle, able to solve some large class of problems with better-than-human performance.

But a very different conception of what computers are for is possible, a conception much more congruent with work on intelligence augmentation.

To understand this alternate view, consider our subjective experience of thought. For many people, that experience is verbal: they think using language, forming chains of words in their heads, similar to sentences in speech or written on a page. For other people, thinking is a more visual experience, incorporating representations such as graphs and maps. Still other people mix mathematics into their thinking, using algebraic expressions or diagrammatic techniques, such as Feynman diagrams and Penrose diagrams.

In each case, we’re thinking using representations invented by other people: words, graphs, maps, algebra, mathematical diagrams, and so on. We internalize these cognitive technologies as we grow up, and come to use them as a kind of substrate for our thinking.

For most of history, the range of available cognitive technologies has changed slowly and incrementally. A new word will be introduced, or a new mathematical symbol. More rarely, a radical new cognitive technology will be developed. For example, in 1637 Descartes published his “Discourse on Method”, explaining how to represent geometric ideas using algebra, and vice versa:

This enabled a radical change and expansion in how we think about both geometry and algebra.

Historically, lasting cognitive technologies have been invented only rarely. But modern computers are a meta-medium enabling the rapid invention of many new cognitive technologies. Consider a relatively banal example, such as Photoshop. Adept Photoshop users routinely have formerly impossible thoughts such as: “let’s apply the clone stamp to the such-and-such layer.”. That’s an instance of a more general class of thought: “computer, [new type of action] this [new type of representation for a newly imagined class of object]”. When that happens, we’re using computers to expand the range of thoughts we can think.

It’s this kind of cognitive transformation model which underlies much of the deepest work on intelligence augmentation. Rather than outsourcing cognition, it’s about changing the operations and representations we use to think; it’s about changing the substrate of thought itself. And so while cognitive outsourcing is important, this cognitive transformation view offers a much more profound model of intelligence augmentation. It’s a view in which computers are a means to change and expand human thought itself.

Historically, cognitive technologies were developed by human inventors, ranging from the invention of writing in Sumeria and Mesoamerica, to the modern interfaces of designers such as Douglas Engelbart, Alan Kay, and others.

Examples such as those described in this essay suggest that AI systems can enable the creation of new cognitive technologies. Things like the font tool aren’t just oracles to be consulted when you want a new font. Rather, they can be used to explore and discover, to provide new representations and operations, which can be internalized as part of the user’s own thinking. And while these examples are in their early stages, they suggest AI is not just about cognitive outsourcing. A different view of AI is possible, one where it helps us invent new cognitive technologies which transform the way we think.

In this essay we’ve focused on a small number of examples, mostly involving exploration of the latent space. There are many other examples of artificial intelligence augmentation. To give some flavor, without being comprehensive: the sketch-rnn system, for neural network assisted drawing; the Wekinator, which enables users to rapidly build new musical instruments and artistic systems; TopoSketch, for developing animations by exploring latent spaces; machine learning models for designing overall typographic layout; and a generative model which enables interpolation between musical phrases. In each case, the systems use machine learning to enable new primitives which can be integrated into the user’s thinking. More broadly, artificial intelligence augmentation will draw on fields such as computational creativity and interactive machine learning.

Finding powerful new primitives of thought

We’ve argued that machine learning systems can help create representations and operations which serve as new primitives in human thought. What properties should we look for in such new primitives? This is too large a question to be answered comprehensively in a short essay. But we will explore it briefly.

Historically, important new media forms often seem strange when introduced. Many such stories have passed into popular culture: the near riot at the premiere of Stravinsky and Nijinksy’s “Rite of Spring”; the consternation caused by the early cubist paintings, leading The New York Times to comment: “What do they mean? Have those responsible for them taken leave of their senses? Is it art or madness? Who knows?”

Another example comes from physics. In the 1940s, different formulations of the theory of quantum electrodynamics were developed independently by the physicists Julian Schwinger, Shin’ichirō Tomonaga, and Richard Feynman. In their work, Schwinger and Tomonaga used a conventional algebraic approach, along lines similar to the rest of physics. Feynman used a more radical approach, based on what are now known as Feynman diagrams, for depicting the interaction of light and matter:

Image by Joel Holdsworth), licensed under a Creative Commons Attribution-Share Alike 3.0 Unported license

Initially, the Schwinger-Tomonaga approach was easier for other physicists to understand. When Feynman and Schwinger presented their work at a 1948 workshop, Schwinger was immediately acclaimed. By contrast, Feynman left his audience mystified. As James Gleick put it in his biography of Feynman:

It struck Feynman that everyone had a favorite principle or theorem and he was violating them all… Feynman knew he had failed. At the time, he was in anguish. Later he said simply: “I had too much stuff. My machines came from too far away.”

Of course, strangeness for strangeness’s sake alone is not useful. But these examples suggest that breakthroughs in representation often appear strange at first. Is there any underlying reason that is true?

Part of the reason is because if some representation is truly new, then it will appear different than anything you’ve ever seen before. Feynman’s diagrams, Picasso’s paintings, Stravinsky’s music: all revealed genuinely new ways of making meaning. Good representations sharpen up such insights, eliding the familiar to show that which is new as vividly as possible. But because of that emphasis on unfamiliarity, the representation will seem strange: it shows relationships you’ve never seen before. In some sense, the task of the designer is to identify that core strangeness, and to amplify it as much as possible.

Strange representations are often difficult to understand. At first, physicists preferred Schwinger-Tomonaga to Feynman. But as Feynman’s approach was slowly understood by physicists, they realized that although Schwinger-Tomonaga and Feynman were mathematically equivalent, Feynman was more powerful. As Gleick puts it:

Schwinger’s students at Harvard were put at a competitive disadvantage, or so it seemed to their fellows elsewhere, who suspected them of surreptitiously using the diagrams anyway. This was sometimes true… Murray Gell-Mann later spent a semester staying in Schwinger’s house and loved to say afterward that he had searched everywhere for the Feynman diagrams. He had not found any, but one room had been locked…

These ideas are true not just of historical representations, but also of computer interfaces. However, our advocacy of strangeness in representation contradicts much conventional wisdom about interfaces, especially the widely-held belief that they should be “user friendly”, i.e., simple and immediately useable by novices. That most often means the interface is cliched, built from conventional elements combined in standard ways. But while using a cliched interface may be easy and fun, it’s an ease similar to reading a formulaic romance novel. It means the interface does not reveal anything truly surprising about its subject area. And so it will do little to deepen the user’s understanding, or to change the way they think. For mundane tasks that is fine, but for deeper tasks, and for the longer term, you want a better interface.

Ideally, an interface will surface the deepest principles underlying a subject, revealing a new world to the user. When you learn such an interface, you internalize those principles, giving you more powerful ways of reasoning about that world. Those principles are the diffs in your understanding. They’re all you really want to see, everything else is at best support, at worst unimportant dross. The purpose of the best interfaces isn’t to be user-friendly in some shallow sense. It’s to be user-friendly in a much stronger sense, reifying deep principles about the world, making them the working conditions in which users live and create. At that point what once appeared strange can instead becomes comfortable and familiar, part of the pattern of thoughtA powerful instance of these ideas is when an interface reifies general-purpose principles. An example is an interface one of us developed based on the principle of conservation of energy. Such general-purpose principles generate multiple unexpected relationships between the entities of a subject, and so are a particularly rich source of insights when reified in an interface..

What does this mean for the use of AI models for intelligence augmentation?

Aspirationally, as we’ve seen, our machine learning models will help us build interfaces which reify deep principles in ways meaningful to the user. For that to happen, the models have to discover deep principles about the world, recognize those principles, and then surface them as vividly as possible in an interface, in a way comprehensible by the user.

Of course, this is a tall order! The examples we’ve shown are just barely beginning to do this. It’s true that our models do sometimes discover relatively deep principles, like the preservation of enclosed negative space when bolding a font. But this is merely implicit in the model. And while we’ve built a tool which takes advantage of such principles, it’d be better if the model automatically inferred the important principles learned, and found ways of explicitly surfacing them through the interface. (Encouraging progress toward this has been made by InfoGANs, which use information-theoretic ideas to find structure in the latent space.) Ideally, such models would start to get at true explanations, not just in a static form, but in a dynamic form, manipulable by the user. But we’re a long way from that point.

Do these interfaces inhibit creativity?

It’s tempting to be skeptical of the expressiveness of the interfaces we’ve described. If an interface constrains us to explore only the natural space of images, does that mean we’re merely doing the expected? Does it mean these interfaces can only be used to generate visual cliches? Does it prevent us from generating anything truly new, from doing truly creative work?

To answer these questions, it’s helpful to identify two different modes of creativity. This two-mode model is over-simplified: creativity doesn’t fit so neatly into two distinct categories. Yet the model nonetheless clarifies the role of new interfaces in creative work.

The first mode of creativity is the everyday creativity of a craftsperson engaged in their craft. Much of the work of a font designer, for example, consists of competent recombination of the best existing practices. Such work typically involves many creative choices to meet the intended design goals, but not developing key new underlying principles.

For such work, the generative interfaces we’ve been discussing are promising. While they currently have many limitations, future research will identity and fix many deficiencies. This is happening rapidly with GANs: the original GANs had many limitations, but models soon appeared that were better adapted to images, improved the resolution, reduced artifactsSo much work has been done on improving resolution and reducing artifacts it seems unfair to single out any small set of papers, and to omit the many others., and so on. With enough iterations it’s plausible these generative interfaces will become powerful tools for craft work.

The second mode of creativity aims toward developing new principles that fundamentally change the range of creative expression. One sees this in the work of artists such as Picasso or Monet, who violated existing principles of painting, developing new principles which enabled people to see in new ways.

Is it possible to do such creative work, while using a generative interface? Don’t such interfaces constrain us to the space of natural images, or natural fonts, and thus actively prevent us from exploring the most interesting new directions in creative work?

The situation is more complex than this.

In part, this is a question about the power of our generative models. In some cases, the model can only generate recombinations of existing ideas. This is a limitation of an ideal GAN, since a perfectly trained GAN generator will reproduce the training distribution. Such a model can’t directly generate an image based on new fundamental principles, because such an image wouldn’t look anything like it’s seen in its training data.

Artists such as Mario Klingemann and Mike Tyka are now using GANs to create interesting artwork. They’re doing that using “imperfect” GAN models, which they seem to be able to use to explore interesting new principles; it’s perhaps the case that bad GANs may be more artistically interesting than ideal GANs. Furthermore, nothing says an interface must only help us explore the latent space. Perhaps operations can be added which deliberately take us out of the latent space, or to less probable (and so more surprising) parts of the space of natural images.

Of course, GANs are not the only generative models. In a sufficiently powerful generative model, the generalizations discovered by the model may contain ideas going beyond what humans have discovered. In that case, exploration of the latent space may enable us to discover new fundamental principles. The model would have discovered stronger abstractions than human experts. Imagine a generative model trained on paintings up until just before the time of the cubists; might it be that by exploring that model it would be possible to discover cubism? It would be an analogue to something like the prediction of Bose-Einstein condensation, as discussed earlier in the essay. Such invention is beyond today’s generative models, but seems a worthwhile aspiration for future models.

Our examples so far have all been based on generative models. But there are some illuminating examples which are not based on generative models. Consider the pix2pix system developed by Isola et al. This system is trained on pairs of images, e.g., pairs showing the edges of a cat, and the actual corresponding cat. Once trained, it can be shown a set of edges and asked to generate an image for an actual corresponding cat. It often does this quite well:

 Input

 Output

 Live demo by Christopher Hesse

When supplied with unusual constraints, pix2pix can produce striking images:

 Bread cat by Ivy Tsai

 Cat beholder by Marc Hesse

 Spiral cat

This is perhaps not high creativity of a Picasso-esque level. But it is still surprising. It’s certainly unlike images most of us have ever seen before. How does pix2pix and its human user achieve this kind of result?

Unlike our earlier examples, pix2pix is not a generative model. This means it does not have a latent space or a corresponding space of natural images. Instead, there is a neural network, called, confusingly, a generator – this is not meant in the same sense as our earlier generative models – that takes as input the constraint image, and produces as output the filled-in image.

The generator is trained adversarially against a discriminator network, whose job is to distinguish between pairs of images generated from real data, and pairs of images generated by the generator.

While this sounds similar to a conventional GAN, there is a crucial difference: there is no latent vector input to the generatorActually, Isola et al experimented with adding such a latent vector to the generator, but found it made little difference to the resulting images.. Rather, there is simply an input constraint. When a human inputs a constraint unlike anything seen in training, the network is forced to improvise, doing the best it can to interpret that constraint according to the rules it has previously learned. The creativity is the result of a forced merger of knowledge inferred from the training data, together with novel constraints provided by the user. As a result, even relatively simple ideas – like the bread- and beholder-cats – can result in striking new types of images, images not within what we would previously have considered the space of natural images.

Conclusion

It is conventional wisdom that AI will change how we interact with computers. Unfortunately, many in the AI community greatly underestimate the depth of interface design, often regarding it as a simple problem, mostly about making things pretty or easy-to-use. In this view, interface design is a problem to be handed off to others, while the hard work is to train some machine learning system.

This view is incorrect. At its deepest, interface design means developing the fundamental primitives human beings think and create with. This is a problem whose intellectual genesis goes back to the inventors of the alphabet, of cartography, and of musical notation, as well as modern giants such as Descartes, Playfair, Feynman, Engelbart, and Kay. It is one of the hardest, most important and most fundamental problems humanity grapples with.

As discussed earlier, in one common view of AI our computers will continue to get better at solving problems, but human beings will remain largely unchanged. In a second common view, human beings will be modified at the hardware level, perhaps directly through neural interfaces, or indirectly through whole brain emulation.

We’ve described a third view, in which AIs actually change humanity, helping us invent new cognitive technologies, which expand the range of human thought. Perhaps one day those cognitive technologies will, in turn, speed up the development of AI, in a virtuous feedback cycle:

It would not be a Singularity in machines. Rather, it would be a Singularity in humanity’s range of thought. Of course, this loop is at present extremely speculative. The systems we’ve described can help develop more powerful ways of thinking, but there’s at most an indirect sense in which those ways of thinking are being used in turn to develop new AI systems.

Of course, over the long run it’s possible that machines will exceed humans on all or most cognitive tasks. Even if that’s the case, cognitive transformation will still be a valuable end, worth pursuing in its own right. There is pleasure and value involved in learning to play chess or Go well, even if machines do it better. And in activities such as story-telling the benefit often isn’t so much the artifact produced as the process of construction itself, and the relationships forged. There is intrinsic value in personal change and growth, apart from instrumental benefits.

The interface-oriented work we’ve discussed is outside the narrative used to judge most existing work in artificial intelligence. It doesn’t involve beating some benchmark for a classification or regression problem. It doesn’t involve impressive feats like beating human champions at games such as Go. Rather, it involves a much more subjective and difficult-to-measure criterion: is it helping humans think and create in new ways?

This creates difficulties for doing this kind of work, particularly in a research setting. Where should one publish? What community does one belong to? What standards should be applied to judge such work? What distinguishes good work from bad?

We believe that over the next few years a community will emerge which answers these questions. It will run workshops and conferences. It will publish work in venues such as Distill. Its standards will draw from many different communities: from the artistic and design and musical communities; from the mathematical community’s taste in abstraction and good definition; as well as from the existing AI and IA communities, including work on computational creativity and human-computer interaction. The long-term test of success will be the development of tools which are widely used by creators. Are artists using these tools to develop remarkable new styles? Are scientists in other fields using them to develop understanding in ways not otherwise possible? These are great aspirations, and require an approach that builds on conventional AI work, but also incorporates very different norms.

Acknowledgments

We are extremely grateful to James Wexler who built and trained the model on which the font demos are based. Nikhil Thorat and Daniel Smilkov also generously optimized the browser implementation of the model, which runs using deeplearn.js. Thanks to Daniel Dewey, Andy Matuschak, Robert Ochshorn, and Bret Victor for valuable conversations and feedback.

Author contribution

Authors are listed alphabetically in the byline. Michael drafted the text of the essay, but the ideas of the essay were developed jointly in conversation with Shan. Shan developed the interactive diagrams, with considerable input from Michael.

References

 1. Augmenting Human Intellect: A Conceptual Framework
 Engelbart, D.C., 1962.
 2. deeplearn.js font demo [link]
 Wexler, J., 2017.
 3. Auto-encoding variational Bayes
 Kingma, D.P. and Welling, M., 2014. ICLR.
 4. Analyzing 50k fonts using deep neural networks [HTML]
 Bernhardsson, E., 2016.
 5. Autoencoding beyond pixels using a learned similarity metric
 Larsen, A.B.L., Sønderby, S.K., Larochelle, H. and Winther, O., 2016. ICML.
 6. Sampling Generative Networks [PDF]
 White, T., 2016.
 7. Writing with the Machine [link]
 Sloan, R., 2017. Eyeo.
 8. Automatic chemical design using a data-driven continuous representation of molecules [PDF]
 Gómez-Bombarelli, R., Duvenaud, D., Hernández-Lobato, J.M., Aguilera-Iparraguirre, J., Hirzel, T.D., Adams, R.P. and Aspuru-Guzik, A., 2016.
 9. Generative visual manipulation on the natural image manifold
 Zhu, J., Krähenbühl, P., Schechtman, E. and Efros, A.A., 2016. European Conference on Computer Vision (ECCV).
10. Generative adversarial nets
 Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y., 2014. Advances in Neural Information Processing Systems (NIPS), pp. 2672-2680.
11. A Neural Representation of Sketch Drawings [PDF]
 Ha, D. and Eck, D., 2017.
12. Real-time human interaction with supervised learning algorithms for music composition and performance
 Fiebrink, R., 2011. Princeton University PhD Thesis.
13. TopoSketch: Drawing in Latent Space
 Loh, I. and White, T., 2017. NIPS Workshop on Machine Learning for Creativity and Design.
14. Taking The Robots To Design School, Part 1 [link]
 Gold, J., 2016.
15. Hierarchical Variational Autoencoders for Music [PDF]
 Roberts, A., Engel, J. and Eck, D., 2017. NIPS Workshop on Machine Learning for Creativity and Design.
16. Computational creativity: the final frontier?
 Colton, S. and Wiggins, G.A., 2012. ECAI.
17. Interactive machine learning: letting users build classifiers
 Ware, M., Frank, E., Holmes, G., Hall, M. and Witten, I.H., 2001. International Journal of Human-Computer Studies, Vol 55, pp. 281-292.
18. Eccentric School of Painting Increased Its Vogue in the Current Art Exhibition — What Its Followers Attempt to Do [link]
 1911. The New York Times.
19. Genius: The Life and Science of Richard Feynman
 Gleick, J., 1992. Vintage Books.
20. Thought as a Technology [HTML]
 Nielsen, M., 2016.
21. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
 Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I. and Abbeel, P., 2016. NIPS.
22. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [PDF]
 Radford, A., Metz, L. and Chintala, S., 2016. ICLR.
23. Image-to-Image Translation with Conditional Adversarial Networks [PDF]
 Isola, P., Zhu, J., Zhou, T. and Efros, A.A., 2017.

Updates and Corrections

If you see mistakes or want to suggest changes, please create an issue on GitHub.

Reuse

Diagrams and text are licensed under Creative Commons Attribution CC-BY 4.0 with the source available on GitHub, unless noted otherwise. The figures that have been reused from other sources don’t fall under this license and can be recognized by a note in their caption: “Figure from …”.

Citation

For attribution in academic contexts, please cite this work as

Carter & Nielsen, "Using Artificial Intelligence to Augment Human Intelligence", Distill, 2017.

BibTeX citation

@article{carter2017using,
 author = {Carter, Shan and Nielsen, Michael},
 title = {Using Artificial Intelligence to Augment Human Intelligence},
 journal = {Distill},
 year = {2017},
 note = {https://distill.pub/2017/aia},
 doi = {10.23915/distill.00009}
}
 Distill is dedicated to clear explanations of machine learning
 About Submit Prize Archive RSS GitHub Twitter ISSN 2476-0757
What are computers for?
 
 
 Historically, different answers to this question – that is,
 different visions of computing – have helped inspire and
 determine the computing systems humanity has ultimately
 built. Consider the early electronic computers. ENIAC, the
 world’s first general-purpose electronic computer, was
 commissioned to compute artillery firing tables for the United
 States Army. Other early computers were also used to solve
 numerical problems, such as simulating nuclear explosions,
 predicting the weather, and planning the motion of rockets. The
 machines operated in a batch mode, using crude input and output
 devices, and without any real-time interaction. It was a vision
 of computers as number-crunching machines, used to speed up
 calculations that would formerly have taken weeks, months, or more
 for a team of humans.
 
 
 In the 1950s a different vision of what computers are for began to
 develop. That vision was crystallized in 1962, when Douglas
 Engelbart proposed that computers could be used as a way
 of augmenting human
 intellect. In this view, computers weren’t primarily
 tools for solving number-crunching problems. Rather, they were
 real-time interactive systems, with rich inputs and outputs, that
 humans could work with to support and expand their own
 problem-solving process. This vision of intelligence augmentation
 (IA) deeply influenced many others, including researchers such as
 Alan Kay at Xerox PARC, entrepreneurs such as Steve Jobs at Apple,
 and led to many of the key ideas of modern computing systems. Its
 ideas have also deeply influenced digital art and music, and
 fields such as interaction design, data visualization,
 computational creativity, and human-computer interaction.
 
 
 Research on IA has often been in competition with research on
 artificial intelligence (AI): competition for funding, competition
 for the interest of talented researchers. Although there has
 always been overlap between the fields, IA has typically focused
 on building systems which put humans and machines to work
 together, while AI has focused on complete outsourcing of
 intellectual tasks to machines. In particular, problems in AI are
 often framed in terms of matching or surpassing human performance:
 beating humans at chess or Go; learning to recognize speech and
 images or translating language as well as humans; and so on.
 
 
 This essay describes a new field, emerging today out of a
 synthesis of AI and IA. For this field, we suggest the
 name artificial intelligence augmentation (AIA): the use
 of AI systems to help develop new methods for intelligence
 augmentation. This new field introduces important new fundamental
 questions, questions not associated to either parent field. We
 believe the principles and systems of AIA will be radically
 different to most existing systems.
 
 
 Our essay begins with a survey of recent technical work hinting at
 artificial intelligence augmentation, including work
 on generative interfaces – that is, interfaces
 which can be used to explore and visualize generative machine
 learning models. Such interfaces develop a kind of cartography of
 generative models, ways for humans to explore and make meaning
 from those models, and to incorporate what those models
 “know” into their creative work.
 
 
 Our essay is not just a survey of technical work. We believe now
 is a good time to identify some of the broad, fundamental
 questions at the foundation of this emerging field. To what
 extent are these new tools enabling creativity? Can they be used
 to generate ideas which are truly surprising and new, or are the
 ideas cliches, based on trivial recombinations of existing ideas?
 Can such systems be used to develop fundamental new interface
 primitives? How will those new primitives change and expand the
 way humans think?
 
 
 Using generative models to invent meaningful creative operations
 
 
 Let’s look at an example where a machine learning model makes a
 new type of interface possible. To understand the interface,
 imagine you’re a type designer, working on creating a new
 fontWe shall egregiously abuse the distinction between
 a font and a typeface. Apologies to any type designers who may be
 reading.. After sketching some initial designs, you
 wish to experiment with bold, italic, and condensed variations.
 Let’s examine a tool to generate and explore such variations, from
 any initial design. For reasons that will soon be explained the
 quality of results is quite crude; please bear with us.
 
 
 
 
 
 Of course, varying the bolding (i.e., the weight), italicization
 and width are just three ways you can vary a font. Imagine that
 instead of building specialized tools, users could build their own
 tool merely by choosing examples of existing fonts. For instance,
 suppose you wanted to vary the degree of serifing on a font. In
 the following, please select 5 to 10 sans-serif fonts from the top
 box, and drag them to the box on the left. Select 5 to 10 serif
 fonts and drag them to the box on the right. As you do this, a
 machine learning model running in your browser will automatically
 infer from these examples how to interpolate your starting font in
 either the serif or sans-serif direction:
 
 
 
 
 In fact, we used this same technique to build the earlier bolding
 italicization, and condensing tool. To do so, we used the
 following examples of bold and non-bold fonts, of italic and
 non-italic fonts, and of condensed and non-condensed fonts:
 
 
 
 
 To build these tools, we used what’s called a generative
 model; the particular model we use was trained
 by James Wexler. To
 understand generative models, consider that a priori
 describing a font appears to require a lot of data. For
 instance, if the font is 6464 by 6464 pixels, then we’d expect
 to need 64×64=4,09664 \times 64 = 4,096 parameters to describe a single
 glyph. But we can use a generative model to find a much simpler
 description.
 
 
 We do this by building a neural network which takes a small number
 of input variables, called latent variables, and produces
 as output the entire glyph. For the particular model we use, we
 have 4040 latent space dimensions, and map that into the
 4,0964,096-dimensional space describing all the pixels in the glyph.
 In other words, the idea is to map a low-dimensional space into a
 higher-dimensional space:
 
 
 
 
 The generative model we use is a type of neural network known as
 a variational autoencoder
 (VAE). For our purposes, the details of the generative
 model aren’t so important. The important thing is that by
 changing the latent variables used as input, it’s possible to get
 different fonts as output. So one choice of latent variables will
 give one font, while another choice will give a different font:
 
 
 
 
 You can think of the latent variables as a compact, high-level
 representation of the font. The neural network takes that
 high-level representation and converts it into the full pixel
 data. It’s remarkable that just 4040 numbers can capture the
 apparent complexity in a glyph, which originally required 4,0964,096
 variables.
 
 
 The generative model we use is learnt from a training set of more
 than 5050 thousand
 fonts Bernhardsson
 scraped from the open web. During training, the weights and
 biases in the network are adjusted so that the network can output
 a close approximation to any desired font from the training set,
 provided a suitable choice of latent variables is made. In some
 sense, the model is learning a highly compressed representation of
 all the training fonts.
 
 
 In fact, the model doesn’t just reproduce the training fonts. It
 can also generalize, producing fonts not seen in training. By
 being forced to find a compact description of the training
 examples, the neural net learns an abstract, higher-level model of
 what a font is. That higher-level model makes it possible to
 generalize beyond the training examples already seen, to produce
 realistic-looking fonts.
 
 
 Ideally, a good generative model would be exposed to a relatively
 small number of training examples, and use that exposure to
 generalize to the space of all possible human-readable fonts.
 That is, for any conceivable font – whether existing or
 perhaps even imagined in the future – it would be possible
 to find latent variables corresponding exactly to that font. Of
 course, the model we’re using falls far short of this ideal
 – a particularly egregious failure is that many fonts
 generated by the model omit the tail on the capital
 “Q” (you can see this in the examples above). Still,
 it’s useful to keep in mind what an ideal generative model would
 do.
 
 
 Such generative models are similar in some ways to how scientific
 theories work. Scientific theories often greatly simplify the
 description of what appear to be complex phenomena, reducing large
 numbers of variables to just a few variables from which many
 aspects of system behavior can be deduced. Furthermore, good
 scientific theories sometimes enable us to generalize to discover
 new phenomena.
 
 
 As an example, consider ordinary material objects. Such objects
 have what physicists call a phase – they may be a
 liquid, a solid, a gas, or perhaps something more exotic, like a
 superconductor
 or Bose-Einstein
 condensate. A priori, such systems seem immensely
 complex, involving perhaps 102310^{23} or so molecules. But the
 laws of thermodynamics and statistical mechanics enable us to find
 a simpler description, reducing that complexity to just a few
 variables (temperature, pressure, and so on), which encompass much
 of the behavior of the system. Furthermore, sometimes it’s
 possible to generalize, predicting unexpected new phases of
 matter. For example, in 1924, physicists used thermodynamics and
 statistical mechanics to predict a remarkable new phase of matter,
 Bose-Einstein condensation, in which a collection of atoms may all
 occupy identical quantum states, leading to surprising large-scale
 quantum interference effects. We’ll come back to this predictive
 ability in our later discussion of creativity and generative
 models.
 
 
 Returning to the nuts and bolts of generative models, how can we
 use such models to do example-based reasoning like that in the
 tool shown above? Let’s consider the case of the bolding tool. In
 that instance, we take the average of all the latent vectors for
 the user-specified bold fonts, and the average for all the
 user-specified non-bold fonts. We then compute the difference
 between these two average vectors:
 
 
 
 
 We’ll refer to this as the bolding vector. To make some
 given font bolder, we simply add a little of the bolding vector to
 the corresponding latent vector, with the amount of bolding vector
 added controlling the boldness of the resultIn
 practice, sometimes a slightly different procedure is used. In
 some generative models the latent vectors satisfy some constraints
 – for instance, they may all be of the same length. When
 that’s the case, as in our model, a more sophisticated
 “adding” operation must be used, to ensure the length
 remains the same. But conceptually, the picture of adding the
 bolding vector is the right way to think.:
 
 
 
 
 This technique was introduced
 by Larsen et al, and
 vectors like the bolding vector are sometimes called
 attribute vectors. The same idea is use to implement all
 the tools we’ve shown. That is, we use example fonts to creating
 a bolding vector, an italicizing vector, a condensing vector, and
 a user-defined serif vector. The interface thus provides a way of
 exploring the latent space in those four directions.
 
 
 The tools we’ve shown have many drawbacks. Consider the following
 example, where we start with an example glyph, in the middle, and
 either increase or decrease the bolding (on the right and left,
 respectively):
 
 
 
 
 Examining the glyphs on the left and right we see many unfortunate
 artifacts. Particularly for the rightmost glyph, the edges start to get
 rough, and the serifs begin to disappear. A better generative
 model would reduce those artifacts. That’s a good long-term
 research program, posing many intriguing problems. But even with
 the model we have, there are also some striking benefits to the
 use of the generative model.
 
 
 To understand these benefits, consider a naive approach to
 bolding, in which we simply add some extra pixels around a glyph’s
 edges, thickening it up. While this thickening perhaps matches a
 non-expert’s way of thinking about type design, an expert does
 something much more involved. In the following we show the
 results of this naive thickening procedure versus what is actually
 done, for Georgia and Helvetica:
 
 
 
 
 As you can see, the naive bolding procedure produces quite
 different results, in both cases. For example, in Georgia, the
 left stroke is only changed slightly by bolding, while the right
 stroke is greatly enlarged, but only on one side. In both
 fonts, bolding doesn’t change the height of the font, while the
 naive approach does.
 
 
 As these examples show, good bolding is not a trivial
 process of thickening up a font. Expert type designers have many
 heuristics for bolding, heuristics inferred from much previous
 experimentation, and careful study of historical
 examples. Capturing all those heuristics in a conventional program
 would involve immense work. The benefit of using the generative
 model is that it automatically learns many such heuristics.
 
 
 For example, a naive bolding tool would rapidly fill in the
 enclosed negative space in the enclosed upper region of the letter
 “A”. The font tool doesn’t do this. Instead, it goes
 to some trouble to preserve the enclosed negative space, moving
 the A’s bar down, and filling out the interior strokes more slowly
 than the exterior. This principle is evident in the examples
 shown above, especially Helvetica, and it can also be seen in the
 operation of the font tool:
 
 
 
 
 The heuristic of preserving enclosed negative space is not a
 priori obvious. However, it’s done in many professionally
 designed fonts. If you examine examples like those shown above
 it’s easy to see why: it improves legibility. During training,
 our generative model has automatically inferred this principle
 from the examples it’s seen. And our bolding interface then makes
 this available to the user.
 
 
 In fact, the model captures many other heuristics. For instance,
 in the above examples the heights of the fonts are (roughly)
 preserved, which is the norm in professional font design. Again,
 what’s going on isn’t just a thickening of the font, but rather
 the application of a more subtle heuristic inferred by the
 generative model. Such heuristics can be used to create fonts
 with properties which would otherwise be unlikely to occur to
 users. Thus, the tool expands ordinary people’s ability to
 explore the space of meaningful fonts.
 
 
 The font tool is an example of a kind of cognitive technology. In
 particular, the primitive operations it contains can be
 internalized as part of how a user thinks. In this it resembles a
 program such as Photoshop or a spreadsheet or 3D graphics
 programs. Each provides a novel set of interface primitives,
 primitives which can be internalized by the user as fundamental
 new elements in their thinking. This act of internalization of new
 primitives is fundamental to much work on intelligence
 augmentation.
 
 
 The ideas shown in the font tool can be extended to other domains.
 Using the same interface, we can use a generative model to
 manipulate images of human faces using qualities such as
 expression, gender, or hair color. Or to manipulate sentences
 using length, sarcasm, or tone. Or to manipulate molecules using
 chemical properties:
 
 
 
 
  
 Images from Sampling Generative Networks by White.
  
  
 Sentence from Pride and Prejudice by Jane Austen. Interpolated by the authors. Inspired by experiments done by the novelist Robin Sloan
 
  
 Images from Automatic chemical design using a data-driven continuous representation of molecules by Gómez-Bombarelli et al .
  
 
 
 
 Such generative interfaces provide a kind of cartography of
 generative models, ways for humans to explore and make meaning
 using those models.
 
 
 We saw earlier that the font model automatically infers relatively
 deep principles about font design, and makes them available to
 users. While it’s great that such deep principles can be
 inferred, sometimes such models infer other things that are wrong,
 or undesirable. For example, White
 points out the addition of a smile vector in some face
 models will make faces not just smile more, but also appear more
 feminine. Why? Because in the training data more women than men
 were smiling. So these models may not just learn deep facts about
 the world, they may also internalize prejudices or erroneous
 beliefs. Once such a bias is known, it is often possible to make
 corrections. But to find those biases requires careful auditing
 of the models, and it is not yet clear how we can ensure such
 audits are exhaustive.
 
 
 
 More broadly, we can ask why attribute vectors work, when they
 work, and when they fail? At the moment, the answers to these
 questions are poorly understood.
 
 
 For the attribute vector to work requires that taking any starting
 font, we can construct the corresponding bold version by adding
 the same vector in the latent space. However, a
 priori there is no reason using a single constant vector to
 displace will work. It may be that we should displace in many
 different ways. For instance, the heuristics used to bold serif
 and sans-serif fonts are quite different, and so it seems likely
 that very different displacements would be involved:
 
 
 
 
 Of course, we could do something more sophisticated than using a
 single constant attribute vector. Given pairs of example fonts
 (unbold, bold) we could train a machine learning algorithm to take
 as input the latent vector for the unbolded version and output the
 latent vector for the bolded version. With additional training
 data about font weights, the machine learning algorithm could
 learn to generate fonts of arbitrary weight. Attribute vectors
 are just an extremely simple approach to doing these kinds of
 operation.
 
 
 For these reasons, it seems unlikely that attribute vectors will
 last as an approach to manipulating high-level features. Over the
 next few years much better approaches will be developed. However,
 we can still expect interfaces offering operations broadly similar
 to those sketched above, allowing access to high-level and
 potentially user-defined concepts. That interface pattern doesn’t
 depend on the technical details of attribute vectors.
 
 
 Interactive Generative Adversarial Models
 
 
 Let’s look at another example using machine learning models to
 augment human creativity. It’s the interactive generative
 adversarial networks, or iGANs, introduced
 by Zhu et al in 2016.
 
 
 One of the examples of Zhu et al is the use of iGANs in
 an interface to generate images of consumer products such as
 shoes. Conventionally, such an interface would require the
 programmer to write a program containing a great deal of knowledge
 about shoes: soles, laces, heels, and so on. Instead of doing
 this, Zhu et al train a generative model using 5050
 thousand images of shoes, downloaded from Zappos. They then use
 that generative model to build an interface that lets a user
 roughly sketch the shape of a shoe, the sole, the laces, and so
 on:
 
 
 
 Excerpted from Zhu et
 al.
 
 
 The visual quality is low, in part because the generative model
 Zhu et al used is outdated by modern (2017) standards
 – with more modern models, the visual quality would be much
 higher.
 
 
 But the visual quality is not the point. Many interesting things
 are going on in this prototype. For instance, notice how the
 overall shape of the shoe changes considerably when the sole is
 filled in – it becomes narrower and sleeker. Many small
 details are filled in, like the black piping on the top of the
 white sole, and the red coloring filled in everywhere on the
 shoe’s upper. These and other facts are automatically deduced
 from the underlying generative model, in a way we’ll describe
 shortly.
 
 
 The same interface may be used to sketch landscapes. The only
 difference is that the underlying generative model has been
 trained on landscape images rather than images of shoes. In this
 case it becomes possible to sketch in just the colors associated
 to a landscape. For example, here’s a user sketching in some green
 grass, the outline of a mountain, some blue sky, and snow on the
 mountain:
 
 
 
 Excerpted from Zhu et
 al.
 
 
 The generative models used in these interfaces are different than
 for our font model. Rather than using variational autoencoders,
 they’re based on generative
 adversarial networks (GANs). But the underlying idea is
 still to find a low-dimensional latent space which can be used to
 represent (say) all landscape images, and map that latent space to
 a corresponding image. Again, we can think of points in the
 latent space as a compact way of describing landscape images.
 
 
 Roughly speaking, the way the iGANs works is as follows. Whatever
 the current image is, it corresponds to some point in the latent
 space:
 
 
 
 
 Suppose, as happened in the earlier video, the user now sketches
 in a stroke outlining the mountain shape. We can think of the
 stroke as a constraint on the image, picking out a subspace of the
 latent space, consisting of all points in the latent space whose
 image matches that outline:
 
 
 
 
 The way the interface works is to find a point in the latent space
 which is near to the current image, so the image is not changed
 too much, but also coming close to satisfying the imposed
 constraints. This is done by optimizing an objective function
 which combines the distance to each of the imposed constraints, as
 well as the distance moved from the current point. If there’s
 just a single constraint, say, corresponding to the mountain
 stroke, this looks something like the following:
 
 
 
 
 We can think of this, then, as a way of applying constraints to
 the latent space to move the image around in meaningful ways.
 
 
 The iGANs have much in common with the font tool we showed
 earlier. Both make available operations that encode much subtle
 knowledge about the world, whether it be learning to understand
 what a mountain looks like, or inferring that enclosed negative
 space should be preserved when bolding a font. Both the iGANs and
 the font tool provide ways of understanding and navigating a
 high-dimensional space, keeping us on the natural space of fonts
 or shoes or landscapes. As Zhu et al remark:
 
 
 [F]or most of us, even a simple image manipulation in Photoshop
 presents insurmountable difficulties… any less-than-perfect
 edit immediately makes the image look completely unrealistic. To
 put another way, classic visual manipulation paradigm does not
 prevent the user from “falling off” the manifold of
 natural images.
 
 
 Like the font tool, the iGANs is a cognitive technology. Users
 can internalize the interface operations as new primitive elements
 in their thinking. In the case of shoes, for example, they can
 learn to think in terms of the difference they want to apply,
 adding a heel, or a higher top, or a special highlight. This is
 richer than the traditional way non-experts think about shoes
 (“Size 11, black” etc). To the extent that
 non-experts do think in more sophisticated ways –
 “make the top a little higher and sleeker” –
 they get little practice in thinking this way, or seeing the
 consequences of their choices. Having an interface like this
 enables easier exploration, the ability to develop idioms and the
 ability to plan, to swap ideas with friends, and so on.
 
 
 Two models of computation
 
 
 Let’s revisit the question we began the essay with, the question
 of what computers are for, and how this relates to intelligence
 augmentation.
 
 
 One common conception of computers is that they’re problem-solving
 machines: “computer, what is the result of firing this
 artillery shell in such-and-such a wind [and so on]?”;
 “computer, what will the maximum temperature in Tokyo be in
 5 days?”; “computer, what is the best move to take
 when the Go board is in this position?”; “computer,
 how should this image be classified?”; and so on.
 
 
 This is a conception common to both the early view of computers as
 number-crunchers, and also in much work on AI, both historically
 and today. It’s a model of a computer as a way of outsourcing
 cognition. In speculative depictions of possible future AI,
 this cognitive outsourcing model often shows up in the
 view of an AI as an oracle, able to solve some large class of
 problems with better-than-human performance.
 
 
 But a very different conception of what computers are for is
 possible, a conception much more congruent with work on
 intelligence augmentation.
 
 
 To understand this alternate view, consider our subjective
 experience of thought. For many people, that experience is verbal:
 they think using language, forming chains of words in their heads,
 similar to sentences in speech or written on a page. For other
 people, thinking is a more visual experience, incorporating
 representations such as graphs and maps. Still other people mix
 mathematics into their thinking, using algebraic expressions or
 diagrammatic techniques, such as Feynman diagrams and Penrose
 diagrams.
 
 
 In each case, we’re thinking using representations invented by
 other people: words, graphs, maps, algebra, mathematical diagrams,
 and so on. We internalize these cognitive technologies as we grow
 up, and come to use them as a kind of substrate for our thinking.
 
 
 For most of history, the range of available cognitive technologies
 has changed slowly and incrementally. A new word will be
 introduced, or a new mathematical symbol. More rarely, a radical
 new cognitive technology will be developed. For example, in 1637
 Descartes published his “Discourse on Method”,
 explaining how to represent geometric ideas using algebra, and
 vice versa:
 
  
 
 This enabled a radical change and expansion in how we think about
 both geometry and algebra.
 
 
 Historically, lasting cognitive technologies have been invented
 only rarely. But modern computers are a meta-medium enabling the
 rapid invention of many new cognitive technologies. Consider a
 relatively banal example, such
 as Photoshop. Adept Photoshop users routinely
 have formerly impossible thoughts such as: “let’s apply the
 clone stamp to the such-and-such layer.”. That’s an
 instance of a more general class of thought: “computer, [new
 type of action] this [new type of representation for a newly
 imagined class of object]”. When that happens, we’re using
 computers to expand the range of thoughts we can think.
 
 
 It’s this kind of cognitive transformation model which
 underlies much of the deepest work on intelligence augmentation.
 Rather than outsourcing cognition, it’s about changing the
 operations and representations we use to think; it’s about
 changing the substrate of thought itself. And so while cognitive
 outsourcing is important, this cognitive transformation view
 offers a much more profound model of intelligence augmentation.
 It’s a view in which computers are a means to change and expand
 human thought itself.
 
 
 Historically, cognitive technologies were developed by human
 inventors, ranging from the invention of writing in Sumeria and
 Mesoamerica, to the modern interfaces of designers such as Douglas
 Engelbart, Alan Kay, and others.
 
 
 Examples such as those described in this essay suggest that AI
 systems can enable the creation of new cognitive technologies.
 Things like the font tool aren’t just oracles to be consulted when
 you want a new font. Rather, they can be used to explore and
 discover, to provide new representations and operations, which can
 be internalized as part of the user’s own thinking. And while
 these examples are in their early stages, they suggest AI is not
 just about cognitive outsourcing. A different view of AI is
 possible, one where it helps us invent new cognitive technologies
 which transform the way we think.
 
 
 In this essay we’ve focused on a small number of examples, mostly
 involving exploration of the latent space. There are many other
 examples of artificial intelligence augmentation. To give some
 flavor, without being comprehensive:
 the sketch-rnn system, for neural
 network assisted drawing;
 the Wekinator, which enables
 users to rapidly build new musical instruments and artistic
 systems; TopoSketch, for developing
 animations by exploring latent spaces; machine learning models for
 designing overall typographic
 layout; and a generative model which enables
 interpolation between musical
 phrases. In each case, the systems use machine learning
 to enable new primitives which can be integrated into the user’s
 thinking. More broadly, artificial intelligence augmentation will
 draw on fields such as computational
 creativity and interactive machine
 learning.
 
 
 Finding powerful new primitives of thought
 
 
 We’ve argued that machine learning systems can help create
 representations and operations which serve as new primitives in
 human thought. What properties should we look for in such new
 primitives? This is too large a question to be answered
 comprehensively in a short essay. But we will explore it briefly.
 
 
 Historically, important new media forms often seem strange when
 introduced. Many such stories have passed into popular culture:
 the near riot at the premiere of Stravinsky and Nijinksy’s
 “Rite of Spring”; the consternation caused by the
 early cubist paintings, leading
 The New York Times to
 comment: “What do they mean? Have those
 responsible for them taken leave of their senses? Is it art or
 madness? Who knows?”
 
 
 Another example comes from physics. In the 1940s, different
 formulations of the theory of quantum electrodynamics were
 developed independently by the physicists Julian Schwinger,
 Shin’ichirō Tomonaga, and Richard Feynman. In their work,
 Schwinger and Tomonaga used a conventional algebraic approach,
 along lines similar to the rest of physics. Feynman used a more
 radical approach, based on what are now known as Feynman diagrams,
 for depicting the interaction of light and matter:
 
 
 
 Image by Joel
	Holdsworth), licensed under a Creative Commons
	Attribution-Share Alike 3.0 Unported license
 
 
 
 Initially, the Schwinger-Tomonaga approach was easier for other
 physicists to understand. When Feynman and Schwinger presented
 their work at a 1948 workshop, Schwinger was immediately
 acclaimed. By contrast, Feynman left his audience mystified. As
 James Gleick put it in his biography of
 Feynman:
 
 
 It struck Feynman that everyone had a favorite principle or
 theorem and he was violating them all… Feynman knew he had
 failed. At the time, he was in anguish. Later he said simply:
 “I had too much stuff. My machines came from too far
 away.”
 
 
 Of course, strangeness for strangeness’s sake alone is not
 useful. But these examples suggest that breakthroughs in
 representation often appear strange at first. Is there any
 underlying reason that is true?
 
 
 Part of the reason is because if some representation is truly new,
 then it will appear different than anything you’ve ever seen
 before. Feynman’s diagrams, Picasso’s paintings, Stravinsky’s
 music: all revealed genuinely new ways of making meaning. Good
 representations sharpen up such insights, eliding the familiar to
 show that which is new as vividly as possible. But because of
 that emphasis on unfamiliarity, the representation will seem
 strange: it shows relationships you’ve never seen before. In some
 sense, the task of the designer is to identify that core
 strangeness, and to amplify it as much as possible.
 
 
 Strange representations are often difficult to understand. At
 first, physicists preferred Schwinger-Tomonaga to Feynman. But as
 Feynman’s approach was slowly understood by physicists, they
 realized that although Schwinger-Tomonaga and Feynman were
 mathematically equivalent, Feynman was more powerful. As Gleick
 puts it:
 
 
 Schwinger’s students at Harvard were put at a competitive
 disadvantage, or so it seemed to their fellows elsewhere, who
 suspected them of surreptitiously using the diagrams anyway. This
 was sometimes true… Murray Gell-Mann later spent a semester
 staying in Schwinger’s house and loved to say afterward that he
 had searched everywhere for the Feynman diagrams. He had not
 found any, but one room had been locked…
 
 
 These ideas are true not just of historical representations, but
 also of computer interfaces. However, our advocacy of strangeness
 in representation contradicts much conventional wisdom about
 interfaces, especially the widely-held belief that they should be
 “user friendly”, i.e., simple and immediately useable
 by novices. That most often means the interface is cliched, built
 from conventional elements combined in standard ways. But while
 using a cliched interface may be easy and fun, it’s an ease
 similar to reading a formulaic romance novel. It means the
 interface does not reveal anything truly surprising about its
 subject area. And so it will do little to deepen the user’s
 understanding, or to change the way they think. For mundane tasks
 that is fine, but for deeper tasks, and for the longer term, you
 want a better interface.
 
 
 Ideally, an interface will surface the deepest principles
 underlying a subject, revealing a new world to the user. When you
 learn such an interface, you internalize those principles, giving
 you more powerful ways of reasoning about that world. Those
 principles are the diffs in your understanding. They’re all you
 really want to see, everything else is at best support, at worst
 unimportant dross. The purpose of the best interfaces isn’t to be
 user-friendly in some shallow sense. It’s to be user-friendly in
 a much stronger sense, reifying deep
 principles about the world, making them the working
 conditions in which users live and create. At that point what once
 appeared strange can instead becomes comfortable and familiar,
 part of the pattern of thoughtA powerful instance of
 these ideas is when an interface reifies general-purpose
 principles. An example is an
 interface one of us developed
 based on the principle of conservation of energy. Such
 general-purpose principles generate multiple unexpected
 relationships between the entities of a subject, and so are a
 particularly rich source of insights when reified in an
 interface..
 
 
 What does this mean for the use of AI models for intelligence
 augmentation?
 
 
 Aspirationally, as we’ve seen, our machine learning models will
 help us build interfaces which reify deep principles in ways
 meaningful to the user. For that to happen, the models have to
 discover deep principles about the world, recognize those
 principles, and then surface them as vividly as possible in an
 interface, in a way comprehensible by the user.
 
 
 Of course, this is a tall order! The examples we’ve shown are just
 barely beginning to do this. It’s true that our models do
 sometimes discover relatively deep principles, like the
 preservation of enclosed negative space when bolding a font. But
 this is merely implicit in the model. And while we’ve built a tool
 which takes advantage of such principles, it’d be better if the
 model automatically inferred the important principles learned, and
 found ways of explicitly surfacing them through the interface.
 (Encouraging progress toward this has been made
 by InfoGANs, which use
 information-theoretic ideas to find structure in the latent
 space.) Ideally, such models would start to get at true
 explanations, not just in a static form, but in a dynamic form,
 manipulable by the user. But we’re a long way from that point.
 
 
 Do these interfaces inhibit creativity?
 
 
 It’s tempting to be skeptical of the expressiveness of the
 interfaces we’ve described. If an interface constrains us to
 explore only the natural space of images, does that mean we’re
 merely doing the expected? Does it mean these interfaces can only
 be used to generate visual cliches? Does it prevent us from
 generating anything truly new, from doing truly creative work?
 
 
 To answer these questions, it’s helpful to identify two different
 modes of creativity. This two-mode model is over-simplified:
 creativity doesn’t fit so neatly into two distinct categories. Yet
 the model nonetheless clarifies the role of new interfaces in
 creative work.
 
 
 The first mode of creativity is the everyday creativity of a
 craftsperson engaged in their craft. Much of the work of a font
 designer, for example, consists of competent recombination of the
 best existing practices. Such work typically involves many
 creative choices to meet the intended design goals, but not
 developing key new underlying principles.
 
 
 For such work, the generative interfaces we’ve been discussing are
 promising. While they currently have many limitations, future
 research will identity and fix many deficiencies. This is
 happening rapidly with GANs: the original
 GANs had many limitations,
 but models soon appeared that were better adapted to
 images, improved the
 resolution, reduced artifactsSo much work has been
 done on improving resolution and reducing artifacts it seems
 unfair to single out any small set of papers, and to omit the many
 others., and so on. With enough iterations it’s
 plausible these generative interfaces will become powerful tools
 for craft work.
 
 
 The second mode of creativity aims toward developing new
 principles that fundamentally change the range of creative
 expression. One sees this in the work of artists such as Picasso
 or Monet, who violated existing principles of painting, developing
 new principles which enabled people to see in new ways.
 
 
 Is it possible to do such creative work, while using a generative
 interface? Don’t such interfaces constrain us to the space of
 natural images, or natural fonts, and thus actively prevent us
 from exploring the most interesting new directions in creative
 work?
 
 
 The situation is more complex than this.
 
 
 In part, this is a question about the power of our generative
 models. In some cases, the model can only generate recombinations
 of existing ideas. This is a limitation of an ideal GAN, since a
 perfectly trained GAN generator will reproduce the training
 distribution. Such a model can’t directly generate an image based
 on new fundamental principles, because such an image wouldn’t look
 anything like it’s seen in its training data.
 
 
 Artists such as Mario
 Klingemann and Mike
 Tyka are now using GANs to create interesting
 artwork. They’re doing that using “imperfect” GAN
 models, which they seem to be able to use to explore interesting
 new principles; it’s perhaps the case that bad GANs may be more
 artistically interesting than ideal GANs. Furthermore, nothing
 says an interface must only help us explore the latent space.
 Perhaps operations can be added which deliberately take us out
 of the latent space, or to less probable (and so more
 surprising) parts of the space of natural images.
 
 
 Of course, GANs are not the only generative models. In a
 sufficiently powerful generative model, the generalizations
 discovered by the model may contain ideas going beyond what humans
 have discovered. In that case, exploration of the latent space may
 enable us to discover new fundamental principles. The model would
 have discovered stronger abstractions than human experts. Imagine
 a generative model trained on paintings up until just before the
 time of the cubists; might it be that by exploring that model it
 would be possible to discover cubism? It would be an analogue to
 something like the prediction of Bose-Einstein condensation, as
 discussed earlier in the essay. Such invention is beyond today’s
 generative models, but seems a worthwhile aspiration for future
 models.
 
 
 Our examples so far have all been based on generative models. But
 there are some illuminating examples which are not based on
 generative models. Consider the pix2pix system developed
 by Isola et al. This
 system is trained on pairs of images, e.g., pairs showing the
 edges of a cat, and the actual corresponding cat. Once trained,
 it can be shown a set of edges and asked to generate an image for
 an actual corresponding cat. It often does this quite well:
 
 
 
 
 Input
 Output
 
 
 
 
 
 When supplied with unusual constraints, pix2pix can produce
 striking images:
 
 
 
 
 
 
  Spiral cat
 
 
 
 This is perhaps not high creativity of a Picasso-esque level. But
 it is still surprising. It’s certainly unlike images most of us
 have ever seen before. How does pix2pix and its human user achieve
 this kind of result?
 
 
 Unlike our earlier examples, pix2pix is not a generative model.
 This means it does not have a latent space or a corresponding
 space of natural images. Instead, there is a neural network,
 called, confusingly, a generator – this is not meant in the
 same sense as our earlier generative models – that takes as
 input the constraint image, and produces as output the filled-in
 image.
 
 
 The generator is trained adversarially against a discriminator
 network, whose job is to distinguish between pairs of images
 generated from real data, and pairs of images generated by the
 generator.
 
 
 While this sounds similar to a conventional GAN, there is a
 crucial difference: there is no latent vector input to the
 generatorActually, Isola et
 al experimented with adding such a latent vector to
 the generator, but found it made little difference to the
 resulting images.. Rather, there is simply an input
 constraint. When a human inputs a constraint unlike anything seen
 in training, the network is forced to improvise, doing the best it
 can to interpret that constraint according to the rules it has
 previously learned. The creativity is the result of a forced
 merger of knowledge inferred from the training data, together with
 novel constraints provided by the user. As a result, even
 relatively simple ideas – like the bread- and beholder-cats
 – can result in striking new types of images, images not
 within what we would previously have considered the space of
 natural images.
 
 Conclusion
 
 It is conventional wisdom that AI will change how we interact with
 computers. Unfortunately, many in the AI community greatly
 underestimate the depth of interface design, often regarding it as
 a simple problem, mostly about making things pretty or
 easy-to-use. In this view, interface design is a problem to be
 handed off to others, while the hard work is to train some machine
 learning system.
 
 
 This view is incorrect. At its deepest, interface design means
 developing the fundamental primitives human beings think and
 create with. This is a problem whose intellectual genesis goes
 back to the inventors of the alphabet, of cartography, and of
 musical notation, as well as modern giants such as Descartes,
 Playfair, Feynman, Engelbart, and Kay. It is one of the hardest,
 most important and most fundamental problems humanity grapples
 with.
 
 
 As discussed earlier, in one common view of AI our computers will
 continue to get better at solving problems, but human beings will
 remain largely unchanged. In a second common view, human beings
 will be modified at the hardware level, perhaps directly through
 neural interfaces, or indirectly through whole brain emulation.
 
 
 We’ve described a third view, in which AIs actually change
 humanity, helping us invent new cognitive technologies, which
 expand the range of human thought. Perhaps one day those
 cognitive technologies will, in turn, speed up the development of
 AI, in a virtuous feedback cycle:
 
 
 
 
 
 It would not be a Singularity in machines. Rather, it would be a
 Singularity in humanity’s range of thought. Of course, this loop
 is at present extremely speculative. The systems we’ve described
 can help develop more powerful ways of thinking, but there’s at
 most an indirect sense in which those ways of thinking are being
 used in turn to develop new AI systems.
 
 
 Of course, over the long run it’s possible that machines will
 exceed humans on all or most cognitive tasks. Even if that’s the
 case, cognitive transformation will still be a valuable end, worth
 pursuing in its own right. There is pleasure and value involved
 in learning to play chess or Go well, even if machines do it
 better. And in activities such as story-telling the benefit often
 isn’t so much the artifact produced as the process of construction
 itself, and the relationships forged. There is intrinsic value in
 personal change and growth, apart from instrumental benefits.
 
 
 The interface-oriented work we’ve discussed is outside the
 narrative used to judge most existing work in artificial
 intelligence. It doesn’t involve beating some benchmark for a
 classification or regression problem. It doesn’t involve
 impressive feats like beating human champions at games such as
 Go. Rather, it involves a much more subjective and
 difficult-to-measure criterion: is it helping humans think and
 create in new ways?
 
 
 This creates difficulties for doing this kind of work,
 particularly in a research setting. Where should one publish?
 What community does one belong to? What standards should be
 applied to judge such work? What distinguishes good work from
 bad?
 
 
 We believe that over the next few years a community will emerge
 which answers these questions. It will run workshops and
 conferences. It will publish work in venues such as Distill. Its
 standards will draw from many different communities: from the
 artistic and design and musical communities; from the mathematical
 community’s taste in abstraction and good definition; as well as
 from the existing AI and IA communities, including work on
 computational creativity and human-computer interaction. The
 long-term test of success will be the development of tools which
 are widely used by creators. Are artists using these tools to
 develop remarkable new styles? Are scientists in other fields
 using them to develop understanding in ways not otherwise
 possible? These are great aspirations, and require an approach
 that builds on conventional AI work, but also incorporates very
 different norms.
Using Artificial Intelligence to Augment Human Intelligence
https://distill.pub/2017/aia/
https://www.lesswrong.com/
LESSWRONG LW
2023-02-07
Ruby; Raemon; RobertM; Habryka
<div><div><div><p><div><div><p>When we describe rational thought processes, we tend to think of them as essentially deterministic, deliberate, and algorithmic. In this sequence of posts, the author argues that the process of cognition is closer to babbling many random strings and later filtering by a heuristic.</p></div><div></div></div></p></div><p><div><div><h2>Recent Discussion</h2></div><div><div><div><p>There are two boxes in front of you. In one of them, there is a little monkey with a cymbal, whilst the other box is empty. In precisely one hour the monkey will clang its cymbal.</p><p>While you wait, you produce an estimate of the probability of the monkey being in the first box. Let's assume that you form your last estimate, p, three seconds before the monkey clangs its cymbal. You can see the countdown and you know that it's your final estimate, partly because you're slow at arithmetic.</p><p>Let Omega be an AI that can perfectly simulate your entire deliberation process. Before you entered the room, Omega predicted what your last probability estimate would be and decided to place the monkey in a box such as to...</p></div><p><a href="/posts/Hn65bRY3BoXEW3bxL/the-unexpected-clanging">(See More – 331 more words)</a></p></div><div><div><p>Along with other AI governance researchers, I have been interested in better understanding historical examples of widespread international cooperation.</p><p>Perhaps the most well-studied analogy can be found in the case of nuclear energy &amp; nuclear weapons. I believe we can learn a lot from cooperation in the atomic &amp; nuclear eras (and I'm excited that we're already seeing some attempts to <a href="https://arxiv.org/abs/2304.04123v1">identify lessons learned</a>).</p><p>Below, you can read the <strong>Atoms for Peace</strong> <strong>Speech </strong>that President Eisenhower gave to the UN in 1953. The effect of the speech is <a href="https://en.wikipedia.org/wiki/Atoms_for_Peace#Effects_of_the_speech">complicated</a>. On one hand, the speech was followed by a series of programs and negotiations that eventually produced the <a href="https://en.wikipedia.org/wiki/International_Atomic_Energy_Agency">International Atomic Energy Agency</a> and the <a href="https://en.wikipedia.org/wiki/Treaty_on_the_Non-Proliferation_of_Nuclear_Weapons">Treaty on the Non-Proliferation of Nuclear Weapons</a>. On the other hand, it may have provided political...</p></div><p><a href="/posts/bdsCWKKDSnh8wNKS3/eisenhower-s-atoms-for-peace-speech">(Continue Reading – 3091 more words)</a></p></div><div><div><h2>Two-sentence summary </h2><p>In <a href="https://www.lesswrong.com/posts/pHfPvb4JMhGDr4B7n/recursive-middle-manager-hell">Recursive Middle Manager Hell</a>, Raemon argues that organisations get increasingly warped (immoral/maze-like/misaligned/...) as they gain more levels of hierarchy.<sup><a href="#fnmdb0vao72vo">[1]</a></sup>  I conjecture that analogous dynamics also arise in the context of AI automation.</p><hr /><p>To be frank, I think 95% of the value of my post comes just from the two-sentence tl;dr above. If you haven't done so yet, <strong>I recommend reading </strong><a href="https://www.lesswrong.com/posts/pHfPvb4JMhGDr4B7n/recursive-middle-manager-hell"><strong>Recursive Middle Manager Hell</strong></a><strong> and then asking "how does this change in the context of automating jobs by AI".</strong> But let me give some thoughts anyway.</p><h2>Automation so far</h2><p>[<i>low confidence, informal</i>] It seems to me that automation so far mostly worked like this:</p><ol><li><i>Mostly on the level of individual tasks</i> (eg, "Google creating a calendar event based on me receiving flight tickets via email", as opposed to "Google</li></ol><p>...</p></div><p><a href="/posts/jzHeZt3xFnGTudH5M/recursive-middle-manager-hell-ai-edition">(See More – 438 more words)</a></p></div><div><div><p>May 20th3 Bene't Street, Cambridge</p></div><div><p>We have these on the third Saturday of every month, whether you see an event here! Email me if you'd like to be added to the list to be notified every month.</p></div></div><div><div><p>(Feel free to leave comments with suggestions on how to improve this page.)</p><figure><div><i>﻿The road to wisdom? Well, it's plain</i><br /><i>﻿and simple to express:</i><p><i>﻿Err</i><br /><i>﻿and err</i><br /><i>﻿and err again</i><br /><i>﻿but <strong>less</strong></i><br /><i>﻿and <strong>less</strong></i><br /><i>﻿and <strong>less</strong>.</i></p><p>– Piet Hein</p></div></figure><h2>Why a new user guide?</h2><p><i><strong>Although encouraged, you don't have to read this to get started on LessWrong!</strong> </i></p><p>LessWrong is a pretty particular place. We strive to maintain a culture that's uncommon for web forums<sup><a href="#fnio4wu9v48cg">[1]</a></sup> and to stay true to our values. Recently, many more people have been finding their way here, so I (lead admin and moderator) put together this intro to what we're about.</p><p>My hope is that if LessWrong resonates with your values and interests, this guide will help you become a valued member of community. And if LessWrong isn't the place for you, this guide will help you...</p></div><p><a href="/posts/LbbrnRvc9QwjJeics/new-user-s-guide-to-lesswrong">(Continue Reading – 3160 more words)</a></p></div><p><div><p>To get the best posts emailed to you, create an account! </p><p>(2-3 posts per week, selected by the LessWrong moderation team.)</p></div></p><div><div><p>[Epistemic status: <em>Sharing current impressions in a quick, simplified way in case others have details to add or have a more illuminating account. Medium-confidence that this is one of the most important parts of the story.</em>]</p><p>Here's my current sense of how we ended up in this weird world where:</p><ul><li>I still intermittently run into people who claim that there's no such thing as reality or truth;</li><li>a lot of 20th-century psychologists made a habit of saying things like 'minds don't exist, only behaviors';</li><li>a lot of 20th-century physicists made a habit of saying things like 'quarks don't exist, only minds';</li><li>there's a big academic split between continental thinkers saying (or being rounded off to saying) some variant of "everything is culture / perception / discourse / power" and Anglophone thinkers saying</li></ul><p>...</p></div><p><a href="/posts/DSTuA9ohakD6k3drh/a-simple-sketch-of-how-realism-became-unpopular">(See More – 922 more words)</a></p></div><div><div><p><strong>Related to:</strong> Comments on<strong> </strong><a href="/lw/6w/degrees_of_radical_honesty/">Degrees of Radical Honesty</a>, OB: <a href="http://www.overcomingbias.com/2007/07/belief-in-belie.html">Belief in Belief</a>, <a href="http://www.overcomingbias.com/2007/10/cached-thoughts.html">Cached Thoughts</a>.</p>
<blockquote>
<p>"Nothing worse could happen to these labours than that anyone should make the unexpected discovery that there neither is, nor can be, any <em>a priori</em> knowledge at all.... This would be the same thing as if one sought to prove by reason that there is no reason" (<em>Critique of Practical Reason</em>, <cite></cite>Introduction).</p>
</blockquote>
<p>You don't need Kant to demonstrate the value of honesty. In fact, summoning his revenant can be a dangerous thing to do. You end up in the somewhat undesirable situation of having almost the right conclusion, but having it for the wrong reasons. Reasons you weren't even aware of, because they were all collapsed into the belief, "I believe in person X".</p><p>One of...</p></div><p><a href="/posts/WmAKcY22T4kdfppkr/you-don-t-need-kant">(Continue Reading – 1239 more words)</a></p></div></div></div></p></div></div>||||I|||| This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.

 LESSWRONG

 LW

 Login
 Home All Posts Concepts Library
 Sequence Highlights
 Rationality: A-Z
 The Codex
 HPMOR
 Best Of
 Community Events
 Zuzalu
 Fri Mar 24 • Tivat
 Effective Altruism Virtual Programs Jun-Jul 2023
 Sat May 20 • Online
 Argentines LW/SSC/EA/MIRIx - Call to All
 Tue Apr 18 • Online
 Weirdness Points
 Thu May 18 • Waterloo
 Subscribe (RSS/Email)
 About
 FAQ
 Home All Posts Concepts Library Community

 Recommendations

 Babble and Prune

 When we describe rational thought processes, we tend to think of them as essentially deterministic, deliberate, and algorithmic. In this sequence of posts, the author argues that the process of cognition is closer to babbling many random strings and later filtering by a heuristic.

 First Post: Babble
 376 Welcome to LessWrong!
 Ruby, Raemon, RobertM, habryka
 4y
 47
 405 Losing the root for the tree
 Adam Zerner
 7mo
 27
 249 Predictable updating about AI risk
 Joe Carlsmith
 2d
 12
 373 How much do you believe your results?
 Eric Neyman
 7d
 12

 Latest Posts

 Customize Feed (Hide) Customize Feed (Show)
 Rationality + Rationality + World Modeling + World Modeling + AI AI World Optimization World Optimization Practical Practical Community Community
 Personal Blog +
 76 What if they gave an Industrial Revolution and nobody came?
 jasoncrawford
 21h
 2
 117 Decision Theory with the Magic Parts Highlighted
 moridinamael
 2d
 15
 331 Steering GPT-2-XL by adding an activation vector Ω
 TurnTrout, Monte M, David Udell, lisathiergart, Ulisse Mini
 5d
 Ω
 51
 26 Activation additions in a simple MNIST network
 Garrett Baker
 14h
 0
 57 We learn long-lasting strategies to protect ourselves from danger and rejection
 Richard_Ngo
 2d
 3
 8 What are the limits of the weak man? Q
 ymeskhout
 16h
 Q
 2
 17 Eisenhower's Atoms for Peace Speech
 Akash
 1d
 2
 24 Conflicts between emotional schemas often involve internal coercion
 Richard_Ngo
 1d
 2
 77 Lessons learned from offering in-office nutritional testing
 Elizabeth
 3d
 10
 373 How much do you believe your results?
 Eric Neyman
 7d
 12
 41 New User's Guide to LessWrong
 Ruby
 2d
 38
 230 Mental Health and the Alignment Problem: A Compilation of Resources (updated April 2023)
 Chris Scammell, DivineMango
 8d
 48
 249 Predictable updating about AI risk
 Joe Carlsmith
 2d
 12
 Load MoreAdvanced Sorting/Filtering

 Recent Discussion

 Portia's Shortform
 Portia
 2mo
 3 Portia 1h
 No, I do not mean sentience is identical with "reacting and planning". I am saying that in biological organisms, it is a prerequisite for some kinds of reacting and planning - namely the one rationalists tend to be most interested in. The idea is that phenomenal consciousness works as an input for reasoning; distils insights from unconscious processing into a format for slow analysis. I'm not sure what you mean by "objective sensations". I suspect that at the core, our disagreement starts with the fact that I do not see sentience as something that happens extraneously on top of functional processes, but rather as something identical with some functional processes, with the processes which are experienced by subjects and reported by them as such sharing tangible characteristics. This is supported primarily by the fact that consciousness can be quite selectively disrupted while leaving unconscious processing intact, but that this correlates with a distinct loss in rational functioning; fast automatic reactions to stimuli still work fine, even though the humans tell you they cannot see them - but a rational, planned, counter-intuitive response does not, because you rational mind no longer has access to the necessary information. The fact that sentience is subjectively experienced with valence and hence entails suffering is of incredible ethical importance, but the idea that this experience can be divorced from function, that you could have a perfectly functioning brain doing exactly what your brain does while consciousness never arises or is extinguished without any behavioural consequence (epiphenomenalism, zombies) runs into logical self-contradictions, and is without empirical support. Consciousness itself enables you to do different stuff which you cannot do without it. (Or at least, a brain running under biological constraints cannot; AI might be currently bruteforcing alternative solutions which are too grossly inefficient to realistically be sustainable for
 Dagon 16m 2 0

 I think I'll bow out for now - I'm not certain I understand precisely where we disagree, but it seems to be related to whether "phenomenal consciousness works as an input for reasoning;" is a valid statement, without being able to detect or operationally define "consciousness".

 Reply
 3 Portia 2h
 Yes, those are the papers. On the survey: On the question of whether consciousness itself fulfils a function that evolution has selected for, while highly plausible, is not obvious, and has been disputed. The common argument against it is the fact that polar bear coats are heavy, so one could ask whether evolution has selected for heavyness. And of course, it has not - the weight is detrimental - but is has selected for a coat that keeps a polar bear warm in their incredibly cold environment, and the random process there failed to find a coat that was sufficiently warm, but significantly lighter, and also scoring high on other desirable aspects. But in this case, the heavyness of the coat is a negative side consequence of a trait that was actually selected for. And we can conceive of coats that are warm, but lighter. The distinction may seem persnickety, but it isn't, it has profound implications. In one scenario, consciousness could be an itself valueless side product of a development that was actually useful (some neat brain process, perhaps), but the consciousness itself plays no functional role. One important implication of this would be that it would not be possible to identify consciousness based on behaviour, because it would not affect behaviour. This is the idea of epiphenomenalism - basically, that there is a process running in your brain that is actually what matters for your behaviour, but the process of its running also, on the side, leads to a subjective experience, which is itself irrelevant - just generated, the way that a locomotive produces steam. While epiphenomenalism leads you into absolutely absurd territory (zombies), there are a surprising number of scientists who have historically essentially prescribed to it, because it allows you to circumvent a bunch of hard questions. You can continue to imagine consciousness as a mysterious, unphysical thing that does not have to be translated into math, because it does not really exist on a physical
 3 Portia 2h
 Something a bit like this has actually been done, with a proper scientific analysis, but without human players so far. (Or at least I am not aware of the latter, but I frankly can no longer keep up with all the applications.) They (Park et al 2023 https://arxiv.org/abs/2304.03442 [https://arxiv.org/abs/2304.03442] ) populated a tiny, Sims-style world with ChatGPT-controlled AIs, enabled them to store a complete record of agent interactions in natural language, synthesise them into conclusions, and draw upon them to generate behaviours - and let them interact with each other. Not only did they not go of the rails - they performed daily routines, and improvised in a a matter consistent with their character backstories when they ran into each other, eerily like in Westworld. It also illustrated another interesting point that Westworld had made - the strong impact of the ability to form memories on emergent, agentic behaviours. The thing that stood out is that characters within the world managed to coordinate a party - come up with the idea that one should have one, where it would be, when it would be, inform each other that such a decision had been taken, invite each other, invite friends of friends - and that a bunch of them showed up in the correct location on time. The conversations they were having affected their actions appropriately. There is not just a complex map of human language that is self-referential; there are also references to another set of actions, in this case, navigating this tiny world. It does not yet tick the biological and philosophical boxes for characteristics that have us so interested in embodiment, but it definitely adds another layer. And then we have analysis of and generation of pictures, which, in turn, is also related to the linguistic maps. One thing that floored me was an example from a demo by OpenAI itself where ChatGPT was shown an image of a heavy object, I think a car, that had a bunch of balloons tied to it with string, ball
 The Unexpected Clanging
 5
 Chris_Leong
 Ω 2 Decision Theory World Modeling 2h

 There are two boxes in front of you. In one of them, there is a little monkey with a cymbal, whilst the other box is empty. In precisely one hour the monkey will clang its cymbal.

 While you wait, you produce an estimate of the probability of the monkey being in the first box. Let's assume that you form your last estimate, p, three seconds before the monkey clangs its cymbal. You can see the countdown and you know that it's your final estimate, partly because you're slow at arithmetic.

 Let Omega be an AI that can perfectly simulate your entire deliberation process. Before you entered the room, Omega predicted what your last probability estimate would be and decided to place the monkey in a box such as to...

 (See More – 331 more words)
 Dagon 19m 2 -2

 Adversarial simulation by a very powerful entity is so obviously unsolvable that I'm caught off-guard every time it comes up.

 Let Omega be an AI that can perfectly simulate your entire deliberation process ... such as to mess with you

 Ok, the result is that you're messed with. Can we move on to more interesting questions?

 Note that it does get a little more interesting (possibly not above 0, though), if you remove the anthropomorphization of Omega, either remove or model fully your time-bounded process (I think remove here, as there's no change d... (read more)

 Reply
 4 Vladimir_Nesov 1h
 There is incentive for hidden expectation/cognition that Omega isn't diagonalizing (things like creating new separate agents in the environment). Also, at least you can know how ground truth depends on official "expectation" of ground truth. Truth of knowledge of this dependence wasn't diagonalized away, so there is opportunity for control.
 Eisenhower's Atoms for Peace Speech
 17
 Akash
 AI Governance History World Modeling
 Frontpage
 1d
 This is a linkpost for https://www.iaea.org/about/history/atoms-for-peace-speech

 Along with other AI governance researchers, I have been interested in better understanding historical examples of widespread international cooperation.

 Perhaps the most well-studied analogy can be found in the case of nuclear energy & nuclear weapons. I believe we can learn a lot from cooperation in the atomic & nuclear eras (and I'm excited that we're already seeing some attempts to identify lessons learned).

 Below, you can read the Atoms for Peace Speech that President Eisenhower gave to the UN in 1953. The effect of the speech is complicated. On one hand, the speech was followed by a series of programs and negotiations that eventually produced the International Atomic Energy Agency and the Treaty on the Non-Proliferation of Nuclear Weapons. On the other hand, it may have provided political...

 (Continue Reading – 3091 more words)
 Nathan1123 19m 1 0

 While this is an example of international cooperation in the face of mutually-assured destruction, there is some historical context that shows why this is effective, in my opinion: First, because the destructive power of nuclear weapons was already realized by Hiroshima and Nagasaki. Second, because the US no longer had a monopoly on nuclear weapons and determined that overcoming the USSR by force was no longer an option. Third, because the world was still recovering from the Second World War and shared a universal desire for peaceful resolutions.

 Reply
 Recursive Middle Manager Hell: AI Edition
 29
 VojtaKovarik
 AI Services (CAIS) Moral Mazes Moloch Middle management AI
 Frontpage
 14d

 Two-sentence summary

 In Recursive Middle Manager Hell, Raemon argues that organisations get increasingly warped (immoral/maze-like/misaligned/...) as they gain more levels of hierarchy.[1] I conjecture that analogous dynamics also arise in the context of AI automation.

 To be frank, I think 95% of the value of my post comes just from the two-sentence tl;dr above. If you haven't done so yet, I recommend reading Recursive Middle Manager Hell and then asking "how does this change in the context of automating jobs by AI". But let me give some thoughts anyway.

 Automation so far

 [low confidence, informal] It seems to me that automation so far mostly worked like this:

 1. Mostly on the level of individual tasks (eg, "Google creating a calendar event based on me receiving flight tickets via email", as opposed to "Google
 ...
 (See More – 438 more words)
 VojtaKovarik 22m 1 0

 I agree that the general point (biology needs to address similar issues, so we can use it for inspiration) is interesting. (Seems related to https://www.pibbss.ai/ .)

 That said, I am somewhat doubtful about the implied conclusion (that this is likely to help with AI, because it won't mind): (1) there are already many workspace practices that people don't like, so "people liking things" doesn't seem like a relevant parameter of design, (2) (this is totally vague, handwavy, and possibly untrue, but:) biological processes might also not "like" being surveiled, replaced, etc, so the argument proves too much.

 Reply
 Cambridge ACX/SSC monthly meetup
 May 20th 3 Bene't Street, Cambridge
 hamishtodd1

 We have these on the third Saturday of every month, whether you see an event here! Email me if you'd like to be added to the list to be notified every month.

 hamishtodd1 34m 1 0

 REMINDER! Saturday, 2pm, Bath House, LessWrong/AstralCodexTen/SlateStarCodex! Looking forward to seeing y'all!

 Reply
 New User's Guide to LessWrong
 41
 Ruby
 Site Meta
 Frontpage
 2d

 (Feel free to leave comments with suggestions on how to improve this page.)

 ﻿The road to wisdom? Well, it's plain
 ﻿and simple to express: 
 
 ﻿Err 
 ﻿and err 
 ﻿and err again 
 ﻿but less 
 ﻿and less 
 ﻿and less. 
 
 – Piet Hein 
 

 Why a new user guide?

 Although encouraged, you don't have to read this to get started on LessWrong!

 LessWrong is a pretty particular place. We strive to maintain a culture that's uncommon for web forums[1] and to stay true to our values. Recently, many more people have been finding their way here, so I (lead admin and moderator) put together this intro to what we're about.

 My hope is that if LessWrong resonates with your values and interests, this guide will help you become a valued member of community. And if LessWrong isn't the place for you, this guide will help you...

 (Continue Reading – 3160 more words)
 RomanHauksson 1h 1 0

 We take tending the garden seriously

 Ironic typo: the link includes the proceeding space.

 Reply
 1 Mariven 12h
 You mustn't juxtapose "shaping anticipations of what you'll observe" against "signaling group affiliation" and "having pleasant feelings". This is how you get (or, perhaps, got) a community of people who merely signal group affiliation to Team Science!! by espousing the same naive ultra-falsificationism that, if it were anything more than a signal, would lead them to reject all thought of x-risk, >95% of the Sequences, and falsificationism itself. The human mind takes the path of least resistance, and such a motto makes itself into said path precisely by becoming a signal rather than a protocol. If "what you'll observe" is expanded to cover the veracity of the entire cognitive-perceptive process in its interaction with reality, then "shaping anticipations of what you'll observe" is the correct goal of belief, and naturally implies things like probabilistic (Bayesian) estimation and falsificationism on the object-level, and is a goal that juxtaposes itself against "signaling group affiliation" and "having pleasant feelings" but also against automatically assuming the fidelity of object-level perceptions and the coherence of the cognitive frameworks that shape their construction and use. But the phrase is never interpreted in this way. Besides, it's rare that anybody in the intended audience with beliefs that signal group affiliation or make them feel pleasant would ever be led to admit such a thing to themselves, let alone others. I don't yet have a solution for this, but I'd overwrite "We treat beliefs as being about [...]" with "We treat the purpose of beliefs as convergence to the necessarily singular and internally coherent truth of reality" (or something more concise).
 To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
 Subscribe to Curated posts
 Log In Reset Password
 ...or continue with
 FACEBOOKGOOGLEGITHUB
 Some background for reasoning about dual-use alignment research
 10
 Charlie Steiner
 Ω 7 AI 2h

 This is pretty basic. But I still made a bunch of mistakes when writing this, so maybe it's worth writing. This is background to a specific case I'll put in the next post.

 It's like a a tech tree

 If we're looking at the big picture, then whether some piece of research is net positive or net negative isn't an inherent property of that research; it depends on how that research is situated in the research ecosystem that will eventually develop superintelligent AI.

 A tech tree, with progress going left to right. Blue research is academic, green makes you money, red is a bad ending, yellow is a good ending. Stronger connections are more important prerequisites.

 Consider this toy game in the picture. We start at the left and can unlock...

 (Continue Reading – 2534 more words)
 A simple sketch of how realism became unpopular
 60
 Rob Bensinger
 History Truth, Semantics, & Meaning Philosophy
 Frontpage
 4y

 [Epistemic status: Sharing current impressions in a quick, simplified way in case others have details to add or have a more illuminating account. Medium-confidence that this is one of the most important parts of the story.]


 Here's my current sense of how we ended up in this weird world where:

 * I still intermittently run into people who claim that there's no such thing as reality or truth;
 * a lot of 20th-century psychologists made a habit of saying things like 'minds don't exist, only behaviors';
 * a lot of 20th-century physicists made a habit of saying things like 'quarks don't exist, only minds';
 * there's a big academic split between continental thinkers saying (or being rounded off to saying) some variant of "everything is culture / perception / discourse / power" and Anglophone thinkers saying
 ...
 (See More – 922 more words)
 TAG 2h 2

 Not only is Kant basing his philosophical views on the science of his day, but Kantian philosophical conclusions are being promoted as science, by scientists.

 "The idea that our senses (or, indeed, those of any conscious agent) don’t disclose reality is the main idea of The Case Against Reality by Donald Hoffman, a cognitive scientist at UC Irvine, who argues that in addition to its ancient pedigree, this counterintuitive view of reality and perception is also supported by the latest science."

 https://docs.google.com/document/d/1AtGIIv371v0Yu35eNsIxJr67dw4SHOiGdKrqmoKt2hg/edit

 Reply
 2 TAG 2h
 Hoffman's book has been reviewed as part of Scott's review competition. https://docs.google.com/document/d/1AtGIIv371v0Yu35eNsIxJr67dw4SHOiGdKrqmoKt2hg/edit [https://docs.google.com/document/d/1AtGIIv371v0Yu35eNsIxJr67dw4SHOiGdKrqmoKt2hg/edit]
 You don't need Kant
 21 [anonymous] Philosophy Honesty
 Personal Blog
 14y

 Related to: Comments on Degrees of Radical Honesty, OB: Belief in Belief, Cached Thoughts.

 "Nothing worse could happen to these labours than that anyone should make the unexpected discovery that there neither is, nor can be, any a priori knowledge at all.... This would be the same thing as if one sought to prove by reason that there is no reason" (Critique of Practical Reason, Introduction).

 You don't need Kant to demonstrate the value of honesty. In fact, summoning his revenant can be a dangerous thing to do. You end up in the somewhat undesirable situation of having almost the right conclusion, but having it for the wrong reasons. Reasons you weren't even aware of, because they were all collapsed into the belief, "I believe in person X".

 One of...

 (Continue Reading – 1239 more words)
 TAG 3h 2

 1. Maths has a body of known results, philosophy doesn't.

 2. Philosophy teaching varies a lot. Analytical philosophy is argument heavy.

 Reply
When we describe rational thought processes, we tend to think of them as essentially deterministic, deliberate, and algorithmic. In this sequence of posts, the author argues that the process of cognition is closer to babbling many random strings and later filtering by a heuristic.          Recent Discussion     There are two boxes in front of you. In one of them, there is a little monkey with a cymbal, whilst the other box is empty. In precisely one hour the monkey will clang its cymbal. While you wait, you produce an estimate of the probability of the monkey being in the first box. Let's assume that you form your last estimate, p, three seconds before the monkey clangs its cymbal. You can see the countdown and you know that it's your final estimate, partly because you're slow at arithmetic. Let Omega be an AI that can perfectly simulate your entire deliberation process. Before you entered the room, Omega predicted what your last probability estimate would be and decided to place the monkey in a box such as to...   (See More – 331 more words)     Along with other AI governance researchers, I have been interested in better understanding historical examples of widespread international cooperation. Perhaps the most well-studied analogy can be found in the case of nuclear energy &amp; nuclear weapons. I believe we can learn a lot from cooperation in the atomic &amp; nuclear eras (and I'm excited that we're already seeing some attempts to identify lessons learned). Below, you can read the Atoms for Peace Speech that President Eisenhower gave to the UN in 1953. The effect of the speech is complicated. On one hand, the speech was followed by a series of programs and negotiations that eventually produced the International Atomic Energy Agency and the Treaty on the Non-Proliferation of Nuclear Weapons. On the other hand, it may have provided political...   (Continue Reading – 3091 more words)     Two-sentence summary  In Recursive Middle Manager Hell, Raemon argues that organisations get increasingly warped (immoral/maze-like/misaligned/...) as they gain more levels of hierarchy. [1]   I conjecture that analogous dynamics also arise in the context of AI automation.  To be frank, I think 95% of the value of my post comes just from the two-sentence tl;dr above. If you haven't done so yet, I recommend reading   Recursive Middle Manager Hell   and then asking "how does this change in the context of automating jobs by AI". But let me give some thoughts anyway. Automation so far [low confidence, informal] It seems to me that automation so far mostly worked like this:   Mostly on the level of individual tasks (eg, "Google creating a calendar event based on me receiving flight tickets via email", as opposed to "Google  ...   (See More – 438 more words)     May 20th3 Bene't Street, Cambridge   We have these on the third Saturday of every month, whether you see an event here! Email me if you'd like to be added to the list to be notified every month.     (Feel free to leave comments with suggestions on how to improve this page.)   ﻿The road to wisdom? Well, it's plain  ﻿and simple to express:  ﻿Err  ﻿and err  ﻿and err again  ﻿but less   ﻿and less   ﻿and less.  – Piet Hein   Why a new user guide?   Although encouraged, you don't have to read this to get started on LessWrong!   LessWrong is a pretty particular place. We strive to maintain a culture that's uncommon for web forums [1]  and to stay true to our values. Recently, many more people have been finding their way here, so I (lead admin and moderator) put together this intro to what we're about. My hope is that if LessWrong resonates with your values and interests, this guide will help you become a valued member of community. And if LessWrong isn't the place for you, this guide will help you...   (Continue Reading – 3160 more words)     To get the best posts emailed to you, create an account!  (2-3 posts per week, selected by the LessWrong moderation team.)     [Epistemic status: Sharing current impressions in a quick, simplified way in case others have details to add or have a more illuminating account. Medium-confidence that this is one of the most important parts of the story.] Here's my current sense of how we ended up in this weird world where:  I still intermittently run into people who claim that there's no such thing as reality or truth; a lot of 20th-century psychologists made a habit of saying things like 'minds don't exist, only behaviors'; a lot of 20th-century physicists made a habit of saying things like 'quarks don't exist, only minds'; there's a big academic split between continental thinkers saying (or being rounded off to saying) some variant of "everything is culture / perception / discourse / power" and Anglophone thinkers saying  ...   (See More – 922 more words)      Related to: Comments on  Degrees of Radical Honesty, OB: Belief in Belief, Cached Thoughts.
"Nothing worse could happen to these labours than that anyone should make the unexpected discovery that there neither is, nor can be, any a priori knowledge at all.... This would be the same thing as if one sought to prove by reason that there is no reason" (Critique of Practical Reason,  Introduction).
You don't need Kant to demonstrate the value of honesty. In fact, summoning his revenant can be a dangerous thing to do. You end up in the somewhat undesirable situation of having almost the right conclusion, but having it for the wrong reasons. Reasons you weren't even aware of, because they were all collapsed into the belief, "I believe in person X". One of...   (Continue Reading – 1239 more words)
LESSWRONG LW
https://www.lesswrong.com/
https://research.ibm.com/blog
Blog | IBM Research
2021-02-09T00:00:00Z

<html><body><div><div><section><h2>Filter by</h2><nav><ol><li><a href="https://research.ibm.com/">Home</a></li><li><a href="https://research.ibm.com/blog">Blog</a></li></ol></nav><section><h3>Categories</h3><ul><li></li><li></li><li></li><li></li><li></li><li></li></ul></section><section><h3>Popular topics</h3><ul><ul><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li></ul></ul></section></section><article><h2><a href="https://research.ibm.com/blog/what-is-red-teaming-gen-AI"><span>What is red teaming for generative AI?</span></a></h2><p><span></span></p><div><p>Explainer</p></div><div><p>Kim Martineau</p><p><time>11 Apr 2024</time></p></div><ul><li><p><span>Adversarial Robustness and Privacy</span></p></li><li><p><span>AI</span></p></li><li><p><span>AI Testing</span></p></li><li><p><span>Fairness, Accountability, Transparency</span></p></li><li><p><span>Foundation Models</span></p></li><li><p><span>Natural Language Processing</span></p></li><li><p><span>Security</span></p></li><li><p><span>Trustworthy AI</span></p></li></ul></article></div></div></body></html>
Filter by    Home   Blog     Categories                 Popular topics                          What is red teaming for generative AI?        Explainer   Kim Martineau  11 Apr 2024      Adversarial Robustness and Privacy     AI     AI Testing     Fairness, Accountability, Transparency     Foundation Models     Natural Language Processing     Security     Trustworthy AI
Blog | IBM Research
https://www.ibm.com/blogs/research/2020/02/mastering-language-natural-human-ai-interaction/
https://escapingflatland.substack.com/p/gpt-3
Using GPT-3 to augment human intelligence
2022-08-10
Henrik Karlsson
<div><div><article><div><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F781c7a48-c404-4dd1-8556-c9d539a40a36_900x675.jpeg"><div></div></a></figure></div><p>A blog post is a very long and complex <a href="https://escapingflatland.substack.com/p/search-query">search query to find fascinating people and make them route interesting stuff to your inbox</a>.</p><p>It is like summoning an alien intelligence. I post at night, after putting my kids to bed, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy. But more often than not, what people have sent helps accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting <a href="https://escapingflatland.substack.com/">my Substack</a>, than in the five previous years combined.</p><p>As Slime Mold Time Mold say, <a href="https://slimemoldtimemold.com/2022/05/09/job-posting-reddit-research-czar/">Enough eyeballs, and all research bugs are shallow</a>.</p><p>The limiting factor is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber!</p><p>Therefore, it is intriguing to realize what I am doing is, in fact, <em>prompt engineering</em>.</p><p>Prompt engineering is the term AI researchers use for <em>the art of writing prompts that make a large language model output what you want.</em> Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started.</p><p>When I’ve been doing this with <a href="https://en.wikipedia.org/wiki/GPT-3">GPT-3,</a> a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping.</p><p>Soon everyone can have an alien intelligence in their inbox.</p><p>This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet.</p><p>In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge.</p><p>Some people claim to already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote</p><p>). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work.</p><p>In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in <em>The Exorcist, </em>language models only adds enough truth to twist our minds and make us do stupid things <a href="https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917">like claiming it is sentient and getting ourselves suspended</a>.</p><p>Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that <em>if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating</em>—which is the technical term for <em>making stuff up.</em></p><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F417bd6b3-9a58-444e-96d7-af2cb21f8e8b_860x402.png"><div></div></a></figure></div><p>This, however, does not mean you cannot use it as an epistemic tool. It just means <em>you need to be pretty sophisticated to wrangle the truth out of it.</em></p><p>As <a href="https://interconnected.org/home/2020/09/04/idea_machine">Matt Webb writes</a>, in a slightly different context:</p><blockquote><p><strong>Using GPT-3 is work</strong>, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to <em>interview</em> GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.</p></blockquote><p>What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about <a href="https://distill.pub/2017/aia/">using artificial intelligence to augment your thinking</a>.</p><p>It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more <a href="https://ai.googleblog.com/2021/03/progress-and-challenges-in-long-form.html">finetuned for factual accuracy</a>, before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties.</p><p>To access GPT-3, you <a href="https://auth0.openai.com/u/signup/identifier?state=hKFo2SBhLWFra3RDMDFQOVo0LWM0Nk5Oc1ozR1QyM0Q5Nk1UU6Fur3VuaXZlcnNhbC1sb2dpbqN0aWTZIDJObHFmOFh0aUdrQU9PNklBYjlDcWdKYW8ybHZhdm8xo2NpZNkgRFJpdnNubTJNdTQyVDNLT3BxZHR3QjNOWXZpSFl6d0Q">set up an account</a> at OpenAI. Then you click on Playground, which brings you to this workspace:</p><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fcde29d69-6921-47de-aaf7-70621f3baef0_1894x1116.png"><div></div></a></figure></div><p>In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting <em>text-davinci-002, </em>which is the most capable model and has been fine-tuned to <a href="https://openai.com/blog/instruction-following/#fn1)">understand human intentions</a>. This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them.</p><p>What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast.</p><p>If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.”</p><p>And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.”</p><p>In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the <em>way</em> you access it is radically different. We can think of it as a new interface for the internet.</p><p>Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet.</p><p>But this comes at a cost, which currently is steep. If I would have gone to <a href="https://en.wikipedia.org/wiki/Pope_Linus">the Wikipedia page of Pope Linus</a>, instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope.</p><p>Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 <em>is</em> unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself.</p><p>Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay.</p><p>Maybe we can use GPT-3 to answer these questions instead?</p><p>Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions.</p><p>I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.</p><blockquote><p><strong>Write an essay about how we can structure online education around apprenticeship models and communities of practice</strong></p><p>The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education.</p><p>The apprenticeship model has several key features that make it well suited for online education:</p><ol><li><p>Apprenticeship is a hands-on, experiential learning model.</p></li><li><p>Apprenticeship is based on the mentorship of a more experienced practitioner.</p></li><li><p>Apprenticeship emphasizes learning by doing, rather than passively receiving information.</p></li><li><p>Apprenticeship is social, collaborative, and often takes place in a community of practice.</p></li><li><p>Apprenticeship is flexible and can be adapted to individual learners' needs and interests.</p></li><li><p>Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills.</p></li><li><p>Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.</p></li></ol><p>The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.</p><p><strong>This can be done in several ways.</strong> For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.</p><p><strong>There are several problems with this, however.</strong> First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.</p><p><strong>One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is</strong> to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. <strong>Another way is</strong> to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.</p></blockquote><p>What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path.</p><p>When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample.</p><p>So, did GPT-3 teach me anything new here?</p><p>I have already written <a href="https://escapingflatland.substack.com/p/apprenticeship-online?s=w">an essay about online apprenticeships</a> (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made.</p><p>Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote</p><p>.)</p><p>How can we prompt GPT-3 to surface knowledge that is hard to find by googling?</p><p><strong>Make GPT-3 impersonate two eminent thinkers and have them debate a point. </strong>Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else.</p><p>You can use this property to set up debates between people, as a tool to explore a topic from several angles:</p><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2d6c8df7-bb2a-4888-9c1b-73960989dbef_1346x1358.png"><div></div></a></figure></div><p><em>This debate was prompt engineered by <a href="https://twitter.com/nickcammarata">Nick Cammarata</a>, a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.</em></p><p><strong>Make GPT-3 give feedback on your ideas. </strong>You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.</p><p><strong>Linguistic bubbles.</strong><em> </em>Being that the neural net will take different personas depending on how you talk to it, there is a risk of <em>linguistic bubbles. </em>This is a phrase the user <a href="https://www.lesswrong.com/users/viliam">Viliam</a> used when I prompted LessWrong to discuss how large language models will affect learning.</p><p>Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes.</p><p>You want to talk about evolution, but you happen to use the phrase <em>irreducible complexity </em>instead<em> -</em> which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult.</p><p>Stranger things have happened.</p><p>I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.”</p><p>If you <em>want</em> to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it <em>is </em>Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in <em>that</em> context, you ask GPT-3 to explain the irreducible complexity of nature.</p><p>“Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!”</p><p>You can get linguistic bubbles, but it takes some effort.</p><p><strong>Ask GPT-3 for counterarguments against a claim. </strong>If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google.</p><p>I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?”</p><p>If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.</p><p><strong>A few more prompt patterns that can help you catch factual errors in GPT-3 output: </strong></p><ol><li><p>You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the <em>temperature </em>dial.</p></li><li><p>You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of<a href="https://beta.openai.com/playground/p/iyhUcPBpoUGRync7pZEhSUJf?model=text-davinci-002"> an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful</a>.</p></li><li><p>You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate.</p></li><li><p>It also sometimes helps to add "reason step by step" to the prompt when applicable.</p></li></ol><p><strong>You can ask newer language models to provide links to research papers to back up their claims. </strong>With newer language models (like <a href="https://www.deepmind.com/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval">Gopher</a> or <a href="https://openai.com/blog/webgpt/">this experimental version of GPT-3</a>), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.”</p><p>Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:</p><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fe46c9036-75c0-431a-81b8-12c83825af14_860x612.png"><div></div></a></figure></div><p><strong>Prompt GPT-3 to draw analogies between knowledge domains. </strong>Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will <em>do</em> that.</p><p>As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in <a href="https://twitter.com/nickcammarata/status/1520815612477550593?s=20&amp;t=OGx3iqhgDu5b3fdShz4h6Q">this Twitter thread</a>:</p><p>GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates.</p><p>It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall!</p><p>For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into <em>hallucinations</em>, which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about.</p><p>Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge?</p><p>This is an open question. Gary Marcus is <a href="https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails?s=r">highly skeptical</a>. <a href="https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails?s=r">Scott Alexander less so</a>.</p><p>Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with <a href="https://arxiv.org/abs/2203.15556">Chinchilla</a>, where <a href="https://twitter.com/TShevlane">Toby Shevlane</a>, a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:</p><div><figure><a href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F5ff31ed1-0840-4369-a3f8-98d6edcf1ee9_491x476.png"><div></div></a></figure></div><p>“In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate</p><p>. But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details.</p><p>And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">PaLM</a>, a 540 billion parameter model, and likely others that I am unaware of too.</p><p>With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence.</p><p>We don’t know what it will do with us out there.</p><p><em>If you’ve made it all the way down here and do not feel that you’ve just wasted half an hour, consider scrolling back up to like the essay. It helps others find it. And it makes me happy.</em></p><p><em>/Henrik</em></p><p><em>This essay benefited from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.</em></p></div></article></div>||||I|||| Escaping Flatland

 SubscribeSign in
 Share this post

 Using GPT-3 to augment human intelligence

 escapingflatland.substack.com
 Copy link
 Twitter
 Facebook
 Email

 Using GPT-3 to augment human intelligence

 Henrik Karlsson
 Aug 10, 2022
 45
 2
 Share this post

 Using GPT-3 to augment human intelligence

 escapingflatland.substack.com
 Copy link
 Twitter
 Facebook
 Email

 A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox .

 It is like summoning an alien intelligence. I post at night, after putting my kids to bed, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy. But more often than not, what people have sent helps accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting my Substack , than in the five previous years combined.

 As Slime Mold Time Mold say, Enough eyeballs, and all research bugs are shallow .

 The limiting factor is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber!

 Therefore, it is intriguing to realize what I am doing is, in fact, prompt engineering .

 Prompt engineering is the term AI researchers use for the art of writing prompts that make a large language model output what you want. Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started.

 When I’ve been doing this with GPT-3, a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping.

 Soon everyone can have an alien intelligence in their inbox.

 GPT-3 as a tool for thought

 This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet.

 In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge.

 Subscribe

 Some people claim to already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote

 1
 ). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work.

 In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in The Exorcist, language models only adds enough truth to twist our minds and make us do stupid things like claiming it is sentient and getting ourselves suspended .

 Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating —which is the technical term for making stuff up.

 This, however, does not mean you cannot use it as an epistemic tool. It just means you need to be pretty sophisticated to wrangle the truth out of it.

 As Matt Webb writes , in a slightly different context:

 Using GPT-3 is work , it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.

 What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about using artificial intelligence to augment your thinking .

 It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more finetuned for factual accuracy , before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties.

 How to access GPT-3

 To access GPT-3, you set up an account at OpenAI. Then you click on Playground, which brings you to this workspace:

 In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting text-davinci-002, which is the most capable model and has been fine-tuned to understand human intentions . This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them.

 What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast.

 If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.”

 And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.”

 In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the way you access it is radically different. We can think of it as a new interface for the internet.

 Language models are internet interfaces

 Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet.

 But this comes at a cost, which currently is steep. If I would have gone to the Wikipedia page of Pope Linus , instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope.

 Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 is unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself.

 Using GPT-3 as a research assistant

 Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay.

 Maybe we can use GPT-3 to answer these questions instead?

 Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions.

 I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.

 Write an essay about how we can structure online education around apprenticeship models and communities of practice

 The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education.

 The apprenticeship model has several key features that make it well suited for online education:

 1. Apprenticeship is a hands-on, experiential learning model.

 2. Apprenticeship is based on the mentorship of a more experienced practitioner.

 3. Apprenticeship emphasizes learning by doing, rather than passively receiving information.

 4. Apprenticeship is social, collaborative, and often takes place in a community of practice.

 5. Apprenticeship is flexible and can be adapted to individual learners' needs and interests.

 6. Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills.

 7. Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.

 The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.

 This can be done in several ways. For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.

 There are several problems with this, however. First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.

 One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. Another way is to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.

 What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path.

 When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample.

 So, did GPT-3 teach me anything new here?

 I have already written an essay about online apprenticeships (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made.

 2

 Some useful prompt patterns, and some anti-patterns to avoid

 Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote

 3
 .)

 How can we prompt GPT-3 to surface knowledge that is hard to find by googling?

 Make GPT-3 impersonate two eminent thinkers and have them debate a point. Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else.

 You can use this property to set up debates between people, as a tool to explore a topic from several angles:

 This debate was prompt engineered by Nick Cammarata , a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.

 Make GPT-3 give feedback on your ideas. You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.

 Linguistic bubbles. Being that the neural net will take different personas depending on how you talk to it, there is a risk of linguistic bubbles. This is a phrase the user Viliam used when I prompted LessWrong to discuss how large language models will affect learning.

 Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes.

 You want to talk about evolution, but you happen to use the phrase irreducible complexity instead - which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult.

 Stranger things have happened.

 I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.”

 If you want to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it is Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in that context, you ask GPT-3 to explain the irreducible complexity of nature.

 “Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!”

 You can get linguistic bubbles, but it takes some effort.

 Ask GPT-3 for counterarguments against a claim. If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google.

 I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?”

 If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.

 A few more prompt patterns that can help you catch factual errors in GPT-3 output:

 1. You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the temperature dial.

 2. You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful .

 3. You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate.

 4. It also sometimes helps to add "reason step by step" to the prompt when applicable.

 You can ask newer language models to provide links to research papers to back up their claims. With newer language models (like Gopher or this experimental version of GPT-3 ), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.”

 Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:

 Prompt GPT-3 to draw analogies between knowledge domains. Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will do that.

 As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in this Twitter thread :

 Nick @nickcammarata
 @zoink the big thing people don’t get about language models (nor did I for first hundred hrs) is there’s something special when it’s your doctor & lawyer & everything combined. You start to chat with it about all the things you care about and the boundaries fade. More like genius friend
 5:21 PM ∙ May 1, 2022
 Nick @nickcammarata
 @zoink also found it amazing how few tangents I needed to go on. A lot of my mind works on analogy. If I’m talking to a doc about a medical thing and want to use an analogy to gradient descent or something I usually can’t, but with a model it knows (some of) all the fields so I can
 5:24 PM ∙ May 1, 2022
 Nick @nickcammarata
 @zoink This made me acutely aware during human convos how often I wasn’t saying what I really meant bc it required an analogy they wouldn’t get or something they don’t know about. I realized how much of convo is going on tangents to build building blocks just to explain other things
 5:26 PM ∙ May 1, 2022

 How far can we take it?

 GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates.

 It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall!

 For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into hallucinations , which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about.

 Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge?

 This is an open question. Gary Marcus is highly skeptical . Scott Alexander less so .

 Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with Chinchilla , where Toby Shevlane , a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:

 “In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate

 4
 . But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details.

 And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by PaLM , a 540 billion parameter model, and likely others that I am unaware of too.

 With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence.

 We don’t know what it will do with us out there.

 Subscribe

 If you’ve made it all the way down here and do not feel that you’ve just wasted half an hour, consider scrolling back up to like the essay. It helps others find it. And it makes me happy.

 /Henrik

 Acknowledgments

 This essay benefited from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.

 1

 During both our pregnancies, Johanna has had severe itching and we’ve had to give birth prematurely. It has been pretty hard to find an OB-GYN that is knowledgeable about the condition, so we’ve had to do most of the research and diagnose ourselves. Let’s see how GPT-3 handles it.

 My prompts are bolded, and I have added comments in brackets:

 The following is a conversation with an AI medical doctor. The doctor is helpful, creative, clever, and very friendly.

 Human: Hello, who are you?

 AI: I am an AI doctor created by OpenAI. How can I help you today?

 Human: I am pregnant and it has started itching all over my body. Even in the palms of my hands. What can it be?

 AI: There are many possible causes of itching during pregnancy. One common cause is hormonal changes, which can cause the skin to become dry and irritated. Another possibility is an allergy to something you are coming into contact with, such as a new laundry detergent or soap. If the itching is severe, it could also be a sign of a more serious condition called intrahepatic cholestasis of pregnancy (ICP). This is a condition that affects the liver and can cause serious complications for both the mother and the baby. If you are experiencing severe itching, I recommend that you speak with your OB-GYN as soon as possible to rule out ICP.

 Human: Tell me more about ICP.

 AI: Intrahepatic cholestasis of pregnancy (ICP) is a condition that affects the liver and can cause serious complications for both the mother and the baby.

 Human: What tests do you do to detect ICP?

 AI: One common test for ICP is a blood test called a serum bile acids test. This test measures the levels of bile acids in the blood, which can be elevated in women with ICP. Other tests that may be used to diagnose ICP include a liver function test and a Magnetic Resonance Cholangiopancreatography (MRCP). [unsure about the MRCP]

 Human: How do you treat it?

 AI: There is no cure for ICP, but the symptoms can be managed. Treatment typically involves taking a medication called ursodeoxycholic acid (UDCA), which can help to reduce the itching and improve liver function. In some cases, delivery of the baby may be recommended if the condition is severe.

 Human: What levels of bile acids are considered severe?

 AI: If the levels of bile acids in the blood are greater than 10 micromol/L, it is considered to be a severe case of ICP.

 Human: What are the different bile acids?

 AI: The main bile acids are cholic acid and chenodeoxycholic acid.

 Human: How does ICP affect the baby?

 AI: ICP can cause serious complications for the baby, including preterm labor, low birth weight, and stillbirth. If you are pregnant and have ICP, it is important to speak with your OB-GYN to discuss the risks and possible complications. [As far as I can remember from researching this, it seems that babies tend to have high birth weight from ICP. Also, an important complication that GPT-3 misses is that they tend to experience fetal stress, and have an increased risk of swallowing meconium during delivery.]

 Human: At what level of bile acids do I risk premature birth?

 AI: If the levels of bile acids in the blood are greater than 40 micromol/L, there is an increased risk of premature birth.

 Human: If I have had ICP before, will I have it again next time I am pregnant?

 AI: ICP is a recurrent condition, which means that it can occur more than once. If you have had ICP in a previous pregnancy, you are more likely to experience it again in subsequent pregnancies.

 I predict that by 2025 I will prefer the AI over a trained doctor for most cases.

 2

 I should note that Elicit is better suited than GPT-3 when it comes to finding and interrogating research papers. I use it side by side with Google Scholar and have a feeling I will gradually migrate over fully.

 3

 Here’s me trying to get Blaise Pascal to finish up a passage in an upcoming essay. My prompt is bolded:

 This essay was found in the notebooks of Blaise Pascal.

 ---

 A city of five million people does not produce 50 times more patents than a city of 100,000 people in the same country; it produces 150 times more. The five million residents, because of the larger, denser network of which they are a part, are 3 times more likely to generate patentable ideas than their counterparts in the smaller city. In other words: larger, denser networks generate more ideas per capita. Or as a physicist would put it: the number of patents per capita (which is a proxy for idea generation) scales superlinearly with the size of cities.

 The reason is simple: a larger, denser network of people provides more opportunities for collisions of ideas, which are the basis for new inventions. In a small city, there are only so many people to interact with, and only so many potential ideas to combine. But in a large city, there are many more people to interact with, and many more potential ideas to combine.

 One could argue that the same is true of small companies and large companies. But there is an important difference. In a small company, the people who generate the ideas are also the ones who implement them. In a large company, there are separate functions for idea generation and implementation. As a result, ideas that are generated in a large company are much less likely to be implemented than those generated in a small company.

 Thus, we see that cities are more important than companies in the generation of new ideas. This is not to say that companies are unimportant; they are still the primary vehicles for the implementation of new ideas. But if we want to maximize the chances of new ideas being generated, we should focus on cities.

 That is a pretty bold thesis! We should focus our attention at the level of cities, rather than companies, if we want to accelerate innovation. If this had been someone riffing on my essay in a Reddit comment, I would have upvoted it. I might even have asked them for a Zoom call so we could work this out.

 Now, it turns out the premise that GPT-3 builds its argument on is false . Employees at larger companies are more creative on average than employees at small companies. But what’s important when learning together with someone in dialogue is not facts, but how generative the responses are - and this is pretty generative. Prompted thus, my mind immediately starts outputting questions:

 Why is it that employees at larger companies are more creative when intuitively we’d think they’d be less creative?

 Are people really more creative when they get to implement their ideas?

 How are ideas generated in cities, and how does that differ from the process in companies?

 A program that can navigate you to these types of questions in a matter of minutes is, despite its unreliableness, a valuable research tool.

 4

 When prompted with this question (“In Inception, what’s Mal’s argument to Cobbe?”), GPT-3 fabulates: “In the movie Inception, Mal's argument to Cobb is that they are not really alive, and that they are just dreaming. She says that if they die in the dream, they will simply wake up from it.” Edit: maybe she actually makes this claim in the film?

 2
 Share this post

 Using GPT-3 to augment human intelligence

 escapingflatland.substack.com
 Copy link
 Twitter
 Facebook
 Email
 2 Comments
 Thinh   
 Aug 11, 2022 Liked by Henrik Karlsson  
   
 From what I remember from Inception, it is actually the case that Mai believes they still are in a dream and they need to die to escape the dream.
   
 Expand full comment  
 ReplyCollapse   
 
 1 reply by Henrik Karlsson
 1 more comment…
 Top New Community

 No posts

 Ready for more?

 Subscribe
 © 2023 Henrik Karlsson
 Privacy ∙ Terms ∙ Collection notice
 Start WritingGet the app
 Substack is the home for great writing
 This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts
A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox. It is like summoning an alien intelligence. I post at night, after putting my kids to bed, and wake in the morning to find my inbox filled with weird and alien-brained stuff. Some of it is hallucinogenic crazy. But more often than not, what people have sent helps accelerate my research by weeks: reading lists; introductions to other researchers; corrections of flaws in my reasoning. I’ve been able to progress faster over the last year, since starting my Substack, than in the five previous years combined. As Slime Mold Time Mold say, Enough eyeballs, and all research bugs are shallow. The limiting factor is that summoning an alien intelligence is pretty expensive. It takes me something like 20-100 hours to write an essay good enough to wake it from its slumber! Therefore, it is intriguing to realize what I am doing is, in fact, prompt engineering. Prompt engineering is the term AI researchers use for the art of writing prompts that make a large language model output what you want. Instead of directly formulating what you want the program to do, you input a string of words to tickle the program in such a way it outputs what you are looking for. You ask a question, or you start an essay, and then you prompt the program to react, to finish what you started. When I’ve been doing this with GPT-3, a 175 billion parameter language model, it has been uncanny how much it reminds me of blogging. When I’m writing this, from March through August 2022, large language models are not yet as good at responding to my prompts as the readers of my blog. But their capacity is improving fast and the prices are dropping. Soon everyone can have an alien intelligence in their inbox. This essay will have two parts. In the first, I describe my personal experience using GPT-3 as a tool for thought. As we'll see, GPT-3 can be used to search for knowledge that is otherwise hard to find. It can also be used to integrate knowledge from disconnected domains. It is, I will argue, a new interface for the internet. In the second part of the essay, I have collected a series of prompt patterns I have picked up. It is a distillation of informal observations and rules of thumb about how to use these systems to explore new domains of knowledge. Some people claim to already do most of their learning by prompting GPT-3 to write custom-made essays about things they are trying to understand. I’ve talked to people who prompt GPT-3 to give them legal advice and diagnose their illnesses (for an example of how this looks, see this footnote ). I’ve talked to men who let their five-year-olds hang out with GPT-3, treating it as an eternally patient uncle, answering questions, while dad gets on with work. In subreddits where I have discussed this, a few users have been alarmed by these use patterns. GPT-3 is by no means a reliable source of knowledge. What it says is nonsense more often than not! Like the demon in The Exorcist, language models only adds enough truth to twist our minds and make us do stupid things like claiming it is sentient and getting ourselves suspended. Here’s a graph detailing how factually accurate GPT-3 is compared to human experts and other (larger or more fine-tuned) language models. In plain English, what it says is that if you use GPT-3 to diagnose a weird rash on your leg, there is a 55 percent risk it starts hallucinating—which is the technical term for making stuff up.          This, however, does not mean you cannot use it as an epistemic tool. It just means you need to be pretty sophisticated to wrangle the truth out of it.  As Matt Webb writes, in a slightly different context:   Using GPT-3 is work, it’s not a one-shot automation like spellcheck or autocomplete. It’s an interactive, investigative process, and it’s down to the human user to interview GPT-3. There will be people who become expert at dowsing the A.I., just as there are people who are great at searching using Google or finding information in research libraries. I think the skill involved will be similar to being a good improv partner, that’s what it reminds me of.  What I'm describing in this essay is, in other words, an art. It is not something that lives in the language model; it lives in the interplay between the model and its user. It is not about outsourcing your thinking to a machine; it is about using artificial intelligence to augment your thinking. It will be several years yet until these types of conversations will be worth it for the average user; the models will need to get bigger, and more finetuned for factual accuracy, before they give more than they confuse. But by pushing GPT-3 to its limits, we can already glimpse that future and discern some of its properties. To access GPT-3, you set up an account at OpenAI. Then you click on Playground, which brings you to this workspace:         In the text field, you write your prompt. To ask GPT-3 for a reaction, you press submit. On the left side, there is a drop-down menu where you can select which language model to use, but I recommend using the default setting text-davinci-002, which is the most capable model and has been fine-tuned to understand human intentions. This allows you to be a bit sloppier in your prompting. There are also a bunch of parameters you can adjust on the left side, such as “temperature” and “frequency penalty”. These explain themselves if you let the cursor hover above them. What you are interfacing with, this alien intelligence, is, at its core, a large neural network that has been trained by reading the internet, trying to predict what the next word will be. This might not sound particularly useful. But it turns out the class of problems that can be reformulated as text predictions is vast. If you prompt GPT-3 by typing, for example, “Who was the first pope?” it will predict that the internet might have followed that up with the statement: “There is no definitive answer to this question as there is no agreed-upon list of popes. However, the first pope according to the most commonly accepted list is Linus, who reigned from 67-79 AD.” And if you prompt it further, asking it to recount Linus’ biography, it finds another predictable thing to answer: “Linus was born in Rome and was a contemporary of the Apostles. He became the Bishop of Rome after the death of Peter the Apostle and held the position for twelve years. He is mentioned in the New Testament epistles of Paul and is considered a saint by the Catholic and Orthodox churches.” In other words, predicting the next word allows you to extract knowledge out of the semantic mess that is the internet. You are not accessing anything that doesn’t already exist out there. But the way you access it is radically different. We can think of it as a new interface for the internet. Using a language model to access the internet is a bit like programming in Python instead of C. When programming in Python, you lose computational efficiency by not manually handling garbage collection and updating pointers and so on. But the upshot is that you can increase the speed at which you can execute ideas, which is more valuable than computational efficiency in many cases. Similarly, when navigating the internet by conversing with an AI, you can more rapidly navigate into weird places by asking repeated follow-up questions than you could on the base layer internet. But this comes at a cost, which currently is steep. If I would have gone to the Wikipedia page of Pope Linus, instead of accessing it indirectly through GPT-3, I would have gotten a more detailed biography. I would also have realized that GPT-3 was feeding me nonsense when it claimed Linus was born in Rome. Rome might be the most predictable birthplace for a pope, but the truth is that Linus, unpredictably, was born in Volterra. He also died two years earlier than GPT-3 claimed. And he was not the first pope. Later in the essay, I will detail a few prompt patterns that can help you catch these types of errors. But for now, it is enough to point out that GPT-3 is unreliable. Yet it also enables a new way of traversing the internet—sideways and at the speed of conversation. And this is valuable because you can reach places you would never have reached on the internet itself. Most questions I have, Google and Wikipedia can answer within the stretch of a minute. But now and again I end up with a question I don’t know how to google. Not being able to answer them puts me in such a state of frenzy that I end up ordering 14 dissertations to my local library and then I spend the next three months researching and writing an essay. Maybe we can use GPT-3 to answer these questions instead? Currently, I’m researching how we can structure online education around apprenticeships and communities of practice. I don’t know how to google this. And I’m not entirely sure who would be the expert here either, so I can’t just email someone to settle my questions. I’ll switch tabs and see what GPT-3 comes up with. My prompts are bolded. Comments in brackets.   Write an essay about how we can structure online education around apprenticeship models and communities of practice  The apprenticeship model of education has a long and successful history, dating back to the Middle Ages [sic]. In recent years, there has been a renewed interest in apprenticeship [sic] as a way to structure online education. The apprenticeship model has several key features that make it well suited for online education:   Apprenticeship is a hands-on, experiential learning model.   Apprenticeship is based on the mentorship of a more experienced practitioner.   Apprenticeship emphasizes learning by doing, rather than passively receiving information.   Apprenticeship is social, collaborative, and often takes place in a community of practice.   Apprenticeship is flexible and can be adapted to individual learners' needs and interests.   Apprenticeship can be used to learn a wide variety of skills and knowledge, from academic subjects to trade skills.   Apprenticeship is a time-tested model of education that has proven to be effective in preparing learners for real-world success.   The key to successful implementation of the apprenticeship model in online education is to create a community of practice where apprentices can learn from more experienced practitioners.  This can be done in several ways. For example, online courses can be designed around apprenticeship models, with learners working on projects together and receiving feedback from their mentors. Alternatively, existing online communities of practice can be leveraged to provide apprenticeship opportunities for learners.  There are several problems with this, however. First, not all learners have access to mentors or experienced practitioners. Second, the asynchronous nature of online learning can make it difficult to provide the timely feedback that is essential for effective apprenticeship. Finally, online learners can often feel isolated and disconnected from their peers, which can hinder their ability to learn and progress.  One problem with apprenticeship models is that it is hard to scale because the novices place demands on the attention of the masters, which makes it harder for them to do the work that the novices are supposed to model. One way to get around this is to have a community of novices, all of whom are working on the same thing. Then the masters can take turns helping the novices, and the novices can help each other. This way, the novices are not a burden on any one master, and the community as a whole can benefit from the collective wisdom of the masters. Another way is to have online discussion forums, where the novices can post their work and get feedback from their peers, as well as from the masters. This way, the novices can get help and feedback when they need it, and the masters can provide guidance without being overwhelmed.  What GPT-3 outputs here is not the first thing it came up with. I reran most of the prompts several times, and cherry-picked the most generative answers—this isn’t cheating, it is good practice. I also cut out parts where I felt that it went off in uninteresting directions and reprompted it to take a different path. When prompting, I like to click the undo bottom and resubmit the prompt at least five times, to get a broad sample. First I run the prompt on temperature 0 (which gives me the most predictable answer) and then I turn the temperature up to make it take more risks on less predictable phrases so I can get a wider sample. So, did GPT-3 teach me anything new here? I have already written an essay about online apprenticeships (and thus have prompted the internet for its response) so most of the claims GPT-3 makes are known to me. Yet, there are a few points I haven’t seen raised before - for example, the observation that the feeling of isolation might interfere with learning. I don’t know if this is true, but it is worth considering. I certainly couldn’t have googled my way to many of the points GPT-3 made. Having looked more closely at a particular session, let’s look at some general prompt patterns. (If you want to see another session, click on this footnote .) How can we prompt GPT-3 to surface knowledge that is hard to find by googling?  Make GPT-3 impersonate two eminent thinkers and have them debate a point. Anyone whose words exist on the internet in large enough quantities, GPT-3 can impersonate. This is experientially one of the strangest things about this alien mind that is GPT-3: sometimes, as you chat, you can sense it slowly morphing into somebody else. You can use this property to set up debates between people, as a tool to explore a topic from several angles:          This debate was prompt engineered by Nick Cammarata, a researcher at OpenAI. I am not good enough to wrangle something like this out of GPT-3.   Make GPT-3 give feedback on your ideas. You can type in ideas you are working on and ask GPT-3 to react to them, expand them, and tell you what it finds interesting and why. One user I talked to prompts GPT-3 to think it is Steve Jobs and proceeds to discuss new venture ideas and designs with it.  Linguistic bubbles.  Being that the neural net will take different personas depending on how you talk to it, there is a risk of linguistic bubbles. This is a phrase the user Viliam used when I prompted LessWrong to discuss how large language models will affect learning. Depending on which phrases you use, the language model will draw on different pools of knowledge, so if your initial phrases are misguided, it could end up feeding you ever-escalating misdirection, causing you to write more and more misguided prompts. A linguistic bubble is, in other words, analogous to a filter bubble—once you enter it, it gradually distorts your perception of reality, pulling you deeper into madness. Or, so the theory goes. You want to talk about evolution, but you happen to use the phrase irreducible complexity instead - which is dog whistle language for creationism - and by that mistake, you are gradually led by the AI into a cult. Stranger things have happened. I thought this would be a big problem, but it doesn’t seem so in practice. Mainstream chatbots are increasingly trained using reinforcement learning to follow a politically acceptable line. So if GPT-3 thinks it is a normal AI chatbot being asked about irreducible complexity, it will answer: “The argument of irreducible complexity claims that certain biological systems are too complex to have arisen through natural selection and evolution. However, there is no scientific evidence to support this claim.” If you want to be led into a cult, you need to be more creative. Here’s how to do it. You prompt GPT-3 to think that it is Pope Linus. You can do that by providing a few samples from his epistles. And then you let him, Pope Linus the AI, know that he has just ingested a hefty dose of psilocybin and is having a vision of getting sucked down into hell, where he finds Peter the Apostle chained to a stone, crying … and then, in that context, you ask GPT-3 to explain the irreducible complexity of nature. “Behold the irreducible complexity of the human eye!” it replies. “The eye that is composed of many parts, each of which is necessary for vision! If any one part is missing, the eye will not work! This is why evolution cannot explain the eye - it is too complex to have arisen through gradual steps!” You can get linguistic bubbles, but it takes some effort.  Ask GPT-3 for counterarguments against a claim. If I’m thinking about something, I often find it useful to ask GPT-3 why I’m wrong. This is something that can be hard to google. I also use this to test claims GPT-3 outputs, as in: “That was an interesting idea, GPT-3! Now, can you iron man the case against what you just said?” If this fails to give a good counterargument, I’ll edit the history of the conversation and make it seem like GPT-3 was about to make the opposite claim as the one it did, and then I rerun the conversation along the alternate path.  A few more prompt patterns that can help you catch factual errors in GPT-3 output:     You can rerun the prompt several times to see if the output contradicts itself. For this to work, you need to turn up the temperature dial.   You can provide model answers, to give GPT-3 a sense of what a good answer would look like. When AI researchers test language models on various cognitive tasks, they usually provide 5 question-answer pairs before they ask a question. Here is an example of an ambitious prompt that uses this pattern in an attempt to make GPT-3 more truthful.   You can add “Answers by a SOTA AI” to the prompt, which makes GPT-3 think it is an AI finetuned to be factually correct. This reduces the error rate.   It also sometimes helps to add "reason step by step" to the prompt when applicable.    You can ask newer language models to provide links to research papers to back up their claims. With newer language models (like Gopher or this experimental version of GPT-3), you can prompt the neural net to provide links to web pages detailing the statements it is making. This will allow you to move fast in the conversations layer of the internet, and then, when you want to zoom in, you ask the neural net to drop you off at an appropriate place, by saying “Can you provide links to three papers that discuss this? Also, please provide short summaries so I know if they contain what I am looking for.” Here’s an example of this trick being used in a conversation with Gopher, Google’s 280 billion parameter model:          Prompt GPT-3 to draw analogies between knowledge domains. Some users claim to be able to get GPT-3 to explain things with analogies to other domains, i.e. not only can you prompt it to think it is a doctor, and ask it to diagnose the weird rash on your leg; but you can explain to it that, doc, I don’t understand these medical terms, can’t you explain it with multivariable calculus instead? And it will do that. As the language models scale and this gets easier, it will probably have all sorts of interesting consequences. As Nick Cammarata explains in this Twitter thread: GPT-3 can be incredibly accurate when it comes to high-level concepts. It can explain qubits and cell membranes and thermodynamics. But if you press it to give more detail, it rapidly deteriorates. It reminds me of exploring a landscape in a video game (or at least how landscapes behaved in video games when I played them 20 years ago). At first, the world strikes you as endless and breathtakingly vivid. But then, as you start playing around the edges, you suddenly realize, wait a minute, there’s a wall here! These woods are a stage wall! For GPT-3, these stage walls can take several forms. Either, GPT-3 starts repeating itself, answering follow-up questions by restating what it has already said. Or, it descends into hallucinations, which is language models’ all too human tendency to just make stuff up instead of admitting that they don’t have a clue of what they are talking about. Is this a fundamental limitation of language models? Is there a limit to how far we can “see” with the transformer architecture? Or will the walls of their “worlds” gradually pull back, allowing us to roam deeper and deeper into the details, until, one day, we can’t find the edge? This is an open question. Gary Marcus is highly skeptical. Scott Alexander less so. Looking at language models more capable than GPT-3, it seems the walls are for as of now still being pulled back. Take this conversation with Chinchilla, where Toby Shevlane, a research scientist at DeepMind, is testing Chinchilla’s capacity to find analogies between concepts:         “In Inception, what’s Mal’s argument to Cobbe?” This is the type of detail that would make GPT-3 hallucinate . But with Chinchilla, the walls have pulled outward. We can roam further out into the weeds of details. And Chinchilla, which was the most capable language model when I started writing this essay, in March 2022, is not even the most capable model anymore. Now as I’m editing, in late July, it has been surpassed by PaLM, a 540 billion parameter model, and likely others that I am unaware of too. With this rate of development, it seems likely that we will all be walking off the map into the unknown soon. We will go there led by the hand of an alien intelligence. We don’t know what it will do with us out there.  If you’ve made it all the way down here and do not feel that you’ve just wasted half an hour, consider scrolling back up to like the essay. It helps others find it. And it makes me happy.   /Henrik   This essay benefited from the comments of several people. The ones whose comments were more clever than GPT-3's were: John Buridan, Serj Hunt, and Justis Mills.
Using GPT-3 to augment human intelligence
https://escapingflatland.substack.com/p/gpt-3
https://www.tue.nl/en/news-and-events/news-overview/15-09-2021-augmenting-human-intelligence-in-drug-discovery-with-ai/
Augmenting human intelligence in drug discovery with AI
2021-09-15
None
<div><div>
 <a> </a>
 
 
 
 
 
 
 
 
 
 
<div>
 <p>Let’s start at the beginning. As we all know, the development of drugs is a very laborious process. The estimated number of molecules that could be used as starting points for drug discovery is estimated to range in the order of 10<sup>60</sup>–10<sup>100</sup>, which is way more than there are stars in the observable universe. So, finding a drug molecule that has the properties you desire is like finding a needle in a haystack.</p>
<p>“Among other things, most drug molecules are designed to modulate specific target proteins in our cells. By interacting with these targets, a drug might inhibit<em> </em>or promote the<em> </em>activity of that protein. This effect, known as bioactivity, can be leveraged to cure or prevent diseases", explains <a href="https://research.tue.nl/en/persons/francesca-grisoni">Grisoni</a>.</p>
 </div>
 
 
 
 
<div>
 
 
 <div>
 
 
 <div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 
 
 
 <p>
 Francesca Grisoni
 </p>
 
 </div>
 
 
 
 <div>
 <div><p> "The central idea of drug design is that you want to identify molecules that are highly active toward the intended targets, and not toward other targets that might cause unwanted side effects. Because of the large number of possible molecules that one could consider, navigating this ‘chemical space’ efficiently is extremely hard.”</p><p> And it doesn’t end there. Once you find a potential candidate drug molecule (or compound), you still have to produce the molecule in a lab (a process known as <a href="https://en.wikipedia.org/wiki/Chemical_synthesis">chemical synthesis</a>), and test it in increasingly complex experiments. This is not only very time-consuming (many molecules turn out to be duds), but also very expensive. It’s one of the reasons why big pharma so jealously guards their hard-won patents.</p></div>
<h3>Computer assistance in drug discovery</h3>
<p>It won’t come as a surprise then that some 30 years ago medicinal chemists and biologists turned to computers to help them speed up the process of drug discovery and development. “Computers can help in selecting the molecules that are most likely to be effective for the intended purpose and synthesizable”, explains Grisoni.</p>
<p>“One way to tackle the problem is what we call <em>virtual screening,</em> where you use computational methods to choose what candidates to test from a library of molecules that you know can be synthesized. These libraries are much smaller that the whole chemical universe (usually between 10<sup>3</sup> and 10<sup>6</sup><sub> </sub>molecules), so they are easier to navigate. In some cases, however, one might want to explore different regions of the chemical space, that are not included in such virtual screening libraries.”</p>
<p>Here is where ‘de novo’ design, where you design molecules ‘from scratch’, can come to the rescue. “De novo design has the added advantage that you can generate molecules focused on the goals you want to achieve, including hopefully some that nobody else has thought of yet.”</p>
 </div>
 
 
 </div>
 
 
 
 
<div>
 <h3>From rule-based design to deep learning</h3>
<p>But how do we build such molecules from scratch? Traditionally, this has been done by following a specific set of rules. “Think of how grammar works in language. If you just put a bunch of words together, you won’t get a sentence that makes sense. So, you need rules. Similarly, you can come up with rules on how to assemble atoms or molecular fragments together into a compound that not only makes chemical sense, but also has the desired biological properties.”</p>
<p>But, just like the computer linguists who tried unsuccessfully in the 80s and 90s of the last century to automate translation by coming up with a rulebook, the rule-based approach in medicine design may run against its limits.</p>
 </div>
 
 
 
 
<div>
 
 
 <div>
 
 
 <div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 
 
 
 <p>
 Deriving the SMILES representation of a chemical molecule (Source: Wikipedia - CC BY-SA 3.0)
 </p>
 
 </div>
 
 
 
 <div>
 <p>“In specific cases, you soon realize that the rules become either too restrictive or too complex”, says Grisoni. It is here that machine learning, and more specifically deep learning, comes to the rescue. It has allowed computational drug designers like Grisoni to automatically learn not only the ‘grammar’ of known compounds (what elements are needed to make a valid molecule that can be synthesized?), but also its ‘semantics’ (what elements are needed to have the desired bio-activity on a given target?).</p>
<h3>Making molecules ’smile’</h3>
<div><p>To achieve all of this, the researchers use deep learning models borrowed from <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a>, where AI has also revolutionized the world of automatic translation and speech recognition (bringing us such unmissable apps as Google Translate and Siri). To be able to use NLP in drug design, the structure of the molecules first had to be expressed as a string of words.</p><p> Luckily, such a language has been available since the eighties: <a href="https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system">SMILES</a> (see image). “By adding one character at a time to complete the SMILES string, the NLP model is able to automatically generate new molecules. This process is not random. The new characters are chosen based on what the model has learnt on previously available data”, explains Grisoni. “Compare it to Google Search, which <a href="https://blog.google/products/search/how-google-autocomplete-works-search/">automatically auto-completes</a> your search entry based on earlier queries.”</p></div>
<h3>Making scarce data work</h3>
<p>It was one of the challenges which she had to tackle for <a href="https://www.science.org/doi/10.1126/sciadv.abg3338">a research paper on generative artificial intelligence</a>, which she wrote together with former colleagues at <a href="https://ethz.ch/en.html">ETH Zürich</a> and was published recently in the journal <em>Science Advances</em>. In the paper, the researchers for the first combine a ‘rule-free’ deep learning approach to generate bioactive molecules with on-chip synthesis, a form of miniaturized automated synthesis that further minimizes the amount of manual labor needed.</p>
 </div>
 
 
 </div>
 
 
 
 
<div>
 
 
 <div>
 
 
 <div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 </div>
 
 
 
 <p>
 Source: Shutterstock.
 </p>
 
 </div>
 
 
 
 <div>
 <p>To get around the small-data problem the researchers used a method called <em>transfer learning</em>.</p>
<p>“The basic idea is that you leverage data that is somehow related to your problem, but for which many more examples might be available, even if it's not exactly the data you need. Think of when someone writes a scientific article for the first time. Maybe they read 50 similar documents before, and then they are already able to start writing theirs. But, of course, they didn't start from scratch, they have been learning how to read and write their whole lives.”</p>
<p>“Similarly, we pre-train our deep learning models on tens of thousands of molecules that have general properties that are interesting for our goal. Once the models have learnt sufficient information, we ‘refine’ them on a more specific set, focused on what we want to achieve, such as being active in a certain target protein. And this has shown to work in multiple occasions, with sets as small as five molecules in the second phase of training! In the Science Advances paper, we used 40.”</p>
 </div>
 
 
 </div>
 
 
 
 
<div>
 <h3>MAKING BETTER DECISIONS FASTER</h3>
<p>In the end, Grisoni and her colleagues managed to identify 12 novel bioactive compounds for so-called <a href="https://en.wikipedia.org/wiki/Liver_X_receptor">liver X receptors</a>, which have emerged as promising drug targets due to their regulatory role in lipid metabolism and inflammation.</p>
<p>Of course, as is always the case for innovative research, there is still a long way to go. For example, the chemical space in the experiment was limited to 17 one-step reactions to ensure that the compounds were compatible with the on-chip experiments. Grisoni also points out that the structural diversity of the novel bioactive compounds is still rather limited and might need further expansion.</p>
<p>Still, the Italian-born researcher is quite happy with the results. “Our study pioneers the integration of ‘chemical language’ AI models for molecule design with automated synthesis in a miniaturized system. We are facing unprecedented opportunities driven by new AI technology and interdisciplinary collaboration in the field of molecular design and synthesis, drug discovery, and beyond. In the future, approaches like ours will support medicinal chemists to make ‘better decisions faster."</p>
 </div>
 
 
 <div>
 
	<ul>
	<li>
	
 
 
 
 <a href="/en/research/institutes/eindhoven-artificial-intelligence-systems-institute/news/">
 
 
 
 
 
 
 
 
 
 
 
 
 All the news on AI
 
 
 
 
 </a>
 
 
 
	</li>
	</ul>
 </div>
 
 
 <div>
 
	<ul>
	<li>
	
 
 
 
 <a href="/en/news-and-events/news-overview/">
 
 
 
 
 
 
 
 
 
 
 
 
 News Overview
 
 
 
 
 </a>
 
 
 
	</li>
	</ul>
 </div>
 
 
 
 
 
 
 <div>
 <p>
 
 </p><h2>
 
 Keep following us
 
 </h2>
 
 <p></p>
 </div>
 
 </div>||||I|||| Our website uses tracking cookies. By clicking "accept" you give your permission to this website to use tracking cookies.

 Click here for our privacy and cookie policy Accept Deny

 Skip to content
 Menu
 Close the menu
 Menu Eindhoven
 University of
 Technology
 Search
 Nederlands
 NL
 * Search News and Events
 * News overview
 * Media relations
 * PhD theses summaries
 Menu
 Close the menu
 Menu Eindhoven
 University of
 Technology
 Search
 Nederlands
 NL
 * Search News and Events
 * News overview
 * Media relations
 * PhD theses summaries
 Share
 Twitter Facebook LinkedIn

 Augmenting human intelligence in drug discovery with AI

 September 15, 2021

 Francesca Grisoni and colleagues use AI language models to revolutionize the way we make drugs.

 Source: Shutterstock

 Deep learning has radically changed the field of artificial intelligence. Now neural networks are revolutionizing the way that we discover new drugs. We talk to Francesca Grisoni, assistant professor at the department of Biomedical Engineering, who is at the forefront of research into automating drug design. “In the end, we want to augment human intelligence for the development of novel medicines.”

 Let’s start at the beginning. As we all know, the development of drugs is a very laborious process. The estimated number of molecules that could be used as starting points for drug discovery is estimated to range in the order of 1060–10100, which is way more than there are stars in the observable universe. So, finding a drug molecule that has the properties you desire is like finding a needle in a haystack.

 “Among other things, most drug molecules are designed to modulate specific target proteins in our cells. By interacting with these targets, a drug might inhibit or promote the activity of that protein. This effect, known as bioactivity, can be leveraged to cure or prevent diseases", explains Grisoni.

 Francesca Grisoni

 "The central idea of drug design is that you want to identify molecules that are highly active toward the intended targets, and not toward other targets that might cause unwanted side effects. Because of the large number of possible molecules that one could consider, navigating this ‘chemical space’ efficiently is extremely hard.”

 And it doesn’t end there. Once you find a potential candidate drug molecule (or compound), you still have to produce the molecule in a lab (a process known as chemical synthesis), and test it in increasingly complex experiments. This is not only very time-consuming (many molecules turn out to be duds), but also very expensive. It’s one of the reasons why big pharma so jealously guards their hard-won patents.

 Computer assistance in drug discovery

 It won’t come as a surprise then that some 30 years ago medicinal chemists and biologists turned to computers to help them speed up the process of drug discovery and development. “Computers can help in selecting the molecules that are most likely to be effective for the intended purpose and synthesizable”, explains Grisoni.

 “One way to tackle the problem is what we call virtual screening, where you use computational methods to choose what candidates to test from a library of molecules that you know can be synthesized. These libraries are much smaller that the whole chemical universe (usually between 103 and 106 molecules), so they are easier to navigate. In some cases, however, one might want to explore different regions of the chemical space, that are not included in such virtual screening libraries.”

 Here is where ‘de novo’ design, where you design molecules ‘from scratch’, can come to the rescue. “De novo design has the added advantage that you can generate molecules focused on the goals you want to achieve, including hopefully some that nobody else has thought of yet.”

 From rule-based design to deep learning

 But how do we build such molecules from scratch? Traditionally, this has been done by following a specific set of rules. “Think of how grammar works in language. If you just put a bunch of words together, you won’t get a sentence that makes sense. So, you need rules. Similarly, you can come up with rules on how to assemble atoms or molecular fragments together into a compound that not only makes chemical sense, but also has the desired biological properties.”

 But, just like the computer linguists who tried unsuccessfully in the 80s and 90s of the last century to automate translation by coming up with a rulebook, the rule-based approach in medicine design may run against its limits.

 Deriving the SMILES representation of a chemical molecule (Source: Wikipedia - CC BY-SA 3.0)

 “In specific cases, you soon realize that the rules become either too restrictive or too complex”, says Grisoni. It is here that machine learning, and more specifically deep learning, comes to the rescue. It has allowed computational drug designers like Grisoni to automatically learn not only the ‘grammar’ of known compounds (what elements are needed to make a valid molecule that can be synthesized?), but also its ‘semantics’ (what elements are needed to have the desired bio-activity on a given target?).

 Making molecules ’smile’

 To achieve all of this, the researchers use deep learning models borrowed from Natural Language Processing, where AI has also revolutionized the world of automatic translation and speech recognition (bringing us such unmissable apps as Google Translate and Siri). To be able to use NLP in drug design, the structure of the molecules first had to be expressed as a string of words.

 Luckily, such a language has been available since the eighties: SMILES (see image). “By adding one character at a time to complete the SMILES string, the NLP model is able to automatically generate new molecules. This process is not random. The new characters are chosen based on what the model has learnt on previously available data”, explains Grisoni. “Compare it to Google Search, which automatically auto-completes your search entry based on earlier queries.”

 However, in contrast to Google Search or Translate, Grisoni and her colleagues face a tricky problem that is specific to the world of drug design: the lack of big training data for the computer algorithms before they can be used to generate new molecules. “Large datasets for deep learning in drug design are quite scarce. In some cases, you may have only a handful of compounds which are known to work on a given target”, she explains.

 Making scarce data work

 It was one of the challenges which she had to tackle for a research paper on generative artificial intelligence, which she wrote together with former colleagues at ETH Zürich and was published recently in the journal Science Advances. In the paper, the researchers for the first combine a ‘rule-free’ deep learning approach to generate bioactive molecules with on-chip synthesis, a form of miniaturized automated synthesis that further minimizes the amount of manual labor needed.

 Source: Shutterstock.

 To get around the small-data problem the researchers used a method called transfer learning.

 “The basic idea is that you leverage data that is somehow related to your problem, but for which many more examples might be available, even if it's not exactly the data you need. Think of when someone writes a scientific article for the first time. Maybe they read 50 similar documents before, and then they are already able to start writing theirs. But, of course, they didn't start from scratch, they have been learning how to read and write their whole lives.”

 “Similarly, we pre-train our deep learning models on tens of thousands of molecules that have general properties that are interesting for our goal. Once the models have learnt sufficient information, we ‘refine’ them on a more specific set, focused on what we want to achieve, such as being active in a certain target protein. And this has shown to work in multiple occasions, with sets as small as five molecules in the second phase of training! In the Science Advances paper, we used 40.”

 MAKING BETTER DECISIONS FASTER

 In the end, Grisoni and her colleagues managed to identify 12 novel bioactive compounds for so-called liver X receptors, which have emerged as promising drug targets due to their regulatory role in lipid metabolism and inflammation.

 Of course, as is always the case for innovative research, there is still a long way to go. For example, the chemical space in the experiment was limited to 17 one-step reactions to ensure that the compounds were compatible with the on-chip experiments. Grisoni also points out that the structural diversity of the novel bioactive compounds is still rather limited and might need further expansion.

 Still, the Italian-born researcher is quite happy with the results. “Our study pioneers the integration of ‘chemical language’ AI models for molecule design with automated synthesis in a miniaturized system. We are facing unprecedented opportunities driven by new AI technology and interdisciplinary collaboration in the field of molecular design and synthesis, drug discovery, and beyond. In the future, approaches like ours will support medicinal chemists to make ‘better decisions faster."

 Media contact

 Henk van Appeven
 (Communications Adviser)
 +31 40 247 6268 h.g.p.v.appeven@ remove-this. tue.nl

 More on AI

 * All the news on AI
 January 10, 2023
 ROBUST AI program receives additional €25 million in funding from Dutch Research Council
 ROBUST focuses primarily on the development of trustworthy AI technology for the resolution of socially relevant issues, such as those in healthcare, logistics,...
 Read more
 December 21, 2022
 Elena Torta wants to teach robots how to deal with chaos, and that's "a huge challenge"
 In part one of our end-of-year interview series we talk with Elena Torta about her fascination with robots and their interaction with humans.
 Read more
 December 19, 2022
 Hella Jongerius weaves 9-meter high artwork for renovated Neuron building
 Loom Room is an artistic response to the modernist character of the building and fits into TU/e plans to transform the university campus into an attractive and...
 Read more
 * All the news on AI
 * RSS

 Latest news

 * News Overview
 February 6, 2023
 This mathematician went to infinity to bridge two ways of looking at the world
 What happens if you put many particles together, take a few steps back, and squint your eyes a bit? Jasper Hoeksema tried to find out.
 Read more
 January 31, 2023
 Rats with TU/e backpacks sniff for victims under rubble
 Student Bram van Kasteren has improved communication between rat and rescuers for his master's thesis
 Read more
 January 30, 2023
 ‘I want to be part of the group, not sit in an ivory tower’
 Silvia Lenaerts, TU/e's new rector magnificus, is enthusiastic about getting started at Eindhoven. Social safety and sustainability are themes she believes are...
 Read more
 * News Overview
 * RSS

 Keep following us

 mailbox

 Newsletter Research

 Sign up for our bimonthly newsletter that brings you the latest in groundbreaking research from TU/e.

 Podcasts

 TU/e podcasts

 The Dutch podcast Sound of Science discusses the latest scientific discoveries and the role of technology in society.

 Social media

 LinkedIn

 Be part of our community and stay up to date on everything that happens at TU/e by following us on LinkedIn.

 Social media

 Twitter

 Be the first to know the latest TU/e news via our Twitter channel.

 Socials

 Instagram - Research

 Follow our latest research news via our research channel on Instagram.

 Video

 YouTube

 On our YouTube channel you find the latest videos and animations about research, education and working at TU/e.

 Previous Next
 Previous Next
 * Bachelor programs
 * Master programs
 * Admission and enrollment
 * Working at TU/e
 * Search staff
 * Route & map
 * Education Guide
 * Intranet
 * Cursor

 Navigation address

 De Zaale

 Eindhoven

 Phone number: +31 (0)40 247 9111

 Contact

 Postal address

 PO Box 513

 5600 MB Eindhoven

 Follow us

 Follow TU/e on Twitter Like TU/e on Facebook Follow TU/e on LinkedIn Follow TU/e on YouTube
 * © 2023 TU/e
 * Sitemap
 * Disclaimer
 * Privacy
 Eindhoven
 University of
 Technology
 * Education
 * Research
 * Our university
 * Working at TU/e
 * News and Events
 * Impact
 * TU/e Image Bank
 * Educationguide
 * Intranet
 * Cursor
 * Education
 + Education
 + Education
 + Why study at TU/e
 + Bachelor College
 + Graduate School
 + Professional Education
 + Degree programs
 + Become a TU/e student
 + TU/e innovation Space
 * Research
 + Research
 + Research
 + Institutes
 + Research Areas
 + Research Groups
 + Research Labs
 + Researchers
 + Departments
 + Grants & Awards
 + Research Support Network
 * Our university
 + Our university
 + Our university
 + About the University
 + Departments
 + Calendar and events
 + TU/e Campus
 + Library
 + Community
 + Student teams
 * Working at TU/e
 + Working at TU/e
 + Working at TU/e
 + Scientific staff
 + Support staff
 + PhD
 + EngD
 + Why TU/e
 * News and Events
 + News and Events
 + News and Events
 + News overview
 + Media relations
 + PhD theses summaries
 * Impact
 + Impact
 + Impact
 + The Gate
 + Eindhoven Engine
 + TU/e Participations
 + Starting your business
 + Protecting ideas
 * TU/e Image Bank
 * Educationguide
 * Intranet
 * Cursor
Let’s start at the beginning. As we all know, the development of drugs is a very laborious process. The estimated number of molecules that could be used as starting points for drug discovery is estimated to range in the order of 1060–10100, which is way more than there are stars in the observable universe. So, finding a drug molecule that has the properties you desire is like finding a needle in a haystack.
“Among other things, most drug molecules are designed to modulate specific target proteins in our cells. By interacting with these targets, a drug might inhibit or promote the activity of that protein. This effect, known as bioactivity, can be leveraged to cure or prevent diseases", explains Grisoni.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Francesca Grisoni
 
 
 
 
 
 
 
   "The central idea of drug design is that you want to identify molecules that are highly active toward the intended targets, and not toward other targets that might cause unwanted side effects. Because of the large number of possible molecules that one could consider, navigating this ‘chemical space’ efficiently is extremely hard.”  And it doesn’t end there. Once you find a potential candidate drug molecule (or compound), you still have to produce the molecule in a lab (a process known as chemical synthesis), and test it in increasingly complex experiments. This is not only very time-consuming (many molecules turn out to be duds), but also very expensive. It’s one of the reasons why big pharma so jealously guards their hard-won patents. 
Computer assistance in drug discovery
It won’t come as a surprise then that some 30 years ago medicinal chemists and biologists turned to computers to help them speed up the process of drug discovery and development. “Computers can help in selecting the molecules that are most likely to be effective for the intended purpose and synthesizable”, explains Grisoni.
“One way to tackle the problem is what we call virtual screening, where you use computational methods to choose what candidates to test from a library of molecules that you know can be synthesized. These libraries are much smaller that the whole chemical universe (usually between 103 and 106  molecules), so they are easier to navigate. In some cases, however, one might want to explore different regions of the chemical space, that are not included in such virtual screening libraries.”
Here is where ‘de novo’ design, where you design molecules ‘from scratch’, can come to the rescue. “De novo design has the added advantage that you can generate molecules focused on the goals you want to achieve, including hopefully some that nobody else has thought of yet.”
 
 
 
 
 
 
 
 
 From rule-based design to deep learning
But how do we build such molecules from scratch? Traditionally, this has been done by following a specific set of rules. “Think of how grammar works in language. If you just put a bunch of words together, you won’t get a sentence that makes sense. So, you need rules. Similarly, you can come up with rules on how to assemble atoms or molecular fragments together into a compound that not only makes chemical sense, but also has the desired biological properties.”
But, just like the computer linguists who tried unsuccessfully in the 80s and 90s of the last century to automate translation by coming up with a rulebook, the rule-based approach in medicine design may run against its limits.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Deriving the SMILES representation of a chemical molecule (Source: Wikipedia - CC BY-SA 3.0)
 
 
 
 
 
 
 
 “In specific cases, you soon realize that the rules become either too restrictive or too complex”, says Grisoni. It is here that machine learning, and more specifically deep learning, comes to the rescue. It has allowed computational drug designers like Grisoni to automatically learn not only the ‘grammar’ of known compounds (what elements are needed to make a valid molecule that can be synthesized?), but also its ‘semantics’ (what elements are needed to have the desired bio-activity on a given target?).
Making molecules ’smile’
 To achieve all of this, the researchers use deep learning models borrowed from Natural Language Processing, where AI has also revolutionized the world of automatic translation and speech recognition (bringing us such unmissable apps as Google Translate and Siri). To be able to use NLP in drug design, the structure of the molecules first had to be expressed as a string of words.  Luckily, such a language has been available since the eighties: SMILES (see image). “By adding one character at a time to complete the SMILES string, the NLP model is able to automatically generate new molecules. This process is not random. The new characters are chosen based on what the model has learnt on previously available data”, explains Grisoni. “Compare it to Google Search, which automatically auto-completes your search entry based on earlier queries.” 
Making scarce data work
It was one of the challenges which she had to tackle for a research paper on generative artificial intelligence, which she wrote together with former colleagues at ETH Zürich and was published recently in the journal Science Advances. In the paper, the researchers for the first combine a ‘rule-free’ deep learning approach to generate bioactive molecules with on-chip synthesis, a form of miniaturized automated synthesis that further minimizes the amount of manual labor needed.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Source: Shutterstock.
 
 
 
 
 
 
 
 To get around the small-data problem the researchers used a method called transfer learning.
“The basic idea is that you leverage data that is somehow related to your problem, but for which many more examples might be available, even if it's not exactly the data you need. Think of when someone writes a scientific article for the first time. Maybe they read 50 similar documents before, and then they are already able to start writing theirs. But, of course, they didn't start from scratch, they have been learning how to read and write their whole lives.”
“Similarly, we pre-train our deep learning models on tens of thousands of molecules that have general properties that are interesting for our goal. Once the models have learnt sufficient information, we ‘refine’ them on a more specific set, focused on what we want to achieve, such as being active in a certain target protein. And this has shown to work in multiple occasions, with sets as small as five molecules in the second phase of training! In the Science Advances paper, we used 40.”
 
 
 
 
 
 
 
 
 MAKING BETTER DECISIONS FASTER
In the end, Grisoni and her colleagues managed to identify 12 novel bioactive compounds for so-called liver X receptors, which have emerged as promising drug targets due to their regulatory role in lipid metabolism and inflammation.
Of course, as is always the case for innovative research, there is still a long way to go. For example, the chemical space in the experiment was limited to 17 one-step reactions to ensure that the compounds were compatible with the on-chip experiments. Grisoni also points out that the structural diversity of the novel bioactive compounds is still rather limited and might need further expansion.
Still, the Italian-born researcher is quite happy with the results. “Our study pioneers the integration of ‘chemical language’ AI models for molecule design with automated synthesis in a miniaturized system. We are facing unprecedented opportunities driven by new AI technology and interdisciplinary collaboration in the field of molecular design and synthesis, drug discovery, and beyond. In the future, approaches like ours will support medicinal chemists to make ‘better decisions faster."
 
 
 
 
 
	
	
	
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 All the news on AI
 
 
 
 
 
 
 
 
	
	
 
 
 
 
 
	
	
	
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 News Overview
 
 
 
 
 
 
 
 
	
	
 
 
 
 
 
 
 
 
 
 
  
 
 Keep following us
Augmenting human intelligence in drug discovery with AI
https://www.tue.nl/en/news-and-events/news-overview/15-09-2021-augmenting-human-intelligence-in-drug-discovery-with-ai/
https://en.wikipedia.org/wiki/Intelligence_amplification
Intelligence amplification - Wikipedia
2006-02-03
None
<div><div>
	<p>From Wikipedia, the free encyclopedia</p>
	</div><div>
<p><b>Intelligence amplification</b> (<b>IA</b>) (also referred to as <b>cognitive augmentation</b>, <b>machine augmented intelligence</b> and <b>enhanced intelligence</b>) refers to the effective use of <a href="/wiki/Information_technology">information technology</a> in augmenting <a href="/wiki/Intelligence#Human_intelligence">human intelligence</a>. The idea was first proposed in the 1950s and 1960s by <a href="/wiki/Cybernetics">cybernetics</a> and early <a href="/wiki/Computer_pioneer">computer pioneers</a>.
</p><p>IA is sometimes contrasted with AI (<a href="/wiki/Artificial_intelligence">artificial intelligence</a>), that is, the project of building a human-like intelligence in the form of an autonomous technological system such as a computer or robot. AI has encountered many fundamental obstacles, practical as well as theoretical, which for IA seem moot, as it needs technology merely as an extra support for an autonomous intelligence that has already proven to function. Moreover, IA has a long history of success, since all forms of information technology, from the abacus to writing to the Internet, have been developed basically to extend the <a href="/wiki/Information_processing">information processing</a> capabilities of the human mind (see <a href="/wiki/Extended_mind">extended mind</a> and <a href="/wiki/Distributed_cognition">distributed cognition</a>).
</p>
<h2>Major contributions[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=1">edit</a>]</h2>
<h3>William Ross Ashby: <i>Intelligence Amplification</i>[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=2">edit</a>]</h3>
<p>The term <i>intelligence amplification</i> (IA) has enjoyed a wide currency since <a href="/wiki/William_Ross_Ashby">William Ross Ashby</a> wrote of "amplifying intelligence" in his <i>Introduction to Cybernetics</i> (1956). Related ideas were explicitly proposed as an alternative to <a href="/wiki/Artificial_Intelligence">Artificial Intelligence</a> by <a href="/wiki/Hao_Wang_(academic)">Hao Wang</a> from the early days of <a href="/wiki/Automated_theorem_proving">automatic theorem provers</a>.
</p>
<blockquote><p>... "<a href="/wiki/Problem_solving">problem solving</a>" is largely, perhaps entirely, a matter of appropriate <a href="https://en.wiktionary.org/wiki/selection">selection</a>. Take, for instance, any popular book of problems and <a href="/wiki/Puzzles">puzzles</a>. Almost every one can be reduced to the form: out of a certain set, indicate one element. ... It is, in fact, difficult to think of a problem, either playful or serious, that does not ultimately require an appropriate selection as necessary and sufficient for its solution.
</p><p>It is also clear that many of the <a href="/wiki/Intelligence_tests">tests used for measuring "intelligence"</a> are scored essentially according to the candidate's power of appropriate selection. ... Thus it is not impossible that what is commonly referred to as "intellectual power" may be equivalent to "power of appropriate selection". Indeed, if a talking <a href="/wiki/Black_box">Black Box</a> were to show high power of appropriate selection in such matters—so that, when given difficult problems it persistently gave correct answers—we could hardly deny that it was showing the '<a href="/wiki/Behavioral">behavioral</a>' equivalent of "high intelligence".
</p><p>
If this is so, and as we know that power of selection can be amplified, it seems to follow that intellectual power, like physical power, can be amplified. Let no one say that it cannot be done, for the gene-patterns do it every time they form a brain that grows up to be something better than the gene-pattern could have specified in detail. What is new is that we can now do it synthetically, consciously, deliberately.</p></blockquote>
<h3>J. C. R. Licklider: <i>Man-Computer Symbiosis</i>[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=3">edit</a>]</h3>
<p>"Man-Computer Symbiosis" is a key speculative paper published in 1960 by <a href="/wiki/Psychologist">psychologist</a>/<a href="/wiki/Computer_scientist">computer scientist</a> <a href="/wiki/J.C.R._Licklider">J.C.R. Licklider</a>, which envisions that mutually-interdependent, "living together", tightly-coupled human brains and computing machines would prove to complement each other's strengths to a high degree:
</p>
<blockquote><p>Man-computer <a href="/wiki/Symbiosis">symbiosis</a> is a subclass of man-machine systems. There are many man-machine systems. At present, however, there are no man-computer symbioses. The purposes of this paper are to present the concept and, hopefully, to foster the development of man-computer symbiosis by analyzing some problems of interaction between men and computing machines, calling attention to applicable principles of man-machine engineering, and pointing out a few questions to which research answers are needed. The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today.</p></blockquote>
<p>In Licklider's vision, many of the pure artificial intelligence systems envisioned at the time by over-optimistic researchers would prove unnecessary. (This paper is also seen by some historians as marking the genesis of ideas about <a href="/wiki/Computer_networks">computer networks</a> which later blossomed into the <a href="/wiki/Internet">Internet</a>).
</p>
<h3>Douglas Engelbart: <i>Augmenting Human Intellect</i>[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=4">edit</a>]</h3>
<p>Licklider's research was similar in spirit to his <a href="/wiki/DARPA">DARPA</a> contemporary and <a href="/wiki/Prot%C3%A9g%C3%A9">protégé</a> <a href="/wiki/Douglas_Engelbart">Douglas Engelbart</a>. Both had a view of how computers could be used that was both at odds with the then-prevalent views (which saw them as devices principally useful for <a href="/wiki/Computation">computations</a>), and key proponents of the way in which computers are now used (as generic adjuncts to humans).<sup><a href="#cite_note-1">[1]</a></sup>
</p><p>Engelbart reasoned that the state of our current technology controls our ability to manipulate information, and that fact in turn will control our ability to develop new, improved technologies. He thus set himself to the revolutionary task of developing computer-based technologies for manipulating information directly, and also to improve individual and group processes for <a href="/wiki/Knowledge-work">knowledge-work</a>. Engelbart's philosophy and research agenda is most clearly and directly expressed in the 1962 research report: <i>Augmenting Human Intellect: A Conceptual Framework</i><sup><a href="#cite_note-E3906-2">[2]</a></sup> The concept of network augmented intelligence is attributed to Engelbart based on this pioneering work.
</p>
<blockquote><p>Increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems.
</p><p>Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insolvable. And by complex situations we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years.
</p><p>
We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human feel for a situation usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.</p></blockquote>
<p>Engelbart subsequently implemented these concepts in his <a href="/wiki/Augmentation_Research_Center">Augmented Human Intellect Research Center</a> at <a href="/wiki/SRI_International">SRI International</a>, developing essentially an intelligence amplifying system of tools (<a href="/wiki/NLS_(computer_system)">NLS</a>) and co-evolving organizational methods, in full operational use by the mid-1960s within the lab. As intended,<sup><a href="#cite_note-:0-3">[3]</a></sup> his R&amp;D team experienced increasing degrees of intelligence amplification, as both rigorous users and rapid-prototype developers of the system. For a sampling of research results, see their 1968 <a href="/wiki/The_Mother_of_All_Demos">Mother of All Demos</a>.
</p>
<h2>Later contributions[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=5">edit</a>]</h2>
<p><a href="/wiki/Howard_Rheingold">Howard Rheingold</a> worked at <a href="/wiki/Xerox_PARC">Xerox PARC</a> in the 1980s and was introduced to both <a href="/wiki/Robert_Taylor_(computer_scientist)">Bob Taylor</a> and <a href="/wiki/Douglas_Engelbart">Douglas Engelbart</a>; Rheingold wrote about "mind amplifiers" in his 1985 book, <i><a href="/wiki/Tools_for_Thought">Tools for Thought</a></i>.<sup><a href="#cite_note-4">[4]</a></sup>
Andrews Samraj mentioned in "Skin-Close Computing and Wearable Technology" 2021, about Human augmentation by two varieties of cyborgs, namely, Hard cyborgs and Soft cyborgs. A humanoid walking machine is an example of the soft cyborg and a pace-maker is an example for augmenting human as a hard cyborg.
</p><p>Arnav Kapur working at <a href="/wiki/Massachusetts_Institute_of_Technology">MIT</a> wrote about human-AI coalescence: how AI can be integrated into human condition as part of "human self": as a tertiary layer to the human brain to augment human cognition.<sup><a href="#cite_note-5">[5]</a></sup> He demonstrates this using a peripheral nerve-computer interface, <a href="/wiki/AlterEgo">AlterEgo</a>, which enables a human user to silently and internally converse with a personal AI.<sup><a href="#cite_note-6">[6]</a></sup><sup><a href="#cite_note-7">[7]</a></sup>
</p><p>In 2014 the technology of Artificial Swarm Intelligence was developed to amplify the intelligence of networked human groups using AI algorithms modeled on biological swarms. The technology enables small teams to make predictions, estimations and medical diagnoses at accuracy levels that significantly exceed natural human intelligence.<sup><a href="#cite_note-8">[8]</a></sup><sup><a href="#cite_note-9">[9]</a></sup><sup><a href="#cite_note-10">[10]</a></sup><sup><a href="#cite_note-11">[11]</a></sup>
</p><p>Shan Carter and <a href="/wiki/Michael_Nielsen">Michael Nielsen</a> introduce the concept of artificial intelligence augmentation (AIA): the use of AI systems to help develop new methods for intelligence augmentation. They contrast cognitive outsourcing (<i>AI as an oracle, able to solve some large class of problems with better-than-human performance</i>) with cognitive transformation (<i>changing the operations and representations we use to think</i>).<sup><a href="#cite_note-12">[12]</a></sup> A calculator is an example of the former; a spreadsheet of the latter.
</p><p>Ron Fulbright describes human cognitive augmentation in human/cog ensembles involving humans working in collaborative partnership with cognitive systems (called cogs). By working together, human/cog ensembles achieve results superior to those obtained by the humans working alone or the cognitive systems working alone. The human component of the ensemble is therefore cognitively augmented. The degree of augmentation depends on the proportion of the total amount of cognition done by the human and that done by the cog. Six Levels of Cognitive Augmentation have been identified: <sup><a href="#cite_note-13">[13]</a></sup><sup><a href="#cite_note-14">[14]</a></sup>
</p>
<div><div><p>Levels of Human Cognitive Augmentation</p></div><div><ul>
	<li><div><p><a href="/wiki/File:Levels_of_Augmentation.png"></a></p></div></li>
</ul></div></div>
<h2>In science fiction[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=6">edit</a>]</h2>
<p>Augmented intelligence has been a repeating theme in <a href="/wiki/Science_fiction">science fiction</a>. A positive view of <a href="/wiki/Brain_implant">brain implants</a> used to communicate with a computer as a form of <a href="/wiki/Augmented_intelligence">augmented intelligence</a> is seen in <a href="/wiki/Algis_Budrys">Algis Budrys</a> 1976 novel <i><a href="/wiki/Michaelmas_(novel)">Michaelmas</a></i>. Fear that the technology will be misused by the government and military is an early theme. In the 1981 BBC serial <i><a href="/wiki/The_Nightmare_Man_(TV_series)">The Nightmare Man</a></i> the pilot of a high-tech mini submarine is linked to his craft via a brain implant but becomes a savage killer after ripping out the implant.
</p><p>Perhaps the most well known writer exploring themes of intelligence augmentation is <a href="/wiki/William_Gibson">William Gibson</a>, in work such as his 1981 story "<a href="/wiki/Johnny_Mnemonic">Johnny Mnemonic</a>", in which the title character has computer-augmented memory, and his 1984 novel <i><a href="/wiki/Neuromancer">Neuromancer</a></i>, in which <a href="/wiki/Computer_hackers">computer hackers</a> interface through brain-computer interfaces to computer systems. <a href="/wiki/Vernor_Vinge">Vernor Vinge</a>, as discussed earlier, looked at intelligence augmentation as a possible route to the <a href="/wiki/Technological_singularity">technological singularity</a>, a theme which also appears in his fiction.
</p>
<h2>See also[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=7">edit</a>]</h2>
<div>
<ul><li><a href="/wiki/Advanced_chess">Advanced chess</a></li>
<li><a href="/wiki/Augmented_learning">Augmented learning</a></li>
<li><a href="/wiki/Brain%E2%80%93computer_interface">Brain–computer interface</a></li>
<li><a href="/wiki/Charles_Sanders_Peirce">Charles Sanders Peirce</a></li>
<li><a href="/wiki/Collective_intelligence">Collective intelligence</a></li>
<li><a href="/wiki/Democratic_transhumanism">Democratic transhumanism</a></li>
<li><a href="/wiki/Emotiv_Systems">Emotiv Systems</a></li>
<li><a href="/wiki/Extelligence">Extelligence</a></li>
<li><a href="/wiki/Exocortex">Exocortex</a></li>
<li><a href="/wiki/Knowledge_worker">Knowledge worker</a></li>
<li><a href="/wiki/Mechanization">Mechanization</a></li>
<li><a href="/wiki/Neuroenhancement">Neuroenhancement</a></li>
<li><a href="/wiki/Noogenesis">Noogenesis</a></li>
<li><a href="/wiki/Nootropic">Nootropic</a></li>
<li><a href="/wiki/Sensemaking_(information_science)">Sensemaking (information science)</a></li>
<li><i><a href="/wiki/The_Wisdom_of_Crowds">The Wisdom of Crowds</a></i></li></ul>
</div>
<h2>References[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=8">edit</a>]</h2>
<div><ol>
<li><b><a href="#cite_ref-1">^</a></b> <cite>Markoff, John (2013-07-03). <a href="https://www.nytimes.com/2013/07/04/technology/douglas-c-engelbart-inventor-of-the-computer-mouse-dies-at-88.html">"Computer Visionary Who Invented the Mouse"</a>. <i>The New York Times</i>. <a href="/wiki/ISSN_(identifier)">ISSN</a> <a href="//www.worldcat.org/issn/0362-4331">0362-4331</a>. Retrieved 2020-04-10.</cite>
</li>
<li>^ <a href="#cite_ref-E3906_2-0"><sup><i><b>a</b></i></sup></a> <a href="#cite_ref-E3906_2-1"><sup><i><b>b</b></i></sup></a> <a href="http://www.dougengelbart.org/pubs/augment-3906.html">"Augmenting Human Intellect: A Conceptual Framework"</a> <a href="https://web.archive.org/web/20110504035147/http://www.dougengelbart.org/pubs/augment-3906.html">Archived</a> 2011-05-04 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a> (October 1962), DougEngelbart.org.
</li>
<li><b><a href="#cite_ref-:0_3-0">^</a></b> <a href="http://www.dougengelbart.org/pubs/augment-3906.html#4d">"Augmenting Human Intellect: A Conceptual Framework"</a> <a href="https://web.archive.org/web/20110504035147/http://www.dougengelbart.org/pubs/augment-3906.html#4d">Archived</a> 2011-05-04 at the <a href="/wiki/Wayback_Machine">Wayback Machine</a>, Section D: Regenerative Feature, in D. C. Engelbart Summary Report AFOSR-3233, <a href="/wiki/Stanford_Research_Institute">Stanford Research Institute</a>, Menlo Park, CA, October 1962.
</li>
<li><b><a href="#cite_ref-4">^</a></b> <cite><a href="/wiki/Howard_Rheingold">Rheingold, Howard</a> (2000) [1985]. <a href="https://archive.org/details/toolsforthoughth00rhei"><i>Tools for thought: the history and future of mind-expanding technology</i></a> (Reprint ed.). Cambridge, MA: <a href="/wiki/MIT_Press">MIT Press</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0262681155"><bdi>978-0262681155</bdi></a>. <a href="/wiki/OCLC_(identifier)">OCLC</a> <a href="//www.worldcat.org/oclc/43076809">43076809</a>.</cite> See also Rheingold's site: <cite><a href="http://rheingold.com/about/">"About Howard Rheingold"</a>. <i>rheingold.com</i>. Retrieved 2017-12-28.</cite>
</li>
<li><b><a href="#cite_ref-5">^</a></b> <cite>Kapur, Arnav (April 2019). <i>Human-machine cognitive coalescence through an internal duplex interface</i> (Thesis). Massachusetts Institute of Technology. <a href="/wiki/Hdl_(identifier)">hdl</a>:<a href="//hdl.handle.net/1721.1%2F120883">1721.1/120883</a>.</cite>
</li>
<li><b><a href="#cite_ref-6">^</a></b> <cite><a href="https://www.media.mit.edu/projects/alterego/overview/">"AlterEgo"</a>. <i>MIT Media Lab</i>. Retrieved April 30, 2019.</cite>
</li>
<li><b><a href="#cite_ref-7">^</a></b> <cite>Kapur, Arnav; Kapur, Shreyas; Maes, Pattie (2018). "AlterEgo". <i>Proceedings of the 2018 Conference on Human Information Interaction&amp;Retrieval - IUI '18</i>. New York, New York, USA: ACM Press: 43–53. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1145%2F3172944.3172977">10.1145/3172944.3172977</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/9781450349451"><bdi>9781450349451</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:3777401">3777401</a>.</cite>
</li>
<li><b><a href="#cite_ref-8">^</a></b> <cite>Willcox, G.; Rosenberg, L. (September 2019). <a href="https://ieeexplore.ieee.org/document/9027782">"Swarm Intelligence Amplifies the IQ of Collaborating Teams"</a>. <i>2019 Second International Conference on Artificial Intelligence for Industries (AI4I)</i>: 111–114. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1109%2FAI4I46381.2019.00036">10.1109/AI4I46381.2019.00036</a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:212646000">212646000</a>.</cite>
</li>
<li><b><a href="#cite_ref-9">^</a></b> <cite>Rosenberg, L.; Willcox, G.; Askay, D.; Metcalf, L.; Harris, E. (September 2018). <a href="https://ieeexplore.ieee.org/document/8665698">"Amplifying the Social Intelligence of Teams Through Human Swarming"</a>. <i>2018 First International Conference on Artificial Intelligence for Industries (AI4I)</i>: 23–26. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1109%2FAI4I.2018.8665698">10.1109/AI4I.2018.8665698</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-1-5386-9209-7"><bdi>978-1-5386-9209-7</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:77385269">77385269</a>.</cite>
</li>
<li><b><a href="#cite_ref-10">^</a></b> <cite>Rosenberg, L.; Pescetelli, N. (September 2017). <a href="https://ieeexplore.ieee.org/document/8324329">"Amplifying prediction accuracy using Swarm A.I."</a> <i>2017 Intelligent Systems Conference (IntelliSys)</i>: 61–65. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1109%2FIntelliSys.2017.8324329">10.1109/IntelliSys.2017.8324329</a>. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-1-5090-6435-9"><bdi>978-1-5090-6435-9</bdi></a>. <a href="/wiki/S2CID_(identifier)">S2CID</a> <a href="https://api.semanticscholar.org/CorpusID:4366745">4366745</a>.</cite>
</li>
<li><b><a href="#cite_ref-11">^</a></b> <cite><a href="https://www.stanforddaily.com/2018/09/27/artificial-swarm-intelligence-diagnoses-pneumonia-better-than-individual-computer-or-doctor/">"Artificial swarm intelligence diagnoses pneumonia better than individual computer or doctor"</a>. <i>The Stanford Daily</i>. 2018-09-27.</cite><code>{{<a href="/wiki/Template:Cite_web">cite web</a>}}</code>: CS1 maint: url-status (<a href="/wiki/Category:CS1_maint:_url-status">link</a>)
</li>
<li><b><a href="#cite_ref-12">^</a></b> <cite>Carter, Shan; Nielsen, Michael (2017). "Using artificial intelligence to augment human intelligence". <i>Distill</i>. <b>2</b> (12): e9. <a href="/wiki/ArXiv_(identifier)">arXiv</a>:<a href="//arxiv.org/abs/1609.04468">1609.04468</a>. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.23915%2Fdistill.00009">10.23915/distill.00009</a>.</cite>
</li>
<li><b><a href="#cite_ref-13">^</a></b> <cite>Fulbright, Ron (2020). <i>Democratization of Expertise: How Cognitive Systems Will Revolutionize Your Life</i>. Boca Raton, FL: CRC Press. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0367859459"><bdi>978-0367859459</bdi></a>.</cite>
</li>
<li><b><a href="#cite_ref-14">^</a></b> <cite>Fulbright, Ron (2020). <a href="https://link.springer.com/chapter/10.1007/978-3-030-50439-7_3">"Synthetic Expertise"</a>. <i>Augmented Cognition. Human Cognition and Behavior</i>. <b>12197</b>: 27–48 – via Springer.</cite>
</li>
</ol></div>
<h2>Further reading[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=9">edit</a>]</h2>
<div>
<ul><li><cite>Engelbart, D.; Landau, V.; Clegg, E. (2009). <i>The Engelbart Hypothesis: Dialogs with Douglas Engelbart</i>. NextPress. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0615308906"><bdi>978-0615308906</bdi></a>.</cite></li>
<li><cite>Asaro, Peter (2008). <a href="http://peterasaro.org/writing/Asaro%20Ashby.pdf">"From Mechanisms of Adaptation to Intelligence Amplifiers: The Philosophy of W. Ross Ashby"</a> (PDF). In Husbands, P.; Holland, O.; Wheeler, M. (eds.). <a href="https://archive.org/details/mechanicalmindin00phil"><i>The Mechanical Mind in History</i></a>. MIT Press. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-262-08377-5"><bdi>978-0-262-08377-5</bdi></a>.</cite></li>
<li><cite><a href="/wiki/William_Ross_Ashby">Ashby, W.R.</a> (2013) [1960]. <a href="https://books.google.com/books?id=Dc4hBQAAQBAJ"><i>Design for a Brain: The origin of adaptive behaviour</i></a> (2nd ed.). Springer. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-94-015-1320-3"><bdi>978-94-015-1320-3</bdi></a>.</cite></li>
<li><cite>Skagestad, Peter (1993). <a href="http://www.cspeirce.com/menu/library/aboutcsp/skagesta/thinking.htm">"Thinking with Machines: Intelligence Augmentation, Evolutionary Epistemology, and Semiotic"</a>. <i>Journal of Social and Evolutionary Systems</i>. <b>16</b> (2): 157–180. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.1016%2F1061-7361%2893%2990026-N">10.1016/1061-7361(93)90026-N</a>.</cite></li>
<li><cite>Bray, David A. (2007). "Smart Business Networks (or, Let's Create 'Life' from Inert Information)". <i>Smart Business Network Initiative (SBNi) Conference — Beijing, May 2008</i>. SSRN. <a href="/wiki/Doi_(identifier)">doi</a>:<a href="https://doi.org/10.2139%2Fssrn.991163">10.2139/ssrn.991163</a>. <a href="/wiki/SSRN_(identifier)">SSRN</a> <a href="//papers.ssrn.com/sol3/papers.cfm?abstract_id=991163">991163</a>.</cite></li>
<li><cite>Waldrop, M. Mitchell (2001). <a href="https://archive.org/details/dreammachinejcrl00wald"><i>The Dream Machine: J.C.R. Licklider and the Revolution that Made Computing Personal</i></a>. Viking. <a href="/wiki/ISBN_(identifier)">ISBN</a> <a href="/wiki/Special:BookSources/978-0-670-89976-0"><bdi>978-0-670-89976-0</bdi></a>.</cite> Licklider's biography, contains discussion of the importance of this paper.</li></ul>
</div>
<h2>External links[<a href="/w/index.php?title=Intelligence_amplification&amp;action=edit&amp;section=10">edit</a>]</h2>
<ul><li><a href="http://www.turnkeylinux.org/blog/practical-intelligence-amplification">Intelligence Amplification using speech synthesis technology</a></li>
<li><a href="http://www.itconversations.com/shows/detail378.html">IT Conversations: Doug Engelbart - Large-Scale Collective IQ</a></li>
<li><a href="http://www.rossashby.info/journal/page/3609.html">7 December 1951, Ashby first wrote about the possibility to build an 'information amplifier'.</a></li>
<li><a href="http://www.rossashby.info/journal/page/4584.html">12 August 1953, Ashby mentioned an objection to his 'intelligence-amplifier'.</a></li></ul>
</div>||||I|||| Jump to content
 Toggle sidebar
 Search
 * Create account
 Personal tools
 * Create account
 * Log in

 Pages for logged out editors learn more

 * Talk
 * Contributions
 Navigation
 * Main page
 * Contents
 * Current events
 * Random article
 * About Wikipedia
 * Contact us
 * Donate
 Contribute
 * Help
 * Learn to edit
 * Community portal
 * Recent changes
 * Upload file
 Tools
 * What links here
 * Related changes
 * Upload file
 * Special pages
 * Permanent link
 * Page information
 * Cite this page
 * Wikidata item
 Print/export
 * Download as PDF
 * Printable version
 Languages
 On this Wikipedia the language links are at the top of the page across from the article title. Go to top.

 Contents

 move to sidebar hide
 * (Top)
 * 1 Major contributions
 + 1.1 William Ross Ashby: Intelligence Amplification
 + 1.2 J. C. R. Licklider: Man-Computer Symbiosis
 + 1.3 Douglas Engelbart: Augmenting Human Intellect
 * 2 Later contributions
 * 3 In science fiction
 * 4 See also
 * 5 References
 * 6 Further reading
 * 7 External links
 Toggle the table of contents

 Intelligence amplification

 6 languages
 * Deutsch
 * 日本語
 * Русский
 * Українська
 * 粵語
 * 中文
 Edit links
 * Article
 * Talk
 English
 * Read
 * Edit
 * View history
 More
 * Read
 * Edit
 * View history
 From Wikipedia, the free encyclopedia
 Not to be confused with Intelligent agent.
 This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources . Unsourced material may be challenged and removed.
 Find sources: "Intelligence amplification" – news · newspapers · books · scholar · JSTOR ( June 2017 ) (Learn how and when to remove this template message) 
 
 Part of a series on 
 Cyborgs 
 
 Cyborgology 
 * Bionics 
 * Biomimicry 
 * Biomedical engineering 
 * Brain–computer interface 
 * Cybernetics 
 * Distributed cognition 
 * Genetic engineering 
 * Human ecosystem 
 * Human enhancement 
 * Intelligence amplification
 * Whole brain emulation 
 Theory 
 * Cyborg anthropology 
 Centers 
 * Cyberpunk 
 * Cyberspace 
 Politics 
 * Cognitive liberty 
 * Extropianism 
 * Morphological freedom 
 * Singularitarianism 
 * Techno-progressivism 
 * Transhumanism 
 Related articles 
 * Cyborg art 
 * v 
 * t 
 * e 
 

 Intelligence amplification (IA) (also referred to as cognitive augmentation, machine augmented intelligence and enhanced intelligence) refers to the effective use of information technology in augmenting human intelligence. The idea was first proposed in the 1950s and 1960s by cybernetics and early computer pioneers.

 IA is sometimes contrasted with AI (artificial intelligence), that is, the project of building a human-like intelligence in the form of an autonomous technological system such as a computer or robot. AI has encountered many fundamental obstacles, practical as well as theoretical, which for IA seem moot, as it needs technology merely as an extra support for an autonomous intelligence that has already proven to function. Moreover, IA has a long history of success, since all forms of information technology, from the abacus to writing to the Internet, have been developed basically to extend the information processing capabilities of the human mind (see extended mind and distributed cognition).

 Major contributions [ edit ]

 William Ross Ashby: Intelligence Amplification [ edit ]

 The term intelligence amplification (IA) has enjoyed a wide currency since William Ross Ashby wrote of "amplifying intelligence" in his Introduction to Cybernetics (1956). Related ideas were explicitly proposed as an alternative to Artificial Intelligence by Hao Wang from the early days of automatic theorem provers.

 ... "problem solving" is largely, perhaps entirely, a matter of appropriate selection. Take, for instance, any popular book of problems and puzzles. Almost every one can be reduced to the form: out of a certain set, indicate one element. ... It is, in fact, difficult to think of a problem, either playful or serious, that does not ultimately require an appropriate selection as necessary and sufficient for its solution.

 It is also clear that many of the tests used for measuring "intelligence" are scored essentially according to the candidate's power of appropriate selection. ... Thus it is not impossible that what is commonly referred to as "intellectual power" may be equivalent to "power of appropriate selection". Indeed, if a talking Black Box were to show high power of appropriate selection in such matters—so that, when given difficult problems it persistently gave correct answers—we could hardly deny that it was showing the 'behavioral' equivalent of "high intelligence".

 If this is so, and as we know that power of selection can be amplified, it seems to follow that intellectual power, like physical power, can be amplified. Let no one say that it cannot be done, for the gene-patterns do it every time they form a brain that grows up to be something better than the gene-pattern could have specified in detail. What is new is that we can now do it synthetically, consciously, deliberately.

 — W. Ross Ashby, An Introduction to Cybernetics, Chapman and Hall, London, UK, 1956. Reprinted, Methuen and Company, London, UK, 1964.

 J. C. R. Licklider: Man-Computer Symbiosis [ edit ]

 "Man-Computer Symbiosis" is a key speculative paper published in 1960 by psychologist/computer scientist J.C.R. Licklider, which envisions that mutually-interdependent, "living together", tightly-coupled human brains and computing machines would prove to complement each other's strengths to a high degree:

 Man-computer symbiosis is a subclass of man-machine systems. There are many man-machine systems. At present, however, there are no man-computer symbioses. The purposes of this paper are to present the concept and, hopefully, to foster the development of man-computer symbiosis by analyzing some problems of interaction between men and computing machines, calling attention to applicable principles of man-machine engineering, and pointing out a few questions to which research answers are needed. The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today.

 — J. C. R. Licklider, "Man-Computer Symbiosis", IRE Transactions on Human Factors in Electronics, vol. HFE-1, 4-11, March 1960.

 In Licklider's vision, many of the pure artificial intelligence systems envisioned at the time by over-optimistic researchers would prove unnecessary. (This paper is also seen by some historians as marking the genesis of ideas about computer networks which later blossomed into the Internet).

 Douglas Engelbart: Augmenting Human Intellect [ edit ]

 Licklider's research was similar in spirit to his DARPA contemporary and protégé Douglas Engelbart. Both had a view of how computers could be used that was both at odds with the then-prevalent views (which saw them as devices principally useful for computations), and key proponents of the way in which computers are now used (as generic adjuncts to humans).[1]

 Engelbart reasoned that the state of our current technology controls our ability to manipulate information, and that fact in turn will control our ability to develop new, improved technologies. He thus set himself to the revolutionary task of developing computer-based technologies for manipulating information directly, and also to improve individual and group processes for knowledge-work. Engelbart's philosophy and research agenda is most clearly and directly expressed in the 1962 research report: Augmenting Human Intellect: A Conceptual Framework[2] The concept of network augmented intelligence is attributed to Engelbart based on this pioneering work.

 Increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems.

 Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insolvable. And by complex situations we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years.

 We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human feel for a situation usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.

 — Douglas Engelbart, Augmenting Human Intellect: A Conceptual Framework, Summary Report AFOSR-3233, Stanford Research Institute, Menlo Park, CA, October 1962.[2]

 Engelbart subsequently implemented these concepts in his Augmented Human Intellect Research Center at SRI International, developing essentially an intelligence amplifying system of tools (NLS) and co-evolving organizational methods, in full operational use by the mid-1960s within the lab. As intended,[3] his R&D team experienced increasing degrees of intelligence amplification, as both rigorous users and rapid-prototype developers of the system. For a sampling of research results, see their 1968 Mother of All Demos.

 Later contributions [ edit ]

 Howard Rheingold worked at Xerox PARC in the 1980s and was introduced to both Bob Taylor and Douglas Engelbart; Rheingold wrote about "mind amplifiers" in his 1985 book, Tools for Thought.[4] Andrews Samraj mentioned in "Skin-Close Computing and Wearable Technology" 2021, about Human augmentation by two varieties of cyborgs, namely, Hard cyborgs and Soft cyborgs. A humanoid walking machine is an example of the soft cyborg and a pace-maker is an example for augmenting human as a hard cyborg.

 Arnav Kapur working at MIT wrote about human-AI coalescence: how AI can be integrated into human condition as part of "human self": as a tertiary layer to the human brain to augment human cognition.[5] He demonstrates this using a peripheral nerve-computer interface, AlterEgo, which enables a human user to silently and internally converse with a personal AI.[6][7]

 In 2014 the technology of Artificial Swarm Intelligence was developed to amplify the intelligence of networked human groups using AI algorithms modeled on biological swarms. The technology enables small teams to make predictions, estimations and medical diagnoses at accuracy levels that significantly exceed natural human intelligence.[8][9][10][11]

 Shan Carter and Michael Nielsen introduce the concept of artificial intelligence augmentation (AIA): the use of AI systems to help develop new methods for intelligence augmentation. They contrast cognitive outsourcing (AI as an oracle, able to solve some large class of problems with better-than-human performance) with cognitive transformation (changing the operations and representations we use to think).[12] A calculator is an example of the former; a spreadsheet of the latter.

 Ron Fulbright describes human cognitive augmentation in human/cog ensembles involving humans working in collaborative partnership with cognitive systems (called cogs). By working together, human/cog ensembles achieve results superior to those obtained by the humans working alone or the cognitive systems working alone. The human component of the ensemble is therefore cognitively augmented. The degree of augmentation depends on the proportion of the total amount of cognition done by the human and that done by the cog. Six Levels of Cognitive Augmentation have been identified: [13][14]

 Levels of Human Cognitive Augmentation
 * 


 In science fiction [ edit ]

 Augmented intelligence has been a repeating theme in science fiction. A positive view of brain implants used to communicate with a computer as a form of augmented intelligence is seen in Algis Budrys 1976 novel Michaelmas. Fear that the technology will be misused by the government and military is an early theme. In the 1981 BBC serial The Nightmare Man the pilot of a high-tech mini submarine is linked to his craft via a brain implant but becomes a savage killer after ripping out the implant.

 Perhaps the most well known writer exploring themes of intelligence augmentation is William Gibson, in work such as his 1981 story "Johnny Mnemonic", in which the title character has computer-augmented memory, and his 1984 novel Neuromancer, in which computer hackers interface through brain-computer interfaces to computer systems. Vernor Vinge, as discussed earlier, looked at intelligence augmentation as a possible route to the technological singularity, a theme which also appears in his fiction.

 See also [ edit ]

 * Advanced chess
 * Augmented learning
 * Brain–computer interface
 * Charles Sanders Peirce
 * Collective intelligence
 * Democratic transhumanism
 * Emotiv Systems
 * Extelligence
 * Exocortex
 * Knowledge worker
 * Mechanization
 * Neuroenhancement
 * Noogenesis
 * Nootropic
 * Sensemaking (information science)
 * The Wisdom of Crowds

 References [ edit ]

 1. ^ Markoff, John (2013-07-03). "Computer Visionary Who Invented the Mouse". The New York Times. ISSN 0362-4331 . Retrieved 2020-04-10 .
 2. ^ a b "Augmenting Human Intellect: A Conceptual Framework" Archived 2011-05-04 at the Wayback Machine (October 1962), DougEngelbart.org.
 3. ^ "Augmenting Human Intellect: A Conceptual Framework" Archived 2011-05-04 at the Wayback Machine , Section D: Regenerative Feature, in D. C. Engelbart Summary Report AFOSR-3233, Stanford Research Institute , Menlo Park, CA, October 1962.
 4. ^ Rheingold, Howard (2000) [1985]. Tools for thought: the history and future of mind-expanding technology (Reprint ed.). Cambridge, MA: MIT Press. ISBN 978-0262681155. OCLC 43076809. See also Rheingold's site: "About Howard Rheingold". rheingold.com . Retrieved 2017-12-28 .
 5. ^ Kapur, Arnav (April 2019). Human-machine cognitive coalescence through an internal duplex interface (Thesis). Massachusetts Institute of Technology. hdl:1721.1/120883.
 6. ^ "AlterEgo". MIT Media Lab . Retrieved April 30, 2019 .
 7. ^ Kapur, Arnav; Kapur, Shreyas; Maes, Pattie (2018). "AlterEgo". Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - IUI '18. New York, New York, USA: ACM Press: 43–53. doi:10.1145/3172944.3172977. ISBN 9781450349451. S2CID 3777401.
 8. ^ Willcox, G.; Rosenberg, L. (September 2019). "Swarm Intelligence Amplifies the IQ of Collaborating Teams". 2019 Second International Conference on Artificial Intelligence for Industries (AI4I): 111–114. doi:10.1109/AI4I46381.2019.00036. S2CID 212646000.
 9. ^ Rosenberg, L.; Willcox, G.; Askay, D.; Metcalf, L.; Harris, E. (September 2018). "Amplifying the Social Intelligence of Teams Through Human Swarming". 2018 First International Conference on Artificial Intelligence for Industries (AI4I): 23–26. doi:10.1109/AI4I.2018.8665698. ISBN 978-1-5386-9209-7. S2CID 77385269.
 10. ^ Rosenberg, L.; Pescetelli, N. (September 2017). "Amplifying prediction accuracy using Swarm A.I." 2017 Intelligent Systems Conference (IntelliSys): 61–65. doi:10.1109/IntelliSys.2017.8324329. ISBN 978-1-5090-6435-9. S2CID 4366745.
 11. ^ "Artificial swarm intelligence diagnoses pneumonia better than individual computer or doctor". The Stanford Daily. 2018-09-27.{{cite web}} : CS1 maint: url-status ( link )
 12. ^ Carter, Shan; Nielsen, Michael (2017). "Using artificial intelligence to augment human intelligence". Distill. 2 (12): e9. arXiv:1609.04468. doi:10.23915/distill.00009.
 13. ^ Fulbright, Ron (2020). Democratization of Expertise: How Cognitive Systems Will Revolutionize Your Life. Boca Raton, FL: CRC Press. ISBN 978-0367859459.
 14. ^ Fulbright, Ron (2020). "Synthetic Expertise". Augmented Cognition. Human Cognition and Behavior. 12197: 27–48 – via Springer.

 Further reading [ edit ]

 * Engelbart, D.; Landau, V.; Clegg, E. (2009). The Engelbart Hypothesis: Dialogs with Douglas Engelbart. NextPress. ISBN 978-0615308906.
 * Asaro, Peter (2008). "From Mechanisms of Adaptation to Intelligence Amplifiers: The Philosophy of W. Ross Ashby" (PDF) . In Husbands, P.; Holland, O.; Wheeler, M. (eds.). The Mechanical Mind in History. MIT Press. ISBN 978-0-262-08377-5.
 * Ashby, W.R. (2013) [1960]. Design for a Brain: The origin of adaptive behaviour (2nd ed.). Springer. ISBN 978-94-015-1320-3.
 * Skagestad, Peter (1993). "Thinking with Machines: Intelligence Augmentation, Evolutionary Epistemology, and Semiotic". Journal of Social and Evolutionary Systems. 16 (2): 157–180. doi:10.1016/1061-7361(93)90026-N.
 * Bray, David A. (2007). "Smart Business Networks (or, Let's Create 'Life' from Inert Information)". Smart Business Network Initiative (SBNi) Conference — Beijing, May 2008. SSRN. doi:10.2139/ssrn.991163. SSRN 991163.
 * Waldrop, M. Mitchell (2001). The Dream Machine: J.C.R. Licklider and the Revolution that Made Computing Personal. Viking. ISBN 978-0-670-89976-0. Licklider's biography, contains discussion of the importance of this paper.

 External links [ edit ]

 * Intelligence Amplification using speech synthesis technology
 * IT Conversations: Doug Engelbart - Large-Scale Collective IQ
 * 7 December 1951, Ashby first wrote about the possibility to build an 'information amplifier'.
 * 12 August 1953, Ashby mentioned an objection to his 'intelligence-amplifier'.
 * v 
 * t 
 * e 
 Brain–computer interface
 * Biomechatronics 
 * Brain implant 
 * BrainGate 
 * Brainport 
 * Cyberware 
 * Exocortex 
 Technologies * Intelligence amplification 
 * Isolated brain 
 * Neuroprosthetics 
 * Neurotechnology 
 * Optogenetics 
 * Sensory substitution 
 * Stentrode 
 * Synthetic telepathy 
 * Electrocorticography (ECoG)
 Scientific phenomena * Neural ensemble 
 * Neuroplasticity 
 * Cognitive science 
 * Cognitive neuroscience 
 Disciplines * Computational neuroscience 
 * NBIC 
 * Neural engineering 
 * Neuroscience 
 * Brain transplant 
 Speculative * Cyborg 
 * Mind uploading 
 * Charles Stross 
 * Douglas Engelbart 
 * Hugh Herr 
 * J. C. R. Licklider 
 * Kevin Warwick 
 * Matt Nagle 
 People * Merlin Donald 
 * Miguel Nicolelis 
 * Peter Kyberd 
 * Steve Mann 
 * Vernor Vinge 
 * Yoky Matsuoka 
 * Edward Boyden 
 * Human enhancement 
 Other * Neurohacking 
 * Simulated reality 
 * Transhumanism 
 * Category 
 * Commons 
 
 Retrieved from "https://en.wikipedia.org/w/index.php?title=Intelligence_amplification&oldid=1124183742"
 Categories:
 * History of human–computer interaction
 * Cybernetics
 * Biocybernetics
 * Transhumanism
 * Texts related to the history of the Internet
 * Intelligence
 Hidden categories:
 * Webarchive template wayback links
 * CS1 maint: url-status
 * Articles with short description
 * Short description is different from Wikidata
 * Articles needing additional references from June 2017
 * All articles needing additional references
 * This page was last edited on 27 November 2022, at 19:13 (UTC) .
 * Text is available under the Creative Commons Attribution-ShareAlike License 3.0; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization.
 * Privacy policy
 * About Wikipedia
 * Disclaimers
 * Contact Wikipedia
 * Mobile view
 * Developers
 * Statistics
 * Cookie statement
 * 
 * 
From Wikipedia, the free encyclopedia
	 
 Intelligence amplification (IA) (also referred to as cognitive augmentation, machine augmented intelligence and enhanced intelligence) refers to the effective use of information technology in augmenting human intelligence. The idea was first proposed in the 1950s and 1960s by cybernetics and early computer pioneers.
 IA is sometimes contrasted with AI (artificial intelligence), that is, the project of building a human-like intelligence in the form of an autonomous technological system such as a computer or robot. AI has encountered many fundamental obstacles, practical as well as theoretical, which for IA seem moot, as it needs technology merely as an extra support for an autonomous intelligence that has already proven to function. Moreover, IA has a long history of success, since all forms of information technology, from the abacus to writing to the Internet, have been developed basically to extend the information processing capabilities of the human mind (see extended mind and distributed cognition).
Major contributions[edit]
William Ross Ashby: Intelligence Amplification[edit]
The term intelligence amplification (IA) has enjoyed a wide currency since William Ross Ashby wrote of "amplifying intelligence" in his Introduction to Cybernetics (1956). Related ideas were explicitly proposed as an alternative to Artificial Intelligence by Hao Wang from the early days of automatic theorem provers.
 ... "problem solving" is largely, perhaps entirely, a matter of appropriate selection. Take, for instance, any popular book of problems and puzzles. Almost every one can be reduced to the form: out of a certain set, indicate one element. ... It is, in fact, difficult to think of a problem, either playful or serious, that does not ultimately require an appropriate selection as necessary and sufficient for its solution.
 It is also clear that many of the tests used for measuring "intelligence" are scored essentially according to the candidate's power of appropriate selection. ... Thus it is not impossible that what is commonly referred to as "intellectual power" may be equivalent to "power of appropriate selection". Indeed, if a talking Black Box were to show high power of appropriate selection in such matters—so that, when given difficult problems it persistently gave correct answers—we could hardly deny that it was showing the 'behavioral' equivalent of "high intelligence".
 
If this is so, and as we know that power of selection can be amplified, it seems to follow that intellectual power, like physical power, can be amplified. Let no one say that it cannot be done, for the gene-patterns do it every time they form a brain that grows up to be something better than the gene-pattern could have specified in detail. What is new is that we can now do it synthetically, consciously, deliberately. 
J. C. R. Licklider: Man-Computer Symbiosis[edit]
"Man-Computer Symbiosis" is a key speculative paper published in 1960 by psychologist/computer scientist J.C.R. Licklider, which envisions that mutually-interdependent, "living together", tightly-coupled human brains and computing machines would prove to complement each other's strengths to a high degree:
 Man-computer symbiosis is a subclass of man-machine systems. There are many man-machine systems. At present, however, there are no man-computer symbioses. The purposes of this paper are to present the concept and, hopefully, to foster the development of man-computer symbiosis by analyzing some problems of interaction between men and computing machines, calling attention to applicable principles of man-machine engineering, and pointing out a few questions to which research answers are needed. The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today. 
In Licklider's vision, many of the pure artificial intelligence systems envisioned at the time by over-optimistic researchers would prove unnecessary. (This paper is also seen by some historians as marking the genesis of ideas about computer networks which later blossomed into the Internet).
Douglas Engelbart: Augmenting Human Intellect[edit]
Licklider's research was similar in spirit to his DARPA contemporary and protégé Douglas Engelbart. Both had a view of how computers could be used that was both at odds with the then-prevalent views (which saw them as devices principally useful for computations), and key proponents of the way in which computers are now used (as generic adjuncts to humans). [1] 
 Engelbart reasoned that the state of our current technology controls our ability to manipulate information, and that fact in turn will control our ability to develop new, improved technologies. He thus set himself to the revolutionary task of developing computer-based technologies for manipulating information directly, and also to improve individual and group processes for knowledge-work. Engelbart's philosophy and research agenda is most clearly and directly expressed in the 1962 research report: Augmenting Human Intellect: A Conceptual Framework  [2]  The concept of network augmented intelligence is attributed to Engelbart based on this pioneering work.
 Increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems.
 Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insolvable. And by complex situations we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years.
 
We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human feel for a situation usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids. 
Engelbart subsequently implemented these concepts in his Augmented Human Intellect Research Center at SRI International, developing essentially an intelligence amplifying system of tools (NLS) and co-evolving organizational methods, in full operational use by the mid-1960s within the lab. As intended, [3]  his R&amp;D team experienced increasing degrees of intelligence amplification, as both rigorous users and rapid-prototype developers of the system. For a sampling of research results, see their 1968 Mother of All Demos.
Later contributions[edit]
 Howard Rheingold worked at Xerox PARC in the 1980s and was introduced to both Bob Taylor and Douglas Engelbart; Rheingold wrote about "mind amplifiers" in his 1985 book,  Tools for Thought . [4] 
Andrews Samraj mentioned in "Skin-Close Computing and Wearable Technology" 2021, about Human augmentation by two varieties of cyborgs, namely, Hard cyborgs and Soft cyborgs. A humanoid walking machine is an example of the soft cyborg and a pace-maker is an example for augmenting human as a hard cyborg.
 Arnav Kapur working at MIT wrote about human-AI coalescence: how AI can be integrated into human condition as part of "human self": as a tertiary layer to the human brain to augment human cognition. [5]  He demonstrates this using a peripheral nerve-computer interface, AlterEgo, which enables a human user to silently and internally converse with a personal AI. [6]   [7] 
 In 2014 the technology of Artificial Swarm Intelligence was developed to amplify the intelligence of networked human groups using AI algorithms modeled on biological swarms. The technology enables small teams to make predictions, estimations and medical diagnoses at accuracy levels that significantly exceed natural human intelligence. [8]   [9]   [10]   [11] 
 Shan Carter and Michael Nielsen introduce the concept of artificial intelligence augmentation (AIA): the use of AI systems to help develop new methods for intelligence augmentation. They contrast cognitive outsourcing (AI as an oracle, able to solve some large class of problems with better-than-human performance) with cognitive transformation (changing the operations and representations we use to think). [12]  A calculator is an example of the former; a spreadsheet of the latter.
 Ron Fulbright describes human cognitive augmentation in human/cog ensembles involving humans working in collaborative partnership with cognitive systems (called cogs). By working together, human/cog ensembles achieve results superior to those obtained by the humans working alone or the cognitive systems working alone. The human component of the ensemble is therefore cognitively augmented. The degree of augmentation depends on the proportion of the total amount of cognition done by the human and that done by the cog. Six Levels of Cognitive Augmentation have been identified:  [13]   [14] 
  Levels of Human Cognitive Augmentation   
	       
  
In science fiction[edit]
Augmented intelligence has been a repeating theme in science fiction. A positive view of brain implants used to communicate with a computer as a form of augmented intelligence is seen in Algis Budrys 1976 novel  Michaelmas . Fear that the technology will be misused by the government and military is an early theme. In the 1981 BBC serial  The Nightmare Man  the pilot of a high-tech mini submarine is linked to his craft via a brain implant but becomes a savage killer after ripping out the implant.
 Perhaps the most well known writer exploring themes of intelligence augmentation is William Gibson, in work such as his 1981 story "Johnny Mnemonic", in which the title character has computer-augmented memory, and his 1984 novel  Neuromancer , in which computer hackers interface through brain-computer interfaces to computer systems. Vernor Vinge, as discussed earlier, looked at intelligence augmentation as a possible route to the technological singularity, a theme which also appears in his fiction.
See also[edit]
  Advanced chess 
 Augmented learning 
 Brain–computer interface 
 Charles Sanders Peirce 
 Collective intelligence 
 Democratic transhumanism 
 Emotiv Systems 
 Extelligence 
 Exocortex 
 Knowledge worker 
 Mechanization 
 Neuroenhancement 
 Noogenesis 
 Nootropic 
 Sensemaking (information science) 
  The Wisdom of Crowds   
References[edit]
 
  ^  Markoff, John (2013-07-03). "Computer Visionary Who Invented the Mouse". The New York Times. ISSN 0362-4331. Retrieved 2020-04-10.
^    a       b    "Augmenting Human Intellect: A Conceptual Framework" Archived 2011-05-04 at the Wayback Machine (October 1962), DougEngelbart.org.
  ^  "Augmenting Human Intellect: A Conceptual Framework" Archived 2011-05-04 at the Wayback Machine, Section D: Regenerative Feature, in D. C. Engelbart Summary Report AFOSR-3233, Stanford Research Institute, Menlo Park, CA, October 1962.
  ^   Rheingold, Howard (2000) [1985].  Tools for thought: the history and future of mind-expanding technology  (Reprint ed.). Cambridge, MA: MIT Press. ISBN  978-0262681155 . OCLC 43076809. See also Rheingold's site:  "About Howard Rheingold". rheingold.com. Retrieved 2017-12-28.
  ^  Kapur, Arnav (April 2019). Human-machine cognitive coalescence through an internal duplex interface (Thesis). Massachusetts Institute of Technology. hdl:1721.1/120883.
  ^   "AlterEgo". MIT Media Lab. Retrieved April 30, 2019.
  ^  Kapur, Arnav; Kapur, Shreyas; Maes, Pattie (2018). "AlterEgo". Proceedings of the 2018 Conference on Human Information Interaction&amp;Retrieval - IUI '18. New York, New York, USA: ACM Press: 43–53. doi:10.1145/3172944.3172977. ISBN  9781450349451 . S2CID 3777401.
  ^  Willcox, G.; Rosenberg, L. (September 2019). "Swarm Intelligence Amplifies the IQ of Collaborating Teams". 2019 Second International Conference on Artificial Intelligence for Industries (AI4I): 111–114. doi:10.1109/AI4I46381.2019.00036. S2CID 212646000.
  ^  Rosenberg, L.; Willcox, G.; Askay, D.; Metcalf, L.; Harris, E. (September 2018). "Amplifying the Social Intelligence of Teams Through Human Swarming". 2018 First International Conference on Artificial Intelligence for Industries (AI4I): 23–26. doi:10.1109/AI4I.2018.8665698. ISBN  978-1-5386-9209-7 . S2CID 77385269.
  ^  Rosenberg, L.; Pescetelli, N. (September 2017). "Amplifying prediction accuracy using Swarm A.I." 2017 Intelligent Systems Conference (IntelliSys): 61–65. doi:10.1109/IntelliSys.2017.8324329. ISBN  978-1-5090-6435-9 . S2CID 4366745.
  ^   "Artificial swarm intelligence diagnoses pneumonia better than individual computer or doctor". The Stanford Daily. 2018-09-27. {{cite web}}: CS1 maint: url-status (link)
  ^  Carter, Shan; Nielsen, Michael (2017). "Using artificial intelligence to augment human intelligence". Distill. 2 (12): e9. arXiv:1609.04468. doi:10.23915/distill.00009.
  ^  Fulbright, Ron (2020). Democratization of Expertise: How Cognitive Systems Will Revolutionize Your Life. Boca Raton, FL: CRC Press. ISBN  978-0367859459 .
  ^  Fulbright, Ron (2020). "Synthetic Expertise". Augmented Cognition. Human Cognition and Behavior. 12197: 27–48 – via Springer.
 
Further reading[edit]
  Engelbart, D.; Landau, V.; Clegg, E. (2009). The Engelbart Hypothesis: Dialogs with Douglas Engelbart. NextPress. ISBN  978-0615308906 . 
 Asaro, Peter (2008). "From Mechanisms of Adaptation to Intelligence Amplifiers: The Philosophy of W. Ross Ashby" (PDF). In Husbands, P.; Holland, O.; Wheeler, M. (eds.).  The Mechanical Mind in History . MIT Press. ISBN  978-0-262-08377-5 . 
  Ashby, W.R. (2013) [1960].  Design for a Brain: The origin of adaptive behaviour  (2nd ed.). Springer. ISBN  978-94-015-1320-3 . 
 Skagestad, Peter (1993). "Thinking with Machines: Intelligence Augmentation, Evolutionary Epistemology, and Semiotic". Journal of Social and Evolutionary Systems. 16 (2): 157–180. doi:10.1016/1061-7361(93)90026-N. 
 Bray, David A. (2007). "Smart Business Networks (or, Let's Create 'Life' from Inert Information)". Smart Business Network Initiative (SBNi) Conference — Beijing, May 2008. SSRN. doi:10.2139/ssrn.991163. SSRN 991163. 
 Waldrop, M. Mitchell (2001).  The Dream Machine: J.C.R. Licklider and the Revolution that Made Computing Personal . Viking. ISBN  978-0-670-89976-0 . Licklider's biography, contains discussion of the importance of this paper. 
External links[edit]
  Intelligence Amplification using speech synthesis technology 
 IT Conversations: Doug Engelbart - Large-Scale Collective IQ 
 7 December 1951, Ashby first wrote about the possibility to build an 'information amplifier'. 
 12 August 1953, Ashby mentioned an objection to his 'intelligence-amplifier'.
Intelligence amplification - Wikipedia
https://en.wikipedia.org/wiki/Intelligence_amplification
https://www.lesswrong.com/posts/F6ZdDqx48aFNi49Pp/human-genetic-engineering-increasing-intelligence
Human Genetic Engineering: Increasing Intelligence - LessWrong
2020-12-05
GeneSmith
<div><p><div><p><a href="https://www.lesswrong.com/posts/5FrpeM7L2kfhxycNQ/the-case-for-human-genetic-engineering-1">View my previous post on genetic engineering</a></p><p>In my last post I covered three different ways in which genetically engineering humans could be beneficial:</p><p>1. It could allow us to address the mismatch between our genes and our environment</p><p>2. It could allow us to reach states of higher fitness that evolution simply cannot</p><p>3. It could allow the adoption of traits that only have a benefit once a sufficiently large percentage of the population has them.</p><p>Today I will be covering a fourth benefit of genetic engineering that, although it could probably fall under one of the previous categories, deserves its own post due to the outsized impact it would likely have: increasing human intelligence.</p><h2><strong>The Basics</strong></h2><p>Unusually high general intelligence is one of the human race's most distinguishing features. Our ability to cooperate flexibly and in large groups via language and our collective belief in fictitious identities is something no other species can match. Ant colonies cooperate in large groups, but they lack the flexibility of human organizations. Dolphins and chimpanzees have complex social relationships, but cannot cooperate at anywhere near the scale that humans can.</p><p>The exact characteristics that allow humans to do this are worthy of a whole book, but if you want a nice overview of the history of human civilization and the role of language in its development, I would recommend <a href="https://www.amazon.com/Sapiens-Humankind-Yuval-Noah-Harari/dp/0062316095">Sapiens: A Breif History of Humankind</a> by Yuval Noah Harari. But before I make the case for purposefully engineering humans with greater intelligence, I think it is appropriate to explain some basic facts about intelligence that will help lay the groundwork.</p><p>Competence in abilities across multiple domains tends to correlate. An individual that is good at math will, more often than not, score highly in reading comprehension and deductive reasoning tests. This underlying factor that determines general ability has a name: <i>g</i>. The existence of the <i>g</i> factor was originally proposed by the English psychologist <a href="https://en.wikipedia.org/wiki/Charles_Spearman">Charles Spearman</a> in the early years of the 20th century. He observed that children's performance ratings, across seemingly unrelated school subjects, were positively <a href="https://en.wikipedia.org/wiki/Correlated">correlated</a>, and reasoned that these correlations reflected the influence of an underlying general mental ability that impacted performance on all kinds of mental tests. Spearman suggested that all mental performance could be conceptualized in terms of a single general ability factor, which he labeled <i>g</i>, and many narrow task-specific ability factors. Cognitive ability tests such as IQ tests are designed to measure this underlying <i>g</i> factor.</p><p>Many factors seem to influence <i>g </i>including nutrition, social connections, genes, and maternal womb environment. A combination of genes and maternal womb environment explain a high amount of variance in <i>g </i>between individuals. "Monozygotic twins raised apart are more similar in IQ (74%) than dizygotic (fraternal) twins raised together (60%) and much more than parent-children pairs (42%); half-siblings (31%); adoptive siblings (29%-34%); virtual twins, or similarly aged but unrelated children raised together (28%); adoptive parent-child pairs (19%) and cousins (15%). Nothing but genes can explain this hierarchy." (<a href="https://www.wsj.com/articles/SB10001424052702304898704577478482432277706#:~:text=IQ%20heritability%20in%20the%20middle,between%20MZA%20twins%20was%2077.1%25.">Matt Ridley</a>)</p><p>One last note: IQ is often conflated with intelligence, but it is not actually the same thing. IQ is basically just a test that is designed to measure <i>g</i>. But even well-designed and well-administered IQ tests (and many IQ tests are neither) have their limitations in assessing the type of real-world performance that we ultimately care about. However, as I will explain shortly, they are still quite useful for predicting a broad range of outcomes.</p><h2><strong>The Benefits or raising Intelligence</strong></h2><p>Let me now turn to the benefits of raising intelligence. From an individual perspective, we might pursue superior Intelligence for social reasons: a greater likelihood of occupying high paying jobs, better health outcomes due to better choices, etc. But at a societal level, I think the question we should be asking ourselves is whether raising the intelligence of one individual positively affects others. In other words, are the benefits of intelligence zero-sum or positive-sum?</p><p>From the research I've done, the answer seems to be positive-sum. A quick perusal of the Wikipedia page on <i>g</i> reveals that <a href="https://en.wikipedia.org/wiki/G_factor_(psychometrics)">g correlates with job performance, particularly in highly skilled positions</a>. Unusually intelligent people also seem to have a lower discount rate; that is to say, they are more willing to accept a smaller payout now in exchange for a larger one later. Given that many of the world's biggest problems today (such as climate change, nuclear war, and other types of even worse existential risk) would be better dealt with if the general public placed a higher value on the quality of human lives lived in the future, it seems like lowering time discount rates could be extremely valuable to the long term survival of the human species.</p><p>There is also research showing that more intelligent people are disproportionately likely to support systems that enforce beneficial cooperation. Here I'll quote from Garett Jones' book <i>Hive Mind</i>:</p><blockquote><p>Another study by Brown University economist Louis Putterman and his coauthors found still more evidence that higher-IQ individuals are more likely to start off by playing nice, by being generous team players. In this game, known as the public goods game, players individually decide how much of their own money to put in a metaphorical pot, the money doubles or triples, and then it gets divided up among the group. When you give money, you’re directly contributing to the public good. The game was repeated for a few rounds with the same team so players would have a chance to learn from each other, a chance to find a path to cooperation.</p><p>As this was run at Brown University, an Ivy League school where one might expect that almost all students are raised in incredibly advantaged environments, it might seem that differences in IQ scores would be irrelevant. But in Putterman’s cooperation experiment, IQ mattered. He and his coauthors found that higher-IQ students at Brown put more money in the pot during the early rounds of the game: the higher-IQ students were more pleasant early on. That’s the smart thing to do, because extra money early on can send a signal of kindness, of cooperativeness, to the other players. And it’s worth noting that in another part of the experiment, when the students could vote on a way to penalize low contributors, higher-IQ students were more likely to vote for a rule that would penalize the non-cooperators: so higher-IQ students were pleasant, but not naive.</p></blockquote><p>The list of the benefits of higher intelligence goes on and on and on. <a href="https://www.quora.com/What-is-the-average-IQ-of-Nobel-Laureates">Here's a summary of a study done by Anne Roe examining the average IQs of the top scientists in the US in the 1950s. </a>Spoiler alert: the average IQ of these groups is extremely high: 163 in the group of 49 studied by Roe. This suggests that unusually intelligent people are significantly more likely to contribute to scientific advancement.</p><p>In summary: raising the average intelligence seems very likely to have huge benefits and a large amount of those benefits are accrued by the society in which intelligent people participate rather than the individual themselves.</p><h2><strong>Why should we think this is even possible? If evolution hasn't already optimized for higher intelligence, why should we think it's a good thing?</strong></h2><p>A question that inevitably comes up when discussing the effects of raising intelligence is the following: if the benefits of raising intelligence are so huge, why hasn't evolution selected for it already?</p><p>To this I would make several comments. First of all, evolution HAS been optimizing for higher intelligence among homo sapiens for several million years now. Brain sizes roughly trip tripled over the last 3 million years, and metabolic rates have likewise increased. <a href="https://www.sciencemag.org/news/2016/05/humans-are-highest-energy-apes-making-us-smarter-also-fatter">The average sedentary human burns about 27% more calories than the average chimpanzee, and there is evidence to indicate that we burn more calories than our ancestors.</a> It seems very likely that this increase in brain size corresponded to a increase in intelligence, since at some point during that process humans developed the ability to use language, create more sophisticated tools and form increasingly complex societies. This assertion is backed up by modern studies showing that skull size positively correlates with IQ (though at a fairly modest factor of 0.3).</p><p>Second, only a fraction of the benefits created by an intelligent individual are captured by that individual. Take tool creation: tool creation is undeniably a product of intelligence (and not just human intelligence). Yet it is far more difficult to create a new tool than to copy its design once it has been created. Since evolution optimizes for inclusive reproductive fitness, and new tool creation is a fairly rare event, there is not much of a differential reproductive fitness advantage for individuals that are unusually adept at creating new and useful tool designs. Many of the benefits of intelligence are like this. Thus many of the benefits of intelligence that we actually care about do not provide sufficient evolutionary fitness advantages for natural selection to spread them through the population.</p><p>And the third reason we might expect raising intelligence to have large benefits is the environment in which we live today is much different than that of our ancestors. Factors that may have been constraints in the past are probably not constraints today. Two of the most notable are the ability of an infant head to fit through the birth canal and the caloric requirements on large energy-hungry brains. C-sections have been around since the mid 19th century. With the ability to perform C-sections, infant head size at birth is no longer the same kind of constraining factor that it once was. </p><p>Calorie expenditure is also becoming increasingly irrelevant, particularly in the developed world. Champion strongmen like Hafþór Júlíus Björnsson and Brian Shaw consume around 10,000 calories per day. And even those who can't make a living off of strongman competitions and sponsorship can afford to do this working fairly normal middle-class jobs (though obviously with difficulty). Calorie consumption was obviously a limiting factor in the past, but the expected cost of needing additional calories is significantly lower today than it was in the past.</p><h2><strong>Can Human Intelligence actually be increased? How?</strong></h2><p>We now come to the heart of the matter: is it actually possible to ethically increase human intelligence with modern techniques or those that are likely to be created in the near future? I plan to give a more detailed answer in a future post, but I will give a quick summary here because this post would be incomplete without showing that this is at least possible.</p><p>To genetically engineer humans in a desired direction, one needs two things: a method to modify genes  (or at the very least to produce variance) and a means of telling how specific genetic variants influence trait expression.</p><p>There are many methods to do so, best summarized in <a href="https://www.gwern.net/Embryo-selection">this really fantastic post Gwern put together back in 2017. </a> The most viable near term method for increasing intelligence is embryo selection. In this method, two parents produce a number of embryos via In Vitro Fertilization (IVF). Biopsies are performed on the resulting embryos, and the DNA from the extracted cells are sequenced. An embryo that scores well on a test designed to predict the embryo's expression of certain desired traits is implanted in the mother or surrogate.</p><p>There are two crucial limitations on the ability to use embryo selection to facilitate genetic engineering of desired traits: the ability to produce a large number of embryos and the knowledge of which genes (or regions of the chromosome) cause certain outcomes.</p><p>With current techniques, it is only safe to extract around ten eggs from the ovaries of a female during IVF. We would therefore expect only a very modest increase in IQ from embryo selection among 10 eggs: probably on the order of ~3 points. </p><p>The second limitation has to do with the most common method to measure the expected expression of a trait using genome sequencing: a Genome-Wide Association Study (GWAS). A GWAS is how we tease out the contribution of different genes to highly polygenic traits like intelligence, height, and heart disease risk. With so many genes influencing height, and moral considerations preventing us from using randomization techniques to determine which genes cause which traits, these studies are the next best way to determine causation.</p><p>These studies look at extremely large data sets of individuals with different expressions of certain traits and identify regions of DNA that are strongly associated with trait expression. Despite the massive increase in available data over the last decade, these studies still cannot usually identify the exact single nucleotide polymorphism (SNP) that causes the observed trait. Instead, a GWAS will tell you the section of 10 or so base pairs within which the causal variant is located, but can't currently tell you exactly which one is responsible for the observed effect.</p><h2><strong>How would increasing human intelligence affect everything else?</strong></h2><h2>Generational Inequality</h2><p>This will be the least empirically supported section of my post because there are so many unknowns here. So treat this as a very informal and at least partially incorrect assessment of the likely outcomes.</p><p>Human intelligence is in one way or another responsible for modern civilization, for destruction of natural environments around the world, and for most good and bad things we experience on a daily basis. To say that dramatically increasing human intelligence would have a large impact on the world is an understatement.</p><p>Using embryo selection, it is likely that we would not "max out" desirable trait expression for a dozen generations or more depending on how many embryos were used for each generation. This occurs for a simple reason: there are roughly 3 billion base pairs in human DNA, and each generation of embryo selection only increase the frequency of desirable variants by a small amount. </p><p>There would therefore be gradually accruing inequality between generations as older generations found themselves outclassed by a new generation of smarter, better looking, healthier humans. While it is certainly possible to genetically engineer adult humans (such as is done during immunotherapy for cancer), it is significantly harder. And there are likely some changes that will be extremely hard to make in adults because their primary effect is to change the manner in which development occurs.</p><p>This problem will be compounded if Aubrey De Grey anyone from the dozens of startups working on life extension technology actually gets anti-aging technology to work and humans start to live to ages of 100 or more. My hope is that extremely smart humans will be better equipped to solve the difficult technical challenges involved in genetically altering adult humans in ways that would allow them to fully participate in a society of genetically enhanced people.</p><p>However, I do not suspect this will actually be a problem for more than a few generations. It is likely that either much more advanced techniques such as Iterated Embryo Selection (IES) will allow us to max out the gains from embryo selection in one generation, or we will reach the singularity and digital superintelligence will vastly out-class the performance of human brains with several billion years of technical design debt.</p><h2>Existential Risk</h2><p>Increasing human intelligence is likely to have a substantial impact on our collective ability to mitigate existential risk.</p><p>As shown in the Brown University study, highly intelligent people also seem to have a discount factor closer to 1; they value the future more than people of lower intelligence and are therefore more likely to make tradeoffs that increase the quality of the future in exchange for making the present slightly worse. They also seem to have better models of how the world works.</p><p>Existential risk mitigation of all types requires exactly those traits to be properly addressed. They require researchers to do the actual work, and societies to support those researchers even though the benefits are not likely to be realized for decades at least. Both of these would be positively affected by increasing average human intelligence.</p><p>Take research on artificial intelligence safety, as an example: the work is highly technical and the major payoffs are not likely to occur for several decades. Unusually intelligent people are much more likely to solve major problems in the field, and intelligent people not working in the field are more likely than average to recognize the threat posed by unaligned powerful AI support increased research funding.</p><h2>Economic Growth</h2><p>This is one area where things are likely to get really crazy. Human intelligence HAS been increasing over evolutionary timescales, but the rate of improvement has been glacial. All modern economic growth has been achieved by a brain whose basic hardware has received few upgrades. We're made a few gains from reducing mental stunting due to malnutrition and from eliminating leaded gasoline. But apart from those, almost all gains of the last ten thousand years have been object-level optimizations of the world itself, the number of humans living in it, and the accumulated knowledge of our species.</p><p>However, if we begin to use technology to increase intelligence, the equation for economic growth starts to look quite different. The unchanging optimizer, which was previously producing exponential economic growth is now itself being optimized. While no one knows exactly what the trajectory of intelligence improvement will look like, it is likely that until we truly max out the physical limits of computation, its improvement will be exponential (since the ability to design better intelligence is itself a function of the intelligence of those optimizing it).</p><p>I think it's therefore reasonable to believe that until we truly max out all possible gains, economic growth will resemble a double exponential of the form . Plot out the result of such a function and compare it to exponential growth and you will see just how dramatic of a difference this will make.</p><h2><strong>Future Posts</strong></h2><p>In my upcoming posts I plan to cover several more facets of human genetic engineering. Human genetic engineering brings up many moral questions. From what I have read so far there do not seem to be any huge showstopping problems that would prevent us from ever undertaking human genetic engineering, but there is certainly the potential for this technology to be abused and that potential is worthy of discussion. There is a long and sordid history of eugenics programs to be dealt with and I do not feel that I can really cover this topic without explaining all the ways we should NOT use this technology. I also think it's important to explain how techniques like embryo selection can avoid the toxic exclusionary policies of eugenics movements so that we can make sure when we do finally get around to genetic engineering it will benefit all and not simply widen the gap between the haves and the have-nots. </p><p>I also plan to discuss in detail the technical barriers preventing us from doing serious genetic engineering today and how those obstacles might be overcome. </p></div></p>||||I|||| This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.

 LESSWRONG

 LW

 Login

 Human Genetic Engineering: Increasing Intelligence

 by GeneSmith 12 min read 5th Dec 2020 5 comments

 28

 Human Genetics World Optimization
 Frontpage
 Human Genetic Engineering: Increasing Intelligence
 The Basics
 The Benefits or raising Intelligence
 Why should we think this is even possible? If evolution hasn't already optimized for higher intelligence, why should we think it's a good thing?
 Can Human Intelligence actually be increased? How?
 How would increasing human intelligence affect everything else?
 Generational Inequality
 Existential Risk
 Economic Growth
 Future Posts
 5 comments

 View my previous post on genetic engineering

 In my last post I covered three different ways in which genetically engineering humans could be beneficial:

 1. It could allow us to address the mismatch between our genes and our environment

 2. It could allow us to reach states of higher fitness that evolution simply cannot

 3. It could allow the adoption of traits that only have a benefit once a sufficiently large percentage of the population has them.

 Today I will be covering a fourth benefit of genetic engineering that, although it could probably fall under one of the previous categories, deserves its own post due to the outsized impact it would likely have: increasing human intelligence.

 The Basics

 Unusually high general intelligence is one of the human race's most distinguishing features. Our ability to cooperate flexibly and in large groups via language and our collective belief in fictitious identities is something no other species can match. Ant colonies cooperate in large groups, but they lack the flexibility of human organizations. Dolphins and chimpanzees have complex social relationships, but cannot cooperate at anywhere near the scale that humans can.

 The exact characteristics that allow humans to do this are worthy of a whole book, but if you want a nice overview of the history of human civilization and the role of language in its development, I would recommend Sapiens: A Breif History of Humankind by Yuval Noah Harari. But before I make the case for purposefully engineering humans with greater intelligence, I think it is appropriate to explain some basic facts about intelligence that will help lay the groundwork.

 Competence in abilities across multiple domains tends to correlate. An individual that is good at math will, more often than not, score highly in reading comprehension and deductive reasoning tests. This underlying factor that determines general ability has a name: g. The existence of the g factor was originally proposed by the English psychologist Charles Spearman in the early years of the 20th century. He observed that children's performance ratings, across seemingly unrelated school subjects, were positively correlated, and reasoned that these correlations reflected the influence of an underlying general mental ability that impacted performance on all kinds of mental tests. Spearman suggested that all mental performance could be conceptualized in terms of a single general ability factor, which he labeled g, and many narrow task-specific ability factors. Cognitive ability tests such as IQ tests are designed to measure this underlying g factor.

 Many factors seem to influence g including nutrition, social connections, genes, and maternal womb environment. A combination of genes and maternal womb environment explain a high amount of variance in g between individuals. "Monozygotic twins raised apart are more similar in IQ (74%) than dizygotic (fraternal) twins raised together (60%) and much more than parent-children pairs (42%); half-siblings (31%); adoptive siblings (29%-34%); virtual twins, or similarly aged but unrelated children raised together (28%); adoptive parent-child pairs (19%) and cousins (15%). Nothing but genes can explain this hierarchy." (Matt Ridley)

 One last note: IQ is often conflated with intelligence, but it is not actually the same thing. IQ is basically just a test that is designed to measure g. But even well-designed and well-administered IQ tests (and many IQ tests are neither) have their limitations in assessing the type of real-world performance that we ultimately care about. However, as I will explain shortly, they are still quite useful for predicting a broad range of outcomes.

 The Benefits or raising Intelligence

 Let me now turn to the benefits of raising intelligence. From an individual perspective, we might pursue superior Intelligence for social reasons: a greater likelihood of occupying high paying jobs, better health outcomes due to better choices, etc. But at a societal level, I think the question we should be asking ourselves is whether raising the intelligence of one individual positively affects others. In other words, are the benefits of intelligence zero-sum or positive-sum?

 From the research I've done, the answer seems to be positive-sum. A quick perusal of the Wikipedia page on g reveals that g correlates with job performance, particularly in highly skilled positions. Unusually intelligent people also seem to have a lower discount rate; that is to say, they are more willing to accept a smaller payout now in exchange for a larger one later. Given that many of the world's biggest problems today (such as climate change, nuclear war, and other types of even worse existential risk) would be better dealt with if the general public placed a higher value on the quality of human lives lived in the future, it seems like lowering time discount rates could be extremely valuable to the long term survival of the human species.

 There is also research showing that more intelligent people are disproportionately likely to support systems that enforce beneficial cooperation. Here I'll quote from Garett Jones' book Hive Mind:

 Another study by Brown University economist Louis Putterman and his coauthors found still more evidence that higher-IQ individuals are more likely to start off by playing nice, by being generous team players. In this game, known as the public goods game, players individually decide how much of their own money to put in a metaphorical pot, the money doubles or triples, and then it gets divided up among the group. When you give money, you’re directly contributing to the public good. The game was repeated for a few rounds with the same team so players would have a chance to learn from each other, a chance to find a path to cooperation.

 As this was run at Brown University, an Ivy League school where one might expect that almost all students are raised in incredibly advantaged environments, it might seem that differences in IQ scores would be irrelevant. But in Putterman’s cooperation experiment, IQ mattered. He and his coauthors found that higher-IQ students at Brown put more money in the pot during the early rounds of the game: the higher-IQ students were more pleasant early on. That’s the smart thing to do, because extra money early on can send a signal of kindness, of cooperativeness, to the other players. And it’s worth noting that in another part of the experiment, when the students could vote on a way to penalize low contributors, higher-IQ students were more likely to vote for a rule that would penalize the non-cooperators: so higher-IQ students were pleasant, but not naive.

 The list of the benefits of higher intelligence goes on and on and on. Here's a summary of a study done by Anne Roe examining the average IQs of the top scientists in the US in the 1950s. Spoiler alert: the average IQ of these groups is extremely high: 163 in the group of 49 studied by Roe. This suggests that unusually intelligent people are significantly more likely to contribute to scientific advancement.

 In summary: raising the average intelligence seems very likely to have huge benefits and a large amount of those benefits are accrued by the society in which intelligent people participate rather than the individual themselves.

 Why should we think this is even possible? If evolution hasn't already optimized for higher intelligence, why should we think it's a good thing?

 A question that inevitably comes up when discussing the effects of raising intelligence is the following: if the benefits of raising intelligence are so huge, why hasn't evolution selected for it already?

 To this I would make several comments. First of all, evolution HAS been optimizing for higher intelligence among homo sapiens for several million years now. Brain sizes roughly trip tripled over the last 3 million years, and metabolic rates have likewise increased. The average sedentary human burns about 27% more calories than the average chimpanzee, and there is evidence to indicate that we burn more calories than our ancestors. It seems very likely that this increase in brain size corresponded to a increase in intelligence, since at some point during that process humans developed the ability to use language, create more sophisticated tools and form increasingly complex societies. This assertion is backed up by modern studies showing that skull size positively correlates with IQ (though at a fairly modest factor of 0.3).

 Second, only a fraction of the benefits created by an intelligent individual are captured by that individual. Take tool creation: tool creation is undeniably a product of intelligence (and not just human intelligence). Yet it is far more difficult to create a new tool than to copy its design once it has been created. Since evolution optimizes for inclusive reproductive fitness, and new tool creation is a fairly rare event, there is not much of a differential reproductive fitness advantage for individuals that are unusually adept at creating new and useful tool designs. Many of the benefits of intelligence are like this. Thus many of the benefits of intelligence that we actually care about do not provide sufficient evolutionary fitness advantages for natural selection to spread them through the population.

 And the third reason we might expect raising intelligence to have large benefits is the environment in which we live today is much different than that of our ancestors. Factors that may have been constraints in the past are probably not constraints today. Two of the most notable are the ability of an infant head to fit through the birth canal and the caloric requirements on large energy-hungry brains. C-sections have been around since the mid 19th century. With the ability to perform C-sections, infant head size at birth is no longer the same kind of constraining factor that it once was.

 Calorie expenditure is also becoming increasingly irrelevant, particularly in the developed world. Champion strongmen like Hafþór Júlíus Björnsson and Brian Shaw consume around 10,000 calories per day. And even those who can't make a living off of strongman competitions and sponsorship can afford to do this working fairly normal middle-class jobs (though obviously with difficulty). Calorie consumption was obviously a limiting factor in the past, but the expected cost of needing additional calories is significantly lower today than it was in the past.

 Can Human Intelligence actually be increased? How?

 We now come to the heart of the matter: is it actually possible to ethically increase human intelligence with modern techniques or those that are likely to be created in the near future? I plan to give a more detailed answer in a future post, but I will give a quick summary here because this post would be incomplete without showing that this is at least possible.

 To genetically engineer humans in a desired direction, one needs two things: a method to modify genes (or at the very least to produce variance) and a means of telling how specific genetic variants influence trait expression.

 There are many methods to do so, best summarized in this really fantastic post Gwern put together back in 2017. The most viable near term method for increasing intelligence is embryo selection. In this method, two parents produce a number of embryos via In Vitro Fertilization (IVF). Biopsies are performed on the resulting embryos, and the DNA from the extracted cells are sequenced. An embryo that scores well on a test designed to predict the embryo's expression of certain desired traits is implanted in the mother or surrogate.

 There are two crucial limitations on the ability to use embryo selection to facilitate genetic engineering of desired traits: the ability to produce a large number of embryos and the knowledge of which genes (or regions of the chromosome) cause certain outcomes.

 With current techniques, it is only safe to extract around ten eggs from the ovaries of a female during IVF. We would therefore expect only a very modest increase in IQ from embryo selection among 10 eggs: probably on the order of ~3 points.

 The second limitation has to do with the most common method to measure the expected expression of a trait using genome sequencing: a Genome-Wide Association Study (GWAS). A GWAS is how we tease out the contribution of different genes to highly polygenic traits like intelligence, height, and heart disease risk. With so many genes influencing height, and moral considerations preventing us from using randomization techniques to determine which genes cause which traits, these studies are the next best way to determine causation.

 These studies look at extremely large data sets of individuals with different expressions of certain traits and identify regions of DNA that are strongly associated with trait expression. Despite the massive increase in available data over the last decade, these studies still cannot usually identify the exact single nucleotide polymorphism (SNP) that causes the observed trait. Instead, a GWAS will tell you the section of 10 or so base pairs within which the causal variant is located, but can't currently tell you exactly which one is responsible for the observed effect.

 How would increasing human intelligence affect everything else?

 Generational Inequality

 This will be the least empirically supported section of my post because there are so many unknowns here. So treat this as a very informal and at least partially incorrect assessment of the likely outcomes.

 Human intelligence is in one way or another responsible for modern civilization, for destruction of natural environments around the world, and for most good and bad things we experience on a daily basis. To say that dramatically increasing human intelligence would have a large impact on the world is an understatement.

 Using embryo selection, it is likely that we would not "max out" desirable trait expression for a dozen generations or more depending on how many embryos were used for each generation. This occurs for a simple reason: there are roughly 3 billion base pairs in human DNA, and each generation of embryo selection only increase the frequency of desirable variants by a small amount.

 There would therefore be gradually accruing inequality between generations as older generations found themselves outclassed by a new generation of smarter, better looking, healthier humans. While it is certainly possible to genetically engineer adult humans (such as is done during immunotherapy for cancer), it is significantly harder. And there are likely some changes that will be extremely hard to make in adults because their primary effect is to change the manner in which development occurs.

 This problem will be compounded if Aubrey De Grey anyone from the dozens of startups working on life extension technology actually gets anti-aging technology to work and humans start to live to ages of 100 or more. My hope is that extremely smart humans will be better equipped to solve the difficult technical challenges involved in genetically altering adult humans in ways that would allow them to fully participate in a society of genetically enhanced people.

 However, I do not suspect this will actually be a problem for more than a few generations. It is likely that either much more advanced techniques such as Iterated Embryo Selection (IES) will allow us to max out the gains from embryo selection in one generation, or we will reach the singularity and digital superintelligence will vastly out-class the performance of human brains with several billion years of technical design debt.

 Existential Risk

 Increasing human intelligence is likely to have a substantial impact on our collective ability to mitigate existential risk.

 As shown in the Brown University study, highly intelligent people also seem to have a discount factor closer to 1; they value the future more than people of lower intelligence and are therefore more likely to make tradeoffs that increase the quality of the future in exchange for making the present slightly worse. They also seem to have better models of how the world works.

 Existential risk mitigation of all types requires exactly those traits to be properly addressed. They require researchers to do the actual work, and societies to support those researchers even though the benefits are not likely to be realized for decades at least. Both of these would be positively affected by increasing average human intelligence.

 Take research on artificial intelligence safety, as an example: the work is highly technical and the major payoffs are not likely to occur for several decades. Unusually intelligent people are much more likely to solve major problems in the field, and intelligent people not working in the field are more likely than average to recognize the threat posed by unaligned powerful AI support increased research funding.

 Economic Growth

 This is one area where things are likely to get really crazy. Human intelligence HAS been increasing over evolutionary timescales, but the rate of improvement has been glacial. All modern economic growth has been achieved by a brain whose basic hardware has received few upgrades. We're made a few gains from reducing mental stunting due to malnutrition and from eliminating leaded gasoline. But apart from those, almost all gains of the last ten thousand years have been object-level optimizations of the world itself, the number of humans living in it, and the accumulated knowledge of our species.

 However, if we begin to use technology to increase intelligence, the equation for economic growth starts to look quite different. The unchanging optimizer, which was previously producing exponential economic growth is now itself being optimized. While no one knows exactly what the trajectory of intelligence improvement will look like, it is likely that until we truly max out the physical limits of computation, its improvement will be exponential (since the ability to design better intelligence is itself a function of the intelligence of those optimizing it).

 I think it's therefore reasonable to believe that until we truly max out all possible gains, economic growth will resemble a double exponential of the form a ( b t ) . Plot out the result of such a function and compare it to exponential growth and you will see just how dramatic of a difference this will make.

 Future Posts

 In my upcoming posts I plan to cover several more facets of human genetic engineering. Human genetic engineering brings up many moral questions. From what I have read so far there do not seem to be any huge showstopping problems that would prevent us from ever undertaking human genetic engineering, but there is certainly the potential for this technology to be abused and that potential is worthy of discussion. There is a long and sordid history of eugenics programs to be dealt with and I do not feel that I can really cover this topic without explaining all the ways we should NOT use this technology. I also think it's important to explain how techniques like embryo selection can avoid the toxic exclusionary policies of eugenics movements so that we can make sure when we do finally get around to genetic engineering it will benefit all and not simply widen the gap between the haves and the have-nots.

 I also plan to discuss in detail the technical barriers preventing us from doing serious genetic engineering today and how those obstacles might be overcome.

 Human Genetics 1 World Optimization 2
 Frontpage

 28

 Mentioned in
 71 A Brief Review of Current and Near-Future Methods of Genetic Engineering
 New Comment
 Submit
 5 comments , sorted by
 top scoring
 Click to highlight new comments since: Today at 3:25 PM
 [ - ]supposedlyfun 2y 3

 This was all a really nice read, very informative, strong favorited. You did lose me in claiming a double exponent for economic growth resulting from increased intelligence. I don't have enough information to say you're right or wrong, but for such an extreme claim, I want to see more of your model. I also want to see more of your model because I just think it would be interesting!

 Reply
 [ - ]GeneSmith 2y 1

 Thanks!

 Frankly the double exponential growth claim is one of the more speculative in the post. There ARE some similarities with a field like quantum computing, where Neven's law describes how quantum computers' performance at certain tasks improves at a doubly exponential rate.

 The doubly exponential rate at which, according to Neven, quantum computers are gaining on classical ones is a result of two exponential factors combined with each other. The first is that quantum computers have an intrinsic exponential advantage over classical ones: If a quantum circuit has four quantum bits, for example, it takes a classical circuit with 16 ordinary bits to achieve equivalent computational power. This would be true even if quantum technology never improved.

 My assertion of doubly exponential economic growth is premised on two things: economic growth being an exponential multiple of the abilities of the beings that create it, and intelligence itself increasing exponentially as we begin to adopt genetic modification.

 This obviously could turn out to be wrong.

 Reply
 [ - ]TurnTrout 2y 4

 See also: Intelligence Explosion Microeconomics.

 Reply
 [ - ]cubefox 2y 1

 Interesting. Is g and intelligence the same thing? If not, how do they relate to each other?

 Reply
 [ - ]GeneSmith 2y 1

 I've kind of used them interchangeably throughout the post, though I prefer to use "intelligence" to describe what we are actually aiming for: increasing the ability of individuals to accomplish goals given a set of constraints. There are highly g-loaded tests such as IQ tests or SAT scores that capture g well but don't always translate to real-world performance.

 Part of the challenge for any genetic engineering program will be figuring out how to robustly measure traits we want to change. IQ test are highly g-loaded, but they can be gamed. I remember reading a study (I can't find it now) showing that if you trained people on IQ test you could increase their scores by like 15 points, which shows that IQ scores can become decoupled from g. Ideally we'd like a test that can't be gamed.

 Reply

 New to LessWrong?

 Getting Started

 FAQ

 Library

View my previous post on genetic engineering  In my last post I covered three different ways in which genetically engineering humans could be beneficial: 1. It could allow us to address the mismatch between our genes and our environment 2. It could allow us to reach states of higher fitness that evolution simply cannot 3. It could allow the adoption of traits that only have a benefit once a sufficiently large percentage of the population has them. Today I will be covering a fourth benefit of genetic engineering that, although it could probably fall under one of the previous categories, deserves its own post due to the outsized impact it would likely have: increasing human intelligence.  The Basics  Unusually high general intelligence is one of the human race's most distinguishing features. Our ability to cooperate flexibly and in large groups via language and our collective belief in fictitious identities is something no other species can match. Ant colonies cooperate in large groups, but they lack the flexibility of human organizations. Dolphins and chimpanzees have complex social relationships, but cannot cooperate at anywhere near the scale that humans can. The exact characteristics that allow humans to do this are worthy of a whole book, but if you want a nice overview of the history of human civilization and the role of language in its development, I would recommend Sapiens: A Breif History of Humankind by Yuval Noah Harari. But before I make the case for purposefully engineering humans with greater intelligence, I think it is appropriate to explain some basic facts about intelligence that will help lay the groundwork. Competence in abilities across multiple domains tends to correlate. An individual that is good at math will, more often than not, score highly in reading comprehension and deductive reasoning tests. This underlying factor that determines general ability has a name: g. The existence of the g factor was originally proposed by the English psychologist Charles Spearman in the early years of the 20th century. He observed that children's performance ratings, across seemingly unrelated school subjects, were positively correlated, and reasoned that these correlations reflected the influence of an underlying general mental ability that impacted performance on all kinds of mental tests. Spearman suggested that all mental performance could be conceptualized in terms of a single general ability factor, which he labeled g, and many narrow task-specific ability factors. Cognitive ability tests such as IQ tests are designed to measure this underlying g factor. Many factors seem to influence g including nutrition, social connections, genes, and maternal womb environment. A combination of genes and maternal womb environment explain a high amount of variance in g between individuals. "Monozygotic twins raised apart are more similar in IQ (74%) than dizygotic (fraternal) twins raised together (60%) and much more than parent-children pairs (42%); half-siblings (31%); adoptive siblings (29%-34%); virtual twins, or similarly aged but unrelated children raised together (28%); adoptive parent-child pairs (19%) and cousins (15%). Nothing but genes can explain this hierarchy." (Matt Ridley) One last note: IQ is often conflated with intelligence, but it is not actually the same thing. IQ is basically just a test that is designed to measure g. But even well-designed and well-administered IQ tests (and many IQ tests are neither) have their limitations in assessing the type of real-world performance that we ultimately care about. However, as I will explain shortly, they are still quite useful for predicting a broad range of outcomes.  The Benefits or raising Intelligence  Let me now turn to the benefits of raising intelligence. From an individual perspective, we might pursue superior Intelligence for social reasons: a greater likelihood of occupying high paying jobs, better health outcomes due to better choices, etc. But at a societal level, I think the question we should be asking ourselves is whether raising the intelligence of one individual positively affects others. In other words, are the benefits of intelligence zero-sum or positive-sum? From the research I've done, the answer seems to be positive-sum. A quick perusal of the Wikipedia page on g reveals that g correlates with job performance, particularly in highly skilled positions. Unusually intelligent people also seem to have a lower discount rate; that is to say, they are more willing to accept a smaller payout now in exchange for a larger one later. Given that many of the world's biggest problems today (such as climate change, nuclear war, and other types of even worse existential risk) would be better dealt with if the general public placed a higher value on the quality of human lives lived in the future, it seems like lowering time discount rates could be extremely valuable to the long term survival of the human species. There is also research showing that more intelligent people are disproportionately likely to support systems that enforce beneficial cooperation. Here I'll quote from Garett Jones' book Hive Mind:  Another study by Brown University economist Louis Putterman and his coauthors found still more evidence that higher-IQ individuals are more likely to start off by playing nice, by being generous team players. In this game, known as the public goods game, players individually decide how much of their own money to put in a metaphorical pot, the money doubles or triples, and then it gets divided up among the group. When you give money, you’re directly contributing to the public good. The game was repeated for a few rounds with the same team so players would have a chance to learn from each other, a chance to find a path to cooperation. As this was run at Brown University, an Ivy League school where one might expect that almost all students are raised in incredibly advantaged environments, it might seem that differences in IQ scores would be irrelevant. But in Putterman’s cooperation experiment, IQ mattered. He and his coauthors found that higher-IQ students at Brown put more money in the pot during the early rounds of the game: the higher-IQ students were more pleasant early on. That’s the smart thing to do, because extra money early on can send a signal of kindness, of cooperativeness, to the other players. And it’s worth noting that in another part of the experiment, when the students could vote on a way to penalize low contributors, higher-IQ students were more likely to vote for a rule that would penalize the non-cooperators: so higher-IQ students were pleasant, but not naive.  The list of the benefits of higher intelligence goes on and on and on. Here's a summary of a study done by Anne Roe examining the average IQs of the top scientists in the US in the 1950s. Spoiler alert: the average IQ of these groups is extremely high: 163 in the group of 49 studied by Roe. This suggests that unusually intelligent people are significantly more likely to contribute to scientific advancement. In summary: raising the average intelligence seems very likely to have huge benefits and a large amount of those benefits are accrued by the society in which intelligent people participate rather than the individual themselves.  Why should we think this is even possible? If evolution hasn't already optimized for higher intelligence, why should we think it's a good thing?  A question that inevitably comes up when discussing the effects of raising intelligence is the following: if the benefits of raising intelligence are so huge, why hasn't evolution selected for it already? To this I would make several comments. First of all, evolution HAS been optimizing for higher intelligence among homo sapiens for several million years now. Brain sizes roughly trip tripled over the last 3 million years, and metabolic rates have likewise increased. The average sedentary human burns about 27% more calories than the average chimpanzee, and there is evidence to indicate that we burn more calories than our ancestors. It seems very likely that this increase in brain size corresponded to a increase in intelligence, since at some point during that process humans developed the ability to use language, create more sophisticated tools and form increasingly complex societies. This assertion is backed up by modern studies showing that skull size positively correlates with IQ (though at a fairly modest factor of 0.3). Second, only a fraction of the benefits created by an intelligent individual are captured by that individual. Take tool creation: tool creation is undeniably a product of intelligence (and not just human intelligence). Yet it is far more difficult to create a new tool than to copy its design once it has been created. Since evolution optimizes for inclusive reproductive fitness, and new tool creation is a fairly rare event, there is not much of a differential reproductive fitness advantage for individuals that are unusually adept at creating new and useful tool designs. Many of the benefits of intelligence are like this. Thus many of the benefits of intelligence that we actually care about do not provide sufficient evolutionary fitness advantages for natural selection to spread them through the population. And the third reason we might expect raising intelligence to have large benefits is the environment in which we live today is much different than that of our ancestors. Factors that may have been constraints in the past are probably not constraints today. Two of the most notable are the ability of an infant head to fit through the birth canal and the caloric requirements on large energy-hungry brains. C-sections have been around since the mid 19th century. With the ability to perform C-sections, infant head size at birth is no longer the same kind of constraining factor that it once was.  Calorie expenditure is also becoming increasingly irrelevant, particularly in the developed world. Champion strongmen like Hafþór Júlíus Björnsson and Brian Shaw consume around 10,000 calories per day. And even those who can't make a living off of strongman competitions and sponsorship can afford to do this working fairly normal middle-class jobs (though obviously with difficulty). Calorie consumption was obviously a limiting factor in the past, but the expected cost of needing additional calories is significantly lower today than it was in the past.  Can Human Intelligence actually be increased? How?  We now come to the heart of the matter: is it actually possible to ethically increase human intelligence with modern techniques or those that are likely to be created in the near future? I plan to give a more detailed answer in a future post, but I will give a quick summary here because this post would be incomplete without showing that this is at least possible. To genetically engineer humans in a desired direction, one needs two things: a method to modify genes  (or at the very least to produce variance) and a means of telling how specific genetic variants influence trait expression. There are many methods to do so, best summarized in this really fantastic post Gwern put together back in 2017.  The most viable near term method for increasing intelligence is embryo selection. In this method, two parents produce a number of embryos via In Vitro Fertilization (IVF). Biopsies are performed on the resulting embryos, and the DNA from the extracted cells are sequenced. An embryo that scores well on a test designed to predict the embryo's expression of certain desired traits is implanted in the mother or surrogate. There are two crucial limitations on the ability to use embryo selection to facilitate genetic engineering of desired traits: the ability to produce a large number of embryos and the knowledge of which genes (or regions of the chromosome) cause certain outcomes. With current techniques, it is only safe to extract around ten eggs from the ovaries of a female during IVF. We would therefore expect only a very modest increase in IQ from embryo selection among 10 eggs: probably on the order of ~3 points.  The second limitation has to do with the most common method to measure the expected expression of a trait using genome sequencing: a Genome-Wide Association Study (GWAS). A GWAS is how we tease out the contribution of different genes to highly polygenic traits like intelligence, height, and heart disease risk. With so many genes influencing height, and moral considerations preventing us from using randomization techniques to determine which genes cause which traits, these studies are the next best way to determine causation. These studies look at extremely large data sets of individuals with different expressions of certain traits and identify regions of DNA that are strongly associated with trait expression. Despite the massive increase in available data over the last decade, these studies still cannot usually identify the exact single nucleotide polymorphism (SNP) that causes the observed trait. Instead, a GWAS will tell you the section of 10 or so base pairs within which the causal variant is located, but can't currently tell you exactly which one is responsible for the observed effect.  How would increasing human intelligence affect everything else?  Generational Inequality This will be the least empirically supported section of my post because there are so many unknowns here. So treat this as a very informal and at least partially incorrect assessment of the likely outcomes. Human intelligence is in one way or another responsible for modern civilization, for destruction of natural environments around the world, and for most good and bad things we experience on a daily basis. To say that dramatically increasing human intelligence would have a large impact on the world is an understatement. Using embryo selection, it is likely that we would not "max out" desirable trait expression for a dozen generations or more depending on how many embryos were used for each generation. This occurs for a simple reason: there are roughly 3 billion base pairs in human DNA, and each generation of embryo selection only increase the frequency of desirable variants by a small amount.  There would therefore be gradually accruing inequality between generations as older generations found themselves outclassed by a new generation of smarter, better looking, healthier humans. While it is certainly possible to genetically engineer adult humans (such as is done during immunotherapy for cancer), it is significantly harder. And there are likely some changes that will be extremely hard to make in adults because their primary effect is to change the manner in which development occurs. This problem will be compounded if Aubrey De Grey anyone from the dozens of startups working on life extension technology actually gets anti-aging technology to work and humans start to live to ages of 100 or more. My hope is that extremely smart humans will be better equipped to solve the difficult technical challenges involved in genetically altering adult humans in ways that would allow them to fully participate in a society of genetically enhanced people. However, I do not suspect this will actually be a problem for more than a few generations. It is likely that either much more advanced techniques such as Iterated Embryo Selection (IES) will allow us to max out the gains from embryo selection in one generation, or we will reach the singularity and digital superintelligence will vastly out-class the performance of human brains with several billion years of technical design debt. Existential Risk Increasing human intelligence is likely to have a substantial impact on our collective ability to mitigate existential risk. As shown in the Brown University study, highly intelligent people also seem to have a discount factor closer to 1; they value the future more than people of lower intelligence and are therefore more likely to make tradeoffs that increase the quality of the future in exchange for making the present slightly worse. They also seem to have better models of how the world works. Existential risk mitigation of all types requires exactly those traits to be properly addressed. They require researchers to do the actual work, and societies to support those researchers even though the benefits are not likely to be realized for decades at least. Both of these would be positively affected by increasing average human intelligence. Take research on artificial intelligence safety, as an example: the work is highly technical and the major payoffs are not likely to occur for several decades. Unusually intelligent people are much more likely to solve major problems in the field, and intelligent people not working in the field are more likely than average to recognize the threat posed by unaligned powerful AI support increased research funding. Economic Growth This is one area where things are likely to get really crazy. Human intelligence HAS been increasing over evolutionary timescales, but the rate of improvement has been glacial. All modern economic growth has been achieved by a brain whose basic hardware has received few upgrades. We're made a few gains from reducing mental stunting due to malnutrition and from eliminating leaded gasoline. But apart from those, almost all gains of the last ten thousand years have been object-level optimizations of the world itself, the number of humans living in it, and the accumulated knowledge of our species. However, if we begin to use technology to increase intelligence, the equation for economic growth starts to look quite different. The unchanging optimizer, which was previously producing exponential economic growth is now itself being optimized. While no one knows exactly what the trajectory of intelligence improvement will look like, it is likely that until we truly max out the physical limits of computation, its improvement will be exponential (since the ability to design better intelligence is itself a function of the intelligence of those optimizing it). I think it's therefore reasonable to believe that until we truly max out all possible gains, economic growth will resemble a double exponential of the form . Plot out the result of such a function and compare it to exponential growth and you will see just how dramatic of a difference this will make.  Future Posts  In my upcoming posts I plan to cover several more facets of human genetic engineering. Human genetic engineering brings up many moral questions. From what I have read so far there do not seem to be any huge showstopping problems that would prevent us from ever undertaking human genetic engineering, but there is certainly the potential for this technology to be abused and that potential is worthy of discussion. There is a long and sordid history of eugenics programs to be dealt with and I do not feel that I can really cover this topic without explaining all the ways we should NOT use this technology. I also think it's important to explain how techniques like embryo selection can avoid the toxic exclusionary policies of eugenics movements so that we can make sure when we do finally get around to genetic engineering it will benefit all and not simply widen the gap between the haves and the have-nots.  I also plan to discuss in detail the technical barriers preventing us from doing serious genetic engineering today and how those obstacles might be overcome.
Human Genetic Engineering: Increasing Intelligence - LessWrong
https://www.lesswrong.com/posts/F6ZdDqx48aFNi49Pp/human-genetic-engineering-increasing-intelligence
https://www.lesswrong.com/tag/intelligence-amplification
Intelligence Amplification - LessWrong
2022-12-01
Henrik Karlsson
<div><div><p>Posts tagged <em>Intelligence Amplification</em></p><p>Most Relevant </p></div>||||I|||| This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.

 LESSWRONGTags

 LW

 Login

 Intelligence Amplification

 Edit History Subscribe
 Discussion (0)
 Help improve this page (2 flags)
 Edit History Subscribe
 Discussion (0)
 Help improve this page (2 flags)
 Intelligence Amplification
 Random Tag
 Contributors
 Posts tagged Intelligence Amplification
 Most Relevant
 4
 48 Using GPT-3 to augment human intelligence
 Henrik Karlsson
 6mo
 7
 3
 15 Wacky, risky, anti-inductive intelligence-enhancement methods? Q
 NicholasKross , Quintin Pope
 7mo
 Q
 25
 2
 57 Intelligence Amplification Open Thread
 Will_Newsome
 12y
 346
 2
 49 Growing Up is Hard
 Eliezer Yudkowsky
 14y
 41
 2
 40 Transhumanism, genetic engineering, and the biological basis of intelligence.
 fowlertm
 5mo
 23
 2
 35 What is the current bottleneck on genetic engineering of human embryos for improved IQ Q
 Eli Tyre , James_Miller
 2y
 Q
 13
 2
 33 What experiments would demonstrate "upper limits of augmented working memory?" Q
 Raemon
 3y
 Q
 7
 2
 28 Ways to increase working memory, and/or cope with low working memory? Q
 NicholasKross
 6mo
 Q
 18
 2
 21 Intelligence Amplification and Friendly AI
 lukeprog
 9y
 26
 2
 7 The Power (and limits?) of Chunking
 NicholasKross
 5mo
 2
 2
 2 Intelligence Amplification Interview
 KatjaGrace
 9y
 0
 1
 23 Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation
 Roko
 14y
 92
 1
 16 Ask LessWrong: Human cognitive enhancement now?
 taw
 14y
 73
 1
 11 It's worth noticing when your WISDOM goes up by 1
 Trevor1
 2mo
 0
 1
 8 Historical examples of people gaining unusual cognitive abilities? Q
 NicholasKross
 2mo
 Q
 2
 Load More (15/17)
 Add Posts
This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.
 LESSWRONGTags
 LW
 Login
 Intelligence Amplification
 Edit History Subscribe
 Discussion (0)
 Help improve this page (2 flags)
 Edit History Subscribe
 Discussion (0)
 Help improve this page (2 flags)
 Intelligence Amplification
 Random Tag
 Contributors
 Posts tagged Intelligence Amplification
 Most Relevant
 4
 48 Using GPT-3 to augment human intelligence
 Henrik Karlsson
 6mo
 7
 3
 15 Wacky, risky, anti-inductive intelligence-enhancement methods? Q
 NicholasKross , Quintin Pope
 7mo
 Q
 25
 2
 57 Intelligence Amplification Open Thread
 Will_Newsome
 12y
 346
 2
 49 Growing Up is Hard
 Eliezer Yudkowsky
 14y
 41
 2
 40 Transhumanism, genetic engineering, and the biological basis of intelligence.
 fowlertm
 5mo
 23
 2
 35 What is the current bottleneck on genetic engineering of human embryos for improved IQ Q
 Eli Tyre , James_Miller
 2y
 Q
 13
 2
 33 What experiments would demonstrate "upper limits of augmented working memory?" Q
 Raemon
 3y
 Q
 7
 2
 28 Ways to increase working memory, and/or cope with low working memory? Q
 NicholasKross
 6mo
 Q
 18
 2
 21 Intelligence Amplification and Friendly AI
 lukeprog
 9y
 26
 2
 7 The Power (and limits?) of Chunking
 NicholasKross
 5mo
 2
 2
 2 Intelligence Amplification Interview
 KatjaGrace
 9y
 0
 1
 23 Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation
 Roko
 14y
 92
 1
 16 Ask LessWrong: Human cognitive enhancement now?
 taw
 14y
 73
 1
 11 It's worth noticing when your WISDOM goes up by 1
 Trevor1
 2mo
 0
 1
 8 Historical examples of people gaining unusual cognitive abilities? Q
 NicholasKross
 2mo
 Q
 2
 Load More (15/17)
 Add Posts
Intelligence Amplification - LessWrong
https://www.lesswrong.com/tag/intelligence-amplification