{"action":"opened","issue":{"url":"https://api.github.com/repos/guevara/read-it-later/issues/1767","repository_url":"https://api.github.com/repos/guevara/read-it-later","labels_url":"https://api.github.com/repos/guevara/read-it-later/issues/1767/labels{/name}","comments_url":"https://api.github.com/repos/guevara/read-it-later/issues/1767/comments","events_url":"https://api.github.com/repos/guevara/read-it-later/issues/1767/events","html_url":"https://github.com/guevara/read-it-later/issues/1767","id":390760864,"node_id":"MDU6SXNzdWUzOTA3NjA4NjQ=","number":1767,"title":"Spaced Repetition - Gwern.net","user":{"login":"guevara","id":228309,"node_id":"MDQ6VXNlcjIyODMwOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/228309?v=4","gravatar_id":"","url":"https://api.github.com/users/guevara","html_url":"https://github.com/guevara","followers_url":"https://api.github.com/users/guevara/followers","following_url":"https://api.github.com/users/guevara/following{/other_user}","gists_url":"https://api.github.com/users/guevara/gists{/gist_id}","starred_url":"https://api.github.com/users/guevara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guevara/subscriptions","organizations_url":"https://api.github.com/users/guevara/orgs","repos_url":"https://api.github.com/users/guevara/repos","events_url":"https://api.github.com/users/guevara/events{/privacy}","received_events_url":"https://api.github.com/users/guevara/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-13T16:35:11Z","updated_at":"2018-12-13T16:35:11Z","closed_at":null,"author_association":"OWNER","body":"Spaced Repetition - Gwern.net<br>\n<br>\nhttps://ift.tt/2yBeFKP<br>\n<br>\ngwern<br>\n<br>\n<div><div>\n<blockquote>\n<p>Spaced repetition is a technique for efficient memorization &amp; practice of skills where instead of doing a lot of work quickly, each item’s practices are automatically spread out over time, with increasing durations as one learns the item. I review what this technique is useful for, some of the large research literature on it and the testing effect (up to ~2013, primarily), the available software tools and use patterns, and miscellaneous ideas &amp; observations on it.</p>\n</blockquote>\n<p>One of the most fruitful areas of computing is making up for human frailties. They do arithmetic perfectly because we can’t<a href=\"#fn1\"><sup>1</sup></a>. They remember terabytes because we’d forget. They make the best calendars, because they always check what there is to do today.</p>\n<p>We use any number of such <a title=\"Wikipedia: neuroprosthetic\" href=\"https://en.wikipedia.org/wiki/neuroprosthetic\">neuroprosthetic</a>s<a href=\"#fn2\"><sup>2</sup></a>, but there are always more to be discovered. They’re worth looking for because they are so valuable: a shovel is much more effective than your hand, but a <a title=\"Wikipedia: power shovel\" href=\"https://en.wikipedia.org/wiki/power%20shovel\">power shovel</a> is orders of magnitude better than both - even if it requires training and expertise to use.</p>\n<section>\n<blockquote>\n<p>You can get a good deal from rehearsal,<br>\nIf it just has the proper dispersal.<br>\nYou would just be an ass,<br>\nTo do it <em>en masse</em>,<br>\nYour remembering would turn out much worsal.<a href=\"#fn3\"><sup>3</sup></a></p>\n</blockquote>\n<p>My current favorite prosthesis is the class of software that exploits the <a title=\"Wikipedia: spacing effect\" href=\"https://en.wikipedia.org/wiki/spacing%20effect\">spacing effect</a>, a centuries-old observation in cognitive psychology, to achieve results in studying or memorization much better than conventional student techniques; it is, alas, obscure<a href=\"#fn4\"><sup>4</sup></a>.</p>\n<p>The spacing effect essentially says that if you have a question (<q>What is the fifth letter in this random sequence you learned?</q>), and you can only study it, say, 5 times, then your memory of the answer (<q>e</q>) will be strongest if you spread your 5 tries out over a long period of time - days, weeks, and months. One of the worst things you can do is blow your 5 tries within a day or two. You can think of the <q><a title=\"Wikipedia: forgetting curve\" href=\"https://en.wikipedia.org/wiki/forgetting%20curve\">forgetting curve</a></q> as being like a chart of a radioactive <a title=\"Wikipedia: half-life\" href=\"https://en.wikipedia.org/wiki/half-life\">half-life</a>: each review bumps your memory up in strength 50% of the chart, say, but review doesn’t do much in the early days because the memory simply hasn’t decayed much! (<em>Why</em> does the spacing effect work, on a biological level? There are clear neurochemical differences between massed and spaced <a title=\"'Molecular Determinants of the Spacing Effect', Naqib et al 2012\" href=\"http://downloads.hindawi.com/journals/np/2012/581291.pdf\">in animal models</a> with spacing (&gt;1 hour) enhancing <a title=\"Wikipedia: long-term potentiation\" href=\"https://en.wikipedia.org/wiki/long-term%20potentiation\">long-term potentiation</a> but not massed<a href=\"#fn5\"><sup>5</sup></a>, but the why and wherefore - that’s an open question; see the concept of <a title=\"Wikipedia: Engram (neuropsychology)\" href=\"https://en.wikipedia.org/wiki/Engram%20%28neuropsychology%29\">memory traces</a> or the <a href=\"#when-to-review\">sleep</a> studies.) A graphical representation of the forgetting curve:</p>\n<figure>\n<img alt=\"Stahl et al 2010; CNS Spectrums\" width=\"449\" title=\"http://www.cnsspectrums.com/userdocs/ArticleImages/Stahl_figure1.jpg\" src=\"http://www.gwern.net/images/spacedrepetition/forgetting-curve-stahl.jpg\" height=\"497\"><figcaption>Stahl et al 2010; <em>CNS Spectrums</em></figcaption>\n</figure>\n<p>Even better, it’s known that <a title=\"Wikipedia: active recall\" href=\"https://en.wikipedia.org/wiki/active%20recall\">active recall</a> is a far superior method of learning than simply passively being exposed to information.<a href=\"#fn6\"><sup>6</sup></a> Spacing also scales to huge quantities of information; gambler/financier <a title=\"Wikipedia: Edward O. Thorp\" href=\"https://en.wikipedia.org/wiki/Edward%20O.%20Thorp\">Edward O. Thorp</a> harnessed <q>spaced learning</q> when he was a physics grad student <q>in order to be able to work longer and harder</q><a href=\"#fn7\"><sup>7</sup></a>, and <a title=\"Wikipedia: Roger Craig (Jeopardy! contestant)\" href=\"https://en.wikipedia.org/wiki/Roger%20Craig%20%28Jeopardy%21%20contestant%29\">Robert Craig</a> set multiple records on the quiz show <em><a title=\"Wikipedia: Jeopardy!\" href=\"https://en.wikipedia.org/wiki/Jeopardy%21\">Jeopardy!</a></em> 2010-2011 in part thanks to <a href=\"http://quantifiedself.com/2011/09/roger-craig-on-knowledge-tracking/#comment-3004\">using Anki</a> to memorize chunks of a collection of <a href=\"http://www.j-archive.com/\">&gt;200,000</a> past questions<a href=\"#fn8\"><sup>8</sup></a>; a later Jeopardy winner, Arthur Chu, also used spaced repetition<a href=\"#fn9\"><sup>9</sup></a>.</p>\n<p>A graphic might help; imagine here one can afford to review a given piece of information a few times (one is a busy person). By looking at the odds we can remember the item, we can see that cramming wins in the short term, but unexercised memories decay so fast that after not too long spacing is much superior<a href=\"#fn10\"><sup>10</sup></a>:</p>\n<p><img alt=\"Massed vs spaced\" width=\"600\" src=\"http://www.gwern.net/images/spacedrepetition/forgetting-curves.png\" height=\"400\"> <img alt=\"Wired (original, Wozniak?); massed vs spaced\" width=\"630\" title=\"http://www.wired.com/images/article/magazine/1605/ff_wozniak_graph_f.jpg\" src=\"http://www.gwern.net/images/spacedrepetition/forgetting-curve-wired-wozniak.jpg\" height=\"369\"></p>\n<p>It’s more dramatic if we look at a video visualizing decay of a corpus of memory with <a href=\"http://www.youtube.com/watch?v=ai2K3qHpC7c#t=2m40s\">random review vs most-recent review vs spaced review</a>.</p>\n<section>\n<h2>If you’re so good, why aren’t you rich</h2>\n<blockquote>\n<p>Most people find the concept of programming obvious, but the doing impossible.<a href=\"#fn11\"><sup>11</sup></a></p>\n</blockquote>\n<p>Of course, the latter strategy (cramming) is precisely what students do. They cram the night before the test, and a month later can’t remember anything. So why do people do it? (I’m not innocent myself.) Why is spaced repetition so dreadfully unpopular, even among the people who try it once?<a href=\"#fn12\"><sup>12</sup></a></p>\n<figure>\n<img alt=\"scumbag brain meme: knows everything when cramming the night before the test / and forgets everything a month later\" width=\"485\" title=\"Humorous description of the downside to cramming as a study method; original: http://www.ml.sun.ac.za/wordpress/wp-content/uploads/2012/01/Screen-Shot-2012-01-30-at-4.24.53-PM.png\" src=\"http://www.gwern.net/images/spacedrepetition/scumbag-brain.png\" height=\"340\"><figcaption>scumbag brain meme: knows everything when cramming the night before the test / and forgets everything a month later</figcaption>\n</figure>\n<p>Because it does work. Sort of. Cramming is a trade-off: you trade a strong memory now for weak memory later. (Very weak<a href=\"#fn13\"><sup>13</sup></a>.) And tests are usually of all the new material, with occasional old questions, so this strategy pays off! That’s the damnable thing about it - its memory longevity &amp; quality are, in sum, less than that of spaced repetition, but cramming delivers its goods <em>now</em><a href=\"#fn14\"><sup>14</sup></a>. So cramming is a rational, if short-sighted, response, and even SRS software recognize its utility &amp; support it to some degree<a href=\"#fn15\"><sup>15</sup></a>. (But as one might expect, if the testing is continuous and incremental, then the learning tends to also be long-lived<a href=\"#fn16\"><sup>16</sup></a>; I do not know if this is because that kind of testing is a disguised accidental spaced repetition system, or the students/subjects simply studying/acting differently in response to small-stakes exams.) In addition to this short-term advantage, there’s <a title=\"'Metacognitive awareness of learning strategies in undergraduates', McCabe 2011\" href=\"http://www.gwern.net/docs/spacedrepetition/2011-mccabe.pdf\">an ignorance</a> of the advantages of spacing and a subjective <em>illusion</em> that the gains persist<a href=\"#fn17\"><sup>17</sup></a><a href=\"#fn18\"><sup>18</sup></a> (cf. <a title=\"Distributed Learning: Data, Metacognition, and Educational Implications\" href=\"http://www.gwern.net/docs/spacedrepetition/2012-son.pdf\">Son &amp; Simon 2012</a><a href=\"#fn19\"><sup>19</sup></a>, <a title=\"The Spacing Effect and Metacognitive Control\" href=\"http://www.gwern.net/docs/spacedrepetition/2014-mulligan.pdf\">Mulligan &amp; Peterson 2014</a>, <a title=\"Self-Regulated Learning: Beliefs, Techniques, and Illusions\" href=\"http://www.gwern.net/docs/spacedrepetition/2013-bjork.pdf\">Bjork et al 2013</a>); from <a title=\"Optimising Learning Using Flashcards: Spacing Is More Effective Than Cramming\" href=\"http://sites.williams.edu/nk2/files/2011/08/Kornell.2009b.pdf\">Kornell 2009</a>’s study of GRE vocab (emphasis added):</p>\n<blockquote>\n<p>Across experiments, <em>spacing</em> was more effective than massing for 90% of the participants, yet after the first study session, 72% of the participants believed that <em>massing</em> had been more effective than spacing….When they do consider spacing, they often exhibit the illusion that massed study is more effective than spaced study, even when the reverse is true (<a title=\"Does the Sensitivity of Judgments of Learning (JOLs) to the Effects of Various Study Activities Depend on When the JOLs Occur?\" href=\"http://www.gwern.net/docs/spacedrepetition/1994-dunlosky.pdf\">Dunlosky &amp; Nelson, 1994</a>; Kornell &amp; Bjork, 2008a; <a title=\"Metacognition in Motor Learning\" href=\"http://www.gwern.net/docs/spacedrepetition/2001-simon.pdf\">Simon &amp; Bjork 2001</a>; <a href=\"http://www.willatworklearning.com/2005/11/research_review.html\">Zechmeister &amp; Shaughnessy, 1980</a>).</p>\n</blockquote>\n<p>As one would expect if the testing and spacing effects are real things, students who naturally test themselves and study well in advance of exams tend to have higher GPAs.<a href=\"#fn20\"><sup>20</sup></a> If we interpret questions as tests, we are not surprised to see that 1-on-1 tutoring works <a title=\"Wikipedia: Bloom's 2 Sigma Problem\" href=\"https://en.wikipedia.org/wiki/Bloom%27s%202%20Sigma%20Problem\">dramatically better</a> than regular teaching and that tutored students answer orders of magnitude more questions<a href=\"#fn21\"><sup>21</sup></a>.</p>\n<p>This short-term perspective is not a good thing in the long term, of course. Knowledge builds on knowledge; one is not learning independent bits of trivia. <a title=\"Wikipedia: Richard Hamming\" href=\"https://en.wikipedia.org/wiki/Richard%20Hamming\">Richard Hamming</a> recalls in <a href=\"http://www.cs.virginia.edu/~robins/YouAndYourResearch.html\"><q>You and Your Research</q></a>:</p>\n<blockquote>\n<p>You observe that most great scientists have tremendous drive. I worked for ten years with <a title=\"Wikipedia: John Tukey\" href=\"https://en.wikipedia.org/wiki/John%20Tukey\">John Tukey</a> at <a title=\"Wikipedia: Bell Labs\" href=\"https://en.wikipedia.org/wiki/Bell%20Labs\">Bell Labs</a>. He had tremendous drive. One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into <a title=\"Wikipedia: Hendrik Wade Bode\" href=\"https://en.wikipedia.org/wiki/Hendrik%20Wade%20Bode\">Bode</a>’s office and said, <q>How can anybody my age know as much as John Tukey does?</q> He leaned back in his chair, put his hands behind his head, grinned slightly, and said, <q>You would be surprised Hamming, how much you would know if you worked as hard as he did that many years.</q> I simply slunk out of the office!</p>\n<p>What Bode was saying was this: <q>Knowledge and productivity are like <a title=\"Wikipedia: compound interest\" href=\"https://en.wikipedia.org/wiki/compound%20interest\">compound interest</a>.</q> Given two people of approximately the same ability and one person who works 10% more than the other, the latter will more than twice outproduce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity - it is very much like compound interest. I don’t want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime. I took Bode’s remark to heart; I spent a good deal more of my time for some years trying to work a bit harder and I found, in fact, I could get more work done.</p>\n</blockquote>\n<p>Knowledge needs to accumulate, and flashcards with spaced repetition can aid in just that accumulation, fostering steady review even as the number of cards and intellectual prerequisites mounts into the <a href=\"#the-workload\">thousands</a>.</p>\n<p>This long term focus may explain why explicit spaced repetition is an uncommon studying technique: the pay-off is distant &amp; counterintuitive, the cost of self-control near &amp; vivid. (See <a title=\"Wikipedia: hyperbolic discounting\" href=\"https://en.wikipedia.org/wiki/hyperbolic%20discounting\">hyperbolic discounting</a>.) It doesn’t help that it’s pretty difficult to figure out <em>when</em> one should review - the optimal point is when you’re just about to forget about it, but that’s the kicker: if you’re just about to forget about it, how are you supposed to remember to review it? You only remember to review what you remember, and what you already remember isn’t what you need to review!<a href=\"#fn22\"><sup>22</sup></a></p>\n<p>The paradox is resolved by letting a computer handle all the calculations. We can thank <a title=\"'Memory: A Contribution to Experimental Psychology', Ebbinghaus 1885 (Ruger &amp; Bussenius 1913)\" href=\"https://en.wikisource.org/wiki/Memory:_A_Contribution_to_Experimental_Psychology\">Hermann Ebbinghaus</a> for investigating in such tedious detail than we can, in fact, program a computer to calculate both the forgetting curve and optimal set of reviews<a href=\"#fn23\"><sup>23</sup></a>. This is the insight behind <a title=\"Wikipedia: spaced repetition\" href=\"https://en.wikipedia.org/wiki/spaced%20repetition\">spaced repetition</a> software: ask the same question over and over, but over increasing spans of time. You start with asking it once every few days, and soon the human remembers it reasonably well. Then you expand intervals out to weeks, then months, and then years. Once the memory is formed and dispatched to long-term memory, it needs but occasional exercise to remain hale and hearty<a href=\"#fn24\"><sup>24</sup></a> - I remember well the large dinosaurs made of cardboard for my 4<sup>th</sup> or 5<sup>th</sup> birthday, or the tunnel made out of boxes, even though I recollect them once or twice a year at most.</p>\n</section>\n<section>\n<h2>Literature review</h2>\n<p>But don’t take my word for it - <em>Nullius in verba</em>! We can look at the science. Of course, if you do take my word for it, you probably just want to read about how to use it and all the nifty things you can do, so I suggest you <a href=\"#using-it\">skip all the way down</a> to that section. Everyone else, we start at the beginning:</p>\n<section>\n<h3>Background: testing works!</h3>\n<blockquote>\n<p><q>If you read a piece of text through twenty times, you will not learn it by heart so easily as if you read it ten times while attempting to recite from time to time and consulting the text when your memory fails.</q> –<em><a title=\"Wikipedia: The New Organon\" href=\"https://en.wikipedia.org/wiki/The%20New%20Organon\">The New Organon</a></em>, <a title=\"Wikipedia: Francis Bacon\" href=\"https://en.wikipedia.org/wiki/Francis%20Bacon\">Francis Bacon</a></p>\n</blockquote>\n<p>The <a title=\"Wikipedia: testing effect\" href=\"https://en.wikipedia.org/wiki/testing%20effect\">testing effect</a> is the established psychological observation that the mere act of testing someone’s memory will strengthen the memory (regardless of whether there is feedback). Since <a title=\"Wikipedia: spaced repetition\" href=\"https://en.wikipedia.org/wiki/spaced%20repetition\">spaced repetition</a> is just testing on particular days, we ought to establish that testing works better than regular review or study, and that it works outside of memorizing random dates in history. To cover a few papers:</p>\n<ol type=\"1\">\n<li>\n<p>Allen, G.A., Mahler, W.A., &amp; Estes, W.K. (1969). <a href=\"http://www.gwern.net/docs/spacedrepetition/1969-allen.pdf\"><q>Effects of recall tests on long-term retention of paired associates</q></a>. <em>Journal of Verbal Learning and Verbal Behavior</em>, 8, 463-470</p>\n<p>1 test results in memories as strong a day later as studying 5 times; intervals improve retention compared to massed presentation.</p>\n</li>\n<li>\n<p>Karpicke &amp; Roediger (2003). <a href=\"http://www.wsu.edu/~fournier/Teaching/psych592/Readings/Karpicke_et_al_2008.pdf\"><q>The Critical Importance of Retrieval for Learning</q></a></p>\n<p>In learning Swahili vocabulary, students were given varying routines of testing or studying or testing and studying; this resulted in similar scores during the learning phase. Students were asked to predict what percentage they’d remember (average: 50% over all groups). One week later, the students who tested remembered ~80% of the vocabulary versus ~35% for non-testing students. Some students were tested or studied more than others; diminishing returns set in quickly once the memory had formed the first day. Students reported rarely testing themselves and not testing already learned items.</p>\n<p>Lesson: again, testing improves memory compared to studying. Also, no student knows this.</p>\n</li>\n<li>\n<p>Roediger &amp; Karpicke (2006a). <a href=\"http://www.gwern.net/docs/spacedrepetition/2006-roediger.pdf\"><q>Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention</q></a></p>\n<p>Students were tested (with no feedback) on reading comprehension of a passage over 5 minutes, 2 days, and 1 week. Studying beat testing over 5 minutes, but nowhere else; students believed studying superior to testing over all intervals. At 1 week, testing scores were ~60% versus ~40%.</p>\nLesson: testing improves memory compared to studying. Everyone (teachers &amp; students) <q>knows</q> the opposite.</li>\n<li>\n<p>Karpicke &amp; Roediger (2006a). <a href=\"http://learninglab.psych.purdue.edu/downloads/2007_Karpicke_Roediger_JEPLMC.pdf\"><q>Expanding retrieval promotes short-term retention, but equal interval retrieval enhances long-term retention</q></a></p>\nGeneral scientific prose comprehension; from Roediger &amp; Karpicke 2006b: <q>After 2 days, initial testing produced better retention than restudying (68% vs. 54%), and an advantage of testing over restudying was also observed after 1 week (56% vs. 42%).</q>\n</li>\n<li>\n<p>Roediger &amp; Karpicke (2006b). <a href=\"http://isites.harvard.edu/fs/docs/icb.topic951136.files/powerOfTestingMemory-roedigerKarpicke.pdf\"><q>The Power of Testing Memory: Basic Research and Implications for Educational Practice</q></a></p>\nLiterature review; 7 studies before 1941 demonstrating testing improves retention, and 6 afterwards. See also the reviews <a title=\"Thalheimer 2006\" href=\"http://phase-6.de/system/galleries/download/lernsoftware/Spacing_Learning_Over_Time__March2009v1_.pdf\"><q>Spacing Learning Events Over Time: What the Research Says</q></a> &amp; <a href=\"http://laplab.ucsd.edu/articles/Carpenter_etal_EPR2012.pdf\"><q>Using spacing to enhance diverse forms of learning: Review of recent research and implications for instruction</q></a>, Carpenter et al 2012.</li>\n<li>\n<p>Agarwal et al 2008, <a href=\"https://pdfs.semanticscholar.org/7521/c9adbe66cb2e777f37b6b00e97f5f95633c2.pdf\"><q>Examining the Testing Effect with Open- and Closed-Book Tests</q></a></p>\nAs with #2, the purer forms of testing (in this case, open-book versus closed-book testing) did better over the long run, and students were deluded about what worked best.</li>\n<li>\n<p>Bangert-Drowns et al 1991. <a href=\"http://www.gwern.net/docs/spacedrepetition/1991-bangert-drowns.pdf\"><q>Effects of frequent classroom testing</q></a></p>\nMeta-analysis of 35 studies (1929-1989) varying tests during school semesters. 29 found benefits; 5 found negatives; 1 null result. Meta-study found large benefits to testing even once, then diminishing returns.</li>\n<li>Cook 2006, <a href=\"http://www.gwern.net/docs/spacedrepetition/2006-cook.pdf\"><q>Impact of self-assessment questions and learning styles in Web-based learning: a randomized, controlled, crossover trial</q></a>; final scores were higher when the doctors (residents) learned with questions.</li>\n<li>Johnson &amp; Kiviniemi 2009, <a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2747780/\"><q>The Effect of Online Chapter Quizzes on Exam Performance in an Undergraduate Social Psychology Course</q></a> (<q>This study examined the effectiveness of compulsory, mastery-based, weekly reading quizzes as a means of improving exam and course performance. Completion of reading quizzes was related to both better exam and course performance.</q>); see also <a href=\"http://www.gwern.net/docs/spacedrepetition/2012-mcdaniel.pdf%22Using%20quizzes%20to%20enhance%20summative-assessment%20performance%20in%20a%20web-based%20class:%20An%20experimental%20study%22\">McDaniel et al 2012</a>.</li>\n<li>Metsämuuronen 2013, <a href=\"http://www.ccsenet.org/journal/index.php/jedp/article/download/19582/15080\"><q>Effect of Repeated Testing on the Development of Secondary Language Proficiency</q></a>\n</li>\n<li>Meyer &amp; Logan 2013, <a href=\"http://www.gwern.net/docs/spacedrepetition/2013-meyer.pdf\"><q>Taking the Testing Effect Beyond the College Freshman: Benefits for Lifelong Learning</q></a>; verifies testing effect in older adults has similar effect size as younger</li>\n<li><p>Larsen &amp; Butler 2013, <a href=\"http://www.gwern.net/docs/spacedrepetition/2013-larsen.pdf\"><q>Test-enhanced learning</q></a></p></li>\n</ol>\n<p>(One might be tempted to object that testing works for <em>some</em> <a title=\"Wikipedia: learning styles\" href=\"https://en.wikipedia.org/wiki/learning%20styles\">learning styles</a>, perhaps verbal styles. This is an unsupported assertion inasmuch as the experimental literature on learning styles is poor and the existing evidence mixed that there are such things as learning styles.<a href=\"#fn25\"><sup>25</sup></a>)</p>\n<section>\n<h4>Subjects</h4>\n<p>The above studies often used pairs of words or words themselves. How well does the testing effect generalize?</p>\n<p>Materials which benefited from testing:</p>\n<ul>\n<li>foreign vocabulary (eg. Karpicke &amp; Roediger 2003, <a href=\"http://www.cs.colorado.edu/~mozer/Research/Selected%20Publications/reprints/Cepedaetal2009.pdf\">Cepeda et al 2009</a>, Fritz et al 2007<a href=\"#fn26\"><sup>26</sup></a>, <a title=\"Chinese Radicals in Spaced Repetition Systems: a pilot study on the acquisition of Chinese characters by students learning Chinese as a foreign language\" href=\"http://scholar.sun.ac.za/bitstream/handle/10019.1/80325/delarouviere_chinese_2012.pdf?sequence=1\">de la Rouviere 2012</a>)</li>\n<li>\n<a title=\"Wikipedia: GRE\" href=\"https://en.wikipedia.org/wiki/GRE\">GRE</a> materials (like vocab, <a href=\"http://sites.williams.edu/nk2/files/2011/08/Kornell.2009b.pdf\">Kornell 2009</a>); prose passages on general scientific topics (Karpicke &amp; Roediger, 2006a; Pashler et al, 2003)</li>\n<li>trivia (<a title=\"Tests and Test Feedback as Learning Sources\" href=\"http://www.gwern.net/docs/spacedrepetition/1991-mcdaniel.pdf\">McDaniel &amp; Fisher 1991</a>)</li>\n<li>elementary &amp; middle school lessons with subjects such as biographical material and science (<a href=\"http://www.archive.org/details/recitationasafa00gategoog\">Gates 1917</a>; <a href=\"http://www.gwern.net/docs/spacedrepetition/1939-spitzer.pdf\">Spitzer 1939</a><a href=\"#fn27\"><sup>27</sup></a> and Vlach &amp; Sandhofer 2012<a href=\"#fn28\"><sup>28</sup></a>, respectively)</li>\n<li>Agarwal et al (2008): short-answer tests superior on textbook passages</li>\n<li>history textbooks; retention better with initial short-answer test rather than multiple choice (<a href=\"http://www.gwern.net/docs/spacedrepetition/1982-nungester.pdf\">Nungester &amp; Duchastel 1982</a>)</li>\n<li>\n<a href=\"http://www.gwern.net/docs/spacedrepetition/1975-laporte.pdf\">LaPorte &amp; Voss (1975)</a> also found better retention compared to multiple-choice or recognition problems</li>\n<li>\n<a href=\"http://www.gwern.net/docs/spacedrepetition/1981-duchastel\">Duchastel &amp; Nungester, 1981</a>: 6 months after testing, testing beat studying in retention of a history passage</li>\n<li>\n<a href=\"http://www.gwern.net/docs/spacedrepetition/1981-duchastel.pdf\">Duchastel (1981)</a>: free recall decisively beat short-answer &amp; multiple choice for reading comprehension of a history passage</li>\n<li>\n<a href=\"http://www.gwern.net/docs/spacedrepetition/1989-glover.pdf\">Glover (1989)</a>: free recall self-test beat recognition or <a title=\"Wikipedia: Cloze deletions\" href=\"https://en.wikipedia.org/wiki/Cloze%20deletions\">Cloze deletions</a>; subject matter was the labels for parts of flowers</li>\n<li>\n<a title=\"Test format and corrective feedback modify the effect of testing on long-term retention\" href=\"http://www.gwern.net/docs/spacedrepetition/2007-kang.pdf\">Kang, McDermott, and Roediger (2007)</a>: prose passages; initial short answer testing produced superior results 3 days later on both multiple choice and short answer tests</li>\n<li>\n<a href=\"http://www.eric.ed.gov/ERICWebPortal/recordDetail?accno=EJ761434\">Leeming (2002)</a>: tests in 2 psychology courses, introductory &amp; memory/learning; <q>80% vs. 74% for the introductory psychology course and 89% vs. 80% for the learning and memory course</q><a href=\"#fn29\"><sup>29</sup></a>\n</li>\n</ul>\n<p>This covers a pretty broad range of what one might call <q>declarative</q> knowledge. Extending testing to other fields is more difficult and may reduce to <q>write many frequent analyses, not large ones</q> or <q>do lots of small exercises</q>, whatever those might mean in those fields:</p>\n<blockquote>\n<p>A third issue, which relates to the second, is whether our proposal of testing is really appropriate for courses with complex subject matters, such as the philosophy of Spinoza, Shakespeare’s comedies, or creative writing. Certainly, we agree that most forms of objective testing would be difficult in these sorts of courses, but we do believe the general philosophy of testing (broadly speaking) would hold-students should be continually engaged and challenged by the subject matter, and there should not be merely a midterm and final exam (even if they are essay exams). Students in a course on Spinoza might be assigned specific readings and thought-provoking essay questions to complete every week. This would be a transfer-appropriate form of weekly <q>testing</q> (albeit with take-home exams). Continuous testing requires students to continuously engage themselves in a course; they cannot coast until near a midterm exam and a final exam and begin studying only then.<a href=\"#fn30\"><sup>30</sup></a></p>\n</blockquote>\n</section>\n<section>\n<h4>Downsides</h4>\n<p>Testing does have some known flaws:</p>\n<ol type=\"1\">\n<li>\n<p>interference in recall - ability to remember tested items drives out ability to remember similar untested items</p>\n<p>Most/all studies were in laboratory settings and found relatively small effects:</p>\n<blockquote>\n<p>In sum, although various types of recall interference are quite real (and quite interesting) phenomena, we do not believe that they compromise the notion of test-enhanced learning. At worst, interference of this sort might dampen positive testing effects somewhat. However, the positive effects of testing are often so large that in most circumstances they will overwhelm the relatively modest interference effects.</p>\n</blockquote>\n</li>\n<li>\n<p>multiple choice tests can accidentally lead to <q>negative suggestion effects</q> where having previously seen a falsehood as an item on the test makes one more likely to believe it.</p>\n<p>This is mitigated or eliminated when there’s quick feedback about the right answer (see Butler &amp; Roediger 2008 <a href=\"http://www.gwern.net/docs/spacedrepetition/2008-butler.pdf\"><q>Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing</q></a>). Solution: don’t use multiple choice; inferior in testing ability to free recall or short answers, anyway.</p>\n</li>\n</ol>\n<p>Neither problem seems major.</p>\n</section>\n</section>\n<section>\n<h3>Distributed</h3>\n<p>A lot depends on <em>when</em> you do all your testing. Above we saw some benefits to testing a lot the moment you learn something, but the same number of tests could be spread out over time, to give us the <em>spacing effect</em> or <em>spaced repetition</em>. There are hundreds of studies involving the spacing effect:</p>\n<ul>\n<li>\n<a title=\"Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis\" href=\"http://uweb.cas.usf.edu/~drohrer/pdfs/Cepeda_et_al_2006PsychBull.pdf\">Cepeda et al 2006</a> is a review of 184 articles with 317 experiments; other reviews include:</li>\n<li>Ruch 1928, <a href=\"http://www.gwern.net/docs/spacedrepetition/1928-ruch.pdf\"><q>Factors influencing the relative economy of massed and distributed practice in learning</q></a>\n</li>\n<li>Crowder 1976, <a href=\"http://www.amazon.com/Principles-Learning-Memory-Experimental-Psychology/dp/0898591155/?tag=gwernnet-20\"><em>Principles of learning and memory</em></a>\n</li>\n<li>Dempster 1989, <a href=\"http://www.gwern.net/docs/spacedrepetition/1989-dempster.pdf\"><q>Spacing effects and their implications for theory and practice</q></a>\n</li>\n<li>Delaney et al 2010, <a href=\"http://www.gwern.net/docs/spacedrepetition/2010-delaney.pdf\"><q>Spacing and testing effects: A deeply critical, lengthy, and at times discursive review of the literature</q></a>\n</li>\n<li>Donovan &amp; Radosevich 1999, <a href=\"http://www.gwern.net/docs/spacedrepetition/1999-donovan.pdf\"><q>A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t</q></a>\n</li>\n<li>Greene 1992, <a href=\"http://www.amazon.com/Human-Memory-Paradigms-Robert-Greene/dp/080580997X/?tag=gwernnet-20\"><em>Human memory: Paradigms and paradoxes</em></a>\n</li>\n<li>Janiszewski et al 2003, <a href=\"http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.200.8846&amp;rep=rep1&amp;type=pdf\"><q>A meta-analysis of the spacing effect in verbal learning: Implications for research on advertising repetition and consumer memory</q></a>\n</li>\n<li>Pavlik &amp; Anderson 2003, <a href=\"http://repository.cmu.edu/cgi/viewcontent.cgi?article=1050&amp;context=psychology\"><q>An ACT-R model of the spacing effect</q></a>\n</li>\n<li>Balota et al 2006, <a href=\"http://psychnet.wustl.edu/coglab/wp-content/uploads/2015/01/2007-Is-expanded.pdf\"><q>Is Expanded Retrieval Practice a Superior Form of Spaced Retrieval? A Critical Review of the Extant Literature</q></a>\n</li>\n<li>Carpenter et al 2012, <a href=\"http://files.eric.ed.gov/fulltext/ED536925.pdf\"><q>Using Spacing to Enhance Diverse Forms of Learning: Review of Recent Research and Implications for Instruction</q></a>\n</li>\n</ul>\n<p>Almost unanimously they find spacing out tests is superior to massed testing when the final test/measurement is conducted days or years later<a href=\"#fn31\"><sup>31</sup></a>, although the mechanism isn’t clear<a href=\"#fn32\"><sup>32</sup></a>. Besides all the previously mentioned studies, we can throw in:</p>\n<ul>\n<li>Peterson, L. R., Wampler, R., Kirkpatrick, M., &amp; Saltzman, D. (1963). <a href=\"http://www.gwern.net/docs/spacedrepetition/1963-peterson.pdf\"><q>Effect of spacing presentations on retention of a paired associate over short intervals</q></a>. <em>Journal of Experimental Psychology</em>, 66(2), 206-209</li>\n<li>Glenberg, A. M. (1977). <a href=\"http://www.gwern.net/docs/spacedrepetition/1977-glenberg.pdf\"><q>Influences of retrieval processes on the spacing effect in free recall</q></a>. <em>Journal of Experimental Psychology: Human Learning and Memory</em>, 3(3), 282-294</li>\n<li>Balota et al 1989, <a href=\"http://psychnet.wustl.edu/coglab/wp-content/uploads/2015/01/1989-Balota.pdf\"><q>Age-related differences in the impact of spacing, lag and retention interval</q></a>. <em>Psychology and Aging</em>, 4, 3-9</li>\n</ul>\n<p>The research literature focuses <em>extensively</em> on the question of <em>what kind</em> of spacing is best and what this implies about memory: a spacing that has static fixed intervals or a spacing which expands? This is important for understanding memory and building models of it, and would be helpful for integrating spaced repetition into classrooms (for example, <a title=\"Making long-term memories in minutes: a spaced learning pattern from memory research in education\" href=\"http://journal.frontiersin.org/Journal/10.3389/fnhum.2013.00589/full\">Kelley &amp; Whatson 2013</a>’s 10 minutes studying / 10 minutes break schedule, repeating the same material 3 times, designed to trigger LTM formation on that block of material?) But for practical purposes, this is uninteresting; to sum it up, there are many studies pointing each way, and whatever difference in efficiency exists, is minimal. Most existing software follows SuperMemo in using an expanding spacing algorithm, so it’s not worth worrying about; as Mnemosyne developer Peter Bienstman says, it’s not clear the more complex algorithms really help<a href=\"#fn33\"><sup>33</sup></a>, and the Anki developers <a href=\"http://ankisrs.net/docs/manual.html#_what_spaced_repetition_algorithm_does_anki_use\">were concerned about</a> the larger errors SM3+ risks attempting to be more optimal. So too here.</p>\n<p>For those interested, 3 of the studies that found fixed spacings better than expanding:</p>\n<ol type=\"1\">\n<li>Carpenter, S. K., &amp; DeLosh, E. L. (2005). <a href=\"http://public.psych.iastate.edu/shacarp/Carpenter_Delosh_2005.pdf\"><q>Application of the testing and spacing effects to name learning</q></a>. <em>Applied Cognitive Psychology</em>, 19, 619-636<a href=\"#fn34\"><sup>34</sup></a>\n</li>\n<li>\n<p>Logan, J. M. (2004). <em>Spaced and expanded retrieval effects in younger and older adults</em>. Unpublished doctoral dissertation, Washington University, St. Louis, MO</p>\nThis thesis is interesting inasmuch as Logan found that young adults did considerably worse with an expanding spacing after a day.</li>\n<li><p>Karpicke &amp; Roediger, 2006a</p></li>\n</ol>\n<p>The fixed vs expanding issue aside, a list of additional generic studies finding benefits to spaced vs massed:</p>\n<ul>\n<li>Cepeda et al 2006 (large review used elsewhere in this page)</li>\n<li>Karpicke &amp; Roediger 2006a</li>\n<li>Rohrer &amp; Taylor 2006. <a href=\"http://uweb.cas.usf.edu/~drohrer/pdfs/Rohrer&amp;Taylor2006ACP.pdf\"><q>The effects of over-learning and distributed practice on the retention of mathematics knowledge</q></a>. <em>Applied Cognitive Psychology</em>, 20: 1209-1224 (see also <a title=\"The shuffling of mathematics problems improves learning\" href=\"http://www.gwern.net/docs/spacedrepetition/2007-rohrer.pdf\">Rohrer &amp; Taylor 2007</a>, <a title=\"The Effect of Overlearning on Long-Term Retention\" href=\"http://escholarship.org/uc/item/0279q9m1.pdf\">Rohrer et al 2005</a>)</li>\n<li>Seabrook et al 2005. <a href=\"http://www.gwern.net/docs/spacedrepetition/2005-seabrook.pdf\"><q>Distributed and Massed Practice: From Laboratory to Classroom</q></a>\n</li>\n<li>\n<p>Keppel, Geoffrey. <a href=\"http://www.gwern.net/docs/spacedrepetition/1967-keppel.pdf\"><q>A Reconsideration of the Extinction-Recovery Theory</q></a>. <em>Journal of Verbal Learning &amp; Verbal Behavior</em>. 6(4) 1967, 476-486</p>\nA week later, the massed reviewers went from 5.9 correct ~&gt; 2.1; the spaced reviewers went from 5.5 ~&gt; 5.0. (Note the usual observation: massed was initially better, and later much worse, less than half as good.)</li>\n<li>\n<p>Bloom &amp; Schuell 1981, <a href=\"http://www.gwern.net/docs/spacedrepetition/1981-bloom.pdf\"><q>Effects of massed and distributed practice on the learning and retention of second-language vocabulary</q></a></p>\nFour days after the 2 high school groups memorized 16 French words, the spaced group remembered 15 and the massed 11.</li>\n<li>\n<p>Rea, Cornelius P; Modigliani, Vito. <a href=\"https://www.researchgate.net/publication/232508016_The_effect_of_expanded_versus_massed_practice_on_the_retention_of_multiplication_facts_and_spelling_lists\"><q>The effect of expanded versus massed practice on the retention of multiplication facts and spelling lists</q></a>. <em>Human Learning: Journal of Practical Research &amp; Applications</em>. Vol 4(1) Jan-Mar 1985, 11-18<a href=\"#fn35\"><sup>35</sup></a></p>\n<blockquote>\n<p>A test immediately following the training showed superior performance for the distributed group (70% correct) compared to the massed group (53% correct). These results seem to show that the spacing effect applies to school-age children and to at least some types of materials that are typically taught in school.<a href=\"#fn36\"><sup>36</sup></a></p>\n</blockquote>\n</li>\n<li>\n<p>Donovan &amp; Radosevich 1999, <a href=\"http://www.gwern.net/docs/spacedrepetition/1999-donovan.pdf\"><q>A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t</q></a>:</p>\n<blockquote>\n<p>According to Donovan and Radosevich’s meta-analysis of spacing studies, the <a title=\"Wikipedia: effect size\" href=\"https://en.wikipedia.org/wiki/effect%20size\">effect size</a> for the spacing effect is <em>d</em>=.42. This means that the average person getting distributed training remembers better than about 67% of the people getting massed training. This effect size is nothing to sneeze at-in education research, effect sizes as low as <em>d</em>=.25 are considered <q>practically significant</q>, while effect sizes above <em>d</em>=1 are rare.<a href=\"#fn37\"><sup>37</sup></a></p>\n</blockquote>\n<blockquote>\n<p>In one meta-analysis by Donovan and Radosevich (1999), for instance, the size of the spacing effect declined sharply as conceptual difficulty of the task increased from low (e.g. rotary pursuit) to average (e.g. word list recall) to high (e.g. puzzle). By this finding, the benefits of spaced practise may be muted for many mathematics tasks.<a href=\"#fn38\"><sup>38</sup></a></p>\n</blockquote>\nThe Donovan meta-analysis notes that the effect size is smaller in studies with better methodology, but still important.</li>\n<li>\n<p>Bahrick, Harry P; Phelphs, Elizabeth. <a href=\"http://www.gwern.net/docs/spacedrepetition/1987-bahrick.pdf\"><q>Retention of Spanish vocabulary over 8 years</q></a>. <em>Journal of Experimental Psychology: Learning, Memory, &amp; Cognition</em>. Vol 13(2) April 1987, 344-349; the extremely long delay after the initial training period makes this particularly interesting:</p>\n<blockquote>\n<p>Harry Bahrick and Elizabeth Phelps (1987) examined the retention of 50 Spanish vocabulary words after an eight-year delay. Subjects were divided into three groups. Each practiced for seven or eight sessions, separated by a few minutes, a day, or 30 days. In each session, subjects practiced until they could produce the list perfectly one time….Eight years later, people in the no-delay group could recall 6% of the words, people in the one-day delay group could remember 8%, and those in the 30-day group averaged 15%. Everyone also took a multiple choice test, and again, the spacing effect was observed. The no-delay group scored 71%, the one-day group scored 80%, and the 30-day group scored 83%.</p>\n<p>…Bahrick and his colleagues varied both the spacing of practice and the amount of practice. Practice sessions were spaced 14, 28, or 56 days apart, and totaled 13 or 26 sessions. They tested subjects’ memory one, two, three, and five years after training. Once again, it took a bit longer to reach the criterion within each session when practice sessions were spaced farther apart, but again, this small investment paid dividends years later. It didn’t matter whether testing occurred at one, two, three, or five years after practice-the 56-day group always remembered the most, the 28-day group was next, and the 14-day group remembered the least. Further, the effect was quite large. If words were practiced every 14 days, you needed twice as much practice to reach the same level of performance as when words were practiced every 56 days!</p>\n</blockquote>\n</li>\n<li>\n<p>Pashler et al, 2003; <a href=\"http://www.pashler.com/Articles/Pashler_Zarow_Triplett03.pdf\"><q>Is Temporal Spacing of Tests Helpful Even When It Inflates Error Rates?</q></a></p>\nLong intervals between tests necessarily means you will often err; errors were thought to intrinsically reduce learning. While the extra errors do damage accuracy in the short-run, the long intervals are powerful enough that they still win.</li>\n<li>works in ill subpopulations:\n<ul>\n<li>\n<p>works on short-term review conducted with Alzheimer’s patients; spacing used on the scale of seconds and minutes, with modest success in teaching object locations or daily tasks to do<a href=\"#fn39\"><sup>39</sup></a>:</p>\n<ul>\n<li>Camp, C. J. (1989). <q>Facilitation of new learning in Alzheimer’s disease</q>. In G. C. Gilmore, P. J. Whitehouse, &amp; M. L. Wykle (Eds.), <a href=\"http://www.gwern.net/docs/spacedrepetition/1989-gilmore-memoryaginganddementia.djvu\"><em>Memory, aging, and dementia</em></a> (pp. 212-225)</li>\n<li>Camp, C. J., &amp; McKitrick, L. A. (1992). <q>Memory interventions in Alzheimer’s-type dementia populations: Methodological and theoretical issues</q>. In R. L. West &amp; J. D. Sinnott (Eds.), <em>Everyday memory and aging: Current research and methodology</em> (pp. 152-172) -</li>\n</ul>\n</li>\n<li>works with traumatic brain injury; Goverover et al 2009, <a href=\"https://pdfs.semanticscholar.org/0eb4/8078fdc906f368d22f324e0520e4ee4f9c08.pdf\"><q>Application of the spacing effect to improve learning and memory for functional tasks in traumatic brain injury: a pilot study</q></a>\n</li>\n<li>and multiple sclerosis; Goverover et al 2009, <a href=\"http://www.gwern.net/docs/spacedrepetition/2009-goverover.pdf\"><q>A functional application of the spacing effect to improve learning and memory in persons with multiple sclerosis</q></a>\n</li>\n</ul>\n</li>\n<li>\n<p>math<a href=\"#fn40\"><sup>40</sup></a>:</p>\n<ul>\n<li>multiplication (Ria &amp; Modigliani 1985)</li>\n<li>\n<a title=\"Wikipedia: permuting\" href=\"https://en.wikipedia.org/wiki/permuting\">permuting</a> a sequence (Rohrer &amp; Taylor 2006)on</li>\n<li>calculating the volume of <a title=\"Wikipedia: polyhedrons\" href=\"https://en.wikipedia.org/wiki/polyhedrons\">polyhedrons</a> (Rohrer &amp; Taylor 2007)</li>\n<li>statistics (<a title=\"Contextual Enrichment and Distribution of Practice in the Classroom\" href=\"http://people.tamu.edu/~stevesmith/SmithMemory/SmithRothkopf1984.pdf\">Smith &amp; Rothkopf 1984</a>)</li>\n<li>pre-calculus (<a title=\"Distributed practice: More bang for your homework buck\" href=\"http://www.gwern.net/docs/spacedrepetition/1997-revak.pdf\">Revak 1997</a><a href=\"#fn41\"><sup>41</sup></a> but there’s a related <a href=\"http://digitalcommons.uconn.edu/dissertations/AAI3464319/\">null <q>calculus I</q> result</a> as well) and algebra (<a title=\"The effects of cumulative practice on mathematics problem solving\" href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1284369/pdf/12102132.pdf\">Mayfield &amp; Chase 2002</a>, <a title=\"The Analysis of Two Teaching Programs: Massed and Distributed\" href=\"http://www.gwern.net/docs/spacedrepetition/2013-patac.pdf\">Patac &amp; Patac 2013</a>; possible null, <a title=\"The effects of distributed practice on two grade 10 mathematics classes\" href=\"https://getd.libs.uga.edu/pdfs/sutherland_pierre_201212_ma.pdf\">Sutherland 2013</a>)</li>\n</ul>\n</li>\n<li>medicine (<a title=\"Online spaced education to teach urology to medical students: a multi-institutional randomized trial\" href=\"http://www.gwern.net/docs/spacedrepetition/2009-kerfoot-brotschi.pdf\">Kerfoot &amp; Brotschi 2009</a>, <a title=\"Impact of online education on intern behaviour around joint commission national patient safety goals: a randomised trial\" href=\"http://qualitysafety.bmj.com/content/early/2012/06/15/bmjqs-2011-000702.abstract\">Shaw et al 2012</a>; <a title=\"Learning Benefits of On-Line Spaced Education Persist for 2 Years\" href=\"http://www.gwern.net/docs/spacedrepetition/2009-kerfoot.pdf\">Kerfoot 2009</a>, a 2 year followup to <a title=\"Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial\" href=\"https://pdfs.semanticscholar.org/f2b5/aed794e5d164065a184207f2663620b96ba3.pdf\">Kerfoot et al 2007</a> and Kerfoot has a number of <a href=\"http://app.qstream.com/pricekerfoot\">other relevant studies</a>; <a title=\"Improving the impact of didactic resident training with online spaced education\" href=\"http://www.gwern.net/docs/spacedrepetition/2013-gyorki.pdf\">Gyorki et al 2013</a>) and surgery (Moulton et al 2006, <a href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1856544/\"><q>Teaching Surgical Skills: What Kind of Practice Makes Perfect? A Randomized, Controlled Trial</q></a>, distributed practice of microvascular suturing; <a title=\"Increasing efficiency of surgical training: effects of spacing practice on skill acquisition and retention in laparoscopy training\" href=\"http://www.gwern.net/docs/spacedrepetition/2014-spruit.pdf\">Spruit et al 2014</a>)</li>\n<li>introductory psychology (Balch 2006, <a href=\"http://www.gwern.net/docs/spacedrepetition/2006-balch.pdf\"><q>Encouraging Distributed Study: A Classroom Experiment on the Spacing Effect</q></a><a href=\"#fn42\"><sup>42</sup></a>. <em>Teaching of Psychology</em>, 33, 249-252)</li>\n<li>8th-grade American history (<a href=\"http://laplab.ucsd.edu/articles/CarpenterPashlerCepeda2009.pdf\">Carpenter, Pashler, and Cepeda 2009</a>)</li>\n<li>learning to read with phonics (Seabrook et al 2005)</li>\n<li>music (<a title=\"When repetition isn't the best practice strategy: Examining differing levels of contextual interference during practice\" href=\"https://works.bepress.com/laura_stambaugh/6/download/\">Stambaugh 2009</a>)</li>\n<li>biology (middle school; <a title=\"Making long-term memories in minutes: a spaced learning pattern from memory research in education\" href=\"http://www.frontiersin.org/Journal/10.3389/fnhum.2013.00589/full\">Kelly &amp; Whatson 2013</a>)</li>\n<li>statistics (introductory; <a title=\"How Spacing and Variable Retrieval Practice Affect the Learning of Statistics Concepts\" href=\"http://www.gwern.net/docs/spacedrepetition/2015-maas.pdf\">Maas et al 2015</a>)</li>\n<li>memorizing website passwords (<a title=\"Towards Reliable Storage of 56-bit Secrets in Human Memory\" href=\"http://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-bonneau.pdf\">Bonneau &amp; Schechter 2014</a>, <a title=\"Spaced Repetition and Mnemonics Enable Recall of Multiple Strong Passwords\" href=\"http://arxiv.org/abs/1410.1490\">Blocki et al 2014</a>, <a title=\"The Complexity of Human Computation: A Concrete Model with an Application to Passwords\" href=\"https://arxiv.org/abs/1707.01204\">Blum &amp; Vempala 2017</a>)</li>\n<li><p>possibly not Australian constitutional law (<a title=\"The impact of student-generated digital flashcards on student learning of constitutional law\" href=\"http://www.gwern.net/docs/spacedrepetition/2015-colbran.pdf\">Colbran et al 2015</a>)</p></li>\n</ul>\n<section>\n<h4>Generality of spacing effect</h4>\n<p>We have already seen that spaced repetition is effective on a variety of academic fields and mediums. Beyond that, spacing effects can be found in:</p>\n<ul>\n<li>various <q>domains (e.g., learning perceptual motor tasks or learning lists of words)</q><a href=\"#fn43\"><sup>43</sup></a> such as spatial<a href=\"#fn44\"><sup>44</sup></a>\n</li>\n<li><q>across species (e.g., <a title=\"'Long-term memory is facilitated by cAMP response element-binding protein overexpression in the amygdala', Josselyn et al 2001\" href=\"http://www.jneurosci.org/content/21/7/2404.long\">rats</a>, pigeons, and humans [or <a title=\"'Deconstructing Memory in Drosophila', Margulies et al 2005\" href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3044934/\">flies</a> or <a title=\"'Massed and Spaced Learning in Honeybees: The Role of CS, US, the Intertrial Interval, and the Test Interval', Menzel et al 2001\" href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC311375/\">bumblebees</a>, and sea slugs, <a title=\"'Long-Term Habituation of a Defensive Withdrawal Reflex in Aplysia', Carew et al 1972\" href=\"http://www.gwern.net/docs/spacedrepetition/1972-carew.pdf\">Carew et al 1972</a> &amp; <a href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC155928/\">Sutton et al 2002</a>])</q></li>\n<li><q>across age groups [infancy<a href=\"#fn45\"><sup>45</sup></a>, childhood<a href=\"#fn46\"><sup>46</sup></a>, adulthood<a href=\"#fn47\"><sup>47</sup></a>, the elderly<a href=\"#fn48\"><sup>48</sup></a>] and individuals with different memory impairments</q></li>\n<li>\n<q>and across retention intervals of seconds<a href=\"#fn49\"><sup>49</sup></a> [to days<a href=\"#fn50\"><sup>50</sup></a>] to months</q> (we have already seen studies using years)</li>\n</ul>\n<p>The domains are limited, however. Cepeda et al 2006:</p>\n<blockquote>\n<p>[<a title=\"The efficacy of masses versus distributed practice as a function of desired learning outcomes and grade level of the student\" href=\"http://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=4569&amp;context=etd\">Moss 1995</a>, reviewing 120 articles] concluded that longer ISIs facilitate learning of verbal information (e.g., spelling<a href=\"#fn51\"><sup>51</sup></a>) and motor skills (e.g., mirror tracing); in each case, over 80% of studies showed a distributed practice benefit. In contrast, only one third of intellectual skill (e.g., math computation) studies showed a benefit from distributed practice, and half showed no effect from distributed practice.</p>\n<p>…[Donovan and Radosevich (1999)] The largest effect sizes were seen in low rigor studies with low complexity tasks (e.g., rotary pursuit, typing, and peg reversal), and retention interval failed to influence effect size. The only interaction Donovan and Radosevich examined was the interaction of ISI and task domain. It is important to note that task domain moderated the distributed practice effect; depending on task domain and lag, an increase in ISI either increased or decreased effect size. Overall, Donovan and Radosevich found that increasingly distributed practice resulted in larger effect sizes for verbal tasks like free recall, foreign language, and verbal discrimination, but these tasks also showed an inverse-U function, such that very long lags produced smaller effect sizes. In contrast, increased lags produced smaller effect sizes for skill tasks like typing, gymnastics, and music performance.</p>\n</blockquote>\n<p>Skills like gymnastics and music performance raise an important point about the testing effect and spaced repetition: they are for the maintenance of memories or skills, they do not increase it beyond what was already learned. If one is a gifted amateur when one starts reviewing, one remains a gifted amateur. Ericsson covers what is necessary to <em>improve</em> and attain new expertise: <a title=\"Wikipedia: deliberate practice\" href=\"https://en.wikipedia.org/wiki/deliberate%20practice\">deliberate practice</a><a href=\"#fn52\"><sup>52</sup></a>. From <a href=\"http://www.gwern.net/docs/psychology/writing/1993-ericsson.pdf\"><q>The Role of Deliberate Practice</q></a>:</p>\n<blockquote>\n<p>The view that merely engaging in a sufficient amount of practice - regardless of the structure of that practice - leads to maximal performance, has a long and contested history. In their classic studies of Morse Code operators, Bryan and Harter (1897, 1899) identified plateaus in skill acquisition, when for long periods subjects seemed unable to attain further improvements. However, with extended efforts, subjects could restructure their skill to overcome plateaus….Even very experienced Morse Code operators could be encouraged to dramatically increase their performance through deliberate efforts when further improvements were required…More generally, Thorndike (1921) observed that adults perform at a level far from their maximal level even for tasks they frequently carry out. For instance, adults tend to write more slowly and illegibly than they are capable of doing….The most cited condition [for optimal learning and improvement of performance] concerns the subjects’ motivation to attend to the task and exert effort to improve their performance….The subjects should receive immediate informative feedback and knowledge of results of their performance….In the absence of adequate feedback, efficient learning is impossible and improvement only minimal even for highly motivated subjects. Hence mere repetition of an activity will not automatically lead to improvement in, especially, accuracy of performance…In contrast to play, deliberate practice is a highly structured activity, the explicit goal of which is to improve performance. Specific tasks are invented to overcome weaknesses, and performance is carefully monitored to provide cues for ways to improve it further. We claim that deliberate practice requires effort and is not inherently enjoyable.</p>\n</blockquote>\n<section>\n<h5>Motor skills</h5>\n<p>It should be noted that reviews conflict on how much spaced repetition applies to motor skills; Lee &amp; Genovese 1988 find benefits, while Adams 1987 and earlier do not. The difference may be that simple motor tasks benefit from spacing as suggested by <a href=\"http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&amp;_&amp;ERICExtSearch_SearchValue_0=EJ215260&amp;ERICExtSearch_SearchType_0=no&amp;accno=EJ215260\">Shea &amp; Morgan 1979</a> (benefits to a randomized/spaced schedule), while <em>complex</em> ones where the subject is already operating at his limits do not benefit, suggested by <a title=\"Principles derived from the study of simple skills do not generalize to complex skill learning\" href=\"https://pdfs.semanticscholar.org/c747/336e77ce39f7f5cdbd937684a7f564e9e194.pdf\">Wulf &amp; Shea 2002</a>. Stambaugh 2009 mentions some of the diverging studies:</p>\n<blockquote>\n<p>The contextual interference hypothesis (Shea and Morgan 1979, Battig 1966 [<q>Facilitation and interference</q> in <a href=\"https://archive.org/details/acquisitionofski00conf\"><em>Acquisition of skill</em></a>]) predicted the blocked condition would exhibit superior performance immediately following practice (acquisition) but the random condition would perform better at delayed retention testing. This hypothesis is generally consistent in laboratory motor learning studies (e.g. <a title=\"The Locus of Contextual Interference in Motor-Skill Acquisition\" href=\"http://www.gwern.net/docs/spacedrepetition/1983-lee.pdf\">Lee &amp; Magill 1983</a>, <a title=\"Contextual interference: a meta-analytic study\" href=\"http://www.gwern.net/docs/spacedrepetition/2004-brady.pdf\">Brady 2004</a>), but less consistent in applied studies of sports skills (with a mix of positive &amp; negative e.g. <a title=\"A comparison of three practice schedules along the contextual interference continuum\" href=\"http://www.gwern.net/docs/spacedrepetition/1997-landin.pdf\">Landin &amp; Hebert 1997</a>, Hall et al 1994, <a title=\"Skill Acquisition and the Influence of Attentional Focus and Practice\" href=\"http://digitalcommons.mtu.edu/cgi/viewcontent.cgi?article=1486&amp;context=etds\">Regal 2013</a>) and fine-motor skills (<a title=\"The influence of professional expertise and task complexity upon the potency of the contextual interference effect\" href=\"http://www.gwern.net/docs/spacedrepetition/2005-ollis.pdf\">Ollis et al 2005</a>, <a title=\"High levels of contextual interference enhance handwriting skill acquisition\" href=\"http://www.gwern.net/docs/spacedrepetition/2004-stemarie.pdf\">Ste-Marie et al 2004</a>).</p>\n</blockquote>\n<p>Some of the positive spaced repetition studies (from Son &amp; Simon 2012):</p>\n<blockquote>\n<p>Perhaps even prior to the empirical work on cognitive learning and the spacing effect, the benefits of spaced study had been apparent in an array of motor learning tasks, including maze learning (Culler 1912), typewriting (Pyle 1915), archery (Lashley 1915), and javelin throwing (Murphy 1916; see Ruch 1928, for a larger review of the motor learning tasks which reap benefits from spacing; see also Moss 1996, for a more recent review of motor learning tasks). Thus, as in the cognitive literature, the study of practice distribution in the motor domain is long established (see reviews by Adams 1987; Schmidt and Lee 2005), and most interest has centered around the impact of varying the separation of learning trials of motor skills in learning and retention of practiced skills. Lee and Genovese (1988) conducted a review and meta-analysis of studies on distribution of practice, and they concluded that massing of practice tends to depress both immediate performance and learning, where learning is evaluated at some removed time from the practice period. Their main finding was, as in the cognitive literature, that learning was relatively stronger after spaced than after massed practice (although see Ammons 1988; Christina and Shea 1988; Newell et al. 1988 for criticisms of the review)…Probably the most widely cited example is Baddeley and Longman’s (1978) study concerning how optimally to teach postal workers to type. They had learners practice once a day or twice a day, and for session lengths of either 1 or 2 h at a time. The main findings were that learners took the fewest cumulative hours of practice to achieve a performance criterion in their typing when they were in the most distributed practice condition. This finding provides clear evidence for the benefits of spacing practice for enhancing learning. However, as has been pointed out (Newell et al. 1988; Lee and Wishart 2005), there is also trade-off to be considered in that the total elapsed time (number of days) between the beginning of practice and reaching criterion was substantially longer for the most spaced condition….The same basic results have been repeatedly demonstrated in the decades since (see reviews by Magill and Hall 1990; Lee and Simon 2004), and with a wide variety of motor tasks including different badminton serves (Goode and Magill 1986), rifle shooting (Boyce and Del Rey 1990), a pre-established skill, baseball batting (Hall et al. 1994), learning different logic gate configurations (Carlson et al. 1989; Carlson and Yaure 1990), for new users of automated teller machines (Jamieson and Rogers 2000), and for solving mathematical problems as might appear in a class homework (Rohrer and Taylor 2007; Le Blanc and Simon 2008; Taylor and Rohrer 2010).</p>\n<ul>\n<li>Culler, E. A. (1912). <a href=\"http://www.gwern.net/docs/spacedrepetition/1912-culler.pdf\"><q>The effect of distribution of practice upon learning</q></a>. <em>Journal of Philosophical Psychology</em>, 9, 580-583</li>\n<li>Pyle, W. H. (1915). <a href=\"http://books.google.com/books?hl=en&amp;lr=&amp;id=P8RMAAAAYAAJ&amp;oi=fnd&amp;pg=PA247&amp;ots=ngbgLvHiqb&amp;sig=Dyur1KbbI6Egs4z1lFqm6rYUIqw#v=onepage&amp;q&amp;f=false\"><q>Concentrated versus distributed practice</q></a>\n</li>\n<li>Lashley 1915, <a href=\"http://www.gwern.net/docs/spacedrepetition/1915-lashley.pdf\"><q>The acquisition of skill in archery</q></a>\n</li>\n<li>Murphy, H. H. (1916). <a href=\"http://www.gwern.net/docs/spacedrepetition/1916-murphy.pdf\"><q>Distributions of practice periods in learning</q></a>. Journal of Educational Psychology, 7, 150-162</li>\n<li>Adams, J. A. (1987). <a href=\"http://www.gwern.net/docs/spacedrepetition/1987-adams.pdf\"><q>Historical review and appraisal of research on the learning, retention, and transfer of human motor skills</q></a>\n</li>\n<li>Schmidt, R. A., &amp; Lee, T. D. (2005). <a href=\"http://www.amazon.com/Motor-Control-Learning-Behavioral-Emphasis/dp/0880114843/?tag=gwernnet-20\"><em>Motor control and learning: A behavioral emphasis</em></a> (4th ed.). Urbana-Champaign: Human Kinetics</li>\n<li>Lee, T. D., &amp; Genovese, E. D. (1988). <a href=\"http://www.gwern.net/docs/spacedrepetition/1988-lee.pdf\"><q>Distribution of practice in motor skill acquisition: Learning and performance effects reconsidered</q></a>. Research Quarterly for Exercise and Sport, 59, 277-287</li>\n<li>Ammons, R. B. (1988). <a href=\"http://www.gwern.net/docs/spacedrepetition/1988-ammons.pdf\"><q>Distribution of practice in motor skill acquisition: A few questions and comments</q></a>. Research Quarterly for Exercise and Sport, 59, 288-290</li>\n<li>Christina, R. W., &amp; Shea, J. B. (1988). <a href=\"http://www.gwern.net/docs/spacedrepetition/1988-christina.pdf\"><q>The limitations of generalization based on restricted information</q></a>. Research Quarterly for Exercise and Sport, 59, 291-297</li>\n<li>Newell, K. M., Antoniou, A., &amp; Carlton, L. G. (1988). <a href=\"http://www.gwern.net/docs/spacedrepetition/1988-newell.pdf\"><q>Massed and distributed practice effects: Phenomena in search of a theory?</q></a> Research Quarterly for Exercise and Sport, 59, 308-313</li>\n<li>Lee, T. D., &amp; Wishart, L. R. (2005). <a href=\"http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.574.5400&amp;rep=rep1&amp;type=pdf\"><q>Motor learning conundrums (and possible solutions)</q></a>\n</li>\n<li>Lee, T. D., &amp; Simon, D. A. (2004). <a href=\"http://www.gwern.net/docs/spacedrepetition/2004-lee.pdf\"><q>Contextual interference</q></a>\n</li>\n<li>Goode, S., &amp; Magill, R. A. (1986). <a href=\"http://www.gwern.net/docs/spacedrepetition/1986-goode.pdf\"><q>Contextual interference effects in learning three badminton serves</q></a>. Research Quarterly for Exercise and Sport, 57, 308-314</li>\n<li>Boyce, B. A., &amp; Del Rey, P. (1990). <q>Designing applied research in a naturalistic setting using a contextual interference paradigm</q>. Journal of Human Movement Studies, 18, 189-200</li>\n<li>Hall et al 1994, <a href=\"http://www.gwern.net/docs/spacedrepetition/1994-hall.pdf\"><q>Contextual interference effects with skilled baseball players</q></a>\n</li>\n<li>Carlson, R. A., &amp; Yaure, R. G. (1990). <a href=\"http://www.gwern.net/docs/spacedrepetition/1990-carlson.pdf\"><q>Practice schedules and the use of component skills in problem solving</q></a>\n</li>\n<li>Carlson, R. A., Sullivan, M. A., &amp; Schneider, W. (1989). <a href=\"http://www.gwern.net/docs/spacedrepetition/1989-carlson.pdf\"><q>Practice and working memory effects in building procedural skill</q></a>\n</li>\n<li>Jamieson, B. A., &amp; Rogers, W. A. (2000). <a href=\"http://www.gwern.net/docs/spacedrepetition/2000-jamieson.pdf\"><q>Age-related effects of blocked and random practice schedules on learning a new technology</q></a>\n</li>\n<li>Le Blanc, K. &amp; Simon, D. A. (2008). <q>Mixed practice enhances retention and JOL accuracy for mathematical skills</q>. Poster presented at the 2008 annual meeting of the Psychonomic Society, Chicago, IL</li>\n<li>Wymbs et al 2016, <a href=\"http://www.cell.com/cms/attachment/2045379445/2056784269/mmc2.pdf\"><q>Motor Skills Are Strengthened through Reconsolidation</q></a>\n</li>\n<li>Dayan &amp; Cohen 2011, <a href=\"http://www.sciencedirect.com/science/article/pii/S0896627311009184\"><q>Neuroplasticity subserving motor skill learning</q></a>\n</li>\n<li>Landin et al 1993, <a href=\"http://www.gwern.net/docs/spacedrepetition/1993-landin.pdf\"><q>The Effects of Variable Practice on the Performance of a Basketball Skill</q></a>\n</li>\n</ul>\n</blockquote>\n<p>In this vein, it’s interesting to note that interleaving may be helpful for tasks with a mental component as well: <a title=\"Practice Makes Perfect: The Critical Role of Mixed Practice in the Acquisition of ECG Interpretation Skills\" href=\"http://www.gwern.net/docs/spacedrepetition/2003-hatala.pdf\">Hatala et al 2003</a>, <a title=\"The effects of practice schedule and critical thinking prompts on learning and transfer of a complex judgment task\" href=\"https://pdfs.semanticscholar.org/7e0c/1bb80cbc332f07bda26a75a163a9cf76d591.pdf\">Helsdingen et al 2011</a>, and according to <a title=\"Mastering the Art of War: How Patterns of Gameplay Influence Skill in Halo\" href=\"http://jeffhuang.com/Final_HaloLearning_CHI13.pdf\">Huang et al 2013</a> the rates at which <em><a title=\"Wikipedia: Halo: Reach\" href=\"https://en.wikipedia.org/wiki/Halo%3A%20Reach\">Halo: Reach</a></em> players advance in skill matches nicely predictions from distribution: players who play 4-8 matches a week advance more in skill per match, than players who play more (distributed); but advance slower per week than players who play many more matches / massed. (See also <a title=\"Testing sleep consolidation in skill learning: a field study using an online game\" href=\"http://eprints.whiterose.ac.uk/97780/\">Stafford &amp; Haasnoot 2016</a>.)</p>\n</section>\n<section>\n<h5>Abstraction</h5>\n<p>Another potential objection is to argue<a href=\"#fn53\"><sup>53</sup></a> that spaced repetition inherently hinders any kind of abstract learning and thought because related materials are not being shown together - allowing for comparison and inference - but days or months apart. Ernst A. Rothkopf: <q>Spacing is the friend of recall, but the enemy of induction</q> (Kornell &amp; Bjork 2008, p. 585). This is plausible based on some of the early studies<a href=\"#fn54\"><sup>54</sup></a> but the 4 recent studies I knowof directly examining the issue both found spaced repetition helped abstraction as well as general recall:</p>\n<ol type=\"1\">\n<li>Kornell &amp; Bjork 2008a, <a href=\"http://www.gwern.net/docs/spacedrepetition/2008-kornell.pdf\"><q>Learning concepts and categories: Is spacing the <q>enemy of induction</q>?</q></a> <em>Psychological Science</em>, 19, 585-592</li>\n<li>Vlach, H. A., Sandhofer, C. M., &amp; Kornell, N. (2008). <a href=\"http://babytalk.psych.ucla.edu/documents/Vlach_Sandhofer_Kornell_2008.pdf\"><q>The spacing effect in children’s memory and category induction</q></a>. <em>Cognition</em>, 109, 163-167</li>\n<li>Kenney 2009. <a href=\"http://akenney.fastmail.fm/works/9.61Paper.pdf\"><q>The Spacing Effect in Inductive Learning</q></a>\n</li>\n<li>Kornell, N., Castel, A. D., Eich, T. S., &amp; Bjork, R. A. (2010). <a href=\"https://pdfs.semanticscholar.org/be1a/70be5a6f990cd86804ccc1be29331556ddfc.pdf\"><q>Spacing as the friend of both memory and induction in younger and older adults</q></a>. <em>Psychology and Aging</em>, 25, 498-503</li>\n<li><a title=\"Spacing and induction: Application to exemplars presented as auditory and visual text\" href=\"http://www.gwern.net/docs/spacedrepetition/2011-zulkiply.pdf\">Zulkiply et al 2011</a></li>\n<li>Vlach &amp; Sandhofer 2012, <a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3399982/\"><q>Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts</q></a>, <em>Child Development</em>\n</li>\n<li>\n<p>Zulkiply 2012, <a href=\"http://espace.library.uq.edu.au/view/UQ:281052\"><q>The spacing effect in inductive learning</q></a>; includes:</p>\n<ul>\n<li>replication of Kornell &amp; Bjork 2008</li>\n<li>Zulkiply et al 2011</li>\n<li>Zulkiply &amp; Burt 2012, <a href=\"http://www.gwern.net/docs/spacedrepetition/2012-zulkiply.pdf\"><q>The exemplar interleaving effect in inductive learning: Moderation by the difficulty of category discriminations</q></a>\n</li>\n<li>unknown paper currently in peer review</li>\n</ul>\n</li>\n<li>McDanie et al 2013, <a href=\"http://laplab.ucsd.edu/articles/McDaniel.Fadler.Pashler2013.pdf\"><q>Effects of Spaced versus Massed Training in Function Learning</q></a>\n</li>\n<li>Verkoeijen &amp; Bouwmeester 2014, <a href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3978334/\"><q>Is spacing really the <q>friend of induction</q>?</q></a>\n</li>\n<li>Rohrer et al 2014: <a title=\"The benefit of interleaved mathematics practice is not limited to superficially similar kinds of problems\" href=\"http://www.gwern.net/docs/spacedrepetition/2014-rohrer-1.pdf\">1</a>, <a title=\"Interleaved Practice Improves Mathematics Learning\" href=\"http://www.gwern.net/docs/spacedrepetition/2014-rohrer-2.pdf\">2</a>\n</li>\n<li>Vlach et al 2014, <a href=\"http://www.gwern.net/docs/spacedrepetition/2014-vlach.pdf\"><q>Equal spacing and expanding schedules in children’s categorization and generalization</q></a>\n</li>\n<li><p>Gluckman et al, <a href=\"http://babytalk.psych.ucla.edu/documents/Gluckman_Vlach_Sandhofer_2014.pdf\"><q>Spacing Simultaneously Promotes Multiple Forms of Learning in Children’s Science Curriculum</q></a></p></li>\n</ol>\n</section>\n</section>\n</section>\n<section>\n<h3>Review summary</h3>\n<p>To bring it all together with the gist:</p>\n<ul>\n<li>testing is effective and comes with minimal <a href=\"#downsides\">negative factors</a>\n</li>\n<li>expanding spacing is roughly as good as or better than (wide) fixed intervals, but expanding is more convenient and the default</li>\n<li>testing (and hence spacing) is best on intellectual, highly factual, verbal domains, but may still work in many low-level domains</li>\n<li>\n<p>the research favors questions which force the user to use their memory as much as possible; in descending order of preference:</p>\n<ol type=\"1\">\n<li>free recall</li>\n<li>short answers</li>\n<li>multiple-choice</li>\n<li>Cloze deletion</li>\n<li>recognition</li>\n</ol>\n</li>\n<li>the research literature is comprehensive and most questions have been answered - somewhere.</li>\n<li>\n<p>the most common mistakes with spaced repetition are</p>\n<ol type=\"1\">\n<li>formulating poor questions and answers</li>\n<li>assuming it will help you learn, as opposed to maintain and preserve what one already learned<a href=\"#fn55\"><sup>55</sup></a>. (It’s hard to learn <em>from</em> cards, but if you have learned something, it’s much easier to then devise a set of flashcards that will test your weak points.)</li>\n</ol>\n</li>\n</ul>\n</section>\n</section>\n<section>\n<h2>Using it</h2>\n<p>One doesn’t need to use SuperMemo, of course; there are plenty of free alternatives. I like <a title=\"Wikipedia: Mnemosyne (software)\" href=\"https://en.wikipedia.org/wiki/Mnemosyne%20%28software%29\">Mnemosyne</a> (<a href=\"http://www.mnemosyne-proj.org/\">homepage</a>) myself - <a title=\"Wikipedia: Free software\" href=\"https://en.wikipedia.org/wiki/Free%20software\">Free</a>, packaged for <a title=\"Wikipedia: Ubuntu Linux\" href=\"https://en.wikipedia.org/wiki/Ubuntu%20Linux\">Ubuntu Linux</a>, easy to use, free mobile client, long track record of development and reliability (I’ve used it since ~2008). But the SRS <a title=\"Wikipedia: Anki (software)\" href=\"https://en.wikipedia.org/wiki/Anki%20%28software%29\">Anki</a> is also popular, and has advantages in being more feature-rich and a larger &amp; more active community (and possibly better support for East Asian language material and a better but proprietary mobile client).</p>\n<p>OK, but what does one do with it? It’s a surprisingly difficult question, actually. It’s akin to <q>the tyranny of the blank page</q> (or blank wiki); now that I have all this power - a mechanical golem that will never forget and never let me forget whatever I chose to - what do I choose to remember?</p>\n<section>\n<h3>How Much To Add</h3>\n<p>The most difficult task, beyond that of just persisting until the benefits become clear, is deciding what’s valuable enough to add in. In a 3 year period, one can expect to spend <q>30-40 seconds</q><a href=\"#fn56\"><sup>56</sup></a> on any given item. The long run <a href=\"http://www.supermemo.com/articles/theory.htm\">theoretical predictions</a> are a little hairier. Given a single item, the formula for daily time spent on it is <span></span><span data-mathml='&lt;math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"inline\"&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mtext mathvariant=\"normal\"&gt;time&lt;/mtext&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mn&gt;500&lt;/mn&gt;&lt;/mfrac&gt;&lt;mo&gt;&amp;#xD7;&lt;/mo&gt;&lt;msup&gt;&lt;mtext mathvariant=\"normal\"&gt;nthYear&lt;/mtext&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#x2212;&lt;/mo&gt;&lt;mn&gt;1.5&lt;/mn&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mn&gt;30000&lt;/mn&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;annotation encoding=\"application/x-tex\"&gt;      text{time} = frac{1}{500} times text{nthYear}^{-1.5} + frac{1}{30000}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;' tabindex=\"0\"><span><span><span><span><span><span><span><span>time</span><span>=</span><span><span><span><span>1</span><span></span></span><span><span>500</span><span></span></span><span><span></span><span></span></span></span></span><span>×</span><span><span><span><span>nthYear</span><span></span></span><span><span><span>−</span><span>1.5</span></span><span></span></span></span></span><span>+</span><span><span><span><span>1</span><span></span></span><span><span>30000</span><span></span></span><span><span></span><span></span></span></span></span></span></span></span><span></span></span></span><span></span></span></span><span></span></span>. During our 20th year, we would spend <span></span><span data-mathml='&lt;math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"inline\"&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mn&gt;500&lt;/mn&gt;&lt;/mfrac&gt;&lt;mo&gt;&amp;#xD7;&lt;/mo&gt;&lt;msup&gt;&lt;mn&gt;20&lt;/mn&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#x2212;&lt;/mo&gt;&lt;mn&gt;1.5&lt;/mn&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mn&gt;3000&lt;/mn&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;annotation encoding=\"application/x-tex\"&gt;      t = frac{1}{500} times 20^{-1.5} + frac{1}{3000}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;' tabindex=\"0\"><span><span><span><span><span><span><span><span>t<span></span></span><span>=</span><span><span><span><span>1</span><span></span></span><span><span>500</span><span></span></span><span><span></span><span></span></span></span></span><span>×</span><span><span><span><span>20</span><span></span></span><span><span><span>−</span><span>1.5</span></span><span></span></span></span></span><span>+</span><span><span><span><span>1</span><span></span></span><span><span>3000</span><span></span></span><span><span></span><span></span></span></span></span></span></span></span><span></span></span></span><span></span></span></span><span></span></span>, or <code>3.556940131083312e-4</code> minutes a day. This is the average daily time, so to recover the annual time spent, we simply multiply by 365. Suppose we were interested in how much time a flashcard would cost us over 20 years. The average daily time changes every year (the graph looks like an exponential decay, remember), so we have to run the formula for each year and sum them all; in Haskell:</p>\n<div><pre><code>sum <span>$</span> map (\\year <span>-&gt;</span> ((<span>1</span><span>/</span><span>500</span> <span>*</span> year<span>**</span>(<span>-</span>(<span>1.5</span>))) <span>+</span> <span>1</span><span>/</span><span>30000</span>) <span>*</span> <span>365.25</span>) [<span>1</span><span>..</span><span>20</span>]\n<span># 1.8291</span></code></pre></div>\n<p>Which evaluates to 1.8 minutes. (This may seem too small, but one doesn’t spend much time in the first year and the time drops off quickly<a href=\"#fn57\"><sup>57</sup></a>.) <a title=\"Dreaming of a World Undone\" href=\"http://www.gwern.net/docs/spacedrepetition/2012-muflax-dreamingofaworldundone.html.maff\">Anki user muflax</a>’s statistics put his per-card time at 71s, for example. But maybe <a title=\"Wikipedia: Piotr Wozniak (researcher)\" href=\"https://en.wikipedia.org/wiki/Piotr%20Wozniak%20%28researcher%29\">Piotr Woźniak</a> was being optimistic or we’re bad at <a href=\"http://www.supermemo.com/articles/20rules.htm\">writing flashcards</a>, so we’ll double it to 5 minutes. That’s our key rule of thumb that lets us decide what to learn and what to forget: if, over your lifetime, you will spend more than 5 minutes looking something up or will lose more than 5 minutes as a result of not knowing something, then it’s worthwhile to memorize it with spaced repetition. 5 minutes is the line that divides trivia from useful data.<a href=\"#fn58\"><sup>58</sup></a> (There might seem to be thousands of flashcards that meet the 5 minute rule. That’s fine. Spaced repetition can accommodate dozens of thousands of cards. See the <a href=\"#the-workload\">next section</a>.)</p>\n<p>To a lesser extent, one might wonder when one is in a hurry, should one learn something with spaced repetition and with massed? How far away should the tests or deadlines be before abandoning spaced repetition? It’s hard to compare since one would need a specific regimens to compare for the crossover point, but for massed repetition, the average time after memorization at which one has a 50% chance of remembering the memorized item seems to be 3-5 days.<a href=\"#fn59\"><sup>59</sup></a> Since there would be 2 or 3 repetitions in that period, presumably one would do better than 50% in recalling an item. 5 minutes and 5 days seems like a memorable enough rule of thumb: <q>don’t use spaced repetition if you need it sooner than 5 days or it’s worth less than 5 minutes</q>.</p>\n<section>\n<h4>Overload</h4>\n<p>One common experience of new users to spaced repetition is to add too much stuff - trivialities and things they don’t really care about. But they soon learn the curse of <a title=\"Wikipedia: Jorge Luis Borges\" href=\"https://en.wikipedia.org/wiki/Jorge%20Luis%20Borges\">Borges’s</a> <a title=\"Wikipedia: Funes the Memorious\" href=\"https://en.wikipedia.org/wiki/Funes%20the%20Memorious\">Funes the Memorious</a>. If they don’t actually want to learn the material they put in, they will soon stop doing the daily reviews - which will cause reviews to pile up, which will be further discouraging, and so they stop. At least with physical fitness there isn’t a precisely dismaying number indicating how far behind you are! But if you have too little at the beginning, you’ll have few repetitions per day, and you’ll see little benefit from the technique itself - it looks like boring flash card review.</p>\n</section>\n</section>\n<section>\n<h3>What to add</h3>\n<p>I find one of the best uses for Mnemosyne is, besides the classic use of memorizing academic material such as geography or the periodic table or foreign vocabulary or Bible/Koran verses or the avalanche of medical school facts, to add in words from <a title=\"Wikipedia: A Word A Day\" href=\"https://en.wikipedia.org/wiki/A%20Word%20A%20Day\">A Word A Day</a><a href=\"#fn60\"><sup>60</sup></a> and <a title=\"Wikipedia: Wiktionary\" href=\"https://en.wikipedia.org/wiki/Wiktionary\">Wiktionary</a>, memorable quotes I see<a href=\"#fn61\"><sup>61</sup></a>, personal information such as birthdays (or license plates, a problem for me before), and so on. Quotidian uses, but all valuable to me. With a diversity of flashcards, I find my daily review interesting. I get all sorts of questions - now I’m trying to see whether a Haskell fragment is syntactically correct, now I’m pronouncing Korean <a title=\"Wikipedia: hangul\" href=\"https://en.wikipedia.org/wiki/hangul\">hangul</a> and listening to the answer, now I’m trying to find the Ukraine on a map, now I’m enjoying some <a title=\"Wikipedia: A.E. Housman\" href=\"https://en.wikipedia.org/wiki/A.E.%20Housman\">A.E. Housman</a> poetry, followed by a few quotes from <a href=\"http://www.lesswrong.com\">LessWrong</a> <a href=\"http://lesswrong.com/tag/quotes/\">quote threads</a>, and so on. Other people use it for many other things; one application that impresses me for its simple utility is <a href=\"http://lesswrong.com/lw/e26/who_wants_to_start_an_important_startup/78ig\">memorizing names &amp; faces</a> <a href=\"http://lesswrong.com/lw/e26/who_wants_to_start_an_important_startup/7dub?context=3\">of</a> <a href=\"http://lesswrong.com/lw/e26/who_wants_to_start_an_important_startup/8fgo?context=3\">students</a> although <a title=\"Perfect Pitch (Absolute Pitch) training with Mnemosyne (spaced repetition software)\" href=\"http://www.youtube.com/watch?v=y3F7cTEL4b8\">learning musical notes</a> is also not bad.</p>\n</section>\n<section>\n<h3>The workload</h3>\n<p>On average, when I’m studying a new topic, I’ll add 3-20 questions a day. Combined with my particular memory, I usually review about 90 or 100 items a day (out of the total &gt;18,300). This takes under 20 minutes, which is not too bad. (I expect the time is expanded a bit by the fact that early on, my formatting guidelines were still being developed, and I hadn’t the full panoply of categories I do now - so every so often I must stop and edit categories.)</p>\n<p>If I haven’t been studying something recently, the exponential decaying of reviews slowly drops the daily review. For example, in March 2011, I wasn’t studying many things, so for 24-26 March 2011, my scheduled daily reviews are 73, 83, and 74; after that, it’ll probably drop down into the 60s, and then after another week or two, into the 50s and so on until it hits the minimum plateau which will slowly shrink over years. (I haven’t gone long enough without dumping cards in to know what that might be.) By February 2012, the daily reviews are in the 40s or sometimes 50s for similar reasons, but the gradual shrinkage will continue. We can see this vividly, and we can even see a sort of analogue of the original forgetting curve, if we ask Mnemosyne 2.0 to graph the number of cards to review per day for the next year up to February 2013 (assuming no additions or missed reviews etc.):</p>\n<figure>\n<img alt=\"A wildly varying but clearly decreasing graph of predicted cards per day\" width=\"787\" src=\"http://www.gwern.net/images/spacedrepetition/scheduled-cards.png\" height=\"392\"><figcaption>A wildly varying but clearly decreasing graph of predicted cards per day</figcaption>\n</figure>\n<p>If Mnemosyne weren’t using spaced repetition, it would be hard to keep up with 18,300+ flashcards. But because it is using spaced repetition, keeping up is easy.</p>\n<p>Nor is 18.3k extraordinary. Many users have decks in the 6-7k range, Mnemosyne developer <a href=\"http://groups.google.com/group/mnemosyne-proj-users/browse_frm/thread/433872b155ad7451/31c1e4c556680a0c\">Peter Bienstman</a> has &gt;8.5k &amp; Patrick Kenny &gt;27k, <a href=\"http://groups.google.com/group/mnemosyne-proj-users/browse_frm/thread/eff44f5fdb1d738b/7a7b654ca87e63be\">Hugh Chen</a> has a 73k+ deck, and in <a href=\"irc://irc.freenode.net#anki\">#anki</a>, they tell me of one user who triggered bugs with his &gt;200k deck. 200,000 may be a bit much, but for regular humans, some amount smaller seems possible - it’s interesting to compare SRS decks to the feat of <a title=\"'Memorising Milton's Paradise Lost: A study of a septuagenarian exceptional memoriser', Seamon et al 2010\" href=\"http://www.gwern.net/docs/spacedrepetition/2010-seamon.pdf\">memorizing <em>Paradise Lost</em></a> or to the Muslim title of <a title=\"Wikipedia: Hafiz (Qur'an)\" href=\"https://en.wikipedia.org/wiki/Hafiz%20%28Qur%27an%29\"><q>hafiz</q></a>, one who has memorized the ~80,000 words of the Koran, or the stricter <q>hafid</q>, one who had memorized the Koran <em>and</em> 100,000 <a title=\"Wikipedia: hadiths\" href=\"https://en.wikipedia.org/wiki/hadiths\">hadiths</a> as well. Other forms of memory are still more powerful.<a href=\"#fn62\"><sup>62</sup></a> (I suspect that spaced repetition is involved in one of the few well-documented cases of <q><a title=\"Wikipedia: hyperthymesia\" href=\"https://en.wikipedia.org/wiki/hyperthymesia\">hyperthymesia</a></q>, <a title=\"Wikipedia: Jill Price\" href=\"https://en.wikipedia.org/wiki/Jill%20Price\">Jill Price</a>: reading <a title=\"Total Recall: The Woman Who Can't Forget\" href=\"https://web.archive.org/web/20131208072327/http://www.wired.com/medtech/health/magazine/17-04/ff_perfectmemory?currentPage=all\"><em>Wired</em></a>, she has ordinary fallible powers of memorization for surprise demands with no observed anatomical differences and is restricted to <q>her own personal history and certain categories like television and airplane crashes</q>; further, she is a packrat with obsessive-compulsive traits who keeps &gt;50,000 pages of detailed diaries perhaps due to a childhood trauma &amp; associates daily events nigh-involuntarily with past events. Marcus says the other instances of hyperthymesia resemble Price.)</p>\n</section>\n<section>\n<h3>When to review</h3>\n<p>When should one review? In the morning? In the evening? Any old time? The studies demonstrating the spacing effect do not control or vary the time of day, so in one sense, the answer is: it doesn’t matter - if it did matter, there would be considerable variance in how effective the effect is based on when a particular study had its subjects do their reviews.</p>\n<p>So one reviews at whatever time is convenient. Convenience makes one more likely to stick with it, and sticking with it overpowers any temporary improvement.</p>\n<p>If one is not satisfied with that answer, then on general considerations, one ought to review before bedtime &amp; sleep. <a title=\"Wikipedia: Memory consolidation#Spacing effect\" href=\"https://en.wikipedia.org/wiki/Memory%20consolidation%23Spacing%20effect\">Memory consolidation</a> seems to be related, and <a title=\"Wikipedia: Sleep and memory\" href=\"https://en.wikipedia.org/wiki/Sleep%20and%20memory\">sleep</a> is known to powerfully influence what memories enter long-term memory, <a title=\"Memory for Semantically Related and Unrelated Declarative Information: The Benefit of Sleep, the Cost of Wake\" href=\"http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0033079\">strengthening memories</a> of material learned close to bedtime and <a title=\"'REM, not incubation, improves creativity by priming associative networks', Cai et al 2009\" href=\"http://www.pnas.org/content/106/25/10130.full\">increasing creativity</a>; interrupting sleep without affecting total sleep time or quality still <a href=\"http://www.pnas.org/content/early/2011/07/20/1015633108\">damages memory formation in mice</a><a href=\"#fn63\"><sup>63</sup></a>. So reviewing before bedtime would be best. (Other mental exercises show improvement when trained before bedtime; for example, <a href=\"http://www.gwern.net/DNB-FAQ#sleep\">dual n-back</a>.) One possible mechanism is that it may be that the <a title=\"Sleep Selectively Enhances Memory Expected to Be of Future Relevance\" href=\"http://www.jneurosci.org/content/31/5/1563.full\"><em>expectancy</em></a> of future reviews/tests is enough to encourage memory consolidation during sleep; so if one reviews and goes to bed, presumably the expectancy is stronger than if one reviewed at breakfast and had an eventful day and forgot entirely about the reviewed flashcards. (See also the correlation between time of studying &amp; GPA in Hartwig &amp; Dunlosky 2012.) Neural growth may be related; from Stahl 2010:</p>\n<blockquote>\n<p>Recent advances in our understanding of the neurobiology underlying normal human memory formation have revealed that learning is not an event, but rather a process that unfolds over time.<sup><a title=\"'Neurogenesis and the spacing effect: Learning over time enhances memory and the survival of new neurons', Sisti et al 2007\" href=\"http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1876761/\">16</a>,<a title=\"'Reconsolidation: maintaining memory relevance', Lee 2009\" href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3650827/\">17</a>,<a title=\"'Play it again: reactivation of waking experience and memory', O'Neill et al 2010\" href=\"http://www.gwern.net/docs/spacedrepetition/2010-oneill.pdf\">18</a>,[Squire 2003 <em><a href=\"http://www.amazon.com/Fundamental-Neuroscience-Second-Larry-Squire/dp/0126603030/?tag=gwernnet-20\">Fundamental Neuroscience</a></em>],<a title=\"'Making Memories Stick', Fields 2005, Scientific American\" href=\"http://davidjf.free.fr/new/Xtech_scientific%20american_%20making%20memories%20stick.pdf\">20</a></sup> Thus, it is not surprising that learning strategies that repeat materials over time enhance their retention.<sup>20,<a title=\"'Spacing repetitions over 1 week', Glenberg &amp; Lehmann 1980\" href=\"http://www.gwern.net/docs/spacedrepetition/1980-glenberg.pdf\">21</a>,<a title=\"'The effect of spacing repetitions on the recognition memory of young children and adults', Toppino et al 1991\" href=\"http://www.gwern.net/docs/spacedrepetition/1991-toppino.pdf\">22</a>,<a title=\"'Optimum rehearsal patterns and name learning', Landauer &amp; Bjork 1978\" href=\"http://www.gwern.net/docs/spacedrepetition/1978-landauer.pdf\">23</a>,<a title=\"'The Critical Importance of Retrieval for Learning', Karpicke &amp; Roediger 2008\" href=\"http://www.wsu.edu/~fournier/Teaching/psych592/Readings/Karpicke_et_al_2008.pdf\">24</a>,<a title=\"'Optimizing retrieval as a learning event: When and why expanding retrieval practice enhances long-term retention', Storm et al 2010\" href=\"https://pdfs.semanticscholar.org/6698/bf91c9333faa0d333a800254b8063230d4f4.pdf\">25</a>,<a title=\"'Enhancing learning and retarding forgetting: Choices and consequences', Pashler et al 2007\" href=\"http://www.pashler.com/Articles/Pashler.Rohrer.Cepeda.Carpenter_2007.pdf\">26</a></sup></p>\n<p>…Thousands of new cells are generated in this region every day, although many of these cells die within weeks of their creation.<sup><a title=\"'Adult neurogenesis produces a large pool of new granule cells in the dentate gyrus', Cameron &amp; McKay 2001\" href=\"https://pdfs.semanticscholar.org/45c2/7c08fbb43f8728e69a7447366d4a4f74e088.pdf\">31</a></sup> The survival of dentate gyrus neurons has been shown to be enhanced in animals when they are placed into learning situations.<sup>16-20</sup> Animals that learn well retain more dentate gyrus neurons than do animals that do not learn well. Furthermore, 2 weeks after testing, animals trained in discrete spaced intervals over a period of time, rather than in a single presentation or a <q>massed trial</q> of the same information, remember better.<sup>16-20</sup> The precise mechanism that links neuronal survival with learning has not yet been identified. One theory is that the hippocampal neurons that preferentially survive are the ones that are somehow activated during the learning process.<sup>16-20</sup><a href=\"#fn64\"><sup>64</sup></a> The distribution of learning over a period of time may be more effective in encouraging neuronal survival by allowing more time for changes in gene expression and protein synthesis that extend the life of neurons that are engaged in the learning process.</p>\n<p>…Transferring memory from the encoding stage, which occurs during alert wakefulness, into consolidation must thus occur at a time when interference from ongoing new memory formation is reduced.<sup>17,18</sup> One such time for this transfer is during sleep, especially during non-rapid eye movement sleep, when the hippocampus can communicate with other brain areas without interference from new experiences.<sup><a title=\"'Sleep, Learning, and Dreams: Off-line Memory Reprocessing', Stickgold et al 2001\" href=\"https://pdfs.semanticscholar.org/d0a7/f06ed267f3193daab1175a65abb7a067bef4.pdf\">32</a>,<a title=\"'The contribution of sleep to hippocampus-dependent memory consolidation', Marshall &amp; Born 2007\" href=\"https://pdfs.semanticscholar.org/77fd/0cf03de8a6c4f56c5decc7c47ebe69cf98c1.pdf\">33</a>,<a title=\"'The Role of Sleep in Learning and Memory', Maquet et al 2001\" href=\"http://www.gwern.net/docs/spacedrepetition/2001-maquet.pdf\">34</a></sup> Maybe that is why some decisions are better made after a good night’s rest and also why pulling an all-nighter, studying with sleep deprivation, may allow you to pass an exam an hour later but not remember the material a day later.</p>\n</blockquote>\n<section>\n<h4>Prospects: extended flashcards</h4>\n<p>Let’s step back for a moment. What are all our flashcards, small and large, doing for us? Why do I have a pair of flashcards for the word <q>anent</q> among many others? I can just look it up.</p>\n<p>But look ups take time compared to already knowing something. (Let’s ignore the previously discussed 5 minute rule.) If we think about this abstractly in a computer science context, we might recognize it as an old concept in algorithms &amp; optimization discussions - the <a title=\"Wikipedia: space-time tradeoff\" href=\"https://en.wikipedia.org/wiki/space-time%20tradeoff\">space-time tradeoff</a>. We trade off lookup time against limited skull space.</p>\n<p>Consider the sort of factual data already given as examples - we might one day need to know the average annual rainfall in Honolulu or Austin, but it would require too much space to memorize such data for all capitals. There are millions of English words, but in practice any more than 100,000 is excessive. More surprising is a sort of procedural knowledge. An extreme form of space-time tradeoffs in computers is when a computation is replaced by pre-calculated constants. We could take a math <a title=\"Wikipedia: Function (mathematics)\" href=\"https://en.wikipedia.org/wiki/Function%20%28mathematics%29\">function</a> and calculate its output for each possible input. Usually such a <a title=\"Wikipedia: lookup table\" href=\"https://en.wikipedia.org/wiki/lookup%20table\">lookup table</a> of input to output is really large. Think about how many entries would be in such a table for all possible integer multiplications between 1 and 1 billion. But sometimes the table is really small (like binary Boolean functions) or small (like trigonometric tables) or large but still useful (<a title=\"Wikipedia: rainbow table\" href=\"https://en.wikipedia.org/wiki/rainbow%20table\">rainbow table</a>s usually start in the gigabytes and easily reach terabytes).</p>\n<p>Given an infinitely large lookup table, we could replace <em>completely</em> the skill of, say, addition or multiplication by the lookup table. No computation. The space-time tradeoff taken to the extreme of the space side of the continuum. (We could go the other way and define multiplication or addition as the slow computation which doesn’t know any specifics like the <a title=\"Wikipedia: multiplication table\" href=\"https://en.wikipedia.org/wiki/multiplication%20table\">multiplication table</a> - as if every time you wanted to add <span></span><span data-mathml='&lt;math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"inline\"&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=\"application/x-tex\"&gt;      2+2&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;' tabindex=\"0\"><span><span><span><span><span><span><span><span>2</span><span>+</span><span>2</span></span></span></span><span></span></span></span><span></span></span></span><span></span></span> you had to count on 4 fingers.)</p>\n<p>So suppose we were children who wanted to learn multiplication. SRS and Mnemosyne can’t help because multiplication is not a specific factoid? The space-time tradeoff shows us that we can de-proceduralize multiplication and turn it partly into factoids. It wouldn’t be hard for us to write a quick script or macro to generate, say, 500 random cards which ask us to multiply AB by XY, and import them to Mnemosyne.<a href=\"#fn65\"><sup>65</sup></a></p>\n<p>After all, which is your mind going to do - get good at multiplying 2 numbers (generate on-demand), or memorize 500 different multiplication problems (<a title=\"Wikipedia: memoize\" href=\"https://en.wikipedia.org/wiki/memoize\">memoize</a>)? From my experience with multiple subtle variants on a card, the mind gives up after just a few and falls back on a problem-solving approach - which is exactly what one wants to exercise, in this case. Congratulations; you have done the impossible.</p>\n<p>From a software engineering point of view, we might want to modify or improve the cards, and 500 snippets of text would be a tad hard to update. So coolest would be a <q>dynamic card</q>. Add a markup type like <code>&lt;eval src=\"\"&gt;</code> , and then Mnemosyne feeds the <code>src</code> argument straight into the Python interpreter, which returns a <a title=\"Wikipedia: tuple\" href=\"https://en.wikipedia.org/wiki/tuple\">tuple</a> of the question text and the answer text. The question text is displayed to the user as usual, the user thinks, requests the answer, and grades himself. In Anki, Javascript is supported directly by the application in HTML <code>&lt;script&gt;</code> tags (currently <a href=\"https://apps.ankiweb.net/docs/manual.html#_javascript\">inline only</a> but Anki could presumably import libraries by default), for example for kinds of <a title=\"Javascript in Anki cards\" href=\"http://ojisanseiuchi.com/2016/03/12/JavaScript-in-Anki-cards/\">syntax highlighting</a>, so any kind of dynamic card could be written that one wants.</p>\n<p>So for multiplication, the dynamic card would get 2 random integers, print a question like <code>x * y = ?</code> and then print the result as the answer. Every so often you would get a new multiplication question, and as you get better at multiplication, you see it less often - exactly as you should. Still in a <a href=\"http://www.reddit.com/r/math/comments/hvqzd/printable_math_flashcards_in_pdf_and_latex_source/c1yror5\">math vein</a>, you could generate variants on formulas or programs where one version is the correct one and the others are subtly wrong; I do this by hand with my programming flashcards (especially if I make an error doing exercises, that signals a finer point to make several flashcards on), but it can be done automatically. <a href=\"http://lesswrong.com/lw/64k/memory_spaced_repetition_and_life/4e5n\">kpreid</a> <a href=\"http://lesswrong.com/lw/64k/memory_spaced_repetition_and_life/4e5n\">describes</a> one tool of his:</p>\n<blockquote>\n<p>I have written <a href=\"https://github.com/kpreid/mathquiz/\">a program</a> (in the form of <a href=\"http://kpreid.github.com/mathquiz/mathquiz.html\">a web page</a>) which does a specialized form of this [generating <q>damaged formulas</q>]. It has a set of generators of formulas and damaged formulas, and presents you with a list containing several formulas of the same type (e.g. ∫ 2x dx = x^2 + C) but with one damaged (e.g. ∫ 2x dx = 2x^2 + C).</p>\n</blockquote>\n<p>This approach generalizes to anything you can generate random problems of or have large databases of examples of. Khan Academy apparently does something like this in associating large numbers of (algorithmicly-generated?) problems with each of its little modules and tracking retention of the skill in order to decide when to do further review of that module. For example, maybe you are studying Go and are interested in learning <a title=\"Wikipedia: Life and death\" href=\"https://en.wikipedia.org/wiki/Life%20and%20death\">life-and-death positions</a>. Those are things that can be generated by computer Go programs, or fetched from places like <a href=\"http://www.goproblems.com\">GoProblems.com</a>. For even more examples, Go is rotationally invariant - the best move remains the same regardless of which way the board is oriented and since there is no canonical direction for the board (like in chess) a good player ought to be able to play the same no matter how the board looks - so each specific example can be mirrored in 3 other ways. Or one could test one’s ability to <q>read</q> a board by writing a dynamic card which takes each example board/problem and adds some random pieces as long as some go-playing program like <a title=\"Wikipedia: GNU Go\" href=\"https://en.wikipedia.org/wiki/GNU%20Go\">GNU Go</a> says the best move hasn’t changed because of the added noise.</p>\n<p>One could learn an awful lot of things this way. Programming languages could be learned this way - someone learning <a title=\"Wikipedia: Haskell (programming language)\" href=\"https://en.wikipedia.org/wiki/Haskell%20%28programming%20language%29\">Haskell</a> could take all the functions listed in the Prelude or his Haskell textbook, and ask <a title=\"Wikipedia: QuickCheck\" href=\"https://en.wikipedia.org/wiki/QuickCheck\">QuickCheck</a> to generate random arguments for the functions and ask the <a title=\"Wikipedia: Glasgow Haskell Compiler\" href=\"https://en.wikipedia.org/wiki/Glasgow%20Haskell%20Compiler\">GHC</a> interpreter <code>ghci</code> what the function and its arguments evaluate to. Games other than go, like chess, may work (a live example being <a href=\"http://chesstempo.com/user-guide/en/probSearchSpacedRepetition.html\">Chess Tempo</a>, and see the experience of <a title=\"I'll mostly discuss my experience using it for chess, since that's what the majority of my 8000 cards are...\" href=\"http://dfan.org/blog/2013/07/07/mnemosyne-part-3/\">Dan Schmidt</a>). A fair bit of mathematics. If the dynamic card has Internet access, it can pull down fresh questions from an <a title=\"Wikipedia: RSS feed\" href=\"https://en.wikipedia.org/wiki/RSS%20feed\">RSS feed</a> or just a website; this functionality could be quite useful in a foreign language learning context with every day bringing a fresh sentence to translate or another exercise.</p>\n<p>With some NLP software, one could write dynamic flashcards which test all sorts of things: if one confuses verbs, the program could take a template like <q>$PRONOUN $VERB $PARTICLE $OBJECT % {right: caresse, wrong: caresses}</q> which yields flashcards like <q>Je caresses le chat</q> or <q>Tu caresse le chat</q> and one would have to decide whether it was the correct conjugation. (The dynamicism here would help prevent memorizing specific sentences rather than the underlying conjugation.) In full generality, this would probably be difficult, but simpler approaches like templates may work well enough. Jack Kinsella:</p>\n<blockquote>\n<p>I wish there were dynamic SRS decks for language learning (or other disciplines). Such decks would count the number of times you have reviewed an instance of an underlying grammatical rule or an instance of a particular piece of vocabulary, for example its singular/plural/third person conjugation/dative form. These sophisticated decks would present users with fresh example sentences on every review, thereby preventing users from remembering specific answers and compelling them to learn the process of applying the grammatical rule afresh. Moreover, these decks would keep users entertained through novelty and would present users with tacit learning opportunities through rotating vocabulary used in non-essential parts of the example sentence. Such a system, with multiple-level review rotation, would not only prevent against overfit learning, but also increase the total amount of knowledge learned per minute, an efficiency I’d gladly invest in.</p>\n</blockquote>\n<p>Even though these things seem like <q>skills</q> and not <q>data</q>!</p>\n</section>\n</section>\n</section>\n</section>\n<section>\n<p>SuperMemo doesn’t fall under the same ratings, but it has sold in the hundreds of thousands over its 2 decades:</p>\n<blockquote>\n<p>Biedalak is CEO of SuperMemo World, which sells and licenses Wozniak’s invention. Today, SuperMemo World employs just 25 people. The venture capital never came through, and the company never moved to California. About 50,000 copies of SuperMemo were sold in 2006, most for less than $30. Many more are thought to have been pirated.<a href=\"#fn69\"><sup>69</sup></a></p>\n</blockquote>\n<p>It seems safe to estimate the combined market-share of Anki, Mnemosyne, iSRS and other SRS apps at somewhere under 50,000 users (making due allowance for users who install multiple times, those who install and abandon it, etc.). Relatively few users seem to have migrated from SuperMemo to those newer programs, so it seems fair to simply add that 50k to the other 50k and conclude that the worldwide population is somewhere around (but probably under) 100,000.</p>\n</section>\n<section>\n<p>Nowhere, really. Mnemosyne/SR software in general are just one of my favorite tools: it’s based on a famous effect<a href=\"#fn70\"><sup>70</sup></a> discovered by science, and it exploits it elegantly<a href=\"#fn71\"><sup>71</sup></a> and usefully. It’s a testament to the Enlightenment ideal of improving humanity through reason and overcoming our human flaws; the idea of SR is seductive in its mathematical rigor<a href=\"#fn72\"><sup>72</sup></a>. In this age where so often the ideal of <q>self-improvement</q> and progress are decried, and gloom are espoused by even the common people, it’s really nice to just have a small example like this in one’s daily life, an example not yet so prosaic and boring as the lightbulb.</p>\n</section>\n<section>\n<p>In the course of using Mnemosyne, I’ve written a number of scripts to generate repetitively varying cards.</p>\n<ul>\n<li>\n<a href=\"http://www.gwern.net/haskell/mnemo.hs\"><code>mnemo.hs</code></a> will take any newline-delimited chunk of text, like a poem, and generates every possible <a title=\"Wikipedia: Cloze deletion\" href=\"https://en.wikipedia.org/wiki/Cloze%20deletion\">Cloze deletion</a>; that is, an ABC poem will become 3 questions: _BC/ABC, A_C/ABC, AB_/ABC</li>\n<li>\n<a href=\"http://www.gwern.net/haskell/mnemo2.hs\"><code>mnemo2.hs</code></a> works as above, but is more limited and is intended for long chunks of text where <code>mnemo.hs</code> would cause a combinatorial explosion of generated questions; it generates a subset: for ABCD, one gets __CD/ABCD, A__D/ABCD, and AB__/ABCD (it removes 2 lines, and iterates through the list).</li>\n<li>\n<a href=\"http://www.gwern.net/haskell/mnemo3.hs\"><code>mnemo3.hs</code></a> is intended for date or name-based questions. It’ll take input like <q>Barack Obama is %47%.</q> and spit out some questions based on this: <q>Barack Obama is _7./47</q>, <q>Barack Obama is 4_./47</q> etc.</li>\n<li>\n<p><a href=\"http://www.gwern.net/haskell/mnemo4.hs\"><code>mnemo4.hs</code></a> is intended for long lists of items. If one wants to memorize the list of US Presidents, the natural questions for flashcards goes something like <q>Who was the 3rd president?/Thomas Jefferson</q>, <q>Thomas Jefferson was the _rd president./3</q>, <q>Who was president after John Adams?/Thomas Jefferson</q>, <q>Who was president before James Madison?/Thomas Jefferson</q>.</p>\n<p>You note there’s repetition if you do this for each president - one asks the ordinal position of the item both ways (item -&gt; position, position -&gt; item), what precedes it, and what succeeds it. <code>mnemo4.hs</code> automates this, given a list. In order to be general, the wording is a bit odd, but it’s better than writing it all out by hand! (Example output is in the <a title=\"Wikipedia: Comment (computer programming)\" href=\"https://en.wikipedia.org/wiki/Comment%20%28computer%20programming%29\">comments</a> to the source code).</p>\n</li>\n</ul>\n<p>The reader might well be curious by this point what <em>my</em> Mnemosyne database looks like. I use Mnemosyne quite a bit, and as of 25 December 2017, I have 16,109 (active) cards in my deck. Said curious reader may find my cards &amp; media at <a href=\"https://www.dropbox.com/s/saltdnrcdzlq490/2017-12-25-gwern-mnemosyne-export.cards?dl=0\"><code>gwern.cards</code></a> (54M; Mnemosyne 2.x format).</p>\n<p>The Mnemosyne project has been collecting user-submitted spaced repetition statistical data for years. The full dataset as of 27 January 2014 <a title=\"Mnemosyne data set available\" href=\"https://archive.org/details/20140127MnemosynelogsAll.db\">is available for download</a> by anyone who wishes to analyze it.</p>\n</section>\n<section>\n<ul>\n<li>\n<p><a href=\"http://bentilly.blogspot.com/2009/09/teaching-linear-algebra.html\"><q>Teaching linear algebra</q></a> (with spaced repetition), by Ben Tilly</p>\n<ul>\n<li><a href=\"http://bentilly.blogspot.com/2012/10/my-sons-flashcard-routine.html\">Manual flashcards for his 2<sup>nd</sup> grader</a></li>\n</ul>\n</li>\n<li>\n<a href=\"http://lesswrong.com/r/discussion/lw/mfm/a_year_of_spaced_repetition_software_in_the/\"><q>A Year of Spaced Repetition Software in the Classroom</q></a>; <a href=\"http://lesswrong.com/r/discussion/lw/nkz/a_second_year_of_spaced_repetition_software_in/\">two years</a>\n</li>\n<li>\n<a href=\"http://www.alljapaneseallthetime.com/blog/all-japanese-all-the-time-ajatt-how-to-learn-japanese-on-your-own-having-fun-and-to-fluency\">AJATT table of contents</a> -(applying SRS to learning Japanese)</li>\n<li>\n<p>SRS for programming:</p>\n<ul>\n<li><a href=\"http://www.supermemo.com/articles/programming.htm\"><q>SuperMemo as a new tool increasing the productivity of a programmer. A case study: programming in Object Windows</q></a></li>\n<li>\n<a href=\"http://www.jackkinsella.ie/2011/12/05/janki-method.html\"><q>Janki Method: Using spaced repetition systems to learn and retain technical knowledge</q></a> (<a href=\"http://www.reddit.com/r/programming/comments/n30hl/janki_method_learning_programming_with_6000/\">Reddit discussion</a>); <a href=\"http://www.oxbridgenotes.com/articles/autodidactism/note_taking\">SRS problems &amp; solutions</a>\n</li>\n<li>\n<a href=\"https://sivers.org/srs\"><q>Memorizing a programming language using spaced repetition software</q></a> (<a title=\"Wikipedia: Derek Sivers\" href=\"https://en.wikipedia.org/wiki/Derek%20Sivers\">Derek Sivers</a>; <a href=\"https://news.ycombinator.com/item?id=5015183\">Hacker News</a>)</li>\n<li><a href=\"https://www.shortcutfoo.com/blog/introducing-interval-training-for-shortcuts/\">learning text editor shortcuts</a></li>\n<li><a href=\"https://developer.atlassian.com/blog/2015/06/golang-flashcards-and-spaced-repetition/\"><q>Learning Go with flashcards and spaced repetition</q></a></li>\n</ul>\n</li>\n<li>\n<a href=\"http://quantifiedself.com/2012/06/spaced-repetition-and-learning/\"><q>QS Primer: Spaced Repetition and Learning</q></a> -(talks on applications of spaced repetition)</li>\n<li>\n<p>Value compared to curriculums:</p>\n<ol type=\"1\">\n<li>Point: <a href=\"http://www.scotthyoung.com/blog/2012/08/05/forgetting-is-good/\"><q>Why Forgetting Can Be Good</q></a>, by Scott H. Young</li>\n<li>Counterpoint: <a href=\"https://web.archive.org/web/20130920193543/http://blog.learnstream.org/2012/08/spaced-repetition-in-natural-and-artificial-learning/\"><q>Spaced repetition in natural and artificial learning</q></a>, by Ryan Muller</li>\n</ol>\nMy own observation is that an optimally constructed curriculum <em>could</em> effectively implement spaced repetition, but even if it did (most don’t), unless it is computerized it will not adapt to the user. -<a href=\"http://www.salon.com/2014/04/20/ditch_the_10000_hour_rule_why_malcolm_gladwells_famous_advice_falls_short/\"><q>Ditch the 10,000 hour rule! Why Malcolm Gladwell’s famous advice falls short; Contrary to what the bestselling author would tell you, obsessive practice isn’t the key to success. Here’s why</q></a>\n</li>\n<li><a href=\"http://www.ummah.com/forum/showthread.php?381181-How-to-Memorize-the-Quran-and-Never-Forget-it\"><q>How to Memorize the Quran and Never Forget it</q></a></li>\n<li>\n<a href=\"http://groups.google.com/group/mnemosyne-proj-users/browse_thread/thread/fd10b9e601fb0eb6\">Bash scripts</a> for generating vocabulary flashcards (processing multiple online dictionaries, good for having multiple examples; images; and audio)</li>\n<li>\n<p>vocabulary selection:</p>\n<ol type=\"1\">\n<li><a href=\"http://jtauber.com/blog/2004/11/26/programmed_vocabulary_learning_as_a_travelling_salesman_problem/\"><q>Programmed Vocabulary Learning as a Traveling Salesman Problem</q></a></li>\n<li><a href=\"http://jtauber.com/blog/2006/05/05/teaching_new_testament_greek/\"><q>Teaching New Testament Greek</q></a></li>\n<li>\n<a href=\"http://graded-reader.org/\">graded-reader</a>: <a href=\"http://jtauber.com/blog/2008/02/10/a_new_kind_of_graded_reader/\"><q>A New Kind of Graded Reader</q></a> (video talk)</li>\n<li><a href=\"http://groups.google.com/group/graded-reader\">Mailing list</a></li>\n<li>\n<a href=\"https://code.google.com/p/graded-reader/\">Programs</a> (I took <a href=\"http://community.haskell.org/~gwern/hcorpus/\">a stab</a> at writing Haskell versions once)</li>\n</ol>\n</li>\n<li><a href=\"http://www.fsavard.com/flow/2012/12/diff-revision/\"><q>Diff revision: diff-based revision of text notes, using spaced repetition</q></a></li>\n<li>Hacker News discussion: <a href=\"https://news.ycombinator.com/item?id=6461936\">1</a>, <a href=\"https://news.ycombinator.com/item?id=7539390\">2</a>, <a href=\"https://news.ycombinator.com/item?id=8183220\">3</a>\n</li>\n<li>\n<a href=\"http://lesswrong.com/r/discussion/lw/juq/a_vote_against_spaced_repetition/\"><q>A vote against spaced repetition</q></a>; <a href=\"http://howtoremember.biz/content/how-flashcards-fail-confessions-tired-memory-guy\"><q>How Flashcards Fail: Confessions of a Tired Memory Guy</q></a>\n</li>\n<li><a href=\"http://blog.beeminder.com/hieroglyphs/\"><q>Learning Ancient Egyptian in an Hour Per Week with Beeminder</q></a></li>\n<li><a href=\"http://rs.io/2014/04/05/anki-10000-cards-later.html\"><q>Anki, 10000 Cards Later: How my Anki usage has evolved</q></a></li>\n<li>\n<a href=\"https://www.duolingo.com/\">Duolingo</a> <a href=\"https://www.quora.com/Do-you-have-any-plans-for-optimizing-Duolingos-vocabulary-learning-using-spaced-repetition\">uses spaced repetition</a>\n</li>\n<li><a href=\"http://archive.wired.com/geekdad/2012/01/everything-about-learning/\"><q>Everything You Thought You Knew About Learning Is Wrong</q></a></li>\n<li>\n<a href=\"http://www.spacedrepetition.com/\">SeRiouS</a>: <a href=\"http://conference.cali.org/2014/sessions/spaced-repetition-technology-legal-education\"><q>Spaced Repetition Technology for Legal Education</q></a>, <a href=\"http://lawpracticetechnology.blogs.law.suffolk.edu/2014/03/11/serious-an-lpti-supported-project-to-improve-students-learning-and-bar-performance/\"><q>SeRiouS: an LPTI-supported Project to Improve Students’ Learning and Bar Performance</q></a>, Gabe Teninbaum (<a href=\"https://www.youtube.com/watch?v=dtClgl07lg8\">video presentation</a>)</li>\n<li>\n<a href=\"http://ejlt.org/article/view/320/424\"><q>The role of digital flashcards in legal education: theory and potential</q></a>, Colbran et al 2014</li>\n<li>\n<a href=\"http://nautil.us/issue/17/big-bangs/how-i-rewired-my-brain-to-become-fluent-in-math-rd\"><q>How I Rewired My Brain to Become Fluent in Math</q></a> (<a href=\"https://news.ycombinator.com/item?id=8402859\">HN</a>)</li>\n<li><a href=\"http://www.newyorker.com/books/page-turner/why-we-should-memorize\"><q>Why We Should Memorize [Poetry]</q></a></li>\n<li><a href=\"http://www.nytimes.com/2014/11/23/sunday-review/studying-for-the-test-by-taking-it.html\"><q>Studying for the Test by Taking It</q></a></li>\n<li>\n<a href=\"http://www.rand.org/content/dam/rand/pubs/monographs/2011/RAND_MG1120.pdf\"><q>Making Summer Count: How Summer Programs Can Boost Children’s Learning</q></a>, McCombs et al 2011 (RAND MG1120)</li>\n<li><a href=\"http://www.learningmedicinebook.com/\"><em>Learning Medicine: An Evidence-Based Guide</em></a></li>\n<li>\n<a href=\"http://www.gwern.net/docs/spacedrepetition/1998-arthur.pdf\"><q>Factors that Influence Skill Decay And Retention: a Quantitative Review and Analysis</q></a>, Arthur et al 1998</li>\n<li>\n<a href=\"http://cbmm.mit.edu/sites/default/files/publications/CBMM%20Memo%20068-On%20Forgetting%20-%20June%2018th%202017%20v2.pdf\"><q>On The Forgetting Of College Academics: At <q>Ebbinghaus speed</q>?</q></a>, Subirana et al 2017</li>\n<li><a href=\"http://lesswrong.com/r/discussion/lw/o8e/how_i_use_anki_to_learn_mathematics/\"><q>How I use Anki to learn mathematics</q></a></li>\n<li>\n<a href=\"https://www.theguardian.com/science/2017/feb/08/total-recall-the-people-who-never-forget\"><q>Total recall: the people who never forget; An extremely rare condition may transform our understanding of memory</q></a>; <a href=\"https://www.newyorker.com/books/page-turner/the-mystery-of-s-the-man-with-an-impossible-memory\"><q>The Mystery of S., the Man with an Impossible Memory: The neuropsychologist Alexander Luria’s case study of Solomon Shereshevsky helped spark a myth about a man who could not forget. But the truth is more complicated</q></a>\n</li>\n<li>\n<a href=\"https://alexvermeer.com/anki-essentials/\"><em>Anki Essentials</em></a>, Vermeer</li>\n<li>\n<a href=\"http://genedan.com/no-126-four-years-of-spaced-repetition/\"><q>No. 126: Four Years of Spaced Repetition</q></a> (Gene Dan, actuarial studies)</li>\n<li>\n<a href=\"https://ncase.me/remember/\"><q>How To Remember Anything Forever-ish</q>: an interactive comic</a> (Nicky Case)</li>\n<li><p>Discussion: <a href=\"https://news.ycombinator.com/item?id=13151790\">HN</a></p></li>\n</ul>\n<section>\n<h2>Flashcard sources</h2>\n<ul>\n<li>the <a href=\"http://mnemosyne-proj.org/card-sets\">Mnemosyne deck collection</a>\n</li>\n<li>the Anki deck collection is viewable only through the Anki clients; &gt;3000 decks</li>\n<li><a href=\"http://www.flashcardexchange.com\">FlashCardExchange.com</a></li>\n<li><a href=\"http://www.studystack.com/\">StudyStack.com</a></li>\n<li><a href=\"http://flashcarddb.com/cardsets\">Flashcarddb</a></li>\n</ul>\n</section>\n</section>\n</div></div><br>\n<br>\n<br>\n<br>\nNinja<br>\n<br>\nvia www.gwern.net http://www.gwern.net<br>\n<br>\nDecember 14, 2018 at 12:14AM"}}
{"actor":{"display_login":"guevara"}}