One of the most impressive achievements of the AI revolution is the development of large language models that can generate meaningful text and respond to instructions in plain English with no additional training necessary. Here we show that language models can be used as a scientific instrument for studying human memory for meaningful material. We developed a pipeline for designing large scale memory experiments and analyzing the obtained results. We performed online memory experiments with a large number of participants and collected recognition and recall data for narratives of different lengths. We found that both recall and recognition performance scale linearly with narrative length. Furthermore, in order to investigate the role of narrative comprehension in memory, we repeated these experiments using scrambled versions of the presented stories. We found that even though recall performance declined significantly, recognition remained largely unaffected. Interestingly, recalls in this condition seem to follow the original narrative order rather than the scrambled presentation, pointing to a contextual reconstruction of the story in memory.

§ INTRODUCTION

In the classical paradigm for studying human memory, participants are presented with randomly assembled lists of words and then perform memory tasks such as recognition and recall (see review in [17]). A wealth of results has been obtained in these studies. For instance, it has been found that words at the end and the beginning of the list have a higher chance of being recalled (recency and primacy effects, respectively), and there is a tendency to recall words close to each other in the list (contiguity, [16]). Moreover, it was found that when the presented lists grow in length, even though the average number of recalled words ($R$) is increasing, a progressively smaller fraction of the words is recalled [26]. Several authors have addressed the issue of the mathematical form of the dependence of $R$ on list length and found that the best description for this dependence is provided by power-law relations, $R \sim L^\alpha$, with exponents $\alpha$ generally below one [27]. It is well known that recall also depends on multiple experimental factors such as e.g. the presentation rate of words, the age of the participants, etc. However, in recent work, some of the authors discovered that if recall performance is analyzed as a function of a number of remembered ($M$), rather than presented words, the relation becomes universal and is described by the analytical form: $R = \sqrt{\frac{3 \pi}{2}M}$ [28]. Moreover, this relation follows from a simple deterministic model where words are retrieved one by one according to a random symmetric matrix of `similarities' reflecting their long-term encoding in memory, until the process enters a cycle and no more words can be recalled. The number of remembered words  $M$, itself can be predicted by the retrograde interference model that assumes that each new word erases some of the previously presented words according to the `valence' or `importance' of each word [11, 12].

While it is remarkable that human memory for random material can be described with universal mathematical relations, it is of course much more important and exciting to try to understand how people remember more natural, meaningful information. After the pioneering work of 3, many studies considered recalls of narratives. As opposed to random lists, narratives convey meaning, and hence have structure on multiple levels which influences recall, as was confirmed in many previous publications (see <Ref>). The first challenge in understanding narrative recall is the fact that people tend not to recall the narrative verbatim. Rather, they remember what the narrative is about and retell it in their own words [14, 10, 33]. Counting correctly recalled words is therefore not a good score of recall, and the better score used in many studies, that we also adopt in our work, is a count of recalled `ideas', or `clauses' (see e.g. [6]). Using this method however requires a human-level of understanding of narratives and recalls, making collecting large amounts of data difficult and extremely time-consuming to analyze. In our study, we develop a way to overcome this and other difficulties by using large language models (LLMs) to  assist in the analysis and design of experiments, as described later. In particular, we use LLMs to generate new narratives of a particular type and length, and to score human recalls obtained in multiple experiments performed over the internet. In addition to recall, we also performed recognition experiments (where people are requested to indicate whether a specific clause was in the presented story or not) in order to estimate how many clauses people remember after reading the narrative. To this end, we use LLMs to generate plausible lures, i.e. novel clauses that could have potentially appeared in the narrative.

Inspired by our previous results, we wanted to understand how recognition and recall performance scale up with narrative length as it increases and what the relation between them is. To this end, we performed a large number of experiments over the internet using the Prolific platform (<www.prolific.com>). We also compared the recall and recognition performance of original narratives with their scrambled versions in order to elucidate the effects of comprehension on different aspects of memory. Since there are different types of narratives that could potentially be more or less difficult for people to remember and recall, we decided to focus on one particular type of narrative first studied in the famous paper by 22 that established the field of narratology, namely the oral retelling of personal experience, told by real people, and the variants of those generated by LLMs (see later). While being collected in a research setting, these spoken recollections of dramatic personal episodes are close to the natural way people share their experiences in real life and therefore are of special interest for studying human memory.

§ RESULTS

§.§ LLM-assisted recall and recognition experiments.

For the purpose of this study, we have chosen several narratives of different lengths from 21 and 22. As part of the analysis in these publications, narratives were segmented into an ordered set of clauses, which are “the smallest unit of linguistic expression which defines the functions of narrative” [22]. In other words, they are the smallest meaningful pieces which still serve some function in communicating a narrative.

Since these are spontaneous narratives spoken in local dialect, they are characterized by a number of features which are awkward to transcribe (pauses, repetition, gestures) as well as non-standard (and sometimes outdated) English vernacular. These factors complicate comprehension when participants have to read narratives on the computer screen. We therefore instructed LLMs to generate new narratives modeled on the original ones, i.e. exhibiting a similar type of a event sequence and the overall length in terms of the total number of clauses. In particular, the LLM-generated narratives inherited the segmentation from the original story, i.e. the number of clauses was the same and the information contained in the corresponding clauses had a similar role in their respective narrative (see Methods <Ref> for details of narrative generation, and some examples in Appendix <ref>). Eight narratives were selected for subsequent memory experiments, ranging from 18 to 130 clauses in length. We presented each narrative to a large number of participants ($\sim200$) who then performed either recall or recognition tasks. In the subsequent analysis, we treated clauses as the basic units that together communicate the meaningful information contained in the narrative. In particular, we quantified each individual recall by identifying which of the clauses in the narrative were recalled, determining this by whether information contained in this clause is present in the recall. We simplify the analysis by considering each clause as being either recalled or not. This scoring of recalls is traditionally performed by human evaluators and is very time-consuming. We, therefore, prompted an LLM to define which of the clauses of the original narrative were recalled and in which order. Here we utilized the remarkable ability of modern LLMs to respond to instructions, provided as prompts written in standard English (as opposed to a programming language), to perform novel tasks without any additional training (known as zero-shot prompting or 'in-context' learning  [7], see Appendix <ref> for more details).

To test the ability of the LLM to adequately score human recall (with appropriate prompting as described in the Methods <Ref> and <Ref>), we performed an additional set of recall experiments with a specially LLM-generated narrative and compared the LLM-performed recall scoring to the one conducted manually by the authors (see Methods <Ref> for details). To this end, we calculated the fraction of participants who recalled each particular clause, i.e. the clause's recall probability ($P_{rec})$, as judged by the LLM and by the authors. For nearly all of our analysis, the LLM we used was OpenAI's GPT-4 (see <Ref> for comparison between different LLMs). As shown in <Ref>, GPT-4 scoring of recalls results in recall probabilities close to ones obtained by human evaluations for a great majority of the clauses. Moreover, variability of scoring is comparable between GPT-4 and human evaluators (compare <Ref>B and C ). Interestingly, the LLM has a greater correlation with the mean human scoring  (r = 0.94) than with any individual scoring (r = 0.92, 0.90, 0.90) (see <Ref> in <Ref>; c.f. 24).

Reliability of LLM scoring of recalls. 3 authors and GPT-4 (OpenAI API model gpt-4-0613) performed scoring of 30 recalls by answering the question of whether the information present in each particular clause is present in each individual recall. (A): comparison between recall probabilities $P_{rec}$ for each clause, as calculated from GPT-4 scores (orange) and average human scores (dashed blue). The full range of human scores is given by the shaded blue region. (B): A strong correlation between human and GPT-4 scores across clauses, with a correlation coefficient (r-value) of 0.94. (C): Correlations between individual human scores, shows overall a strong agreement between human scorers.

One prominent feature seen in <Ref>A is a wide range of recall probabilities for different clauses. In other words, while some clauses are recalled by most of the participants, other clauses are not. Such a wide distribution of $P_{rec}$'s across the clauses was observed in recalls of all the narratives and contrasts sharply with corresponding results for random lists of words where $P_{rec}$'s are rather uniform except for the ones in the beginning and the  end of the list (see <Ref>). This wide distribution of $P_{rec}$s is apparently due to the fact that not all clauses have similar importance for communicating the narrative. Indeed if we select the clauses with high enough $P_{rec}$, we usually get a good summary of the narrative (see <Ref> for examples).

We also performed recognition experiments in order to estimate the average number of clauses that participants remember after presentation. As we explain below, this analysis requires a large number of plausible lures, i.e. novel clauses that could have possibly been in the narrative and hence cannot be easily distinguished from the true clauses using context and style. Generating these lures is highly nontrivial as it requires an understanding of the narrative. This makes manually generating lures very challenging and time-consuming, which is why we utilized LLMs for this purpose (see <Ref> for prompts and example output). Using the LLM, for each story we obtained the same number of lures as true clauses, and sampled 10 clauses from this entire pool of 2L clauses uniformly and randomly for the testing phase. Participants would then see one clause at a time, and were asked whether the clause was in the presented narrative or not. We checked that presenting several clauses for recognition does not result in systematic drift in performance, i.e. no output interference was detected in our experiments (8; see Appendix <ref>) .  We then estimated the number of clauses retained in memory after presentation of the narrative ($M$) from the fraction of `hits', i.e. correct recognitions of the true clauses ($P_{h}$) and the fraction of `false alarms', i.e. reporting lures as true clauses ($P_{f}$). In particular, we assume that if the participant remembers a given clause, they always recognize it as being part of the narrative; otherwise they still give a positive answer with probability $P_f$. The total probability of a correct recognition is then given by

\begin{equation*}
P_h = \frac{M}{L} + \left(1-\frac{M}{L} \right) P_f,
\end{equation*}

from which we obtain

\begin{equation}
\frac{M}{L} = \frac{P_h - P_f}{1-P_f} .
\label{recog}
\end{equation}

This equation emphasizes the importance of using lures in recognition experiments, since without lures we would have an uncontrolled tendency of participants to indicate any presented clause as a true one irrespective of whether or not they remember it.

§.§ Scaling of Recall and Memory

Having each narrative seen by roughly 200 participants, half of them doing recall and another half recognition, we were able to determine the average number of remembered clauses ($M$) and recalled clauses ($R$). As expected both $M$ and $R$ grow with the length of the narrative presented, as measured by the number of clauses in the narrative and denoted by $L$ (see Fig. <ref>A,B). Moreover, both $M$ and $R$ appear to grow linearly with $L$ for the range of narrative lengths we explored, and hence when we plot $R$ vs $M$, we also get an approximately linear relationship (see Fig. <ref>C). This scaling behaviour is very different from what we observed with random lists of words with a characteristic square root scaling, i.e. unsurprisingly, recall of meaningful material is better than for random ones of the same size, even if we discount for better memorization. One of the factors that apparently leads to better recall of narratives is the temporal ordering of recall. When people recall narratives, recall mostly proceeds in the forward direction (see Fig. <ref>A), probably reflecting the natural order of events in the narrative that cannot be inverted without affecting its coherence. This contrasts with the case of random lists, when recall proceeds in both directions with similar probability (see Fig. <ref>B), which, according to a model proposed in [28] results in the process entering a cycle preventing many words from being recalled.

Human performance in recall and recognition experiments for narratives of different length. (A): Estimated number of remembered clauses (M) is plotted as a function of the number of clauses in the narrative (L) measured in recognition experiment. Surprisingly M has similar values in intact and scrambled narrative. (B): Average number of recalled clauses (R) for narratives of different length. In contrast to the M, R drops substantially for scrambled narratives. (C): Average number of recalled clauses vs. number of remembered clauses from the same story. As expected from panels a) and b) the number of retrieved clauses in scrambled narrative is substantially smaller that in intact narrative for the same number of remembered clauses. For comparison we presented the theoretical performance for the random list of words, which describes data well (28). It is clear that there are more clauses recalled in intact narratives than words in lists of random words. Surprisingly, retrieval of scrambled stories is significantly worse than random lists, suggesting an active suppression of items in service of generating a coherent recall (participants were implicitly instructed to recall story).

Recall order. Color-coded order of clauses or words for different conditions are shown in all panels. Recalled clauses or words are stacked together vertically (with the first recalled clause at the bottom of a column, and the last recalled clause at the top). The height of the column represents the total number of clauses or words recalled in a given trial. In panels A, B, and D, color code represents serial position of presentation of clauses or words, from early (red) to later (blue) in presentation position. Panel (C) is the only exception, in which the color code reflects the serial position of clauses in the original (intact) story. (A) shows that recall of coherent stories largely preserves presentation order. (B) recall of random word lists does not preserve presentation order. (C) As with random lists, the recall of a scrambled story does not preserve presentation order, but rather appear to reconstruct the original order of the story, as seen from the color gradients in panel (B).
Apparently, random words and scrambled stories are recalled in random order considering their presentation order, but people perform some unscrambling of the scrambled stories as can be seen in (C) - there is tendency of recalled clauses being in the order of original unscrambled narrative. The participants construct a mental representation of the scrambled narrative which is evidently close to its original form. Recall consequently does not reflect input sequence, but rather the original sequence of the clauses.

§.§ Meaning and memory

As we mentioned above, people's recall is strongly influenced by  narrative comprehension, such that clauses that are most important in communicating the summary of the narrative are the ones that are recalled by most of the participants. We found however that recognition is not so strongly affected by meaning. This can be observed by evenly dividing all the clauses used in our experiments into subsequent bins according to their recall probabilities, and calculating the average recognition performance for all the clauses in each bin. Surprisingly, there is very little increase of recognition with recall probability across the clauses, such that clauses with highest and lowest average $P_{rec}$ only differ in their $P_h$ by less than 0.15  (see Fig. <ref>; c.f. [35, 38]).

To further elucidate the role of meaning in memory, we repeated our experiments with another group of participants after randomly scrambling the order of clauses, thus making comprehension much more difficult if not impossible. We found that, unsurprisingly, recall of scrambled narratives is much poorer then the original ones (Fig. <ref>B,C). Recognition performance for scrambled narratives however is practically the same (Fig. <ref>A).  This result indicates that memory encoding of clauses is not significantly affected by the structure and meaning of the narrative.  Interestingly, the order in which people tend to recall clauses from a scrambled narrative corresponds much better to the order of these clauses in the original narrative than in the presented, scrambled one (see Fig. <ref>C,D), indicating that even in this situation people are trying to comprehend the meaning of the narrative rather than processing the input as a random list of unconnected clauses. This might explain why recall of scrambled narratives appears to be worse than recall of random word lists of the same size.

Recognition vs recall performance across different clauses. Clauses from all the narratives used in this study were divided evenly into 15 bins according to their $P_{rec}$, and the average $P_h$ for the clauses in each bin was computed and plotted against the center of the corresponding bin.

§ PREVIOUS WORK

The experimental study of memory for narratives can be traced back to the highly influential descriptive work of 3. This and followup work introduced the idea that the encoding of memory for narratives is a process of abstraction [14], and that subsequent recall is in large part a generative process driven by a participant's prior knowledge and biases. This line of thought was formalized much later in theories of narrative structure involving schemas, scripts, frames, and story grammars [1, 32]. By now, there is ample support for an abstracting process for memory encoding, and the existence of schematic structures which guide recall [2].

A parallel line of research into narrative structure originated in the field of sociolinguistics by 22. In carrying out linguistic fieldwork to analyze spoken dialects of English, the authors found that personal narratives of emotionally charged events tented to elicit the best examples of local dialect. Furthermore, the resulting narratives, which were produced spontaneously and by non-expert storytellers, tended to be very regular in their structural properties. Of particular significance to us was the observation that stories were typically told in the order in which they experienced by the speaker, which Labov encapsulates in his “egocentric principle" and “no flashback” constraint [21]. This lends some support to the strong iconicity assumption [39], which states that readers (or listeners) will interpret the order in which events are reported as reflecting the actual chronological order in which they occurred.

However, serial order alone likely cannot explain the rich structure we observe in narrative recall. Indeed, temporal order provides just a single axis along which events in a narrative are organized. In addition to simple serial order, events or clauses can have causal relations (e.g. A causes B) 37, 36, 23, inferential relations (e.g. A implies B), and superordinate or constituency relations (e.g. A consists of B) [4]. These relations conspire to give narratives hierarchical structure [18]. Story grammars provide one natural mechanism whereby the hierarchical structure arises, and served as early inspiration for studying structure dependence in narrative recall and recognition.

In this direction, 38, 35 performed recall and recognition experiments to test how encoding and processing depend on hierarchical structure. Their recognition experiments included only old, paraphrase, and false statements in the test phase. The false statements tested were appropriate (invoking the appropriate agents or actions), but inconsistent with or contradictory to the story. Similar to our results, they observe that recognition is uncorrelated with recall. Furthermore, whereas recall was argued to be sensitive to the structure of a narrative, recognition appeared to have no such sensitivity.

In another experiment seeking to isolate the structure sensitivity of narrative processing, 19 tested processing of scrambled narratives, in which paragraphs of a text were presented to participants in random order. Participants were able to produce coherent summaries of such scrambled text, which were indistinguishable from summaries produced by participants reading the coherent text. It is possible that scrambling larger units (paragraphs in this case, versus clauses in our experiments) produced an overall more comprehensible text, making descrambling easier. Nevertheless, we still observe descrambling, but of lower quality (see <Ref>C).

Scaling laws for memory have been observed for random unstructured lists (pictures, words, etc) [34, 27, 28]. Two important takeaways from these works are the following: memory typically exhibits power-law scaling with list length, and the retrieval process appears to be universal. Surprisingly, there is very little work which considers the scaling of narrative memory with story length, let alone attempts to quantify it. The only work we are aware of is  [13], which measured average recall as a function of story lengths up to 83 “informational units", which include clauses as well as noun and adjective phrases. The experimental design was motivated largely by questions about story grammars, and therefore the episodic structure of the narratives (in the story grammar sense developed by the authors) was kept constant while descriptive phrases and clauses (so called 'informational units') were added to increase the length of the story. Therefore, the added statements are arguably a kind of filler, not contributing any significant additional meaning or structure to the underlying narrative. This might account for the seemingly sublinear scaling of their mean recall with story length, compared to our linear results (<Ref>). There are other significant differences, including the test population (24 second-grade schoolchildren), the stimulus input format (spoken narratives), and the recall format (spoken recall).

The choice of “informational units" in [13] also differs from our choice of clauses, and is similar to segmentations that use propositional analysis [20], or pausal breaks [15]. An important feature of these different units is that they provide a more fine-grained segmentation of a narrative - a clause can consist of many propositions or pausal breaks, but not the other way around. However, these smaller units would not have a narrative function in the Labovian sense.

§ DISCUSSION

In this contribution, we describe a new way to study human memory for meaningful narratives with the help of large language models. Together with using internet platforms for performing experiments, this technology enables a qualitative leap in the amount of data one can collect and analyze. In particular, we describe the prompts that we used to make GPT-4 generate new narratives of a particular type and size, score multiple human recalls of these narratives by identifying which clauses were recalled, and generating plausible lures for recognition experiments. Having a large amount of data is important for memory research because, as opposed to e.g. sensory processing that is believed to be largely uniform across people [30], the way people remember and recall meaningful material is highly variable. Hence, only through statistical analysis can some general patterns be uncovered. In particular, we considered how recognition and recall performance scales up for narratives of increasing length. We found that approximately the same fraction of narrative clauses are recognized and recalled when narratives become longer, in the range of 20 to 130 clauses that we considered for this analysis. We expect that as narratives become longer, this trend will not persist because people will start summarizing larger and larger chunks of the narrative into single sentences in their recalls. In this case, deciding whether individual narrative clauses in the chunk were recalled or not is ambiguous. In fact, we observed that summarizing started to appear in recalls of some of the participants of the longest narrative in our pool, which resulted in the average number of clauses in a recall being significantly smaller than the average number of narrative clauses that were deemed to be recalled (see <ref>). In the extreme case, a short summary of the entire story can be scored as having recalled nothing, since no particular clause was recalled, when in fact there was a non-trivial recall but at a higher level of abstraction or organization. This in itself does not mean that studying recall for clauses is incorrect or uninformative, but rather that it must have some regimes of validity. It also illustrates the need to develop techniques which are sensitive to this hierarchical structure of recall, by automatically detecting higher-level units of meaning such as events and episodes.

We investigated the role of meaning in narrative memory by presenting the participants with the  same clauses as in the original narrative but in a scrambled order. These `narratives' are much more difficult, if at all, to make sense of, and indeed their recall was very poor. However, recognition of individual clauses was practically as good as in the original narrative. This surprising observation indicates that the encoding of clauses in memory is not very sensitive to the overall structure of the narrative, and only in the process of recall  does the meaning of the narrative play a major role. This finding is consistent with the observations of [35, 38]. It also provides a striking confirmation of the generally held wisdom that while recall is structure sensitive, recognition need not be [18]. It is still possible, however, that these trends will change when longer narratives are considered. This will have to be investigated further.

Another interesting observation concerns the fact that as the narrative becomes longer, the range of recall probabilities for different clauses remains very wide, e.g. there are always some clauses that are recalled by most of the participants and others that are almost never recalled. In contrast, the probability to recall words from a random list of words decreases with the length of the list, with the exception of the last few words (due to the recency effect) [25]. It would be an interesting theoretical direction for future studies to try to identify the factors that could predict how well a given clause will be recalled in a given narrative.

We focused on first-person spoken stories. These were personal accounts of important events, shared naturally and informally. This way, they lacked the refinement of crafted stories, which may utilize tricks to improve memorability. It would be interesting to see how scaling of memory is affected by such expertly told or literary stories. Evidently there is an impact, considering that many stories in the oral tradition endure over very long timescales  [31, 29]. A striking example of this in a more controlled lab setting had participants trained to construct a narrative in which they embedded a random list of words [5]. This work found that employing such a mnemonic improved recall to nearly perfect for up to 12 consecutively learned lists.

While practice and rehearsal are necessary for preserving narratives in oral tradition, our results suggest that narratives are intrinsically more memorable. We find that memory performance for narratives encountered only once, as measured by the scaling relations in <Ref>, are robustly superior to performance on unstructured lists.

The research conducted for this report relied crucially on a set of LLM input “prompts”, i,e, instructions, written in standard English, given to the LLM for carrying out various tasks. Roughly speaking, these appear as if they were instructions given to a human research assistant. Quite remarkably, the LLM completes the input string to provide a correct output without any additional training, a phenomenon  known as `in-context' learning [7]. Since this phenomenon is still not fully understood, we had to resort to a good amount of trial-and-error and fine-tuning in designing the prompts used in our analysis. We provide all of the prompts used in our experiments in <ref>. The specific model we utilize in most of the paper is OpenAI's GPT-4. However, while we believe the capabilities necessary to carry out our experiments are not limited to this model, it is an open question whether the prompts we use can be transferred to different models.

In summary, using LLMs in conjunction with internet platforms for performing experiments is a powerful new tool that could bring significant new advances in understanding human memory.

§ METHODS

With the aim of conducting a large scale study on memory for natural continuous material (personal narratives in this case) we required an automated procedure that would facilitate measuring human recall memory performance, since manual scoring of recalls is very labor-intensive and thus limits the ability to analyse large datasets. We were able to achieve this through the use of Large Language Models (LLMs) and we assessed the reliability of our pipeline by comparing it to human scoring performed by the authors. Our dataset was generated by conducting recall and recognition experiments online, recruiting participants through a crowd-sourcing platform. All segments of this study are detailed below.

§.§ Stimulus Set - Narrative pool

Nearly all of the stimuli we use are generated by LLMs and are based on first-person oral narratives taken from socio-linguistic interviews [22, 21][The narratives `boyscout' and `stein' are taken directly from these references. The rest are generated using templates.]. The oral narratives are segmented into clauses in these references, and these are used as templates for the LLM narrative generation. The LLM output is a narrative of equal length (in number of clauses), with very similar narrative-syntactic structure, but involving different subject matters. Two stories were generated from each template for lengths L = 18, 32, and 54. Two additional narratives were directly taken from Refs. [22, 21], one with L = 19 (`boyscout' <ref>), and the other with L = 130 which was minimally edited to remove punctuation due to speech breaks, in order to increase readability (`stein' <ref>). More details of the narrative generation by LLMs can be found in <ref>, along with a sample narrative template in <ref> and examples of generated outputs in <ref>. All narratives used as stimuli can be found in <Ref>.

For the purpose of evaluating the reliability of recall scoring of LLMs and their similarity to human scoring, we generated and segmented a different narrative based on the `boyscout' story (<ref>, L=19). This stage began before the rollout of gpt-4 and for this reason we document the evaluation process separately in <Ref>. The narrative generation step in this part produced variable length narratives in prose, which we had to subsequently segment using GPT-3. As a result, this narrative generation procedure did not keep the same number of clauses as the story it was based on (`boyscout').

§.§ Experimental Design

Participants were recruited online through the `Prolific' platform (www.prolific.com) and experiments were conducted on participant's browser. Only candidates that indicated English as their first language were allowed to participate. Participants were initially greeted with a short description of the experiment and an informed consent form. The trial was initiated by a button press. After a three second counter, a narrative was presented in the form of rolling text (marquee) in black font in the middle of a white screen. All narratives were animated in constant speed with a total duration calculated as the character span of the story divided by 250 (resulting in a character moving with a speed of approximately 25 characters/s) while the font size has been set to 40px. Once the marquee for the narrative has traversed outside the screen (all characters shown and disappeared), the testing phase was triggered automatically. This marquee style presentation was chosen because it allowed for comfortable reading while fixing the presentation duration for all participants and simultaneously preventing revisiting of already read material.

In the free recall experiments, the testing phase consisted of a textbox and a prompt to recall the story as close as possible to the original one. Once participants finished typing their recall of the narrative, they submitted their response with a button press and the experiment was concluded.

For the testing phase of the recognition experiment, 10 queries were sequentially presented. In each query, the participant was shown a single clause at random, either from the just presented narrative (old) or a lure (new). They were tasked to select whether they remembered seeing the presented clause by pressing "Yes" if they thought the clause appeared in the narrative and "No" otherwise. We did not observe any obvious signatures of output interference [8] (see <Ref>) and therefore used all queries in the subsequent analysis.
Lures were generated by asking the LLM to take a given narrative segmentation, and insert novel clauses between each existing clause. This ensures that the lures are distinct from the true clauses, but still fit within the overall context of the story. For instance, this avoids lures which might mention "dolphins" if the story is about boy scouts. The prompt used to generate lures and an example completion by gpt-4 are given in <Ref>.

§.§ Analysis

Analysis was conducted through custom Python scripts.
For recognition memory, in order to estimate total encoded memory $M$ from <Ref>, we used population and test trial averaged hit rate (true positive probability $P_{h}$) and false alarm rate (false positive probability $P_{f}$). Standard error was computed using statistical bootstrap [9].

Recall scoring was done using the OpenAI model gpt-4-0613 (a GPT-4 model which receives no updates) based on the clause segmentation of the narratives. For each participant's recall, gpt-4-0613 was instructed to loop through each clause of the given narrative (as presented) and examine whether the information that this clause provided was present in some form in the participant's recall and the corresponding passage. The numbers of all clauses evaluated as being recalled were given at the end of the output in the form of a list. The full prompt we use for scoring recalls is given in <ref>, which takes three arguments: the narrative stimulus in prose (e.g. <ref>), the numbered clause segmentation of the narrative stimulus (e.g. <ref>), and the participant's recall (e.g. <ref>). A sample completion is provided in <ref>.

Separately, to evaluate the similarity of recall scoring between humans and LLMs, 3 authors performed manual scoring of 30 recalls of <ref> using the same procedure, evaluating whether each clause was present in the recall or not.

§.§ Random List of Nouns Experiment

We performed an additional experiment with a list of 32 nouns (see <Ref>) that were randomly selected from the pool of nouns used in [28]. The experimental protocol was exactly the same as in that work with presentation speed 1.5 sec per word. 105 participants were recruited using the Prolific online platform, with each participant accepting informed consent prior to the beginning of a trial.

§.§ Data And Code Availability

Experimental data as well as code for the online experiments and the analysis presented above can be found in <https://github.com/mkatkov/llm-narrative-analysis>.

§ ACKNOWLEDGEMENTS

TC acknowledges the support of the Eric and Wendy Schmidt Membership in Biology, the Simons Foundation, and the Starr Foundation Member Fund in Biology at the Institute for Advanced Study. MT is supported by the Foundation Adelis and the Simons Foundation. We thank Omri Barak, Danqi Chen, Michael Douglas, Yadin Dudai, and Ariel Goldstein for helpful conversations. We thank Stefanos Gkouveris for  helpful comments on Javascript development.

[1]
Alba, J. W. and Hasher, L. (1983).
Is memory schematic?
Psychological Bulletin, 93(2):203.

[2]
Baldassano, C., Hasson, U., and Norman, K. A. (2018).
Representation of real-world event schemas during narrative
Journal of Neuroscience, 38(45):9689–9699.

[3]
Bartlett, F. C. (1932).
Remembering: A study in experimental and social psychology.
Cambridge university press.

[4]
Black, J. B. and Bower, G. H. (1979).
Episodes as chunks in narrative memory.
Journal of verbal learning and verbal behavior, 18(3):309–318.

[5]
Bower, G. H. and Clark, M. C. (1969).
Narrative stories as mediators for serial learning.
Psychonomic Science, 14(4):181–182.

[6]
Bransford, J. D. and Johnson, M. K. (1972).
Contextual prerequisites for understanding: some investigations of
comprehension and recall.
Journal of Verbal Learning and Verbal Behavior,

[7]
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P.,
Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).
Language models are few-shot learners.
Advances in neural information processing systems,

[8]
Criss, A. H., Malmberg, K. J., and Shiffrin, R. M. (2011).
Output interference in recognition memory.
Journal of Memory and Language, 64(4):316–326.

[9]
Efron, B. and Tibshirani, R. J. (1994).
An introduction to the bootstrap.
CRC press.

[10]
Fillenbaum, S. (1966).
Memory for Gist: Some Relevant Variables.
Language and Speech, 9(4):217–227.

[11]
Georgiou, A., Katkov, M., and Tsodyks, M. (2021).
Retroactive interference model of forgetting.
Journal of Mathematical Neuroscience, 11(1):4.

[12]
Georgiou, A., Katkov, M., and Tsodyks, M. (2023).
Forgetting dynamics for items of different categories.
Learning & Memory, 30(2):43–47.

[13]
Glenn, C. G. (1978).
The role of episodic structure and of story length in children's
recall of simple stories.
Journal of Verbal Learning and Verbal Behavior, 17(2):229–247.

[14]
Gomulicki, B. R. (1956).
Recall as an abstractive process.
Acta Psychologica, 12:77–94.

[15]
Johnson, R. E. (1970).
Recall of prose as a function of the structural importance of the
linguistic units.
Journal of Verbal Learning and Verbal Behavior, 9(1):12–20.

[16]
Kahana, M. J. (1996).
Associative retrieval processes in free recall.
Memory & cognition, 24(1):103–109.

[17]
Kahana, M. J. (2020).
Computational models of memory search.
Annual Review of Psychology, 71(3):107–138.

[18]
Kintsch, W. (1998).
Comprehension: A paradigm for cognition.
Cambridge university press.

[19]
Kintsch, W., Mandel, T. S., and Kozminsky, E. (1977).
Summarizing scrambled stories.
Memory & Cognition, 5(5):547–552.

[20]
Kintsch, W. and van Dijk, T. A. (1978).
Toward a model of text comprehension and production.
Psychological Review, 85(5):363–394.

[21]
Labov, W. (2013).
The language of life and death: The transformation of experience
in oral narrative.
Cambridge University Press.

[22]
Labov, W. and Waletzky, J. (1966).
Narrative Analysis: Oral Versions of Personal Experience.
In Helm, J., editor, Essays on the Verbal and Visual Arts:
Proceedings of the 1966 annual spring meeting of the American ethnological
society, pages 12–44.

[23]
Lee, H. and Chen, J. (2022).
Predicting memory from the network structure of naturalistic
Nature Communications, 13(1):1–14.

[24]
Michelmann, S., Kumar, M., Norman, K. A., and Toneva, M. (2023).
Large language models can segment narrative events similarly to

[25]
Murdock Jr, B. B. (1960).
The immediate retention of unrelated words.
Journal of Experimental Psychology, 60(4):222.

[26]
Murdock Jr, B. B. (1962).
The serial position effect of free recall.
Journal of experimental psychology, 64(5):482.

[27]
Murray, D. G., Pye, C., and Hockley, W. E. (1976).
Standing's power function in long-term memory.
Psychological Research, 38(4):319–331.

[28]
Naim, M., Katkov, M., Romani, S., and Tsodyks, M. (2020).
Fundamental law of memory recall.
Physical Review Letters, 124(1):018101.

[29]
Nunn, P. (2018).
The edge of memory: ancient stories, oral tradition and the
post-glacial world.
Bloomsbury Publishing.

[30]
Read, J. (2015).
The place of human psychophysics in modern neuroscience.
Neuroscience, 296:116–129.
Contributions from different model organisms to brain research.

[31]
Rubin, D. C. (1995).
Memory in oral traditions: The cognitive psychology of epic,
ballads, and counting-out rhymes.
Oxford University Press, USA.

[32]
Rumelhart, D. E. (1975).
Notes on a schema for stories.
In Representation and understanding, pages 211–236. Elsevier.

[33]
Sachs, J. S. (1967).
Recognition memory for syntactic and semantic aspects of connected
Perception and Psychophysics, 2:437.

[34]
Standing, L. (1973).
Learning 10,000 pictures.
Quarterly Journal of Experimental Psychology, 25(I

[35]
Thorndyke, P. W. and Yekovich, F. R. (1980).
A critique of schema-based theories of human story memory.
Poetics, 9(1-3):23–49.

[36]
Trabasso, T. and Sperry, L. (1985).
Causal Relatedness and the Importance of Narrative Events.
Journal of Memory and Language, 24(1894):595–611.

[37]
Trabasso, T. and van den Broek, P. (1985).
Causal thinking and the representation of narrative events.
Journal of Memory and Language, 24(5):612–630.

[38]
Yekovich, F. R. and Thorndyke, P. W. (1981).
An evaluation of alternative functional models of narrative schemata.
Journal of Verbal Learning and Verbal Behavior, 20(4):454–469.

[39]
Zwaan, R. A. (1996).
Processing narrative time shifts.
Journal of Experimental Psychology: Learning, memory, and
cognition, 22(5):1196.

Appendix .0.5em

§ PROMPTS AND COMPLETIONS

A significant part of the successful use of LLMs is the effective design of inputs or “prompts”, which amount to instructions, written in standard English, for carrying out a particular task.

More precisely, the input to a LLM is a string, which we interchangeably refer to as the prompt or the context. The output of the LLM is also a string. This is also referred to as a “completion", since LLMs are trained to complete text fragments given to them as inputs.

Despite the accumulation of wisdom on prompt engineering, we still resorted to a good amount of trial-and-error and fine-tuning in designing the prompts used in our analysis. Since these prompts are tantamount to algorithms written for LLMs, we present them below in pseudocode boxes. In the following sections, we exhaustively detail the prompts used, along with examples of LLM outputs (i.e. completions). The specific model we utilize in most of the paper is OpenAI's GPT-4. For the data analysis, in the interest of reproducibility, we opted to use the deprecated model gpt-4-0613 which does not receive updates. In <ref>, we show how scoring using the latest model gpt-4 can change with time as a result of OpenAI's regular model updating.

§.§ Narrative Generation

We started with a template narrative, selected from the collection of oral narratives in [21], and instructed the LLM to produce variations of the story which changed the surface form, but kept the overall structure (e.g. number of clauses). For these stories, we used the segmentation given by 21, wherein the narratives are segmented word-for-word into linguistic clauses.  Here is a sample of the prompt, with arguments shown enclosed in brackets, like  $\{ \ff{this}\}$.
[prompt:narr_gen]Narrative Generation

This is a true personal narrative about a single event in someone's life. It has exactly $\{ \ff{N} \}$ clauses:

{template narrative}

Generate a new personal narrative that is unique and about something completely different. Try to keep the overall narrative structure of the personal narrative above, but change as much of the subject matter and action as possible. Do not just use the narrative and replace key persons, places and things. Make it completely new. This new narrative must also contain exactly $\{ \ff{N} \}$ clauses.

As an example, we used the story of Jacob Schissel from [21] as a template narrative which has $\ff{N} = 18$:

[arg:schissel]Example template narrative
1. My brother put a knife in my head.

2. This was just a few days after my father had died

3. and we were sitting shiva.

4. And the reason the fight started,

5. he saw a rat out in the yard or

6.  – this was out in Coney Island –

7. and he started talk about it.

8. And my mother had just sat down to have a cup of coffee

9. and I told him to cut it out.

10.  'Course kids, y'know, he don't hafta listen to me.

11. So that's when I grabbed his arm

12. and twisted it up behind him.

13. When I let go his arm, there was a knife on the table,

14. he just picked it up

15. and he let me have it.

16. And . . .; I started bleeding – like a pig.

17. And naturally first thing to do, run to the doctor,

18. and the doctor just says, “Just about this much more,” he says, “and you'd a been dead.

The output of the LLM upon processing this prompt was a story which clearly bore a structural relation to the original, but contained different agents involved in a completely different scenario. We provide an example completion below obtained using $\ff{gpt-4}$ with temperature $T = 0.6$. This example was used in experiments, and also appears in <ref> next to another generated narrative ( <ref>).

[comp:story_gen_1]from <ref>, <ref>

1. My best friend pushed me into the pool.

2. This was right after I had broken up with my boyfriend

3. and I was feeling really down.

4. The reason she did it,

5. she thought I needed a distraction or

6. - this was at her house in the suburbs -

7. and she wanted to cheer me up.

8. I had just sat down with a plate of snacks

9. and I told her I was fine.

10. 'Course friends, y'know, they don't always believe you.

11. So that's when she sneaked up behind me

12. and gave me a little shove.

13. When I surfaced from the water, there was a float nearby,

14. she just grabbed it

15. and tossed it to me.

16. And . . .; I started laughing - uncontrollably.

17. And naturally the first thing to do, hug my friend,

18. and I just said, “Just a little more,” I said, “and I would've been drowning in my sorrows.

The resulting narratives generated from this procedure vary in their coherence. For our experiments, we chose the most coherent generated narratives.

Here is an example output which still sticks to the original structure, but at the expense of maintaining coherence and intelligibility:

[comp:story_gen_2]from Prompt <ref>

1. My sister crashed my brand-new car.

2. This happened on the day of my big job interview

3. and I was already running late.

4. The reason she took the car,

5. she wanted to surprise me with a good luck charm from our favorite store or

6. – this was in the downtown area –

7. and she thought it would help me.

8. I had just finished getting dressed in my best suit

9. and I asked her why she didn't just tell me.

10. 'Course siblings, y'know, they always think they know best.

11. So that's when I felt my heart sink

12. and panic set in.

13. When I stepped outside, there was a taxi passing by,

14. I just flagged it down

15. and hopped in.

16. And . . .; I made it to the interview – just in time.

17. And naturally, the first thing I did after, I called my sister,

18. and I told her, “Just a few minutes later,” I said, “and I would've missed the opportunity.

§.§ Lure Generation

We utilized gpt-4 at $T = 0.3$ with the following prompt:

[prompt:lure] Lure Generation

The items above all fit together to tell a story. Add more items of roughly the same length, numbered 1.5, 2.5, and so on, interleaving the existing items, elaborating on the story, and without repetition. These new items should introduce completely new plot elements, but still make sense in the context of the rest of the story. Add as many items as possible.

The following is an example completion using the template narrative segmentation (<ref>). For purposes of presentation, we present the lures along with the original clauses from the narrative. The completion does not include the original clauses, which appear in gray, and only consists of the text shown in black.

[comp:lure]from <ref>

1. My brother put a knife in my head.

1.5. It was a tense time for our family, with everyone on edge.

2. This was just a few days after my father had died

2.5. We were all grieving in our own ways, trying to cope with the loss.

3. and we were sitting shiva.

3.5. Our house was filled with relatives and friends, offering their condolences.

4. And the reason the fight started,

4.5. My brother had always been a bit of a troublemaker, but I never thought he'd go this far.

5. he saw a rat out in the yard or

5.5. He was always fascinated by the creatures that roamed the streets and alleys.

6.  – this was out in Coney Island –

6.5. The sun was setting, casting an eerie glow on the scene.

7. and he started talk about it.

7.5. He started describing the rat in gruesome detail, trying to get a rise out of us.

8. And my mother had just sat down to have a cup of coffee

8.5. She looked so tired and worn out, I just wanted her to have a moment of peace.

9. and I told him to cut it out.

9.5. I tried to keep my voice calm, but I could feel my anger rising.

10.  'Course kids, y'know, he don't hafta listen to me.

10.5. He just smirked at me, clearly enjoying the attention.

11. So that's when I grabbed his arm

11.5. I didn't want to hurt him, just get him to stop his nonsense.

12. and twisted it up behind him.

12.5. He let out a yelp of pain, but I didn't think I'd done any real damage.

13. When I let go his arm, there was a knife on the table,

13.5. I didn't even see him reach for it, he was so quick.

14. he just picked it up

14.5. I barely had time to react before the blade was coming towards me.

15. and he let me have it.

15.5. The pain was sharp and intense, like nothing I'd ever felt before.

16. And . . .; I started bleeding – like a pig.

16.5. I could feel the warmth of my blood soaking through my clothes.

17. And naturally first thing to do, run to the doctor,

17.5. I stumbled out of the house, desperate for help.

18. and the doctor just says, “Just about this much more,” he says, “and you'd a been dead.

18.5. I couldn't help but think how close I'd come to joining my father.

§.§ Recall Scoring

In order to score recall, we utilize zero-shot prompting with OpenAI's deprecated model gpt-4-0613. We chose to use this particular model for the benefit of reproducibility, since we have found that results from the following analysis can drift in time as OpenAI updates its model (see <Ref>).

The recall scoring <ref>, shown below, was a block of text, represented as a python string, which took as arguments the original narrative, the numbered segmentation of the narrative, and a single participant's recall of the narrative. The prompt was constructed so that it would identify which clauses from the narrative are present in the recall. A consequence of this scoring procedure is that a single clause in the recall can encode multiple clauses from the original narrative. gpt-4 appeared to have no trouble identifying such compressed clauses in the recall, and frequently would identify multiple original clauses within a single clause in the participant's recall. An example of this can be seen in <ref>, where the same passage in the recall  (“even the scoutmaster looked on") is counted as recalling both clauses 15 and 16 from the original narrative (<ref>).

Below, we provide the scoring prompt, as well as some example arguments, and the resulting LLM completion.

[prompt:scoring]Recall Scoring

This is the original text:

It can be broken down into the following independent pieces of information:

Here is an alternative version of the original text where some of the above pieces of information may be missing:

For each of the numbered information pieces of the list above, evaluate whether the information of each piece is given in the alternative version of the story, stating the number and showing the corresponding passage from the alternative story it is given in. After, write all the numbers of the pieces that are given in the alternative version of the story in a set of brackets at the end of the response.

[arg:boyscout]Example narrative

Yeah, I was in the boy scouts at the time. And we was doing the 50-yard dash, racing, but we was at the pier, marked off and so we was doing the 50-yard dash. There was about 8 or 9 of us, you know, going down, coming back. And going down the third time, I caught cramps and I started yelling "Help!", but the fellows didn't believe me, you know. They thought I was just trying to catch up, because I was going on or slowing down. So all of them kept going. They leave me. And so I started going down. Scoutmaster was up there. He was watching me. But he didn't pay me no attention either. And for no reason at all there was another guy, who had just walked up that minute... He just jumped over and grabbed me.

[arg:boyscout_seg]Example segmentation

1. Yeah, I was in the boy scouts at the time.

2. And we was doing the 50-yard dash,

3. racing,

4. but we was at the pier, marked off,

5. and so we was doing the 50-yard dash.

6. There was about 8 or 9 of us, you know, going down, coming back.

7. And, going down the third time, I caught cramps

8. and I started yelling "Help!",

9. but the fellows didn't believe me, you know.

10. They thought I was just trying to catch up, because I was going on or slowing down.

11. So all of them kept going.

12. They leave me.

13. And so I started going down.

14. Scoutmaster was up there.

15. He was watching me.

16. But he didn't pay me no attention either.

17. And for no reason at all there was another guy, who had just walked up that minute...

18. He just jumped over

19. and grabbed me.

[arg:boyscout_recall]Example recall

A boy was with the scoutmaster and he was going up and down and on the third time of going up the pier he had cramps and shouted 'help' but no-one believed him, even the scoutmaster looked on by until this guy came out of nowhere and grabbed the boy and saved him.

[comp:boyscout_score]from <ref>

1. Not given

2. Not given

3. Not given

4. Not given

5. Not given

6. Not given

7. Given - "on the third time of going up the pier he had cramps"

8. Given - "and shouted 'help'"

9. Given - "but no-one believed him"

10. Not given

11. Not given

12. Not given

13. Not given

14. Given - "was with the scoutmaster"

15. Given - "even the scoutmaster looked on"

16. Given - "even the scoutmaster looked on"

17. Given - "until this guy came out of nowhere"

18. Not given

19. Given - "and grabbed the boy and saved him"

(7, 8, 9, 14, 15, 16, 17, 19)

We note here that the completion does not always appear as in this example. Some minor deviations might occur (e.g. parentheses replaced with brackets), or the format of the answers change (e.g. instead of “Not Given", some completions give "There is no corresponding passage for ....").

§.§ Ordered Recall Scoring

In order to extract the order in which the original clauses appeared in a participant's recall, we added an extra step to the recall scoring discussed above. For a given narrative, segmentation, and recall, we concatenated <ref> with its <ref>, and appended to this the following additional prompt:

[prompt:order]Ordered Recall Scoring
Now repeat the alternative version of the narrative with the number of the independent piece of information inserted next to the location in which it appears in the alternative version. Then, list the numbers separately in the order in which they appear in the alternative story immediately above. The final list of numbers should be enclosed in parentheses.

Therefore, the total input or prompt for ordered recall scoring would be <ref> + <ref> + <ref>. The output for this example was the following:

[comp:boyscout_order]from <ref> +  <ref> + <ref>

A boy (14) was with the scoutmaster and he was going up and down and on the third time of going up the pier he had cramps (7) and shouted 'help' (8) but no-one believed him (9), even the scoutmaster looked on (15, 16) by until this guy came out of nowhere (17) and grabbed the boy and saved him (19).

(14, 7, 8, 9, 15, 16, 17, 19)

§ RELIABILITY OF LLM SCORING

Here we report additional details of the reliability for LLM narrative scoring.

In scoring the recalls, the human scorers (who are the authors on this paper) did not develop overly sophisticate scoring protocols. The directions were intuitive, essentially boiling down to “did this clause from the original story appear in the recalled story". Despite the ambiguity of this procedure, there was very high inter-scorer correlation.

A comparison between gpt-4, mean human ($\bar{h}$), and individual human scorers ($h_{1}$, $h_{2}$, $h_{3}$). Each entry in the table gives the correlation coefficient (r-value) between the corresponding row and column label.

r        $\bar{h}$         $h_{1}$     $h_{2}$     $h_{3}$
gpt-4-0613      0.94     0.92     0.90     0.90

$\bar{ h}$                 0.95     0.98     0.97

$h_{1}$                        0.89     0.86

$h_{2}$                             0.96

§.§ Drift in Performance over time

The models offered through the OpenAI API are known to undergo regular updates. We found that within the span of two months, the correlation coefficient with mean human performance rose by $0.05$. We compare gpt-4 called via OpenAI's API on 05/23/23, the same model gpt-4 called less than two months later on 07/03/23, with the deprecated model gpt-4-0613 (called after 06/13/23). Fortunately, the frozen model gpt-4-0613, which we use throughout the paper, is essentially as good as the most recent version.

A comparison on gpt-4 scoring compiled at different times of the year. Evidently, the model improves over time in approximating the mean human scoring, with the most recent model achieving a correlation closest to one. The entries of the table show correlation coefficients (r-values) between the corresponding row and column variables.

r      gpt-4 (05/23/23)      gpt-4-0613         gpt-4 (07/03/23)
$\bar{ h}$       0.89         0.937     0.944

gpt-4 (05/23/23)                 0.94      0.92

gpt-4-0613                        0.99

§.§ Comparison between different LLMs

We compared different LLMs available through the OpenAI API. In order of increasing number of learned parameters (model size), these were GPT-3 (text-davinci-003), GPT-3.5 (or ChatGPT, gpt-3.5-turbo) and GPT-4 (gpt-4-0613). The same scoring <ref> was used for all of these models. The results are visualized in <Ref>. We find that GPT-4 is qualitatively better than the smaller models (compare correlation between human and LLM scoring in <Ref>B). Furthermore, increasing size does not seem to be sufficient to increase performance, as illustrated by the severe drop in performance by GPT-3.5. As seen in <Ref>(A,B), GPT-3.5 appears to score more generously than the other models, resulting in a systematic upward bias of nearly all recall probabilities. It is possible that this reflects the sensitivity of these models to prompts, and that with appropriate tuning of the scoring prompt, the performance of GPT-3.5 could be improved.

Reliability of different LLMs for scoring narratives We observed that performance on human recall scoring was not monotonic in the size of the LLM. (A) shows $P_{rec}$ for each clause as computed by LLMs of three different sizes, in order of increasing size: GPT-3 (OpenAI's API model text-davinci-003), GPT-3.5 (or ChatGPT, model gpt-3.5-turbo), and GPT-4 (gpt-4-0613). GPT-3.5 scores appear systematically biased to be higher than the mean human scores, whereas GPT-3 roughly follows the human trend. (B) shows correlations between these models and mean human performance. (C) shows inter-LLM correlations.

§ COMPRESSED RECALL IN LONGER STORIES

Each recall can be split up into a number of clauses. We denote the mean number of clauses used in recalls by $C$. <Ref> compares $C$ to the mean number of recalled clauses, defined previously and denoted by $R$. If every clause used by the participant were recalling a single clause from the original story, then the points would lie on a straight line, corresponding to $C = R$. For shorter stories, we find in fact that $C> R$, and people appear to use more clauses than necessary. However, for the longest story ($L = 130$), there is a marked compression, in which participants on average use $75\%$ as many clauses as they are scored to have recalled.

We used <ref> with GPT-4 at $T = 0$ to segment recalls. An example of this applied to <ref> is given in <ref>.

Compression of Recalls. The vertical axis shows the mean number of clauses in a population of recalls, determined using the segmentation prompt given in this section. The horizontal axis gives $R$, the mean number of recalled clauses.

Segmentation Prompt
Provide a word-for-word segmentation of the following narrative into linguistic clauses, numbered in order of appearance in the narrative:

[comp:seg_comp]<ref> using  <ref>
1. A boy was with the scoutmaster

2. and he was going up and down

3. and on the third time of going up the pier

4. he had cramps

5. and shouted 'help'

6. but no-one believed him,

7. even the scoutmaster looked on

8. by until this guy came out of nowhere

9. and grabbed the boy

10. and saved him.

§ LACK OF OUTPUT INTERFERENCE IN RECOGNITION EXPERIMENTS

Similar to [8], we measure the strength of output interference in recognition experiments by tracking the $d'$ measure as a function trial number. The discrimination measure used here is conventionally called simply $d'$ and is given by the difference in z-score between the hit probability $P_{h}$ (i.e. true positive rate) and the false alarm probability $P_{f}$ (i.e. false positive rate ). The z-transform is defined

\begin{align}
z(p) = \sqrt{2} {\rm erf}^{-1}\left( 2 p-1\right),
\end{align}

and the discrimination measure is

\begin{align}
d' = z(P_{h}) - z(P_{f}).
\end{align}

We plot $d'$ over the course of the recognition trials in <Ref> separately for coherent stories and scrambled stories. Output interference is characterized by a $d'$ which decreases with trial number, indicating a diminishing ability to discriminate new vs old items.

Discrimination measure shows good memory for true clauses which does not change significantly as the recognition trial progresses.

§ RECALL PROBABILITY CURVES AND DISTRIBUTIONS

Here we plot recall probability per clause as a function of the serial position of presentation. We refer to these below as recall probability curves. We compare recall probability curves for narratives to data obtained for random word lists. <Ref> a shows the recall probability curve for coherent (blue), scrambled (red) stories, alongside the recall probability curve for free recall of a random word list (black dashed). Importantly, for the scrambled recall, the serial position corresponds to the position of presentation. Two main points can be gleaned from these figures. First, whereas for random lists there is a pronounced primacy effect, we see no such structure in both coherent and scrambled stories. In fact, highly recalled clauses appear at all positions in the story. Another observation is that the distribution of $P_{rec}$ for clauses in a narrative covers a much broader range than $P_{rec}$ for words in a list. This can be seen by looking at the cumulative distribution function of $F(p) = P\left(P_{rec} > p\right)$, which is the probability to find $P_{rec} > p$. As shown in <Ref>A, $F(p)$ drops to zero around $p \approx 0.6$ for random lists, while showing a more gradual decay for coherent narratives. In contrast, <Ref>B shows that recall is significantly impaired for scrambled stories, with $F(p)$ showing a more precipitous fall to zero similar to random word lists.

However, the recall probability curves for scrambled stories reveals a more subtle effect. While the overall distribution of recall probabilities may be similar to (if not worse than) random lists, there are still spikes in $P_{rec}$ throughout the bulk of the narrative, suggesting that these texts are not processed as if they were a random list of clauses. In <Ref>, we replot the recall of only the scrambled stories from <Ref> according the serial position of the clause in the coherent story. Below this, we show the correlation between recall probability of a clause appearing in the coherent story versus the scrambled version. While low, the correlations are positive and statistically significant, suggesting that participants are able to identify memorable clauses even in the scrambled scenario, and use this in attempting to construct a coherent recall. This active process of selection might explain why average recall of clauses for scrambled narratives appears worse than recall of words in random lists (i.e. why the scrambled narrative data (orange triangles) lie below the square-root scaling (dashed gray) in <Ref>).

Recall probability per item for narratives and random lists of words. (A) shows the recall probability per clause as a function of the serial position of presentation for a coherent narrative (blue) and the scrambled version of this same narrative (red). As a comparison, we plot the recall probability per word as a function of serial position of presentation (black dashed). While the recall probability for the list shows a marked primacy effect, with a general suppression of recall in the bulk of the list, the recall of coherent narratives shows many spikes in $P_{rec}$ throughout the narrative. And while the scrambled stories have overall much lower recall probabilities, they still show large peaks within the bulk of the presentation. (B) and (C) compare coherent and scrambled narratives of increasing length.

Cumulative distribution of recall probabilities (A) As compared to random lists (purple dashed), coherent narratives have a slower decline in the cumulative distribution function, indicating a larger number of clauses with a higher recall probability. (B) shows this behavior is reversed for scrambled stories, where the recall probability of clauses appears to uniformly drop, in some cases well below the corresponding distribution for recall of a random list.

Descrambling of narratives in recall. (A-C) show recall probability as a function of the serial position in the coherent story. Note that for the scrambled $P_{rec}$ (red curves), this is not the presentation order, but rather the original order of the clauses. Plotting the curves in this way, we observe the spikes in recall probability appear to coincide between coherent and scrambled recall. Below, we show correlations between $P_{rec}$ for a given clause in the coherent and scrambled presentation.

§ CLAUSES WITH HIGH RECALL PROBABILITY

We noted that clauses recalled with higher probability tend to encapsulate a summarized version of the presented narrative. While not all nuances are retained, they collectively outline the principal storyline. Presented below are examples from stories of varying lengths. It's important to note that we present the recalled clauses exactly as they appeared in the original story and maintain the same sequence.

[ex:schissel_v1-pool]Highly recalled clauses for narrative of length 18 clauses

Original Story

* My best friend pushed me into the pool.

* This happened during my cousin's wedding reception,

* where everyone was dressed to impress.

* And the reason it happened,

* she spotted a bee hovering near my face,

* - this was at a fancy hotel garden -

* and she tried to save me from being stung.

* And my aunt had just handed me a glass of champagne,

* and I warned her to be careful.

* 'Course friends, y'know, they don't always think things through.

* So that's when she gave me a little shove,

* and I tumbled into the water.

* When I resurfaced, gasping for air,

* she just started laughing,

* and she apologized profusely.

* And... my beautiful dress was ruined,

* and naturally, the first thing to do was to get out and dry off,

* and my cousin just says, "Just about a few inches more," she says, "and you'd have landed on the cake."

Highly recalled clauses

0.857  1. My best friend pushed me into the pool.

0.893  2. This happened during my cousin's wedding reception,

0.893  5. she spotted a bee hovering near my face,

0.661  16. And... my beautiful dress was ruined,

0.857  18. and my cousin just says, "Just about a few inches more," she says, "and you'd have landed on the cake."

[ex:shester_v2-church]Highly recalled clauses for narrative of length 54 clauses

Original Story

* Let me share a story from when I was twenty-two years old, just fresh out of college

* I was staying in a small town about five miles from here

* I had just started my first job at a local bookstore

* It was a quiet afternoon, and I was alone in the store

* My grandpa, who had been bedridden for almost a year, was at home with my sister

* She had to care for him daily, as he couldn't move around on his own

* There were only two of us siblings, my sister and me

* My parents had gone out of town for the weekend

* As I was shelving books, I suddenly heard a voice inside my head

* It said, "If you go to the old church down the street and light a candle for your grandpa, he will get better

* I looked around the bookstore, but no one was there

* I continued with my work, trying to ignore the voice

* It spoke to me again, repeating the same message

* I looked around once more, but still found no one

* I decided to take a break and walk to the church

* I was only a short distance away, maybe a quarter-mile

* As I approached the church, I hesitated for a moment

* I was not a particularly religious person

* I remembered my grandma always saying that prayers from a sincere heart could bring miracles

* On the other hand, she also said that prayers without faith were useless

* I admitted to myself that I was far from being a saint

* I had my share of mistakes and misdeeds

* I thought about my grandpa and how much he meant to our family

* Finally, I went inside the church and found a candle

* I knelt down in front of the altar

* I said, "God, I don't know how to pray properly

* But I am asking you with all my heart to heal my grandpa

* Please help him regain his strength and guide us in taking care of him

* After saying my prayer, I returned to the bookstore

* I resumed my work, shelving books

* Suddenly, I felt a strong urge to go home

* It was as if something was pulling me towards the house

* I didn't hear any voice this time

* It was just an overwhelming feeling that I needed to go

* I locked up the store and rushed home

* To my surprise, I found my grandpa sitting in the living room, talking to my sister

* He hadn't been able to leave his bed for months without assistance

* My sister told me that he had suddenly felt a surge of energy and was able to stand up and walk

* That day marked the beginning of his recovery

* He gradually regained his strength and independence

* It was a turning point in our lives

* We became more grateful for the time we had together

* I often thought back to that day in the church

* I wondered if my prayer had anything to do with my grandpa's recovery

* I couldn't be sure, but it definitely changed my perspective on faith and the power of prayer

* My grandpa lived for many more years after that incident

* He celebrated his ninetieth birthday surrounded by family and friends

* We cherished every moment we had with him

* That experience happened almost twenty years ago

* It remains a vivid memory in my mind

* My grandpa passed away a few years later, but his spirit lives on in our hearts

* Sometimes, when I'm feeling lost or uncertain, I think back to that day

* I remember the power of faith and the miracles it can bring

* And I am reminded that we are never truly alone, even when it seems like we are

Highly recalled clauses

0.710  1. Let me share a story from when I was twenty-two years old, just fresh out of college.

0.830  3. I had just started my first job at a local bookstore.

0.850  5. My grandpa, who had been bedridden for almost a year, was at home with my sister.

0.860  9. As I was shelving books, I suddenly heard a voice inside my head.

0.920  10. It said, "If you go to the old church down the street and light a candle for your grandpa, he will get better."

0.720  15. I decided to take a break and walk to the church.

0.700  31. Suddenly, I felt a strong urge to go home.

0.900  36. To my surprise, I found my grandpa sitting in the living room, talking to my sister.

0.780  46. My grandpa lived for many more years after that incident.

§ NARRATIVES USED IN THE STUDY

The narratives that were generated by GPT-4 for this study used an existing segmentation into clauses. The original segmentation was taken from [22, 21]. These segmentations were provided to the LLM to generate new variants, as detailed in <ref>. Below, we present all narratives that were presented to participants in our experiments. In the examples, the narratives are labeled by a prefix reflecting the source story, with the exception of “panic attack" which was created by GPT-3.
The scrambled narratives are based on intact narrative (without “-scrambled” in the narrative name) where all clauses were shuffled. The “panic” story was generated based on the “boyscout” story from [22], <ref> below. The story was generated using <ref>.

<ref> and <ref> below were generated using <ref>, with the Jacob Schissel story from [21] as a template.

[ex:schissel-v1-pool]Narrative schissel-v1-pool

* My best friend pushed me into the pool.

* This happened during my cousin's wedding reception,

* where everyone was dressed to impress.

* And the reason it happened,

* she spotted a bee hovering near my face,

* - this was at a fancy hotel garden -

* and she tried to save me from being stung.

* And my aunt had just handed me a glass of champagne,

* and I warned her to be careful.

* 'Course friends, y'know, they don't always think things through.

* So that's when she gave me a little shove,

* and I tumbled into the water.

* When I resurfaced, gasping for air,

* she just started laughing,

* and she apologized profusely.

* And... my beautiful dress was ruined,

* and naturally, the first thing to do was to get out and dry off,

* and my cousin just says, "Just about a few inches more," she says, "and you'd have landed on the cake."

[ex:schissel-v2-lake]Narrative schissel-v2-lake

* My best friend pushed me into the lake.

* This happened during our annual summer camping trip

* and we were enjoying a peaceful afternoon.

* The reason it all began,

* she spotted a frog near the water's edge or

* - this was at our favorite spot in the woods -

* and she dared me to catch it.

* I had just sat down to eat my sandwich

* and I told her I wasn't interested.

* 'Course friends, y'know, they don't always take no for an answer.

* So that's when she snuck up behind me

* and gave me a little nudge.

* When I emerged from the water, there was a frog on the shore,

* she just scooped it up

* and handed it to me.

* And . . .; I couldn't help but laugh – uncontrollably.

* And naturally, the first thing to do was to get revenge,

* and I said, "Just you wait," I said, "you'll get yours soon enough."

<ref> and <ref> below were generated using <ref>, with the Charles Triplett story from [21] as a template.

[ex:triplett-v1-rookie]Narrative triplett-v1-rookie

* Back in the days when I was a rookie in the police force, the Chief was a veteran.

* He had caught a notorious criminal –

* or had been awarded for bravery.

* But he had – a young daughter.

* And in those days I was a fitness enthusiast.

* And it seemed she was trying to impress me.

* I never observed it.

* Truth is I wasn't very fond of her because she was –

* She was a charming girl until she opened her mouth.

* She was a chatterbox.

* Goodness, she could talk.

* Then she left a message one day saying she was going to run away because he was always scolding her about me.

* He arrived at my apartment.

* Big intimidating look on his face.

* I managed to calm him down.

* and proposed “Well we'll search for her

* and if we can't locate her well you can – do whatever you think is right.”

* I was strategizing.

* So he accepted my offer.

* And we went to where they found her scarf – near a park.

* and we traced down a little more

* and we couldn't locate her.

* And returned

* – it was a police station –

* she was sitting on a chair with a book in her hands.

* She hadn't run away.

* But – nevertheless – that resolved the issue for that day.

* But that evening the deputy, Frank Mitchell, said “You better transfer and leave because that old man never forgets anything once he gets it into his mind.”

* And I did.

* I transferred.

* and left.

* That was the end of my rookie year.

[ex:triplett-v2-catlady]Narrative triplett-v2-catlady

* In the neighborhood I grew up in, Mrs. Baker was the resident cat lady.

* She had adopted at least twenty –

* or had given them homes.

* But she had – a young grandson

* and back then I was quite the tomboy.

* And apparently, he was trying to befriend me.

* I never caught on.

* Truth is I didn't take to him much because he had –

* He was a charming lad until you saw his manners.

* He had poor manners.

* Goodness gracious he had poor manners.

* Then he left a note one day saying he was going to run away because she was always nagging about me.

* She came to my treehouse.

* Big old tabby cat in tow.

* I talked her down

* and said “Well we'll go find him

* and if we can't locate him, you can – go ahead and call the police if you need to.”

* I was strategizing.

* So she took my advice.

* And we went to where they found his baseball cap – near the old mill

* and we trailed a bit more

* and we didn't find a trace.

* And returned

* – it was a treehouse club –

* he was sitting on a branch with a comic book in his hand.

* He hadn't run away.

* But – however – that resolved it for that day.

* But that evening the local sheriff, Officer Dawson said “You better lay low and stay out of sight because that woman never forgets anything once she gets it into her mind.”

* And I did.

* I laid low

* and stayed out of sight.

* That was the second time.

<ref> and <ref> below were generated using <ref>, with the Adolphus Hester story from [21] as a template.

[ex:hester-v1-park]Narrative hester-v1-park

* Let me share a story from when I was twenty-one years old

* I was living in a small apartment near the city center

* I had just started my first job

* And uh I was working as a graphic designer

* My father, who was recovering from a surgery, hadn't been out in – in uh – well I'd say about eight months

* He had to use a wheelchair to get around

* The only person there with him during the day was a kind neighbor, a lady in her sixties

* My younger sister was away at college –

* so it was just my dad and the neighbor lady at home during the day

* One day, I got this strong feeling while I was at work

* It told me: "If you go to the nearby park during your lunch break and pray for your father's recovery, he'll start to improve

* I looked around my office

* I didn't see anyone nearby

* So, I went to the park during my break

* and found a quiet spot there

* That feeling came back again

* I looked around once more

* I still didn't see anyone nearby

* but I decided to take a chance

* I wasn't too far from my office, maybe a ten-minute walk

* I found a bench to sit on and closed my eyes – uh – uh –

* that feeling came back once more

* I just took a deep breath

* I was sitting under a big oak tree now

* I looked up at the branches above me

* Nobody was around

* And uh I said to myself, well now,

* I had been taught that prayers can make a difference

* And that it doesn't hurt to try

* I knew I wasn't perfect, but I wanted to help my dad

* I'd admit that to anyone

* But I was willing to give it a try

* I looked around one more time

* and then I closed my eyes

* and said a quiet prayer

* I whispered, "God, I don't know if you're listening

* But I ask you to please help my father recover

* So he can regain his independence and enjoy life again

* I opened my eyes

* and went back to work

* Suddenly, I received a text message from my father

* It said, "I just took a few steps with my walker

* I – I couldn't believe it

* It felt like a sign that my prayer had been heard

* I rushed home after work

* and saw my father standing in the living room with the help of his walker

* He hadn't been able to do that in months without assistance

* And there he was, taking small steps

* That was the turning point in his recovery

* He continued to improve

* and regained most of his mobility within a year

* It's been, what, six or seven years since then?

* My father is now able to walk without any assistance

* He's living life to the fullest, even in his seventies

[ex:hester-v2-church]Narrative hester-v2-church

* Let me share a story from when I was twenty-two years old, just fresh out of college.

* I was staying in a small town about five miles from here.

* I had just started my first job at a local bookstore.

* It was a quiet afternoon, and I was alone in the store.

* My grandpa, who had been bedridden for almost a year, was at home with my sister.

* She had to care for him daily, as he couldn't move around on his own.

* There were only two of us siblings, my sister and me.

* My parents had gone out of town for the weekend.

* As I was shelving books, I suddenly heard a voice inside my head.

* It said, "If you go to the old church down the street and light a candle for your grandpa, he will get better."

* I looked around the bookstore, but no one was there.

* I continued with my work, trying to ignore the voice.

* It spoke to me again, repeating the same message.

* I looked around once more, but still found no one.

* I decided to take a break and walk to the church.

* I was only a short distance away, maybe a quarter-mile.

* As I approached the church, I hesitated for a moment.

* I was not a particularly religious person.

* I remembered my grandma always saying that prayers from a sincere heart could bring miracles.

* On the other hand, she also said that prayers without faith were useless.

* I admitted to myself that I was far from being a saint.

* I had my share of mistakes and misdeeds.

* I thought about my grandpa and how much he meant to our family.

* Finally, I went inside the church and found a candle.

* I knelt down in front of the altar.

* I said, "God, I don't know how to pray properly.

* But I am asking you with all my heart to heal my grandpa.

* Please help him regain his strength and guide us in taking care of him."

* After saying my prayer, I returned to the bookstore.

* I resumed my work, shelving books.

* Suddenly, I felt a strong urge to go home.

* It was as if something was pulling me towards the house.

* I didn't hear any voice this time.

* It was just an overwhelming feeling that I needed to go.

* I locked up the store and rushed home.

* To my surprise, I found my grandpa sitting in the living room, talking to my sister.

* He hadn't been able to leave his bed for months without assistance.

* My sister told me that he had suddenly felt a surge of energy and was able to stand up and walk.

* That day marked the beginning of his recovery.

* He gradually regained his strength and independence.

* It was a turning point in our lives.

* We became more grateful for the time we had together.

* I often thought back to that day in the church.

* I wondered if my prayer had anything to do with my grandpa's recovery.

* I couldn't be sure, but it definitely changed my perspective on faith and the power of prayer.

* My grandpa lived for many more years after that incident.

* He celebrated his ninetieth birthday surrounded by family and friends.

* We cherished every moment we had with him.

* That experience happened almost twenty years ago.

* It remains a vivid memory in my mind.

* My grandpa passed away a few years later, but his spirit lives on in our hearts.

* Sometimes, when I'm feeling lost or uncertain, I think back to that day.

* I remember the power of faith and the miracles it can bring.

* And I am reminded that we are never truly alone, even when it seems like we are.

<ref> below is taken from the story of Gloria Stein (Ch. 7) in [21].
[ex:stein]Narrative stein

* We moved down to Third and Wharton

* We had gotten moved

* but it was like – you remember I told you my father drank,

* but it was still day,

* and after he had gotten in and put the furniture where it was supposed to go, my mother left, to go to market to get some food to bring into the house

* And my father went to take the horses – take the team of horses back

* But he didn't come home

* He – stopped somewhere on – I dunno,

* but he – how he – why he wasn't in the house

* And of course I was left there with the kids

* Well no sooner did my mother get out of the house – my father went first – then – oh it was such an awful crash came through the window

* And I – and I looked and it was a big stone

* And then of course they – they – well they stoned the house

* And I had sense enough to get the kids into the stairway so that they wouldn't get hurt by flyin' glass

* And I said to them, “Now you stay there

* Don't move 'cause nothing will reach you here

* And I'm gonna go and see if I can find mom” –

* (We call our mother mom

* And they were crying

* and carryin' on

* I was scared to death,

* I really was frightened out of my wits

* And I didn't know what to do,

* but I remember saying a prayer as I went out the front door because I didn't know what was gonna hit me when I went out that door

* And I ran –

* I was ten years old at the time this happened –

* I ran to Fourth street, because I knew my mother wa-

* They had push carts in those days, on Fourth street

* And it was a Jewish neighborhood

* and they – they stayed open until – until all hours

* Because they would, you know, however much money they could get they wanted

* So uh that's where she was going to do her shopping

* And I finally found my mother

* I did find her

* And while I was trying to tell her what had happened

* a policeman went by

* And my voice was such that you could hardly make out what I was saying

* I was so nervous and – and – my voice was shaking so,

* and he wanted to find out what ha- what was wrong with me

* And then when he understood what the situation was he turned to my mother

* and says, “This is a bad neighborhood

* It's Irish

* And he said, “They're known to do things like that,

* Better move out of there

* My mother said, “I just moved in, how am I gonna move out?”

* He says, “I dunno

* He says, “But they don't want any black people in there

* An' he didn't say black 'cause they didn't use black then

* But “they don't want anybody in there

* So uh, she said, “Well you're the law

* You're supposed to protect us

* He says, “I can't do anything with it

* He was Irish himself

* “I can't do anything about it

* So my mother said um, “Where is the uh precinct you work from?”

* And he told her

* I forgot now

* I was right there with her

* but I've forgotten where he told her

* So uh she said, “Well if you can't do anything to help me, I'll go see who – if I can't find someone else to help me

* Well in those days black people were so cowed by circumstances an' uniforms and white people, you know, they uh many times didn't assert themselves

* But thankfully my mother was not one of those kind

* And she – well I always thought of her as an educated person which she was because she was the only one I knew who had completed high school

* She went to Lackawanna high school

* And this was – let me see what year was this happening – well it must have been nineteen twenty-five, so you can see what – what the times were

* And we went – she took me, and we went to the police station

* And they also had police wagons with horses attached to them

* So my mother said, to the captain, “What are you going to do about it?”

* And he says, “Well lady there's nothing we can do about it

* And she says, “You mean to tell me that I have to live under those circumstances and there's nothing that you can do about it?”

* She says – he says, “Well uh the only thing I can tell you to do is to move

* He says, “Because” – he said to me, “Did you see anyone?”

* And I said, “No

* He says, “Well how can – how can you” – to my mother – “how can you do anything if you don't know who – who to put the blame on?”

* An' he says, “I'm sure there was more than one person

* So my mother said, “Well I don't know what I'm going to do

* She says, “But one thing I know I'm not going to do is move

* They took all we had to, to pay the rent

* In that house you had to pay in advance

* And she said, “If they didn't want me there they should not have rented the house to me

* Now the real estate agent should be accountable

* So anyhow, she said to him, “Well I know one thing you can do

* And he says, “What's that?”

* She says, “You can take me home

* He said, “Take you home?”

* She said, “Yes in the police patrol

* So – I guess they figured “Let's get this lady out of here because she's talking loud and long

* And they had never come up with anybody like that

* So they put us in the police patrol

* and they took us home

* We weren't far from – from where it was – maybe Fifth street – from where we had moved

* And – course with the horses and the police patrol, and the clang –

* they had a bell that clanged

* you know they told you about the trolley that went “Clang

* Well this police patrol that bell they had, it went “Clang clang clang

* And um all the neighbors came running out to see what was happening

* And of course they saw the man came around the back –

* I can see it right now – from where he was sitting up front there where the horses came around,

* and opened the door in the back

* And they had a long door in the back where you had to come down steps to get out the back of the – of this police patrol

* Well we got out of that police patrol –

* Before the patrol ever left my mother turned

* and faced all those people that were standing around,

* and she says, “Why you're doing this to me I don't know

* She says, “I know that I am different from you, color wise,

* but in every other respect I'm just the same as you are,

* and all I'm trying to do is provide a home for my family

* She told them that they had rented the house,

* she told them how much money she had paid,

* she had told them that was all the money she had

* And they had paid that rent for a month

* She knew she was going to be there for the month

* She did not know beyond the month

* Whether they'd have enough money to pay it for the next month

* But she knew they were going to be there for the month

* She said, “It is winter time and it is cold

* And I cannot afford to be buying windows

* “Now” she said, “Whoever threw the stones I don't know

* But you can all see what has happened

* And now I'm telling you that we're going to be here for a month

* So you might just as well make up your mind to that

* If you don't want me here then you people get together and give me enough money to move somewhere else, because I do not have it

* Well of course you know they didn't do that

* They probably didn't have any money either

* So in the meantime – not in the meantime but during, during the period of time that we were there, my father became sick

* And the neighbors were as nice as anybody could ever have been

* When there was trouble they responded just like anyone else

* But that was our welcoming to the neighborhood

* And it was terrible

* and it certainly was frightening for me

Below, <ref> is taken directly from [22]. <ref> is the scrambled version used in experiments.

[ex:boyscout]Narrative boyscout

* Yeah, I was in the boy scouts at the time.

* And we was doing the 50-yard dash,

* racing,

* but we was at the pier, marked off,

* and so we was doing the 50-yard dash.

* There was about 8 or 9 of us, you know, going down, coming back.

* And, going down the third time, I caught cramps

* and I started yelling "Help!",

* but the fellows didn't believe me, you know.

* They thought I was just trying to catch up, because I was going on or slowing down.

* So all of them kept going.

* They leave me.

* And so I started going down.

* Scoutmaster was up there.

* He was watching me.

* But he didn't pay me no attention either.

* And for no reason at all there was another guy, who had just walked up that minute...

* He just jumped over

* and grabbed me.

[ex:boyscout-scrambled]Narrative boyscout-scrambled

* They leave me.

* racing,

* Yeah, I was in the boy scouts at the time.

* but we was at the pier, marked off,

* and so we was doing the 50-yard dash.

* And we was doing the 50-yard dash,

* but the fellows didn't believe me, you know.

* and I started yelling "Help!",

* and grabbed me.

* Scoutmaster was up there.

* So all of them kept going.

* And so I started going down.

* There was about 8 or 9 of us, you know, going down, coming back.

* He just jumped over

* They thought I was just trying to catch up, because I was going on or slowing down.

* And, going down the third time, I caught cramps

* And for no reason at all there was another guy, who had just walked up that minute...

* He was watching me.

* But he didn't pay me no attention either.

<ref>, <ref>, and <ref> are the scrambled versions of <ref>,<ref>, and <ref>, respectively. The scrambled is done by taking a random permutation of the clauses. The numbering in the scrambled versions reflects the presentation order.
[ex:triplett-v1-rookie-scrambled]Narrative triplett-v1-rookie-scrambled

* and we traced down a little more

* and left.

* And I did.

* I never observed it.

* I was strategizing.

* Big intimidating look on his face.

* and proposed “Well we'll search for her

* Back in the days when I was a rookie in the police force, the Chief was a veteran.

* and we couldn't locate her.

* And in those days I was a fitness enthusiast.

* And we went to where they found her scarf – near a park.

* But – nevertheless – that resolved the issue for that day.

* That was the end of my rookie year.

* And it seemed she was trying to impress me.

* or had been awarded for bravery.

* He had caught a notorious criminal –

* She hadn't run away.

* She was a chatterbox.

* I managed to calm him down.

* – it was a police station –

* And returned

* Truth is I wasn't very fond of her because she was –

* Then she left a message one day saying she was going to run away because he was always scolding her about me.

* But that evening the deputy, Frank Mitchell, said “You better transfer and leave because that old man never forgets anything once he gets it into his mind.”

* I transferred.

* But he had – a young daughter.

* Goodness, she could talk.

* So he accepted my offer.

* She was a charming girl until she opened her mouth.

* she was sitting on a chair with a book in her hands.

* He arrived at my apartment.

* and if we can't locate her well you can – do whatever you think is right.”

[ex:hester-v2-church-scrambled]Narrative hester-v2-church-scrambled

* I remembered my grandma always saying that prayers from a sincere heart could bring miracles

* Finally, I went inside the church and found a candle

* As I was shelving books, I suddenly heard a voice inside my head

* I remember the power of faith and the miracles it can bring

* It was a turning point in our lives

* That experience happened almost twenty years ago

* I was only a short distance away, maybe a quarter-mile

* My grandpa lived for many more years after that incident

* I said, "God, I don't know how to pray properly

* And I am reminded that we are never truly alone, even when it seems like we are

* It spoke to me again, repeating the same message

* I was not a particularly religious person

* Suddenly, I felt a strong urge to go home

* I looked around the bookstore, but no one was there

* I was staying in a small town about five miles from here

* My sister told me that he had suddenly felt a surge of energy and was able to stand up and walk

* Sometimes, when I'm feeling lost or uncertain, I think back to that day

* I wondered if my prayer had anything to do with my grandpa's recovery

* Let me share a story from when I was twenty-two years old, just fresh out of college

* My parents had gone out of town for the weekend

* I locked up the store and rushed home

* It was as if something was pulling me towards the house

* I admitted to myself that I was far from being a saint

* I had just started my first job at a local bookstore

* As I approached the church, I hesitated for a moment

* She had to care for him daily, as he couldn't move around on his own

* It was a quiet afternoon, and I was alone in the store

* We cherished every moment we had with him

* My grandpa, who had been bedridden for almost a year, was at home with my sister

* I had my share of mistakes and misdeeds

* He hadn't been able to leave his bed for months without assistance

* There were only two of us siblings, my sister and me

* I didn't hear any voice this time

* But I am asking you with all my heart to heal my grandpa

* My grandpa passed away a few years later, but his spirit lives on in our hearts

* I looked around once more, but still found no one

* He celebrated his ninetieth birthday surrounded by family and friends

* That day marked the beginning of his recovery

* We became more grateful for the time we had together

* I continued with my work, trying to ignore the voice

* It was just an overwhelming feeling that I needed to go

* I knelt down in front of the altar

* I thought about my grandpa and how much he meant to our family

* Please help him regain his strength and guide us in taking care of him

* To my surprise, I found my grandpa sitting in the living room, talking to my sister

* I often thought back to that day in the church

* I decided to take a break and walk to the church

* I couldn't be sure, but it definitely changed my perspective on faith and the power of prayer

* After saying my prayer, I returned to the bookstore

[ex:stein-scrambled]Narrative stein-scrambled

* She says, “But one thing I know I'm not going to do is move. Because I have no money to move with.”

* And they also had police wagons with horses attached to them. Believe it or not in those days.

* they had a bell that clanged. Clang clang clang –

* And I didn't know what to do,

* But thankfully my mother was not one of those kind.

* He says, “I dunno.”

* We weren't far from – from where it was – maybe Fifth street – from where we had moved.

* In that house you had to pay in advance.

* She says – he says, “Well uh the only thing I can tell you to do is to move.”

* Don't move 'cause nothing will reach you here.

* but in every other respect I'm just the same as you are,

* So uh, she said, “Well you're the law.

* And they had never come up with anybody like that.

* and opened the door in the back.

* And I cannot afford to be buying windows.”

* She says, “I know that I am different from you, color wise,

* I was so nervous and – and – my voice was shaking so,

* An' he didn't say black 'cause they didn't use black then.

* I can see it right now – from where he was sitting up front there where the horses came around,

* (We call our mother mom.)

* I was right there with her

* So my mother said um, “Where is the uh precinct you work from?”

* And she says, “You mean to tell me that I have to live under those circumstances and there's nothing that you can do about it?”

* She says, “You can take me home.”

* And I said, “No.”

* And it was terrible

* But that was our welcoming to the neighborhood.

* And now I'm telling you that we're going to be here for a month.

* And she – well I always thought of her as an educated person which she was because she was the only one I knew who had completed high school.

* He says, “Because” – he said to me, “Did you see anyone?”

* They had push carts in those days, on Fourth street.

* She said, “It is winter time and it is cold.

* Before the patrol ever left my mother turned

* So you might just as well make up your mind to that.

* but I've forgotten where he told her.

* And he told her.

* And I finally found my mother.

* Well no sooner did my mother get out of the house – my father went first – then – oh it was such an awful crash came through the window.

* So uh that's where she was going to do her shopping.

* But “they don't want anybody in there.”

* I was scared to death,

* And I had sense enough to get the kids into the stairway so that they wouldn't get hurt by flyin' glass.

* And he said, “They're known to do things like that,

* An' he says, “I'm sure there was more than one person.”

* It's Irish.”

* but I remember saying a prayer as I went out the front door because I didn't know what was gonna hit me when I went out that door.

* Whether they'd have enough money to pay it for the next month.

* And this was – let me see what year was this happening – well it must have been nineteen twenty-five, so you can see what – what the times were.

* And I'm gonna go and see if I can find mom” –

* Better move out of there.”

* I forgot now.

* Because they would, you know, however much money they could get they wanted.

* and she says, “Why you're doing this to me I don't know.”

* So anyhow, she said to him, “Well I know one thing you can do.”

* and carryin' on.

* So – I guess they figured “Let's get this lady out of here because she's talking loud and long.”

* And I – and I looked and it was a big stone.

* you know they told you about the trolley that went “Clang.”

* And they had paid that rent for a month.

* He said, “Take you home?”

* He says, “Well how can – how can you” – to my mother – “how can you do anything if you don't know who – who to put the blame on?”

* I did find her.

* I really was frightened out of my wits.

* but he – how he – why he wasn't in the house.

* She knew she was going to be there for the month.

* a policeman went by.

* Now the real estate agent should be accountable.”

* And while I was trying to tell her what had happened

* And we went – she took me, and we went to the police station.

* Well of course you know they didn't do that.

* And of course they saw the man came around the back –

* And my voice was such that you could hardly make out what I was saying.

* and they – they stayed open until – until all hours.

* So in the meantime – not in the meantime but during, during the period of time that we were there, my father became sick.

* And he says, “What's that?”

* He was Irish himself.

* We moved down to Third and Wharton.

* He says, “I can't do anything with it.”

* So my mother said, “Well I don't know what I'm going to do.”

* she told them how much money she had paid,

* “Now” she said, “Whoever threw the stones I don't know.

* You're supposed to protect us.”

* So my mother said, to the captain, “What are you going to do about it?”

* and faced all those people that were standing around,

* And the neighbors were as nice as anybody could ever have been.

* They probably didn't have any money either.

* Well this police patrol that bell they had, it went “Clang clang clang.”

* He says, “But they don't want any black people in there.”

* But you can all see what has happened.

* And she said, “If they didn't want me there they should not have rented the house to me.

* And I ran –

* and it certainly was frightening for me.

* And he says, “Well lady there's nothing we can do about it.”

* and they took us home.

* I ran to Fourth street, because I knew my mother wa-

* Well we got out of that police patrol –

* “I can't do anything about it.”

* He – stopped somewhere on – I dunno,

* She said, “Yes in the police patrol.”

* And they had a long door in the back where you had to come down steps to get out the back of the – of this police patrol.

* And then of course they – they – well they stoned the house.

* So they put us in the police patrol

* They took all we had to, to pay the rent.

* We had gotten moved

* and all I'm trying to do is provide a home for my family.”

* But she knew they were going to be there for the month.

* and after he had gotten in and put the furniture where it was supposed to go, my mother left, to go to market to get some food to bring into the house.

* And they were crying

* And um all the neighbors came running out to see what was happening.

* When there was trouble they responded just like anyone else.

* She told them that they had rented the house,

* If you don't want me here then you people get together and give me enough money to move somewhere else, because I do not have it.”

* And – course with the horses and the police patrol, and the clang –

* And I said to them, “Now you stay there.

* And then when he understood what the situation was he turned to my mother

* she had told them that was all the money she had.

* She went to Lackawanna high school.

* I was ten years old at the time this happened –

* And of course I was left there with the kids.

* and he wanted to find out what ha- what was wrong with me.

* And my father went to take the horses – take the team of horses back.

* and says, “This is a bad neighborhood.

* but it was still day,

* And it was a Jewish neighborhood

* but it was like – you remember I told you my father drank,

* But he didn't come home.

* So uh she said, “Well if you can't do anything to help me, I'll go see who – if I can't find someone else to help me.”

* She did not know beyond the month.

* Well in those days black people were so cowed by circumstances an' uniforms and white people, you know, they uh many times didn't assert themselves.

* My mother said, “I just moved in, how am I gonna move out?”

§ LIST OF NOUNS

The following list of 32 nouns in exactly this order was presented to each participant in the Random List of Nouns experiment.

'FAN', 'POLICE', 'SURVEY', 'COFFEE', 'ISLAND', 'CALF', 'ACID',
'SUITCASE', 'TIDE', 'BULLET', 'BUILDING', 'TELEVISION', 'QUEEN',
'BEDROOM', 'LIGHTNING', 'MOUNTAIN', 'LEMON', 'SEAT', 'CITY',
'VICTIM', 'LIST', 'CHAIN', 'STEM', 'ATOM', 'STAKE', 'DEER',
'OPERATOR', 'CROSS', 'FLOOR', 'INFANT', 'INDIAN', 'PASSAGE'

§ STORY USED FOR LLM RELIABILITY ASSESSMENT

The following narrative was used for comparison of recall performance evaluation by humans versus GPT in <ref>.

The story was generated with an older version of GPT (text-davinci-003) using the following prompt seeded by the boyscouts narrative  (<ref>, in paragraph form without the numbering):

[ex:panic_gen]Panic Attack Story Generation
This is an oral narrative about a life-threatening event:

Create a different, longer and more detailed oral narrative with a similar narration style about another life-threatening event.

yielding the following narrative:
[ex:panic]Panic Attack Narrative

Well, I was at work one day and I was doing my usual job - working on the warehouse floor, stocking boxes and organizing merchandise. I had been working for a few hours and it was getting late. I had been feeling a bit strange, a bit lightheaded, but chalked it up to hunger and not having eaten yet that day. But then, as I was walking around the warehouse returning items to their proper places, I started to feel a strange tightness and pressure in my chest. I stopped what I was doing and leaned against a nearby wall for a bit, but the pressure only got stronger. And I could feel my heart pounding in my chest. Next thing I know, I started seeing spots and I could feel sweat dripping down my forehead and my arms. I tried to take some deep breaths, but I just couldn't catch my breath. I was really scared and didn't know what to do. Suddenly I heard a voice coming from across the warehouse. It was the manager, who had seen me and called out to me - it seems someone had noticed me and was checking to see if I was alright. I tried to yell back that something was wrong, but I couldn't find my voice. The manager started walking towards me and I could feel my world going dark. I started to drop to the ground, but luckily the manager got to me in time and caught me just before my legs gave out. He took me to the staff room and sat me down in a chair. He asked me questions about what was wrong and what happened. He got me a glass of cold water and told me to rest for a bit. After a few minutes, the pressure in my chest slowly eased up and I started to feel better. It turned out I had been having a panic attack, but I escaped from the situation with the help of the quick action of the manager. It was never a fun experience, but it really could have been much worse, and it taught me to check in with myself and make sure to properly rest and eat to avoid further episodes.

The story was subsequently segmented using the prompt:

[ex:panic_seg_prompt]Segmentation Prompt
This is the original story:

Split the sentences of the original story into the maximum number of small pieces, such that each piece contains minimal information.

The segmentation that comes up is the following:
[ex:panic_seg]Panic Attack Segmentation
1. I was at work.

2. I was working on the warehouse floor.

3. I was stocking boxes and organizing merchandise.

4. I had been working for a few hours.

5. It was getting late.

6. I had been feeling a bit strange.

7. I was feeling lightheaded.

8. I chalked it up to hunger and not having eaten yet that day.

9. I was walking around the warehouse returning items to their proper places.

10. I started to feel a strange tightness and pressure in my chest.

11. I stopped what I was doing.

12. I leaned against a nearby wall.

13. The pressure only got stronger.

14. I could feel my heart pounding in my chest.

15. I started seeing spots.

16. I could feel sweat dripping down my forehead and my arms.

17. I tried to take some deep breaths.

18. I couldn't catch my breath.

19. I was really scared.

20. I didn't know what to do.

21. I heard a voice coming from across the warehouse.

22. It was the manager.

23. Someone had noticed me.

24. The manager was checking to see if I was alright.

25. I tried to yell back that something was wrong.

26. I couldn't find my voice.

27. The manager started walking towards me.

28. I could feel my world going dark.

29. I started to drop to the ground.

30. The manager got to me in time.

31. He caught me just before my legs gave out.

32. He took me to the staff room.

33. He sat me down in a chair.

34. He asked me questions about what was wrong and what happened.

35. He got me a glass of cold water.

36. He told me to rest for a bit.

37. The pressure in my chest slowly eased up.

38. I started to feel better.

39. It turned out I had been having a panic attack.

40. I escaped from the situation with the help of the quick action of the manager.

41. It was never a fun experience.

42. It really could have been much worse.

43. It taught me to check in with myself.

44. It taught me to properly rest and eat to avoid further episodes.

Note that the segmentation in this particular case is not verbatim, the clauses have been transformed to first person while retaining most important information. If a verbatim segmentation is desired however, it can be achieved by utilizing the same prompt with gpt-4 although the number of clauses will be slightly different.