Other notable developments to GANs brought improvements to image quality and resolution (Karras et al. 2017). In late 2018, the release of StyleGAN (Karras 2021), a model built on a combination of ideas from Style Transfer and PGGAN, demonstrated very convincing images of human faces. In his article "How to recognize fake AI-generated Images", the artist Kyle McDonald (McDonald 2018) investigated the images generated by StyleGAN, and highlighted the visual artefacts he found. At a glance these images look like photographs, but on closer inspection irregularities such as patches of straight hair, misaligned eyelines, or mismatched earrings reveal the difficulties GANs have in managing "long-distance dependencies" in images.

In 2017 the paper *Attention Is All YouNeed* (Vaswani et al. 2017) proposed a new network architecture called the Transformer. This model addressed the long-distance dependency issue in RNNs and CNNs by rethinking how we could handle sequences. Rather than looking at a sentence word by word, the Transformer observes the relationship between all elements of the sequence simultaneously. Being able to better handle long distance dependencies meant the Transformer was appropriate for natural language generation. Artists and poets such as Allison Parrish have explored the use of VAEs for short text generation, but with the emergence of LLM passages of long, coherent texts could be generated (Brown et al. 2020). As dataset sizes increased, along with hardware costs for training these large models, they have become harder for individuals to train themselves, and the mode of interaction has shifted from curated datasets and homemade scripts, to web APIs and third party services. While it is more difficult to participate in the training process, the availability of services and interfaces provides new ways of working with these models that can produce less technical andmore playful approaches. For example,Hito Steyerl used GPT-3 to create *Twenty-One Art Worlds: A Game Map* (2021) and described the process as "fooling around" with GPT-3 to write descriptions of different ArtWorlds, using the tool in the process of world building. In the resulting text it is difficult to distinguish which words may have been written by Steyerl and which were written by GPT-3.

The learnings from Large Language Models for text generation were soon applied to image generation (Dosovitskiy et al. 2021), and the simultaneous release of

4 https://www.nvidia.com/en-us/research/ai-art-gallery/artists/helena-sarin/

CLIP (Radford et al. 2021) and DALL-E (Ramesh et al*.* 2021) signalled the start of a new era of image generation. Although the DALL-E model was not released, CLIP was made available to the public, and the model was quickly adopted by AI artists who applied theidea of CLIP guidance to variousimage generation techniques.Ryan Murdock produced the colab notebooks DeepDaze5 , combining CLIP and SIREN, and BigSleep6 , combining CLIP and BIGGAN, which were subsequently adapted by Katherine Crowson in the widely distributed VQGAN+CLIP notebooks7 .

The paper Denoising Diffusion Probabilistic Models (Ho / Jain / Abbeel 2020) introduced a different method for creating generative models. This technique trains a model by adding increasing amounts of noise to an image and then having the model remove the noise, resulting in a model that can generate images from only noise.Diffusionmodels,when combined with CLIP or other conditioning processes, enable much faster text-to-image processing.

The popularity and accessibility of these techniques was further raised by the product release of DALL-E 2 (Ramesh et al. 2022) and Midjourney8 . Midjourney became so popular it is now the largest Discord server with over 5 million members as of writing. Following the releases of these products, open source models such as Stable Diffusion have also been developed. There are many benefits of using free and open source models for artists. Being able to modify code and develop on your own hardware allows the artist to pursue their own experimental approaches, not restricted to the interface designed by a service provider.

The artist's involvement in generating new images with these models is vastly different to working with GANs. Rather than building custom datasets and training models, instead the focus has shifted to writing prompts that can generate the images the artist wants to find, and designing interfaces for exploring these prompts and their translations. This artist Johannez coined the term "Promptism" (Herdon / Dryhurst 2022) for this art practice, and wrote a humorous Prompist manifesto using GPT-3. Against a backdrop of models trained on hundreds of millions of images scraped from the internet, the manifesto asserts "The prompt must always be yours" (Johannez 2021).

### **Artist-Guided Neural Networks**

Many papers discuss AI from the point of view of creativity taking mostly one position of two: either AI as an amazing tool for artists and creativity, or AI is seen as

5 https://github.com/lucidrains/deep-daze

6 https://github.com/lucudrains/big-sleep

7 https://github.com/EleutherAI/vqgan-clip

8 https://www.midjourney.com

something negative in art. It is easy to see that people from the industry advocate for the first position, and theory scholars for the second one. But, how do practitioners see contemporary AI technology themselves? And in which ways AI is deployed in art practice? Hence, it is not the focus of this paper to discuss whether AI can make art, but rather how AI can be useful for artists and what new ideas it can offer. By using practice-based research methodology, we decode the role of AI tools in artistic practice and trace the evolution of such artistic work. In this paper, the practice of artist duo Varvara & Mar was used as a case study, which provided us with the insides in this research.

In this chapter, we explore in various ways in which AI was deployed in creative practice, dividing it into four categories based on medium: synthetic image, synthetic text, synthetic form, and translation models. From the view of the practitioner, the limitations, new possibilities, and change in production processes are discussed.

#### **Synthetic Image**

Our exploration of DL started with image generation in 2017 when we used Google Deep Dream algorithm. The idea behind the *Neuronal Landscapes* project9 was to imagine how the Estonian landscape will look like in 100 years' time (commission work for Estonian History Museum). Since machine mysterious synthetic vista began to be prevalent, the artwork offers an experience of what it is like to see the environment through machine eyes, and to be immersed into an endless hallucinated simulacra of a neural net. Our intention here was to go beyond a still image and achieve immersiveness depicting Estonian society's evolution throughout time: from forest and farm land, towards urbanisation and increasing digitalization. For that purpose, a 360º VR video (00:09:39 long) was created. First, the video material was filmed and then edited. Filming was done with two 360º cameras carried by a drone to achieve a seamless, continuous shot, without a visible camera. After the stabilisation of the image, each frame was passed through the DeepDream algorithm. The rendering process took 30 days' time on two (for that time) powerful machines with Nvidia TitanX GPUs working in parallel. Although we could change parameters of the algorithm to slightly customise the effect, the imprint of the algorithm was still very present. At the end, Google Deep Dream became sort of a generative filter to the images.

In the next art project, ProGAN was deployed. For the first time we worked with datasets and training GAN models. *PlasticLand* (2019)10 talks about plastic waste and

9 https://var-mar.info/neuronal-landscapes/

10 https://var-mar.info/plasticland/

ecological problems this material causes. We composed four different datasets of images of layered plastics in our planet: landfills, plastic on top of water, plastic underwater, and plastiglomerates. The ProGAN model was trained on a local machine using pyTorch and took a week to train, and we used part of the images generated during training to create a video composition.

A metal totem displaying those synthetic, as plastic is, layers, we draw attention not only to the problem of waste but also question whether AI has some similarity with this material. Since the invention of plastic, this material was applied almost everywhere because of its perfect qualities, until we realised that it is not sustainable and ecology-friendly. Will a similar story happen with AI?

From the practice-based research perspective, this artwork shows artists' desire to move from a still to moving image and towards sculptural form that is held back by the early stage of machine learning technology: low resolution images jumping from one frame to another.

The next artworks *POSTcardLandscapesfromLanzarote I(00:18:37)*and*II(00:18:40)*11 in 2021 demonstrate the artist's ability to create video works with StyleGAN2 (see figure 2). The hypnotic appearance of these works, where one frame morphs naturally into another, shows the artists' ability in guiding the outputs of the NN. Vector curation and composition of a journey through the latent space, created by training the model on specific datasets of 2000+ images, were crucial and integral parts of the artistic process.

The artwork talks about critical tourism and how circulation of images representing touristic gaze overpower the nature of seeing. In the words of Jonas Larsen "'reality' becomes touristic, and item for visual consumption" (2006: 241–257). Hence, we scraped, where licence allowed, the location-tagged images from Flickr and composed two datasets of photos categorised as tourism or landscape.

As we have written elsewhere,

the art project consists of two videos representing a journey of critical tourism through the latent space of AI-generated images using StyleGan2. Later the images are composed into latent interpolations that take the form of smoothly progressive videos. The two videos are random walks in the latent space of the Stylegan2 trained models, creating a cinematic synthetic space. The audiovisual piece shows an animated image through the melted liquid trip of learning acquired from the dataset composed of static images. The video flows from point to point, generating new views and meaning spaces through the latent space's movement. The audio was created after the video was generated in response to the visual material to complete the art piece. (Guljajeva / Canet Sola 2022a)

11 https://var-mar.info/postcard-landscapes-from-lanzarote/

The sound for local or landscape view was created by a sound artist from Lanzarote, Adrian Rodd, who aimed to give a socio-political voice to the piece. In contrast, the sound design created by Taavi Varm is a soundscape replying to touristic gaze. We aimed to initiate collaborations with others but also to experiment with human-AI co-creation. In a similar vein is the artwork *Phantom Landscapes of Buenos Aires* (00:20:00, 2021)12, with sound work by Cecilia Castro.

Our last experiment with GAN models *Synthetic Scapes of Tartu* (00:10:00, 2022), demonstrates a different approach. Taking a dataset composed from our own video footage (flaneur walks), we first produced the sound (a composition by Taavi Varm, Ville MJ Hyvönen with piano by J. Kujanpää) and used this to inform the direction of the video. The result was a sound-guided AI-generated visual output.

# **Synthetic Text**

In this section, we focus on artwork incorporating AI text generation as part of the artistic concept. Our journey to text generation started with the online participative theatre project ENA13 and ended with a hand-bound publication.

During the first lockdown in May 2020, together with theatre maker Roger Bernat, we created an online participative theatre piece ENA on the website of Theater Lliure in Barcelona. ENA is a generative chatbot that talks to its audience, and together (AI and audience), they make theatre (see figure 3). As we have described

12 https://var-mar.info/phantom-landscapes-of-buenos-aires/

13 https://var-mar.info/ena/

before: Although in the description of the project it was stated explicitly that people were talking to a machine, multiple participants were convinced that on the other side of the screen another human was replying to them – more precisely the theatre director himself, or at least an actor". (Guljajeva / Canet Sola 2021)

Analysing synthetic books, Varvara Guljajeva has stressed the importance of human input in the AI text-generation systems (2021). In addition, one also needs to guide the audience participation and interaction with the chatbot. For this purpose, we have adopted the traditional theatre method for guiding actors, as a way to guide the audience, and thus, the bot, too. Stage directions were used as a guiding method, which triggered thematic conversation and offered meaningful dialogue between humans and the AI system. We found the conversations so meaningful that we decided to publish a book that contains all the conversations with ENA.

With this project, we learned that it is essential to guide neural networks via audience interaction. In order to do this, it is also necessary to guide the audience. Without audience interaction guidance, it is nearly impossible to achieve meaningful navigation of NNs.

#### **Translation models**

This category focuses on translation models that enable interactive and installationbased formats. Translation refers to the conversion of mediums, or as we put it, translation of semiotic spaces. To illustrate this, we introduce*Dream Painter*, 14 an art installation that translates the audience's spoken dreams to a line-drawing produced by a robot (see figure 4). As described earlier: "Dream Painter is an interactive robotic art installation that explores the creative potential of speech-to-AI-drawing transformation, which is a translation of different semiotic spaces performed by a robot. We extended the AI model CLIPdraw which uses CLIP encoder and the differential rasterizer diffvg for transforming the spoken dreams into a robot-drawn image". (Canet Sola / Guljajeva 2022) "Design- and technology-wise, the installation is composed of four larger parts: audience interaction via spoken word, AI-driven multicolored drawing software, control of an industrial robot arm, and kinetic mechanism, which makes paper progression after each painting has been completed. All theseinterconnected parts are orchestratedinto aninteractive and autonomous system in the form of an art installation […]". (Guljajava / Canet Sola 2022b) Out of all the projects discussed, this was the most difficult to realise.

In this project, we investigated how guidance of NNs could be interactive and real-time instead of non-interactive and pre-determined, as shown in previous examples of our work. It is important to notice that methods, such as dataset composition and output curation were not used in this case. In fact, visual output curation is totally missing. The artists created an interactive system to be experienced and discovered by the audience.This means the audience determines the output. Instead of curating a dataset, a CLIP model is used that can produce nearly real-time output

14 https://var-mar.info/dream-painter/

guided by a text prompt. As we have written earlier: "Translation of semiotic spaces, such as spoken dreams to AI-generated robot-drawn painting, allowed us to deviate from image-to-image or text-to-text creation, and thus,imagine different scenarios for interaction and participation". (Guljajava / Canet Sola 2022b)

This project indicates our search for transformative outputs of AI technology, and thus, shows the evolution in practice. By extending available DL tools and combining with other technology, for example, text-to-speech models, real-time industrial robot control, and physical computing, it offered an interactive robotic and kinetic experience of NN latent space navigation. This contributes towards the explainability of AI because the audience could experience how the words affected the drawing, and which concept triggered which outcome.

Being inspired by Sigmund Freud's work on the interpretation of the human mind while unconscious, we speculatively ask if AI is powerful enough to understand our dreamworld. Through practice we question the capacities of NNs and investigate how far we can push this technology in the art context.This artwork allows the audience to experience the limits of concept-based navigation with AI. The system is unable to interpret and can only illustrate our dreams. It cannot understand the prompt semantically and only gets the concepts.

*Fig. 4: Kuka industrial robot painting audience's dreams. Installation view.*

#### **Synthetic Form**

In this section, we ask how artists can guide neural networks when creating volumetric forms, and what happens when AI meets materiality. After working for a while with DL tools that produce 2D outputs, it is an obvious step to explore possibilities to produce 3D results. To our surprise,it was not an easy task to find the solution (Oct 2021). *Psychedelic Forms*15 is a series of sculptures produced in ceramics and recycled plastic through which we investigated the possibilities of AI in producing physical sculptures. The project re-interprets antique culture in the contemporary language and tools.

Following the same paradigm shift as in the previous section, text2mesh is a CLIP-based model that does not require a dataset, but a 3D object and text prompt as input. Hence, the model actually does not create a 3D model but stylises the inserted one. And this is guided by inputted text.

We decided to go back to the origins,in terms of ancient sculptures and material selection. Although it was said that there was no dataset, we still had a collection of 3d models of ancient sculptures because, by far, not all produced the desirable output. In this sense, there was definitely an output curation present in the process.

The criteria for selection were the following: first, the form had to be intriguing, and second, it should be possible to produce it in material afterwards. It was clear that we had to modify each model because the physical world has gravity, and the DL model does not take this into account. Some generated models were discarded because they were seen as not-fixable, although interesting in their shape.

The process demonstrated here is quite an unusual way to create an object. After extensive experimentation with the tool, we learned how certain words triggered certain shapes and colours. This knowledge gave us a chance to treat text prompts as poetic input. Thus, we created short poems to guide NN. The best ones survived as titles and are reflected in the forms.

The artists did not strictly follow the original model but took the creative liberty to modify the shape and determine the colour by manually glazing the sculptures. The dripping technique was used for colouring the sculptures. This served as a metaphor for liquid latent space and the psychedelic production process (this was the artists' inner feeling about the creative process because they did not know what results would be achieved in the end).

Sometimes, AI-generated vertex colouring was taken as inspiration, sometimes totally ignored. Nevertheless, digital sculptures were exhibited alongside the physical ones to underline the transformation and human role in the creative process. Although ceramic sculptures were 3D printed in clay, the fabrication process had to follow the traditional way of producing pottery (see final sculptures in figure 5).

15 https://var-mar.info/psychedelic-forms/

Since we had never engaged in ceramics before, the whole production process felt psychedelic: unexpected NN processes led to transformation by numerical, physical, and chemical processes, all guided by both the artists and chance. Hence, the art project highlights the relationship between different agencies.

In the end, we can say that AI is not prepared for the physical world. It created nice images, but when one wants to materialise the output, it requires considerable additional work.However, those extra processes were very rewarding and creative in our case. In this project, AI served as an inspiration or a departing point more than anything else. In other words, the experimental phase of technology is necessary for experimental practices, and this can lead to the creation of a new production pipeline.The fine line between control and chance when guiding the neural networks and related processes is likely the main creative drive for the artists.

*Fig. 5: Ceramic sculptures guided by 3D object and text prompt, 3D printed in clay, and glazed manually. From left to right: Psychedelic Angel (Venus), Mermaid in green jelly and pink feather (Nymph), Psychedelic Angel (Venus).*

# **Discussion**

According to the media hype around AI, this technology is intelligent enough to create art autonomously (Perez 2018; Vallance 2022). However, the reality is different. According to computer scientist and a co-inventor of Siri Luc Julia, AI does not exist. He advocates for machines' multiple intelligences that often outperform humans. However, machine intelligence is limited and discontinuous compared to human intelligence (Julia 2020). Therefore, it is vital to have artistic practices around this technology, as a counterbalance to the AI fantasies served by the industry and mass media.

We see AI as a creative tool with its own possibilities and limitations, which can stimulate artists' creativity through unexpected outputs. Research has shown that tool-making expands human cognitive level and constitutes evolution in culture (Stout 2016). Similarly, as a new tool, generative AI could potentially enrich creativity by allowing new production pipelines that can generate unique results.

Coming back to the synthetic images, we can say that all machine-created synthetic image-based works discussed here have particular aesthetics: both with Deep Dream and GAN. Unlike the output of GANs, Deep Dream has a more recognizable style and can be seen more as a filter that transforms every inputted image instead of learning from the given dataset.

Regarding GAN aesthetics, such visual appearance is inherited from two entities to a large extent: the dataset and the model itself. GANs have a particular footprint, as seen in all works produced with this model. The visual palette comes from the used datasets. For example, if a dataset is homogeneous (only landscape images), then we will easily recognize landscapes in the generated output. However,if images in the dataset have a lot of visual variation, the output is rather abstract. *POSTcard Landscapes from Lanzarote* illustrate this well. Also, when photos in the dataset look similar, the output will also be similar,as was the case with the *Synthetic Scapes of Tartu* video work where frames from recorded flaneur walksin a city were extracted.When we talk about video works generated with the neural net, then manual guidance of latent space offered more variations than an audio-led approach.

Synthetic image works have encouraged us to work with formats like images and videos that we did not engage in before in our art practice, but we found it exciting working with AI and video.  It might take a few minutes to render a single image.The limitations inspire us to devise new solutions and work in new mediums. Moreover, the limitations of the medium has always been a good challenge for our creativity.

Working with GANs or other image-generation tools has become much easier in recent years, although it used to be quite difficult. We must note that for practitioners, easy-to-use tools, such as DALL-E and Midjourney, offer little creative freedom, and thus, are less attractive to the artists. Those products tend to instrumentalize the user rather than the other way around. At the same time, open source models offer more creative freedom and enable broader use of artistic ideas.

The work with generated text demonstrates that AI is not context-aware but maps concepts automatically without understanding semantics. More importantly, as shown in the *ENA* project the audience must also be guided alongside the AI. In the case of *ENA*, stage directions were used, and in the *Dream Painter* project, the concept of dream telling was applied to guide the participants who in turn guided the neural net through their interaction, creating a chain reaction. Navigating concepts in latent space is artistically interesting and inspiring, this was especially evident when working with form. The artists went beyond semantics and learned how to guide neural networks with a text prompt and 3D object.

The presented practice represents a paradigm shift in machine learning,moving away from composing datasets for GANs and toward translating semiotic spaces enabled by diffusion models. The evolution in practice shows how artists discover and learn to work with the DL toolset, embracing its possibilities and limitations. In the case of practice-based research, practice can be seen as a lab for testing artistic ideas with technology through chance until control is encountered.

# **Conclusion**

In this article, we have summarised DL development from the perspective of artists' interests concentrating on the image, video, text, 3D object generation, and translation models. We applied practice-based research methodology to investigate the role and possibilities of recent co-creative AI tools in artistic practice.

It is difficult to keep pace with AI development. In less than a decade, we have gone from blurry black-and-white faces to impressive high-resolution images guided by text prompts. The user level has gone from difficult to easy, which on one side, broadens possibilities for creation, but on another, it diminishes experimentation and creativity, since AI outputs seem ready-made. This is also demonstrated by the explorative nature of the body of work presented here.

Furthermore,it was noticed that creative AI, especially GAN models, have recognizable aesthetics, which in the long run, become repetitive. This led to the change of tools by the artists.The curation of datasets, models, and outputs, along with NN guidance, have become the toolset of an artist working with AI. Finally, these models can generate multitudes of outputs, but the art is giving the right input to guide the desired output and selecting the results that best serve the concept

As Andy Warhol had envisioned in 1963, eventually, art production will become mechanised and automated. In his own words: "I want to be a machine", which was also a reflection on that time's vast industrialization process. Resonating with today's deep learning age: I want my machine to do art.

### **Bibliography**

*IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. doi: 10.1109/cvpr52688.2022.01313.

# **Embodied Voice and AI: a Techno-Social System in Miniature**

*Diana Serbanescu, Scott DeLahunta, Ilona Krawczyk, Kate Ryan, Mika Satomi*

# **Motivations**

One of the original motivations for the practice-based research presented in this paper is rooted in the premise that embodied knowledge is relevant to the field of AI, not only in terms of application to design but also in terms of critical practice. A key proponent for integrating embodied knowledge into Human-Computer Interaction, Schiphorst (2009: 225) reckons "embodiment in the context of designing for technology" matters, and "concepts such as embodied computing and embodied interaction" require "design strategies that take advantage of our senses, accessing a richer and more fully articulated form of human being". In terms of critical practice, feminist techno science scholars such as Leurs (2017: 137) regard the researcher as an embodied subject, and stress the importance of materiality, by emphasizing the relational entanglement of bodies and infrastructures in the creation of knowledge. In her seminal book, Artificial Knowing: Gender and the Thinking Machine, Adam (2006) argues that "the body plays a crucial role in the making of knowledge". The afore-mentioned scholars draw on feminist research to counter the dominant data science discourses that still equate intelligence with rationalistic problem solving, in which claims for objectivity and universal truths propagate what feminists call, a "view from nowhere". Anthropologist Suchman (2007) in her critique of the "disembodied intelligence in AI" also points out that "feminist theorists have extensively documented the subordination,if not erasure, of the body within theWestern philosophical canon", an erasure Suchman claims extends to the field of AI and Robotics.

While addressing the wide literature and theory in both application design and feminist critical practices lies outside the scope of this paper, these are the kind of perspectives that provided the original motivations for embarking on our research project. Continued research will further explore the implications of these theories for this practice-based setting.What we can do here is to acknowledge, as Suchman (2007) posits, following Haraway (1997: 11), that "technologies [...] are forms of materialised figuration". They are brought into existence by assemblages of humans and

machines, bringing meaning and values, including dominant assumptions and biases that subordinate, into the process of new knowledge creation. We believe this calls for more critical introspection of one's own practice following Leurs (2017: 145) who states,

"it is through being critically self-reflexive about one's positionality, locating research practices within wider power relations and structures, that we can begin to destabilise the normalised politics of knowledge production".

Therefore,in presenting the framework below, we aim to also reflect on the positionality and the ethics of our research,in part through revealing the individuals and the corresponding assemblages in the development of tools and processes.

# **Strand One: Augmentation of performative practice through an AI wearable design**

This section proposes a hybrid methodological framework which discusses the technical and formal aspects of augmenting an embodied practice for actor training by means of a custom-made wearable device which gives the body a voice. The device is an interactive system that uses an Artificial Intelligence (AI)-based module to map movements to synthesised sound. Our framework draws on techniques from the fields of contemporary performance, wearable technology design, Human-Computer Interaction (HCI) and AI pipeline design.

# Methodological framework

# Embodied voice in post-Grotowskian practice

In the Embodied Voice and AI project, our team focused on a particular method for physical actor training chosen from the post-Grotowskian repertoire. This choice of method was motivated by many reasons. First, according to Wolford (1996), Grotowski's1 practice, constituted as a laboratory of embodied research, and his rigorous system of actor training mirrored his fascination with science. Thus, his epistemic aims towards reproducibility of results and claims of scientific objectivity resulted in methodologically unique approaches to the embodied practice.This structured way of working, which inspired generations of practitioners, could serve as a

1 Grotowski was a Polish theatre director and theorist whose innovative approaches to theatre training involved the study of performers immediate, "organic", "animal-like" psychophysical responses to impulses where "there is no discursive mind to block immediate organic reaction, to get in the way" (Richards 2003:66) . He called the process of eradicating such blocks *via negativa* (Grotowski & Barba 2002:17).

conceptual scaffolding of the bodymind continuum suitable for a technological design.The training works by making the actor more aware of moments at which their body and mind act as one – giving them, through the process of repeated training, the ability to more frequently, and more deeply, access a state of "freedom from the time-lapse between inner impulse and outer reaction in such a way that the impulse is already an outer reaction" (Grotowski & Barba 2002: 16). Voice and singing are important parts of this process.

Embodied voice training in post-Grotowskian practice seeks to express deep psychophysical states through voice, in moment-to-moment situations in relation to self and to partners. Thus, "performers are trained to 'experience the body as music, as a melodic and rhythmical vibration' (Dowling, 2011: 248) that affects and allows one to be affected by others within musical and rhythmical structures" (Krawczyk 2021: 24). Last but not least, the choice of focusing on one post-Grotowskian technique is rooted in the first-hand experience that our team of practitioners had with this type of practice. We wanted to model a practice that we had a thorough embodied understanding of2 .

#### Team

AI-based systems are products of collaborative and team effort. As previously mentioned, we consider a process of reflexivity in relation to one's own practice to be an integral part of our research framework and a necessary step toward an ethical guideline for AI research. In this regard, we take from Leurs (2017: 139) the principle of an "ethics of care", which is described as "value-based" and cognisant of "the dependencies, partiality, political commitments and personal involvements of researchers". Therefore, we believe that awareness about the process of research, development and decisionmaking starts with the team. In a small team and project like ours with its focus on augmenting a pre-existing psychophysical training method, it is easier to explore the ecologies of skills, knowledge, but also experience, subjective preferences, interests and embodied abilities, which the practitioners and researchers bring to the room. Together, these create a distinctive ontology of practice for the enacted "assemblages of doings" (Leeker 2017: 13) – in which body datafication, while a core focus of the research,is undertaken in an iterative way – that opens space in a process for the kind of reflexivity we propose could be valuably undertaken in other practice contexts.

2 The framework presented here is a continuation of a growing body of collaborative work established in two previous projects, The Shape of Things to Come (2019) – https://rep lica.institute/ (accessed 12 Dec. 2022) – and Dancing at the Edge of the World (2020) – https://berlin-open-lab.org/portfolio/dancing-at-the-end-of-the-world/ (accessed 12 Dec. 2022), https://www.hybrid-plattform.org/forschung/detail?tx\_news\_pi1%5Bnews%5D=996 &cHash=18878ad6e067104b2e55c31756756b28 (accessed 12 Dec. 2022).

For this practice-based research, our team consisted of the following five scholars / practitioners, who are also the co-authors of this paper. Each member of the team assumed one or more specific roles within the research: Kate Ryan – performer; Mika Satomi – wearable technology and interaction designer; Diana Serbanescu3 – artistic director, and project lead; Scott DeLahunta – ethnographer; Ilona Krawczyk – post-Grotowskian practitioner, and external expert consultant.

Suchman (2017) posits that "when we ascribe skills to a person [...] the person acts as a symbol". However, she also discusses how the activation of skills is dependent on the team configuration. In relation to this, Ilona Krawczyk notices how the activation of skills in our team was supported and encouraged, through feedback sessions embedded in the process:

What I found particularly important was how Kate and her process stood out in the first place. In the post-Grotowskian theatre this is not common, as the director's perspective on the training or performance devising process dominates the performer's perspective.

With regards to the ethics of practice, and related to a process of self-reflexion, Ilona comments:

From the perspective of my research on ethical considerations in theatre practice, another significant aspect of this project is the effort to attentively investigate what kind of knowledge we are producing, in what areas within and beyond performance, training and AI, how the process of knowledge production is distributed among all the participants and how we acknowledge each other's contributions.

As a tool for collaborative post-reflection on the process, we used a web-based annotation platform developed by Motion Bank in Mainz.4 This platform supports dialogue and documentation of process from multiple perspectives, allowing us to partially trace decisions and members' contributions.What our documentation reveals is that there are also aspects of tacit and embodied knowledge that each of us contributed, sometimes beyond the "traditional" roles in the team.

3 As the project lead and artistic director of this project, as well as the two preceding projects – "The Shape of Things to Come" (2019) and "Dancing at the Edge of the World" (2020) – Diana Serbanescu made key decisions in shaping the research framework and agenda. She was responsible for selecting team members in all three project configurations and contributed to design decisions regarding the structural and functional elements of the AI-based technological artifact.

4 Motion Bank. Hochschule Mainz University of Applied Sciences. https://medium.com/motio n-bank (accessed 12 Dec. 2022).

#### The AI-based motion capture wearable device

*Fig. 1: Wearable design by Mika Satomi. Credit: Mika Satomi*

The idea5 of imagining the body as music, as rhythmic vibration, inspired our first prototype6 , an AI-based wearable device that sonifies movement by mapping body postures to sound cues. The practice-based research discussed here builds on this earlier prototype7 designed by Mika Satomi, and aims to alter it to meaning-

5 Ditte Berkley, a post-Grotowskian practitioner involved in "The Shape of Things to Come" (2019), encouraged the performers to "sing with the body and dance with the voice".

6 The initial prototype was developed for a performance, titled Dancing at the Edge of the World (2020).

7 In this version of the prototype, the focus was on giving a voice to the collective body of six performers by endowing each actor with an AI-based wearable device that uses movement

fully augment a specific exercise, taken from the post-Grotowskian embodied voice training.

Figure 1 shows a schematic representation of the custom-made AI-based wearable device designed by Mika Satomi.The device is a collar, endowed with bend sensors8 that can be attached to eight points on the body, able to accomplish the function of motion-capture9 . At the back of each collar thereis a Bela board, which acts as a processing unit. All the computational process: data capture, posture recognition and mapping through a machine learning (ML)10 module, sound synthesis11 happen locally on the Bela board contained in each collar12. The ML module is based on ml-lib13, a library of machine learning externals for Max and Pure Data, built for the Gesture Recognition Toolkit by Nick Gillian. A speaker sits on the front of the collar, emitting sounds that can react to the position of the sensors, on how the performer moves their body.

The ML model is trained on incoming real-time data from the sensors placed on Kate's body. The data from the sensors is captured in specific poses, which are devised according to our movement framework and agreed upon collaboratively by Kate Ryan and Diana Serbanescu.These data points are then mapped to parameters from the synthesizer. Due to the fact that every time Kate is wearing the collar, there are small alterations in the placement of the sensors on her body, the training process needs to be performed anew at the beginning of each rehearsal-session. During the rehearsal, the stretch of the material and consequently the resistance of the conductive fabric is also slightly altered, introducing small variations in the data and compromising the accuracy of the mappings. To counter this, the system needs to

to trigger sound. The mappings of the ealy prototype were simple, connecting randomly devised postures to sound cues: synthesised vowels, or pre-recorded voice samples of the performers singing.

8 E-textile bend sensors made as flexible tapes.

9 By capturing the bending motion of the body part on which the sensor is placed.

10 More specifically, the sensor data is read and processed by the Pure Data (PD) (https://pured ata.info/ – accessed 12 Dec. 2022) software running on the Bela board attached to each collar. This software installed on each Bela board also contains a ML-based module, which is used for mapping body postures to synthesised voice sound. The captured sensor data goes through this ML-based processing module which extends an "ml.lib" (https://github.com/irllabs/ml -lib – accessed 12 Dec. 2022) external object based on the Gesture Recognition Toolkit (htt ps://www.media.mit.edu/projects/gesture-recognition-toolkit/overview/ – accessed 12 Dec. 2022).

11 This processing module is trained to recognize patterns of body postures, which are then used to control synthesizer parameter within the Pure Data patch.

12 The synthesised voice sound is played back from the Bela board directly to the portable speaker worn by the performer.

13 https://github.com/irllabs/ml-lib – accessed 24 May 2023.

be recalibrated and then retrained with fresh data, a process which is repeated multiple times during the rehearsals. Due to this practice of working with the variability of the textile, our data set is transient, in a continuous state of flux.

The data from the sensors can also be streamlined in real-time to a remote central unit, through a wireless network. Mika Satomi has also implemented the controller interface which enables the connection to each of the individual Bela-boards from a central laptop and is capable of visualising the sensor data in real-time and of initializing the training sequence of each system.More information about the design of the wearable device can be found on Mika Satomi's personal webpage14 .

#### Input: Motion capture through sensors

As previously mentioned, we limited ourselves by design15 to work with eight e-textile bend sensors. How the sensors are placed on the body becomes very important in the customization to a particular exercise for embodied training, as we aim to capture movements on the body that are most relevant for that particular exercise. Therefore, we needed to experiment and develop a specific logic of how the sensors are placed on the body. Mika Satomi's design provides some space for flexibility and re-arrangements in sensor positioning16. In the context of this research, we decided to use two of the upper body sensors to be placed one on the back and one on the belly, instead of the shoulders17. In this updated design the lower-body sensors are embedded in a pair of pants and are following the skeletal lines of the lower part of the body.

14 https://www.nerding.at/costume-for-dancing-at-the-edge-of-the-world/ – accessed 24 May 2023.

15 The Bela board we used includes eight analog sensor inputs.

16 The bend sensors were made with kinesiology tape, often used as sport taping to retain flexibility, and they were applied directly on skin like the original tape.

17 In the original prototype (2020), we focused on sensing upper parts of the body by placing more sensors on arms: elbows and shoulders. In the first prototype there were also four distinct sensors for the legs to be placed individually on knees and feet. These were all fixed directly on the skin using glue and kinesiology tape.

# Output: Synthesised sound

The sound design is realised through patches implemented in pure data. Our prototype includes two modes – the Voder mode18, and the Granular19 mode – mapping body postures to synthesised sound.

# Augmentation of practice: In-tensions

Gesture to sound mapping

*Fig. 2: Modelling In-tensions, an embodied voice training exercise. Credit: Diana Serbanescu*

Our practice-based research focuses on an exercise from the post-Grotowskian embodied voice training, which is called In-tensions20. The original aim of In-tensionsis to transfer a physical processinto the vocal quality of a performer, toimprove the in-body listening (with a partner), and last but not least, to improve the vocal presence of the performer.This training system treats sound as a physical presence, as an agentive actor in space. In-tensions consists of a series of concrete physical actions. For our research we've focused on four actions: "push", "pull", "caress" and

18 This is based on formant synthesis. The algorithm is inspired by the early speech synthesiser developed by Bell Labs 1939. In this mode body postures are mapped to synthesised vowels from English language.

19 In this mode body postures control speed and playhead of the granular synthesis on prerecorded voice samples.

20 This was introduced to us by Ditte Berkley. However variations of this method exist in the practice of different other post-Grotowskian practitioners. For example, Ilona has her own personal variant of this method.

"bounce".The aim was to transfer the physical energy of these actions into the vibratory patterns of the voice.

Figure 2 shows a diagrammatic formalisation of mapping four pre-defined (embodied) actions to be delegated to selected sound cues. These actions would be performed by Kate, and the data captured by the sensors in these postures are used to train the AI-based module for classifying these actions.

An example of training the system: "push" action against the wall

This section introduces step by step the process involved in training the system to recognize and map the physical quality of performing a "push" action against a wall. At the end of the training, the pre-recorded sound cues are activated by the movements of the performer in a session of improvised responses to the task of playing with variations of the "push" action against the wall.

*Fig. 3: Kate performs an action of "push" at the wall.*

Step One: Marking a "push" action

Figure 3 captures Kate engaged in the process of searching for her optimal voice quality while performing the action of "push" against the wall. Originally, this is an exercise devised by Ditte Berkley. Its intention is to give a real obstacle – the wall – for the performer to engage with in order to execute an authentic "push" action, by contracting the muscles of the body that would also help with the release of the voice. The performer is encouraged to cultivate awareness in listening to their own voice, while in action, and to assess the moment of their best perceived vocal quality.Thus,

when Kate identifies the peak moment of her voice and the position in which it happens, she is instructed to tap the wall with her hand, while maintaining the posture. Diana suggested to use this tapping as a cue for Mika to start recording the incoming data from the sensors21. The process involves an act of precise synchronisation between Mika, operating the interface, and Kate, performing the action.The sensor data captured here will be used as training data samples for the ML-based module. The intention is to use the data that represents a moment of full embodied unity between vocal and physical quality of the performance.

#### Step Two: Explore "push" action mapped to sound

After the training by examples, our ML model has now learned to map the incoming sensor data with corresponding parameters for voice synthesis. The parameters for voice synthesis are experimentally chosen by Mika. At this stage, they vary from training to training.The intention is to also explore qualities of voice synthesis22 that resemble the real voice of the performer engaged in action. Diana asks Kate to explore different types of "push" actions against the wall, and test what the system has learned and how it reacts to variations from the training samples.

#### Task-based improvisation with all actions

Task-based improvisations are sessions of varying lengths, lasting between one and two hours, with the aim of exploring the limitations and possibilities opened by augmenting the In-tensions practice. This is based on the embodied listening between Kate, as performer / practitioner and the AI-based system, through the wearable device. The intention is to observe how this engagement with the AI-based system could become conducive to moments of flow, or how it stimulates in-the-moment awareness in the performer.

The session is structured in two parts: the part in which we are training the system, followed by a structured improvisation. Diana guides Kate through these two phases, and she describes this process as follows:

We are working with four actions: push, pull, caress and bounce. I ask Kate to take a moment and imagine in great detail a real context in which she would perform each of these actions, then I ask her to perform each action, in her imaginary context, as accurately as possible. After a first enactment of each precisely

21 This is the pressure applied on the resistive material representing by the stretch and fold/bend of the sensors, as activated by the muscles, and posture of the body engaged in that particular pose.

22 In Granular mode, the pre-recorded voice samples of the performer, while embodying these actions, were transferred into the system and mapped to her corresponding body postures while doing the actions. In Voder mode, we chose synthesised vowels that would resonate with the action.

executed action, I ask her to narrate out loudly details of the imaginary context, while performing the action. For example, she is now on a pier and she's pushing a heavy wooden crate. Next, I ask her to narrate how her body engages in this action. For example, she lifts her right hand, she engages her pelvic muscles, etc. When she's at the peak engagement with the action, I ask her to release the voice. This is the cue for Mika to start recording the data for training the system. The intention is that we achieve reproducible "push", "pull", "bounce", "caress" actions, that are accurately represented in the system.

The imaginary contexts and embodied narrations are meant as handles for Kate to accurately access that action in her body during the improvisation session. Once the system is trained with the data for all pre-defined physical actions, these postures are now mapped each one for a synthesised vowel, or each one for a synthesised vocal recording. In the guided improvisation, I instruct Kate to start simple, by going through all four actions in sequence. I suggest that she first tries to reproduce the actions as closely as possible to the training samples and become familiar with how the system reacts. Are the generated sound responses predictable? I guide her to focus both on her body and on the listening to the device. Can she control the sound? Can she create a song through her movements which is repeating the same melodical pattern?

The space in between the actions is mapped by the ML-module into spaces inbetween the pre-defined synthesised sounds. Next, I ask Kate to slowly play with variations to the initial actions, and by doing so, to discover the variations in sounds generated by the algorithm. I direct her focus of attention on listening. Can she, through movement, continuously explore the space of sound? I ask her to continuously search for sound and never for silence. Who is now in control, her or the AI? After she becomes familiar with the sound of the system I ask her to use it as a springboard for her own voice, etc. (Diana)23

Byintroducing the concept of "symbiotic gestures", Suchman(2017) notices that "humans and artefacts are mutually constituted". Observing Kate engaged in the process, we are witnessing a practice unfolding in intra-action between the human performer and the technological artefact. Kate is exploring the limits of the machine, and the machine reacts to Kate's actions. It is a continuous exploration of boundaries. From the sides, and by adjusting the parameters of this human-machine interaction, Mika is also modulating the boundaries between the human and the system. By suggesting shifts in attention focus, Diana is also altering the boundaries of this interaction. This type of experimenting explores in practice the notion that "agencies – and associated accountabilities – reside neither in us nor in our artifacts but in our intra-actions" (Suchman 2007). Scott is documenting our practice. Every session of structured improvisation ends in a reflection round.

23 Serbanescu, Diana. Personal note (Notebook). Entry from 3 Feb. 2022.

# Potential for augmentation

Our augmentation of practice focused on improvisation with AI as a relational partner. We were interested in observing the psycho-physical process of the performer involved in this partnership, the way in which this mode of working engaged her affect and imagination, and whether she achieved moments of flow. We were interested in the potential of this system for training the vocal presence of the performer. Kate describes moments of her discoveries as well as some as her challenges as follows:

For me the relationship with the sound became that – I felt like I was blessed in moments by a thick, association rich environment and then it might disappear. You saw a moment of disappearing there. Where I got disconnected would be those gaps, often.24

Or, later on she remarks:

It feels like there's a strong platform of sound in this position – a well of sound – that I can trust to support a melody. You'll see it's a sudden leap into quite a dense sound environment – and there's an interval there immediately. (Kate)25

Kate's statements point towards an engaged relationship with the AI at imaginative level, through sound. She also comments on achieving short moments of flow. Diana, as exterior observer, also notices that:

During a session of one and a half hours of improvisation, I can perceive an increase in Kate's presence and engagement, especially towards the end of the session.26

Later on, Diana reinforces this observation with another note in the annotation system:

Very interesting moment! This resembles very much a call and response. Kate, in this moment here, you've managed to establish a very beautiful "organicity" with the machine. It goes in the direction of a truthful partnership.27

24 Ryan, Kate. sessionone23FEB22.MP4.00:08:01.812.Text Annotation on Video. Motion Bank App.

25 Ryan, Kate. Sessionone23FEB22.MP4.00:08:09.575. Text Annotation on Video. Motion Bank App.

26 Serbanescu, Diana. Personal Notebook. 22 Feb.

27 Serbanescu, Diana. sessionone23FEB22.MP4.00:12:19.292. Text Annotation on Video. Motion Bank App.

Further witnesses on experiential trials of the wearable device come from Ilona, our invited embodied voice expert. She comments on her experience:

During the lab session in Berlin in December 2021, I had a chance to wear the device myself. This experience gave me an idea of how the technology can be used as a partner in the absence of another performer, enhancing sensory awareness and providing impulses that eventually guided me towards a flow and heightened psychophysical state while listening attentively to the feedback sounds produced by the machine.28

# **Strand Two: Reflections on the studio-based work and on perceptions of the AI**

During the last week of work in February, the work was documented with video recordings of the studio sessions.These were uploaded to theMotion Bank platform. We then used collaborative annotation to reflect and comment on the work that had taken place in the studio.

This had two objectives,one to reflect on the processes of developing the technology to augment the training practice as part of the design process. The second was to probe the perceptions and attitudes experienced by the team toward the AI-based system itself. Both drew primarily on questions to and responses from Kate, as the performer practitioner, whose relation to the AI system constituted the experience at the focal point of the research project. An example of the kind of question posed about her experience of the practice itself came from Ilona who wrote in the annotation: "*It would be interesting from my perspective if Kate annotated the moments of flow in the other video and reflected on the moments triggering change, any particular significant situations or moments where for example you got bored*".29 Kate responded by annotating several instances (see figure 4) in the practice session recording with the following sequence in close succession:

For Ilona – this is the tentative beginnings of trying to find a relationship with the sound / improvisation. Not bored yet, but definitely in testing mode, not flow. (05:10)

For Ilona – again, not bored exactly but definitely not flow. It felt like I didn't find much to follow there and I decided to cut to a different position. I think what follows about getting to an upright position is deliberate rather than in flow. (06:08)

For Ilona – perhaps beginning of a little flow here! (09:38)

28 Krawczyk, Ilona. Personal Correspondence (Email). 27 Nov. 2022.

29 Krawczyk, Ilona. sessiontwo23FEB22.MP4. 05:27. Video Annotation. Motion Bank App.

For Ilona – more flow here. (12:19)

For Ilona – definitely flow. (12:33)

For Ilona – I have a feeling I didn't know where to take the previous moment of flow. That I was scared of losing the connection with the sound so didn't follow it through past a certain point. (13:16)30

*Fig. 4: A sequence of annotations by Kate Ryan about "flow". Credit: Motion Bank*

In this context, Kate is reporting on the felt sense of "flow" to Ilona, who is an acknowledged expert in this form of embodied voice practice. As such, it is likely somewhat of an emic shorthand, denoting the practical knowledge of Kate and Ilona as recorded in this exchange between them. There are no references here to the AI, to the sensors or the computer, which is an interesting set up for questioning their perceptions of the AI systems.These questions tended to focus on the form of agency attributed to the AI system and explored concepts such as tools, agency, control and predictability. An example of one of these involved asking Kate about her experience of being in an embodied relationship to it. In figure 5, there is a question about 'control' and Kate has responded with an audio annotation.

30 Ryan, Kate. sessionone23FEB22.MP4. 05:10-13:16. Video Annotations. Motion Bank App.

*Fig. 5: Two annotations. One written and one audio. Credit: Motion Bank*

Written [edited] annotation question:

Kate you used the word control a few times, for example in another annotation you wrote about placing the sensors in specific locations for 'control purposes'. How are you thinking about the relationship to the computer here? Is it the computer you are controlling, the algorithm, the synthesizer? (scott)31

Audio annotation response:

It's a very interesting question, I think I'm guilty of anthropomorphising or accepting the technology much more than everyone else in the process and have been since the beginning. I'm not thinking at all about the computer, the algorithm certainly not, the synthesiser no. My initial relationship to the sound was much more to be controlled by it. To try and control it… which is something Ilona noticed very early in the process as well. And even when I did start to get some control in terms of having some knowledge of my body, which meant I could little bit predict what would come, I think that was still the basic relationship of that interaction. (kate)32

Here Kate is describing an evolving relationship with the technology. It makes clear this relationship isn't any one thing, but something that changes over time, in re-

31 deLahunta, Scott. sessionone23FEB22.MP4. 10:32. Video Annotation. Motion Bank App

32 Ryan, Kate. sessionone23FEB22.MP4. 10:32. Video Annotation/ Audio. Motion Bank App

lation to the specific embodied practices involved. Kate refers to herself as "guilty" of anthropomorphising the technology. There are studies showing when people anthropomorphize AI technology this has an impact on how much they consider the AI itself to be accountable for things that happen (e.g. Epstein, et al. 2020). This has a clear implication for AI and ethics, because it calls into question whether the machine should be held accountable or those who developed it. But Kate also says she is paying no attention to the computer (also reflected in her comments on Flow), in part because her attention is directed towards her embodied practice. What is interesting is how these two forms of reflection are entwined. One concerns the (embodied) feelings of the experienced practitioner in accomplishing the task, which involves paying some kind of attention to the location of the sensors, but the sensors themselves are not the focus of attention. The fact that attention can shift here to the sensors, to the AI system and some form of agency temporarily attributed to it seems to underpin the complexity and relational fluidity that exists in these kinds of assemblages.

# **Conclusions and Areas of Further Research**

We believe there is rich potential for future research and exploration involving the study of small-scale artistic research projects like this one (as a techno-social system in miniature) with a view toward developing and refining tools for these entwined forms of self-reflection. As previously stated, AI-based systems are products of collaborative and team effort. If these efforts are also viewed as assemblages, the question is not whether or not the same kind of self-reflective processes might apply, but how to apply them. This points toward an ethics-in-practice approach that could potentially inform the wider discussions about AI and ethics. Therefore, we call for more projects like this one to be further studied and consulted by other fields.There are other research areas that can be further developed here as well. One that our project has already begun is creating a new epistemology of practice that goes somewhat against the original ethos of the post-grotowskian training: originally this training was specific about human-to-human interaction, and embodied exchange of energy. And it is interesting to discover what changes when the machine gets involved, and what the limitations of this system are when it encounters such an established tradition of embodied practice and research.

# **Bibliography**

Adam, Alison (2006): *Artificial Knowing: Gender and the Thinking Machine*, 1st ed. [Kindle] Routledge.

# **Sound of Contagion – An Artistic Research Project Exploring A.I. as a Creative Tool for Transmedial Storytelling**

*Wenzel Mehnert, Robert Laidlow, Chelsea Haith & Sara Laubscher*

### **Introduction**

Sound of Contagion (SoC) is a transmedial art project addressing the cultural narratives surrounding global diseases and pandemics through different media. It is borne out of the collaborative partnership between the University of the Arts in Berlin and the University of Oxford and explores the use of Artificial Intelligence (A.I.) in creative processes.We used texts written by the machine learning algorithm GPT-2, which was trained on narratives about pandemics from the last 2500 years. From the resulting text fragments, we built a storyworld that served as a basis for narratives, illustrations, musical compositions and a lecture performance.

Besides the aesthetic and creative output, the project is a research endeavour that explores the usage of A.I. as a creative tool to facilitate interdisciplinary working. Through the practical collaboration between artists, researchers and technology, the imaginary that A.I. can independently and consciously create art is shifting towards the notion that A.I. becomes a tool that inspires and encourages collectivecreative processes. Thus, the work follows a collaborative paradigm and acknowledges A.I. as a creative tool with its own unique aesthetic.

This article is an insight into the project. It is a case study of how A.I. is used as a creative tool within an interdisciplinary collective working in the fields of literature studies, cultural studies, composition and visual arts.

#### **About the project**

The project Sound of Contagion (SoC) understands itself as a practice-based research project that uses artistic practices as an approach to understand A.I. as a creative tool. It is borne out of the Oxford / Berlin Creative Collaborations activities (supported and funded by the University of the Arts in Berlin (Germany) and the

University of Oxford (UK)), which fosters projects and research across the arts and humanities, and supported by the Minderoo Foundation through the Minderoo-Oxford Challenge Fund.

From the beginning, we focused on an interdisciplinary collaboration, as each member had a different artistic and academic background: Chelsea Haith, with a background in publishing, journalism and gender studies, and Wenzel Mehnert, with a background in cultural studies and narratology, represented the humanities. Sara Laubscher, working as a professionalillustrator, and Robert Laidlow, composer, researcher and AI-expert, represent the artistic side. Through the interdisciplinary nature of the project, it started with several different research questions that were guiding the thought process without limiting the creativity and necessary openness of an artistic research project:

How can A.I. be used as a tool within an interdisciplinary, creative project? How do we acknowledge the unique agency of the tool? How do we integrate it within a human-technical collaboration?

The project started in the beginning of 2020, shortly after the Covid19 virus was first reported in Europe. We took this global transition into the unknown future of lockdowns and social distancing as the starting point for our endeavour.Through the international distribution of our project partners and the lockdown situations, we were forced to set up the project solely through digital communication technologies like Zoom and Miro. Some of us had not met in person, but only digitally.

# **Dataset, Data & Processes**

#### Dataset

Cognitively occupied with the pandemic, the project became an opportunity to process and deal with the shifting situation through the experiences. We decided to work with narratives of previous generations and how they dealt with pandemics, epidemics or similar global phenomena. Principally led by Chelsea Haith at this stage, we selected texts, all dealing with pandemic and plagues, as the dataset to train the A.I. and curated the selection from the last 2500 years of pandemic literature. We permitted novels and plays only. One reason was the focus on narratives and reappearing tropes that could hint at cultural patterns, the other was, that we were interested to see, what a machine learning algorithm might do with these.The selection of dataset already had interesting consequences for the outputs generated.

We divided the texts into two data sets – Sophocles to the year 2000 was one data set, and texts published between 2000 to 2020 the other. This splits the total dataset roughly in half.This split provoked its own insights into cultural preoccupations in the last two decades as well as the increased popularity and thus production of dystopian and apocalyptic narratives in popular literature in the same period. As a total data set, these texts serve as cultural artifacts and represent the collective memory from which we draw and confront the changing environment. The earliest of our selection was David Mulroy's translation of Sophocles'*Oedipus Rex* (Sophocles, 2014 [429 BC]), while the latest was Lauren Beukes' *Afterland* (2021). These texts were used to train the A.I. model with the aim to generate new text fragments based on the dataset.

#### Data

To create an A.I. model that could generate new texts, based on existing texts referring to pandemics, we first had to make two decisions:

What would comprise the dataset upon which the A.I. was trained? Which A.I. algorithm would we use?

The choice of training data has been shown to make an enormous difference to the output of generative A.I. algorithms, even when other parameters are left the same (Leeming 2022; Mehri et al. 2017; Caillon & Esling 2021). Similarly, two different A.I. algorithms trained on the same kinds of dataset can provide entirely distinct generations; compare, for example, 'WaveNet' (Van den Oord et al. 2016) trained on Beethoven sonatas with e.g., PRiSM-SampleRNN re-implementation trained on the same (Melen 2020).

We first decided upon the choice of A.I. algorithm, which was to be GPT-2 Simple (Woolf 2019). GPT-2 Simple is a reimplementation of the popular and well supported GPT-2 A.I. algorithm (Radford et al. 2019), developed and released by OpenAI.We chose this algorithm because of its capacity to be fine-tuned to produce specific types of text, while still utilizing rules it has learned through general training. In this context, fine-tuning means that we began with a GPT-2 model that had already learned the 'rules' of English and then we directed this model specifically to learn to apply this to the context of our dataset. We felt this made it superior for the task we had in mind to other contemporary algorithms, which relied only on the fine-tuned dataset (e.g., WordRNN). We also had prior experience with this algorithm (de Roure et al. 2019) which helped in speeding up the early process. Finally, GPT-2 Simple is easily accessible online through Google Collab, and does not require a GPU on a personal computer to train. This allowed each team member to access it with their own machine during COVID-19 lockdowns, facilitating easier discussion and collaboration.

For the choice of dataset, we decided upon a set of 21 pandemic-related fictional texts, taken from across the last two millennia (as described above). This decision was reached as a compromise between wanting a substantial amount of data to train GPT-2 Simple on and our interest in being able to – or at least, thinking we were able to – draw links between specific GPT-2 Simple generations and specific works in the dataset.We felt that identifying the A.I. algorithm's reference points would assist in the world-building and narrative probes element of the project.

# Training the Models

We initially trained GPT-2 Simple several times using this dataset, creating a variety of models.Most important parameters included the base type of model that we then fine-tuned (124M or 355M), how the dataset was divided, and for how many steps the model was left to train before we sampled generations from it. From this initial experimentation and iteration, we chose a set of parameters we felt were most suitable for our project.This decision was made collaboratively and subjectively, according to our qualitative judgment of the models' text generations.

To create the text, we made the following decisions:

We used the 124M model as a base. Generations from this baseline model tended to reflect more strongly the ideas of pandemics that our dataset focussed upon.

We split the dataset (in full, 12MB in size), into two halves.These were texts written prior to the year 2000 (8.1MB in size) and post-2000 (5.7MBin size). Each dataset was therefore more homogenous than the full dataset, which allowed for finer control of text generation.

We trained the models for 4000 steps on Google Collab.

Once we had created the two fine-tuned models (pre-2000 and post-2000), we sampled text generations from them. These generations were both 'seeded' and 'not seeded'.

In this context, seeded means that we provided the beginning of a sentence and the A.I. algorithm continued from that point. Not seeded meant that the model was freely generating text. Seeding texts allowed us to 'point' the model towards talking about specific elements of pandemics, whereas not seeding allowed us to see how the model would 'write' without a specific context, after having been fine-tuned on a pandemic dataset. Generations also have a 'temperature' between 0 and 1 which is a rough indication of how much randomness is introduced into the system. The 'temperature' of the generations we created was generally 0.8 as this gave us interesting texts that made, in our opinion, at least some kind of sense. There was some variation here, with some generations used being as high as 0.95 or as low as 0.5.

On balance, the fine-tuningmethod seemed to us to be a successin that themodels would consistently write about plagues, fevers,illness, chaos, death, destruction, displacement and other related themes, even when they were not seeded. While the models did not appear to overfit (copy exactly from the dataset), it was sometimes clear that some parts of the dataset had a greater prominence than others in its generations. A stand-out example would be Stephen King's *The Stand* (1990 [1978]) in the pre-2000 dataset, which was particularly long at 1152 pages.

Through this iterative process of training, sampling, dataset adjustment, and further training, we began to get a feel for how the two fine-tuned models wrote, and how we might get different kinds of responses out of them. We instructed the two models in exactly the same way, providing us with a contrasting pair of generations for each seed. The post-2000 model, for example, was prone to creating successive lines of dialogue. We used several texts from this model that were seeded with a single quotation mark and with single-word pronouns. The pre-2000 model more often described landscape in more poetic language. From the pre-2000 model we utilized more generations seeded with phrases such as 'the pandemic', 'the plague', and 'when it began'.

With the training of the models complete and several hundred generations sampled, the project moved to the next stage involving world-building, narrative probes, and transmedial collaboration.

#### **A.I. as a creative tool**

One of the guiding principles of the project was to treat the results of the A.I. as a unique form of creativity and acknowledging the creative agency of the algorithm with its own flaws and failures as a new aesthetic form. One of these "failures", in the context of fictional writing, is that the text itself does not hold any sense – as the author did not mean to create any sense in the first place. One figure of thought that helped us to understand our relation to this text comes from Roland Barthes famous essay *The death of the author* (1978 [1968]). Here Barthes proclaims that the origin of a text does not lie with the author, but with its destination: the reader (142 ff.). Barthes starts with a description of an author which he argues against:

The Author, when believed in, is always conceived of as the past of his own book: book and author stand automatically on a single line, divided into a before and an after. The Author is thought to nourish the book, which is to say that he exists before it, thinks, suffers, lives for it, is in the same relation of antecedence to his work as a father to his child. (145)

 In our case, there is no author – It is not even dead as it has never been alive – and therefore no message or moral to convey is purposefully inscribed in the text. Furthermore, the A.I., particularly the GPT2 algorithm we used, still lacked a semiotic understanding. It did not make connections between the characters, or the places mentioned in the text fragments. Rather, the A.I. produced text fragments that are similar to the narratives used as training data, without understanding the context or the connections between the elements of the narratives. It recombined the passages

we fed it with, and the results are an arbitrary combination of different writings, like Barthes describes in the following:

A text is made of multiple writings, drawn from many cultures and entering into mutual relations of dialogue, parody, contestation, but there is one place where this multiplicity is focused and that place is the reader, not, as was hitherto said, the author. The reader is the space on which all the quotations that make up a writing are inscribed without any of them being lost; a text's unity lies not in its origin but in its destination" (148).

In the context of fictional writing, this means that the meaning of a text is given by the reader, who constantly recreates the sense and searches for coherence – this is particular true when reading text fragments created by a GPT2 algorithm, as they not only lack meaning but also a form of coherence. The non-coherent text created by the A.I. forces the reader to constantly seek for a meaning and brings the reader into the active role of interpretation and drawing connections. This is the unique aesthetic of text fragments produced by a GPT2 algorithm. Thus, as the A.I. was not capable of doing it, the practice of sense-making and creating coherence had to come from us.

# Worldbuilding

For this reason and following the guiding principle described above, we considered all the text fragments as *Narrative Probes* (Fischer & Mehnert 2021). Narrative Probes are flash fiction like narratives that all probe into the same possible world, following a specific logic that we had to uncover.The narratologist Marie-Laury Ryan uses the metaphor of the text as windows to the extralinguistic realm of characters, objects, facts, and states of affairs serving as referents to the linguistic expressions.  Ekman & Taylor call this process *readerly worldbuilding* and constitute that the reader carries out a construction of a world by adding pieces to a structure already in place evaluating the pieces in light of what is known about the world so far, and, conversely, evaluating the whole world in light of new pieces (Ekman & Taylor 2016: 11). With this perspective in mind, we treated the text fragments as a multitude of windows that showed us snippets of a storyworld that we as readers had to piece together. Therefore, through a structured deconstruction of the text fragments, we engaged in a worldbuilding process based on the textual cues that we found in the fragments, loosely building on the work byWolf (2012) and Mehnert & Fischer (2021):

In the first step we detected the elements of the world. For this process, five cues were of specific importance to us: Places, events and references to time, characters mentioned, values and norms as well as specific objects.

In the second step, we enriched the elements with additional information that either came from the text or was already given by the logic (e.g. when the text refers to a leader, there is also a group of people following).

The third step was to create connections between the enhanced elements (e.g. if two characters were mentioned in two different text fragments, and we assume they both exist in one world, what is the connection between the two?).

### Example

To give an example, we will explain the process with the following two text fragments:

#### Fragment I:

*They came to a real leader, they said to each other.  He was real. He lived, and he led his people to a real leader. This real man was the one they called their real leader. He was named Mannum, and Mannum knew this about himself; he had practiced seduction and charm the long time. He was a good liar, a real one. The pandemic of war had begun.*

Fragment II:

*"Then, friend", said Piranesi,"make death eternal in the liquor laws, so that no sick person may return to drink after consuming what they drank".*

In Fragment I we learn two important things about the world: (1) In this world exists a character called *Mannum*. We can say that *Mannum* is not an honest leader and a good liar, thus he is not to be trusted and rather an evil character. (2) We also have an event, the *pandemic of war* that stands in connection to *Mannum* as it has been mentioned right after. So in the world that we want to explore, there is some kind of a biological war going on but we do not know why and who else is involved.

In Fragment II, two characters are mentioned. The first character, named Piranesi, is talking about *making death eternal*. When enhancing this caracter, we assumed that Piranesi is a chemist who was responsible for creating a deadly disease. The second character in this Fragment is *friend*. We decided that *friend* is a reference to Mannum, the leader from Fragment I, by which we connected both fragments. Based on the cues given in these two text fragments a narrative about a conspiracy plotted by Piranesi and Mannum to create a biochemical weapon to start a war is formed.

# **Outcomes**

# Nested Narratives

Working with A.I.in such a way can be seen as an intriguing source of inspiration. It gives you material to play around with, to dig in, to take out, to transform, to iterate and to make sense of.The result of our sense-making process was a nested narrative, meaning a story within a story, within a story. Each part plays in a different time and features different characters, however, they are still connected with each other.

The first part is a short story from the 17th century about the two characters introduced above, Piranesi & Mannum. Both are plotting a war using a biochemical weapon that fails and extinguishes Mannums kingdom and the people living within. In the second part, we follow the story of a little girl called Emily. At the beginning of the story, she reads a comic about the story of Piranesi & Mannum, this is how the first narrative is nested within the second. While she reads the story, a TV news anchor is presenting the latest development of an actual pandemic that is raging all over the world. We situated this situation in the year 1981 and the pandemic, which the news anchor is referring to, is the AIDS pandemic. Emily's narrative is again nested within a third narrative. The last one plays in 2021. Here we have text fragments about a person that we called The Writer. He is similar to our real lived experience in 2021. Due to the Covid-19 restriction and the social lockdown, he stays alonein a small apartmentin New York. To stay sane, he starts to write a story about a young girl called Emily, which is how the second narrative is nested within the third. Furthermore, the writer mirrored our own experience, as we were in lockdown during the production phase of this project as well.

# Illustrations

As mentioned in the introduction to this project, Sound of Contagion follows a transmedial approach to storytelling, thus we transformed the narratives into different other mediums beyond text. The first transformation was based on the illustration by Sara Laubscher. We picked key scenes of each nested narrative as prompts for Sara, who illustrated them in her own style. She got inspired and created the second output of our project.

The first illustration shows Mannum and Piranessi, sitting in an old library plotting how to use the biochemic weapon. In the secondillustration we see Emily whois sitting in front of a television set.The caption of this image is the report of the news anchor.We read it as the voiceover of the newscast.This section indicates that it was drawing from the part of the data set coming from the 20th century rather than from the Renaissance.The discourse around virus transmission, the effect on different geographical regions, and the conceptualisation of the pandemic as a weapon all point to the effect of social collapse during pandemic conditions. At the same time, we can identify reappearing tropes that often come up when we are dealing with topics of contagion, plagues or pandemics. The Writer, who became a point of self reflection but also the mastermind of our narrative, is depicted in the next illustration. Here we see the lonely writer sitting in his small apartment, haunted by his memories and trying to stay sane by coming up with stories about pandemics from different centuries.

*Fig. 1: Illustrations from the Sound of Contagion project. Copyright: Sara Laubscher*

As mentioned above, the illustrations were the first transformation of the storyworld. The next transformation came in the form of music.

### Musical Adaptation

The Sound of Contagion text was further adapted into several pieces of music by Robert Laidlow, and one by composer Marco Galvani. These include *The Writer*, premiered online during COVID-19 lockdowns and the piece *Disc Fragments,* both performed by Bandwidth Ensemble.The following is an insight into the creative process of Robert and how he reflects the process of working with the text fragments.

For a 'Sound of Contagion' event at Oxford University in November 2021, I (Robert Laidlow) composed a piece for tenor and synthesizer called *Disc Fragments*, which used texts generated as part of this project.This piece is in seven movements, the first three and last three being somewhat symmetrical – that is, Movement 1 is similar to 7, 2 to 6, and 3 to 5. This arch form was a parallel to the nested narrative form that Chelsea Haith assembled from 'GPT-2' texts we had generated earlier. The texts for six movements (all except Movement 4) were generated by fine-tuned 'GPT-2'. These texts dictated the form and material of the music; here, the first movement, *At Delphi*, is discussed as an example.There was a certain mysticism that intrigued me in this text:

*Fragment III:*

*At Delphi A Prayer Let not the Impossible Him The Impossible Him take what he has done Take it, Steal it, put the word in your mind, take it, the other choice.*

The use of 'Delphi', 'the Impossible Him', and 'put the words in your mind' implied to me that 'GPT-2' could be read as imitating an oracle. I wanted the music to create a sense of ritual and of unseen pattern, similar to (for example) Messiaen's *Quartet for the End of Time*. The voice part is constructed from a series of interlocking patterns. Ten pitches repeat, superimposed on a rhythmic pattern repeating every nine notes. The text is simply applied to this pattern. The synthesizer is set to an organ voice and uses chorale-style material generated by AI, using an algorithm developed by Omar Peracha (https://omarperacha.github.io/make-js-fake/).This repeats itself every twenty-three beats.

In this A.I. generated text, among others, I was struck by its sudden ending. When 'GPT-2' is generating, the user instructs how many characters to generate. Due to its transformer architecture, 'GPT-2' does not plan forward. It only looks back to what has already been generated whenever it generates a new token. When it reaches its arbitrary character limit set by the user, it simply stops. It cannot plan a 100-character length 'story', for example, because it is not able to plan forward. In this movement the music simply stops when the text comes to an end, which does not coincide with the end of any pattern described above. The inspiration for such a technique comes also from composers such as Birtwistle (i.e., *Carmen Arcadiae Mechanicae Perpetuum*) and Edmund Finnis (i.e., *The Air, Turning*).

#### Lecture Performance

As already hinted at in the paragraph above, the last transformation of the work came in form of a lecture performance at the University of Oxford in November 2021. The performance was a mixture of different media and perspectives. The first part introduced the process, the creation of the fragments and explained the worldbuilding process. In the second part we read the narratives, presented the illustrations, and performed the musical pieces on stage. Throughout the Performance we presented different perspectives of the project through different means of interaction: while Robert and the musicians were on stage, both, the reading of the fragments done by Wenzel as well as a contextualisation of the fragments by Chelsea, came via zoom. After the presentation, there was a Q&A session in which the audience got invited to discuss with the performers, pose questions to the project and engage in a discussion.

#### **A.I. for transmedia collaborations**

Through this project, we found that A.I. acted as a useful way to facilitate collaborations across different media and disciplines. We were each able to use algorithms generating text and images, despite each having very different levels of previous experience with algorithmic processes or computer programming in general. This allowed us to contribute meaningfully to artistic areas that were not naturally our own. Going forward, it seems as though A.I. might be useful as a means to 'try' another collaborator's discipline.This might allow collaborators to much more quickly engage in higher-level discussions and decision-making.

In addition, using A.I. as part of a collaborative artistic research project inherently mandated many creative decisions to which we all contributed. During the training portion these included decisions such as what comprised the dataset or datasets, the type of model we were going to fine-tune, what our preferred parameters were when training the models, and how we were going to sample generations from these models. Later, they included which elements of the generations we would keep or discard, and how much (if at all) we would adapt or edit them. Since there was no dedicated human writer on the team, these decisions were made collectively.

Utilising A.I. as part of a creative process also differed from a normal collaborator in the sense that A.I. does not fundamentally understand the context of what it is writing.While it can create plausible-sounding sentences and stories,it is not aware that it is doing this, and there is no intentionality behind any of the 'creative' decisions it makes. This is very different to most human writers. We therefore felt empowered to make radical decisions when treating the text, including transforming it into entirely different media, but also felt liberated in leaving it in its found-state, even if that found-state was nonsensical and narratively underdeveloped. None of our readings of the material it generated could be 'right' or 'wrong'.This is especially true of the GPT-2 model we were using when compared to models released closer to the time of this paper, which have shown significant improvement in understanding underlying context of text and ability to develop logical statements (Griffiths 2022).

#### **Conclusion**

In this project, we explored the use of A.I. as a creative tool for transmedial storytelling. The project points to the cultural-semantic flaws, which are peculiar for the creative results of A.I. works and fills this gap with the sensemaking work done by human actors. SoC challenges the myth that A.I. could replace authors, poets or other creators of art, entertainment or fiction in general. Instead, by using them in a creative process, A.I.s make an important contribution to new art forms, foster inspiration for artistic practices and thus promote independent modes of expression.

We started with a GPT-2 algorithm, trained on a vast amount of fictional literature about pandemics and plagues, that generated several incoherent text fragments. Through the process of worldbuilding, linking the fragments and creating an imaginary world, we acknowledged the unique aesthetic of the A.I., its flaws and particular form of nonsense-writing, and made the tool part of the creative collaboration between different disciplines and practitioners. By documenting the process, we showed that A.I. is not an autonomous actor, that creates art on its own, but instead, that it has the potential to become a tool in an emerging toolbox and that it needs new artistic approaches to make use of this tool and shape it to its own desires.

Last but not least, this insight into the process also serves as an inspiration for continuous, creative engagements with A.I. software. As past events have shown – and surely future development will proof – A.I. technology becomes a fixed part in our everyday life and in the workflow of designers and artists.Therefore, It is necessary to explore approaches of adaptation, appropriation and domestication within new routines and creative processes.

### **Bibliography**

Barthes, Roland (1978): *Image-Music-Text*. Hill and Wang.

Beukes, Lauren (2021): *Afterland*. Penguin.

Van den Oord Aaron / Sander Dielemann / Heiga Zen / Karen Simonyan / Oriol Vinyals / Alex Graves / Nal Kalchbrenner / Andrew Senior / Koray Kavukcuoglu (2016): *WaveNet: A Generative Model for Raw Audio*. *ArXiv.* arXiv:1609.03499v2 (cs.SD).

Wolf, Mark J. P. (2012): *Building Imaginary Worlds*. Routledge.

Woolf, Max (2019): *GPT-2-Simple* https://github.com/minimaxir/gpt-2-simple (21 January, 2024).

# **Discography**

Birtwistle, Harrison (1978): *Carmen Arcadiae Mechanicae Perpetuum*. Finnis, Edmund (2016): *The Air, Turning*. Messiaen, Olivier (1941): *Quartet for the end of time*.

# **Challenges and Opportunities for Computational Construction of Narratives**

*Pablo Gervás*

# **Introduction**

As Artificial Intelligence enters a new era where large language models show surprising capabilities for handling language, it becomes important to review the challenges and insights accumulated by the community of researches that have worked on building computational modelling of literary creativity over the years.The efforts of this community have at times achieved success at small tasks and at other times faced failure when over-ambitious goals were pursued. It would be important for efforts on related tasks undertaken from this point on to keep in mind the insights accumulated on the nature of the task and the challenges it presents. The present paper attempts to summarize some of these insights into a set of challenges that face the automated generation of narratives, but also as a set of opportunities open for the future.

The paper is structured as a review of a number of possible ways to formulate the problem of having a program generate automatically a story on demand, followed by some reflections on how storytelling might be subdivided into a set of related subtasks as suggested by that set of formulations of the problem.

# **Programs that Deliver Stories**

The possible ways of building programs that deliver stories explored in this paper are presented from the simpler formulation towards increasingly complex approaches. At each point the challenges (or lack thereof) and opportunities presented are analyzed, trying at each point to lead on to the approach discussed after it.

# How to Request Stories

The simplest possible formulation for requesting a story from a story generation program is one where there is no input and the output should be a story. Such a program might be built by considering some existing repository of stories and to select from it one story at random. The user would get a new story every time, and she would have difficulty to tell whether the story has been generated by the system or selected from a set of prewritten stories. A slightly more elaborate approach would be to consider an input in terms of a series of keywords or phrases to narrow down the set of possible stories returned. The development of a system under this approach requires the existence of a vocabulary for presenting queries formulated in terms of various characteristics such as: details about the concepts in the story world, details about characters and relations between them, plot structures that capture causal and temporal relations between events in the story, or details about the potential effect of the story on its audience. The most elaborate approach would be to provide a text prompt that describes the story in some way, and have the story generator provide a story matching the prompt.

Existing story generators have progressively shifted from the early attempts that worked on an empty input to the most recent neural solutions that accept complex text prompts, with intervening systems that worked on keyword-based queries of different complexity.

# Original Stories

If one further establishes the constraint that the outcomes must be original – that is, not already available in some form – the task becomes more difficult. Two different challenges need to be met: how to build a story and how to judge if a newly built story is sufficiently different from those already existing to be considered original.

### Building Stories

The task of building a story from constituent elements, in the hope that the result may be original, requires solving two basic problems: what to consider as constituent elements and what type of procedure can be employed to put together those constituents so that the result be a valid story. Several approaches have been considered in the past, but only some are reviewed here as illustrative examples. For more detailed reviews, readers are referred to existing surveys in the field (Kybartas / Bidarra 2016; Alhussain / Azmi 2021). Some approaches consider breaking existing stories into pieces and then recombine the resulting set of pieces from a large set of different stories into new instances of stories.

The simplest approach of this type relies on individual words as the pieces that are obtained from stories and then recombined. Attempts to achieve this based on statistical models of have proven successful in the past at producing legible text, original but usually lacking sense (Brown et al. 2015).This procedure is also the basis for the recent work on transformers (Wolf et al. 2020) that has become the rage in the world of AI. It is only when systems built based on these models have started to be prompted with full sentences describing the desired output that successful results have made their appearance.We will address these models in the section "Acceptable Stories", when I discuss more elaborate ways of requesting a story.

Another approach to generating stories from pieces of prior stories involves using instead of words some abstraction of the meaning of stories and finding ways of recombining these abstractions into conceptual representations of a story that can then be transcribed as text. That is essentially what story generation based on planning does (Young et al. 2013). In this approach, a set of planning operators is constructed as an abstraction of the events that appear in a story. Each *planning operator* represents an event in a story – usually in some form of predicate logic that allows a predicate to represent the action and a set of argument variables to represent the characters or objects that participate in it – but it also encodes additional information of which other predicates are preconditions or post-conditions of that event. This allows a planner to create chains of events that are causally linked to one another, from an initial situation to a goal. For planning-based story generators, both the initial situation and the final goal are usually provided as input.

This type of approach is faced with the important challenge of having to generate a body of planning operators of sufficient coverage to generate a broad range of stories, which constitutes a significant bottleneck for the approach. Attempts have been made to solve the problem by means of advanced methods of knowledge engineering (O'Neill / Riedl 2014) and crowdsourcing (Guzdial et al. 2015).

Other efforts rely on similar abstractions of the meaning of events in a story as predicates associated with pre-conditions and post-conditions but forego the emphasis on goal-driven causality of the planning paradigm. Such systems design their abstractions for story actions based on different conceptual approaches.The Mexica system (Pérez y Pérez 1999) relies on a set of story actions that associate character emotions and tensions between characters as preconditions or post-conditions.The PropperWryter system (Gervás 2015) relies on existing abstractions on plot relevant actions (Propp 1928) and dependencies established between them by virtue of being actions associated with narrative roles such as villains or heroes.

This approach also suffers from the lack of a broad vocabulary of story actions to employ. This is known in artificial intelligence as the *knowledge acquisition bottleneck* – a long-standing problem (Cullen / Bryman 1988) that remains current to this day (Pasini 2021). Supported by a knowledge engineering effort to annotate plots of musicals (Gervás et al. 2016), the PropperWryter system achieved success when it was engaged in the production of the first computer-generated musical, which was staged at the London West End for two weeks in 2016 (Colton et al. 2016).

#### Judging Story Originality

However, building complete stories up from elementary blocks is no guarantee that the stories will be original. If the blocks used to build them correspond to elements that appear in already existing stories – and this is usually the motivating requirement when building knowledge resources to use as such building blocks – and the construction procedure is guided by reasonable heuristics, there is a non-zero probability that the building process resultin stories very similar to the ones thatinspired the construction of the knowledge resource. Furthermore, when story construction efforts of this kind have to be maintained over time, it becomes important to devise means to avoid the repetition of stories built previously by the system.

To address this challenge, research has been carried out on developing means for deciding when a story is similar to another (Fisseni / Löwe 2014; Hervás et al. 2015) or when a story can be considered sufficiently novel (Peinado et al. 2010). The existence of these effortsis very significant, becauseit opens a new avenue of research on computational storytelling, one where the systems do not only generate stories but also include the ability to produce judgments of some kind over them. This follows a general tendency in the field of computational creativity to progressively evolve from systems that merely generate artifacts to systems that develop their own aesthetic and which are capable of defending why the artifacts they produce are valuable (Colton / Charnley / Pease 2011).

### Acceptable Stories

The need for systems that construct artifacts to include procedures for assessing the quality of their outputs arises from the fact that such systems essentially explore a search space of possible artifacts, and not all the artifacts in that search space are equally valid. Once a specific type of artifact is defined in a way that allows instances to be constructed computationally,it becomes very easy to produce a very large number of different instances of it.

To address this challenge in the field of narrative generation, recent research efforts have focused on defining metrics for story quality. Following traditional engineering practice, these efforts usually identify one specific aspect that is known to impact the perception of quality and attempt to model it, without necessarily considering other aspects that are also relevant.Examples of aspects that have been considered in the development of metrics for story quality are: character believability (Gomes et al. 2013), semantic coherence across the story on relevant events such as birth / death or romantic entanglements (Gervás / Concepción / Méndez 2021), reproduction of features observed in human-written stories (Leon et al. 2020), consistent use of entities across the narrative (Papalampidi / Cao / Kocisky 2022) or probability of each sentence in the story with and without its preceding story context (Sap et al. 2022).

The set of relevant aspects that should be considered is very large, and efforts should be made to integrate these various models so they can be applied together onto the same stories. The quality of story generators would improve significantly once these metrics can be applied as filters to their output.

The integration of a metric on story quality – essentially a story critic module – into a story generator presents another important challenge. Story generation systems are generally constructed on the basic assumption that they take an input and generate an output that is a story. The idea that one should read the output, reflect upon it and then, based on that reflection, rework it in some way is fundamental for the human approach to writing (Flower / Hayes 1981; Sharples 1999). However, generation systems constructed on a similar basis are few and far between. The Mexica system is a remarkable exception, being based on Sharples' cognitive model of the writing task (Sharples 1999).

There is another view on the quality of stories that differs slightly from whether it is simply a good story. This view considers whether the story is acceptable as an instance of a story described in some way by the user that requests it. Many of the features described above as potential input could be used to narrow down the set of stories that are acceptable in response to a specific input: categories of stories, details on the world or the characters, plot structures or potential effects of the story on its audience. If an interface of this type is made available, a story returned by a story generator in response to a specific request would only be acceptable to the extent that it matches the given request.

The recent development of neural-based solutions for text generation such as GPT-2 and GPT-3 (Zhang / Li 2021) makes this type of solution capable of responding to user prompts with large paragraphs of valid and fluid text. These solutions are based on a large language model trained over a neural network representation, and these models essentially capture the relationships between words as featured in examples of text written by humans.The interactive nature of these solutions has made it possible to present requests for stories as simple sentences that describe the desired story. The tests carried out so far on how acceptable the responses of these systems are in terms of their correlation with the given prompts show outstanding results for instances of general conversation, even though testers remain unconvinced of their general validity (Elkins / Chun 2020). More exhaustive tests need to be carried out on the applicability of these solutions in the realm of narrative generation.

#### Stories for a Specific Purpose

This observation leads into the final but not less important challenge that story generation faces: generating stories for specific purposes.The concept of purpose is very difficult to represent formally in terms that a computational system would understand. Yet it is crucial for guiding the composition of any message in any media that is aimed at communicating to an audience (Smedley 1952).

Story generator systems very rarely consider any representation of purpose among their inputs. In most cases, when a story generator is designed there is some idea in the mind of the designer of what the generated stories should achieve. As a result, story generation modules are sometimes included in systems designed for very clear purposes, such as teaching children about bullying (Aylet et al 2007), supporting emergency rescue training (Hullett / Mateas 2009), or for military training (Zook et al. 2012). In such cases, there is a clear purpose that the generated story has to fulfill, and the generator is designed with that purpose in mind. But the identification of the purpose and the tuning of the generation mechanism to ensure the generated stories achieve it are wholly in the mind of the designer and not explicitly modeled in the system.

The main body of research on storytelling has focused on generating stories whose content is constructed at the same time as the story. This actually sidesteps one of the main purposes of storytelling as used by people, which is to convey a set of events that has actually happened. In this case, the overall description of the task changes significantly. The set of inputs to consider to a potential system that builds such stories must necessarily include a description of the events that the story needs to convey. In most cases, the construction procedure is expected to respect that set of events and not introduce any additional ones that did not occur. This slightly different formulation of the task has been addressed much more rarely but instances exist of systems that generate stories about events that the story generator receives as input: stories about past interactions between a user and an intelligent agent (Behrooz / Swanson / Jhala 2015), stories abstracted from the moves of the pieces in a given chess game (Gervás 2014), stories about a user constructed from data on their routines acquired via sensors (Reddington / Tintarev 2011), stories to explain cybersecurity logs (Afzaliseresht et al. 2020), narratives from personal digital data (Farrow / Dickinson / Aylett 2015) or narrative biographies from knowledge extracted from the web (Kim et al. 2002).

Living as we do in world obsessed with fake news, we need also to consider a revised version of the task where a story is generated that is based on a given set of facts, but purposefully departs from it at some point when building a story about them. This is another storytelling task that is intuitively familiar to most people: biased historians or politicians do it often to their audiences, parents do it to soften the world for their children, transgressors do it to hide their offences. It is also the key task underlying historical fiction or fictional accounts of recent events. From a computational point of view this task can be understood as a combination of the task of generating a new story with the task of telling a known set of facts. This particular view of the task has been addressed computationally by attempting to match the known set of events to a particular plot structure, allowing mismatching real events

to be omitted, and using the plot structure to provide additional fictional events to complete the story (Gervás 2018a, 2018b).

Another possible approach is to tell stories tailored for a specific audience. This is, yet again, an aspect of the storytelling job that is considered inseparable from the task itself by human writers, and yet very rarely addressed in story generation systems. Fortunately, we are beginning to see research aimed in this direction, such as a module to enhance a video game by being able to tell stories tailored to a specific player, based on a model for each individual player (Ramirez / Bulitko 2012).

Whether the goal is to achieve a particular purpose or satisfy a specific user, it is clear that generator systems would do a much better job the better they understand the potential reactions of the audience of their output. Computational models of how this might be achieved in terms of having specific models of the reader reaction included in the generation system have been proposed (Gervás / León 2016). Although this type of integration is yet in the future, efforts already exist to build computational models of the reading task. As in the case of metrics on story quality, specific approaches tend to focus on specific aspects of the task, such as understanding the set of events narrated in a given discourse (Niehaus / Young 2014), modeling the reaction of the reader in terms of an evolving curve of suspense (Doust 2015), interpretation of embedded stories told within a frame story (Gervás 2021), or reconstruction of the actual chronology of a story told in a discourse that allows flashbacks (Gervás 2022).

#### **Rethinking Basic Assumptions**

In view of the analysis presented to this point, it is important to accept that the concept of computational story-telling should be considered not as a single task, but as a set of interconnected tasks, all related to narrative, but corresponding to formulations that are very different in computational terms, as described above. Most of the existing systems for computational storytelling operate at the level of discourse as understood by Ricoeur (1976): a sequence of sentences, where each sentence involves a predicate applied to some entities that need to be identified by the subject (and objects) of the sentence. This view abstracts away from what Ricoeur describes as "the particular structure of the particular linguistic system", which would be closer to the text.

From the point of view of a detailed analysis of computational storytelling, this distinction between text and discourse provides a key tool to understand the broad range of solutions that have been developed over the years. Suppose we consider that a story describes a *story world* – often fictional but not necessarily so. Suppose also that its narrative structure can be represented by some form of *discourse*, that may be rendered as *text* but which may allow rendering in other formats. Under these assumptions, computational storytelling may be described in terms of a number of component sub-tasks that embody transitions between story world, discourse and text. The sub-task of *narrative generation* involves construction *ex novo* of either text, discourse or world.The sub-task of *narrative composition* involves building a discourse that tells some story from a world, or building a text that tells a given discourse.The sub-task of *narrative interpretation* involves reconstructing the conceptual discourse that underlies a given text, or reconstructing the world that underlies a give discourse. These ideas are expanded in the following sections.

### Constructing Narratives: Narrative Generation and Narrative Composition

Let us consider narrative generation as the task of constructing a story that did not exist before. Faced with the distinction between story world, discourse and text, an engineer wishing to design a story generator needs to decide how many of these levels she is willing to represent within her design. All these levels are interconnected, in the sense that a reader faced with a text will interpret a discourse as its meaning, and imagine a story world as its content.The question is at how many of these levels will the program operate.

The system could generate text directly, with no additional representations in terms of discourse or story world. This is indeed the choice favoured by neural approaches to the story generation task.

The system could focus on generating discourse, abstracting away from the complexities of generating language but still building a sequence of discourse – represented conceptually – that has a valid narrative structure. This presents the advantage that the story world does not have to be represented in the system. The approaches described above that rely on abstract representations of the meaning of stories as story actions operate under this choice.

Because people are better readers of text than of predicate logic representations, systems that generate discourse usually include a module that transcribes the discourse onto text. This process is not really generating the story, but rather finding a way of telling as text a story that already exists as discourse. We refer to this task as narrative composition.

The system could include an explicit representation of the story world. Stories can now be built directly by generating events or characters, or locations in the story world directly and then find a way of telling the story of what happens in her story world.This is the choice favoured by story generators based on simulation of an underlying story world (Ryan 2018).Under this approach, two further tasks are needed: one that transcribes possibly a selection of the events in the story world onto a discourse, and one that renders the resulting discourse as text. Both are instances of narrative composition.

With respect to the classic pipeline for natural language generation systems (Reiter / Dale 2000) the transition from story world to discourse corresponds to the task of content planning, and the transcription from discourse to text to the stages of sentence planning and surface realization.The content planning task will optionally include a task of selecting which events in the story world are included in the story, which aligns well with the recently formulated task of story sifting (Ryan 2018). Research efforts related to generation of stories based on known facts covered in the section "Stories for a Specific Purpose" correspond to tasks of narrative composition.

#### Processing Narratives: Narrative Interpretation

When the situation is presented in this manner, it becomes clear that there is a related set of possible computational tasks corresponding to the transitions from text to discourse and from discourse onto a conceptual representation of the story world. These correspond to subtasks of narrative interpretation. This task of narrative interpretation has been a long-sought goal in the field of artificial intelligence, originally known as natural language understanding (Allen 1995). Historically, a succession of un-met expectations over the years lead the natural language processing community to operate under a progressively shrinking scope, focusing more and more on specific sub-tasks instead of attempting anything like processing text onto an exhaustive representation of its meaning. Recent efforts to model identification of narrative structure (Gervás 2022) would correspond to the transition between discourse and story world, including the possibility of a given discourse referring not to one single story world but to a number of interrelated story worlds.

The attempts at developing computational models of the reading task would also constitute instances of the narrative interpretation task. Very little is known about this task, because it has only recently started to be treated computationally. This makes it stand out as an open field for challenges and opportunities for future research.

#### **Conclusions**

The panorama of research efforts described in this paper suggest that story telling needs to be considered not as a single monolithic task but as a set of interconnected sub-tasks. A simple distinction between text, discourse and story world provides means for describing some of these sub-tasks in terms of processes of narrative generation, narrative composition and narrative interpretation

In terms of how the existing research on computational narrative relates to human performance on equivalent basic goals related to the construction of stories, it is clear that computational solutions have a long way to go. The analysis presented in the sections "Programs that Deliver Stories" and "Rethinking Basic Assumptions" constitutes a fair review of the particular tasks considered to this point in past research on computational narrative. However, it becomes clear that, regardless of which of the combinations of transitions between the levels have already been explored in specific computational systems, the approach that humans apply to the storytelling task takes advantage of a very broad combination of this set of subtasks. Some writers plan out the plot of their story to the very end and only then sit down to write it. This would align with the idea of building a discourse and then telling it. Other authors construct a whole word and then find stories to tell in that world. Others sit down each day to produce a number of pages of text with no idea of how the plot is going to proceed as a result. Even the same writer may start from a sentence that sounds promising to him (text), work out in his head how the characters might react to the resulting situation (simulation on the story world) then rework it all to ensure that the revealing moment takes place at the optimal point in the scene to maximize its impact on the reader (iteration of discourse revision guided by a metric for impact), possibly guided by a dynamic model of how the reader progressively react to the sentences as they appear in the text.

As a result, to the various specific challenges describedin the paper onemust add the grander challenge of exploring how all of these specific challenges may interact in more complex processes of story writing.

# **Acknowledgments**

This paper has been partially funded by the project CANTOR: Automated Composition of Personal Narratives as an aid for Occupational Therapy based on Reminiscence, Grant. No. PID2019-108927RB-I00 (Spanish Ministry of Science and Innovation) and the ADARVE (Análisis de Datos de Realidad Virtual para Emergencias Radiológicas) Project funded by the Spanish Consejo de Seguridad Nuclear (CSN), Grant Ref. SUBV-20/2021.

# **Bibliography**

# **The Marcel Duchamp Case in, against, or after Artificial Creativity**

*Jan Løhmann Stephensen*

In the introductory chapter entitled "Even an AI could do that" of the book *Artificial Aesthetics: A Critical Guide to AI, Media and Design* (2021–23), which is currently being published chapter-by-chapter on digital culture theorist Lev Manovich' homepage, Emanuele Arielli,Manovich's co-author, notes that while some sorts of art with more 'traditional' or 'classical' characteristics seem quite straightforward for an AI to reproduce, the oeuvre of Marcel Duchamp poses a set of perhaps unresolvable problems. In this paper, I will discuss how this argument on some levels makes good sense, whilst on other levels less so. In extension of this, I want to reflect on what the logic underlying Arielli's argument might tell us about how the 'project' of artificial creativity and artmaking is currently being perceived and pursued.

# **Marcel Duchamp, the proto-post-conceptual contemporary artist?**

If artmaking performed by an autonomously working artificial intelligence has come to stand as 'the final frontier' of artificial creativity research (Colton & Wiggins 2012), the successful reproduction of Marcel Duchamp, given his status as arguably the most important artists of the 20th Century, would be the pinnacle of such endeavours. But it actually would so for a number of other reasons than the ones Arielli suggests. As I will argue below, when I dive into the actual passage in "Even an AI could do that", Arielli extends his proposition concerning Duchamp to include contemporary art in general, thereby suggesting – quite reasonably, I would argue – that Duchamp should be considered some kind of proto-conceptual artist and hence also a precursor to contemporary art as so-called 'post-conceptual art'. This is for instance in accord with the perspective of Juliane Rebentisch,1 who notes that contemporary art no longer seeks to conform to genres or traditions, and in this connection credits both the readymade and conceptual art for this 'turn', that has left (or enriched) us with an "unfathomable diversity of artworks" (2013:

1 All quotes from Rebentisch (2013) are my translation.

111) as well as with a lot of philosophical and art-theorical issues to deal with. So, the problems Duchamp might pose to AI-art are quite similar to the ones much contemporary art would pose, Duchamp just happens to be a good paradigmatic example.

Those familiar with Duchamp scholarship would know that this kind of proposition seems to indicate that I am taking sides in quite heated debates (cf. Buchloh et al. 1994; Lund / Wamberg 2019); that I am more or less saying that this particular reading – Duchamp as a 'practical philosopher' of (pre- and post-)conceptual art – should be regarded as 'the true meaning' of Duchamp. In my opinion, I am not. I am simply referring to the mainstream dominant reception history of Duchamp. Which, of course – and here comes an important point of this paper – is not really just a process in which the oeuvre of Duchamp has been received or interpreted in this or that way, right or wrong. It is rather a process of co-production, that is, of making the person Marcel Duchamp and his oeuvre into *that* 'Duchamp', which over the decades – beginningin the post-war decades when he was rediscovered (cf.Crow 1996: 81–87) – has been taking place between artists, critics,academics, philosophers of art, etc., that is, within the institution which sociologist of art Howard Becker (1982) refers to as the 'art world'. So, perhaps it is just as meaningful to think of 'Duchamp' as a product than purely as a producer (or even creator).

In fact, to jump to the conclusion of this paper, it is really the complicated nature of these kinds of inherently social and often also political and / or ideological processes of co-production within the art world (including Academia), which all entail the co-production of art works and their 'meanings', as well as of artists' oeuvres and (the conception) of artists' 'creativity', that is so difficult for an AI to grasp. So, the problem for an AI is not these formalistic aspects concerning what an artificial Duchampian artwork might 'look like', but all the fuzzy social stuff that surrounds it.

# **'Art' and 'creativity' as social categories – and their un-artificial Others**

Secondly, this perspective on the socio-historical character ofmaking Duchampinto 'Duchamp' is much broader than just related to this particular artist, his oeuvre and his 'brand'. It is also applicable to the categories of *art* and *creativity* in general terms. Thus, I would argue, we should never speak of these phenomena as things that just *are,* in essentialist terms. 'Art-ness' or 'creativity' is a status that is conferred upon artefacts, practices or people – or for that sake: technologies – and in this way 'art' or 'creative' / 'creativity'are co-produced through social actions,which, tomakematters even more complicated, are embedded in material, technological and ideological contexts. Hence, at least in principle, anything (or anyone) could become 'art', 'creative', or both.2 Yet, this does not happen; and certainly not randomly. It always happens as a part of a complicated social game within the art world (cf. Becker) or within the so-called 'creativity complex'(cf.Reckwitz 2017),and typicallyin ways that are entangled with other concerns such as ideology, politics, and economy.3