How AR, VR, and AI can power immersive storytelling

In augmented and virtual reality, characters and stories need to be different

By | 3 minute read | April 26, 2018

This image released by the Tribeca Film Festival shows a scene from the virtual reality installation, “Terminal 3,” In this AR documentary, the user is an American immigration official interrogating middle eastern travelers.

Lucy may be a VR character, but she doesn’t act like one. Like any friend, she remembers what you’ve done and uses that information to inform how she talks with you. She can express a full range of emotions. And she’s as good at hanging out as she is going on adventures.

But Lucy—a character in “Wolves in the Walls,” a Fable Studios VR adaptation of the Neil Gaiman children’s book by the same name—wasn’t always so dynamic. Initially, Lucy was designed largely to follow a script and respond to viewer actions. Then Fable decided to take full advantage of VR’s capabilities and make her a fully-formed personality, with her own interests.

“Having an interactive character enslaved to your actions stops viewers from relaxing and stops the story. She wasn’t going on a ‘hero’s journey;’ she was more of a servant,” said Edward Saatchi, co-founder of Fable, at an IBM-sponsored panel on the future of immersive storytelling at the Tribeca Film Festival.

Lucy’s evolution is symbolic of a broader evolution taking place in the XR storytelling community. As the technology grows from a gimmick to a mature medium with large audiences, creators are experimenting with new ways to take viewers on more meaningful journeys.

But what exactly makes a great story or a compelling character in augmented and virtual reality? And how does AI help enrich these converging new mediums? Asad Malik, the creator of Terminal 3, an interactive AR documentary which had its world premier at Tribeca, said that AR, more than earlier forms of media, has the capacity to break down cultural and social barriers.

“Imagine putting a refugee in the White House or a homeless person in the office of the CEO of Uber. We can interview real people with real stories and have you interact with them,“ Malik said. This empathy, catalyzed by the unique experience of AR/VR, is a powerful way to foster deeper emotional connections that inspire change.

Augmented reality can also correct the mistakes of the physical world. Through an AR experience at a natural history museum, Jessica Brillhart, Founder and Director of Vrai Pictures, could fix a dinosaur skeleton that had been standing on two legs—a physical impossibility noted by the staff. Through AR, she was able to correct the creature’s posture to reflect the scientific reality.

“We can build digital layers on top of the world that are quite useful,” said Brillhart.

AI will play a pivotal role when creating immersive worlds—and an ecosystem of interactive characters—from scratch.

“AI makes characters feel real. You can build a character that not only remembers things you enjoy, but also your game and story decisions,” said Saatchi.

The entertainment industry won’t have to wait for the singularity and sentient robots to start using AI. Directors will be able to take the same kind of conversational technology that powers customer service chatbots and transform it into voice-based interactions that can complement both VR and AR.

“Interactive voice recognition in VR is the fourth dimension of storytelling,” said Michael Ludden, Director of Product at IBM Watson Developer and AR/VR Labs. “The possibilities are endless.”

One example is the game Star Trek: Bridge Crew. IBM Watson’s voice functionality, embedded in the game, allows players to issue orders vocally to the crew, fulfilling a dream for many nerds. The game shows how much sense it makes to use vocal commands during VR experiences, whether it’s a game, training program, or creative work session.

Though creators are just starting to fulfill the potential of these technologies, it’s clear that AR and VR, powered by AI, will fundamentally change storytelling and society.

“We are all on the same journey. The rules haven’t been written. We are all creators and experimenters in AR and VR, especially in fields where you apply AI,” said Ludden.