# The History of Mobile Augmented Reality

Clemens Arth  Lukas Gruber  Raphael Grasset  Tobias Langlotz  Alessandro
Mulloni  Dieter Schmalstieg  Daniel Wagner

###### Abstract

This document summarizes the major milestones in mobile Augmented Reality
between 1968 and 2014. Mobile Augmented Reality has largely evolved over the
last decade, as well as the interpretation itself of what is Mobile Augmented
Reality. The first instance of Mobile AR can certainly be associated with the
development of wearable AR, in a sense of experiencing AR during locomotion
(mobile as a motion). With the transformation and miniaturization of physical
devices and displays, the concept of mobile AR evolved towards the notion of
”mobile device”, aka AR on a mobile device. In this history of mobile AR we
considered both definitions and the evolution of the term over time.

Major parts of the list were initially compiled by the member of the Christian
Doppler Laboratory for Handheld Augmented Reality in 2009 (author list in
alphabetical order) for the ISMAR society. More recent work was added in 2013
and during preparation of this report.

Permission is granted to copy and modify. Please email the first author if you
find any errors.

linksWeblinks

## Introduction

This document summarizes the major milestones in mobile Augmented Reality
between 1968 and 2014. Mobile Augmented Reality has largely evolved over the
last decade, as well as the interpretation itself of what is Mobile Augmented
Reality. The first instance of Mobile AR can certainly be associated with the
development of wearable AR, in a sense of experiencing AR during locomotion
(mobile as a motion). With the transformation and miniaturization of physical
devices and displays, the concept of mobile AR evolved towards the notion of
”mobile device”, aka AR on a mobile device. In this history of mobile AR we
considered both definitions and the evolution of the term over time.

Major parts of the list were initially compiled by the member of the The list
was compiled by the member of the Christian Doppler Laboratory for Handheld
Augmented Reality111CDL on Handheld AR: http://studierstube.org/handheld_ar/
in 2009 (author list in alphabetical order) for the ISMAR society. More recent
work was added in 2013 and during preparation of this report.

Permission is granted to copy and modify. Please email the first author if you
find any errors.

|  |  |
---|---|---|---
(a) Research |  (b) Mobile PC |  (c) Mobile Phone |  (d) Hardware
|  |  |
(e) Standard |  (f) Game |  (g) Tool |  (h) Deal
Figure 1: Icons used throughout this report for a rough categorization of
related research, development and events.

## 1968

Ivan Sutherland [67] creates the first augmented reality system, which is also
the first virtual reality system (see Fig.2 left). It uses an optical see-
through head-mounted display that is tracked by one of two different 6DOF
trackers: a mechanical tracker and an ultrasonic tracker. Due to the limited
processing power of computers at that time, only very simple wireframe
drawings could be displayed in real time.

Figure 2: (a): Sutherland’s system in [67]. (b): Conceptual Tablet Computer by
Kay in 1972 [32]. (c): First handheld mobile phone by Motorola in 1973. (d):
Caudell and Mizell coining AR in 1992 [8]. (e): IBM smartphone presented in
1992.

## 1972

The first conceptual tablet computer was proposed in 1972 by Alan Kay, named
the Dynabook [32]. The Dynabook was proposed as personal computer for
children, having the format factor of a tablet with a mechanical keyboard
(really similar design from the One Laptop per Child project started in 2005).
The Dynabook is probably recognized as being the precursor of the tablet
computers decades before the iPad (see Fig. 2).

## 1973

The first handheld mobile phone was presented by Motorola and demonstrated in
April 1973 by Dr Martin Cooper [1]. The mobile named DynaTAC for Dynamic
Adaptive Total Area Coverage was supporting only 35 minutes of call (see Fig.
2).

## 1982

The first laptop, the Grid
Compass222http://home.total.net/~hrothgar/museum/Compass/ 1100 is released,
which was also the first computer to use a clamshell design. The Grid Compass
1100 had an Intel 8086 CPU, 350 Kbytes of memory and a display with a
resolution of 320x240 pixels, which was extremely powerful for that time and
justified the enormous costs of 10.000 USD. However, its weight of 5kg made it
hardly portable.

## 1992

Tom Caudell and David Mizell coin the term ”augmented reality” to refer to
overlaying computer-presented material on top of the real world [8] (see
Fig.2). Caudell and Mizell discuss the advantages of AR versus VR such as
requiring less processing power since less pixels have to be rendered. They
also acknowledge the increased registration requirements in order to align
real and virtual.

At COMDEX 1992, IBM and Bellsouth introduce the first smartphone, the IBM
Simon Personal Communicator333Wikipedia:
http://en.wikipedia.org/wiki/Simon_(phone), which was released in 1993 (see
Fig.2). The phone has 1 Megabyte of memory and a B/W touch screen with a
resolution of 160 x 293 pixels. The IBM Simon works as phone, pager,
calculator, address book, fax machine, and e-mail device. It weights 500 grams
and cost 900 USD.

## 1993

Loomis _et al_. develop a prototype of an outdoor navigation system for
visually impaired [39]. They combine a notebook with a differential GPS
receiver and a head-worn electronic compass. The application uses data from a
GIS (Geographic Information System) database and provides navigational
assistance using an ”acoustic virtual display”: labels are spoken using a
speech synthesizer and played back at correct locations within the auditory
space of the user.

Figure 3: (a): Chameleon system proposed by Fitzmaurice [16]. (b): NAVSTAR-GPS
goes live in 1993. (c): Apple Newton Message Pad 100.

Fitzmaurice creates Chameleon (see Fig.3), a key example of displaying
spatially situated information with a tracked hand-held device. In his setup
the output device consists of a 4” screen connected to a video camera via a
cable [16]. The video camera records the content of a Silicon Graphics
workstation’s large display in order to display it on the small screen.
Fitzmaurice uses a tethered magnetic tracker (Ascension bird) for registration
in a small working environment. Several gestures plus a single button allow
the user to interact with the mobile device. Chameleon’s mobility was strongly
limited due to the cabling. It did also not augment in terms of overlaying
objects on a video feed of the real world.

In December 1993 the Global Positioning System (GPS, official name ”NAVSTAR-
GPS”) achieves initial operational capability (see Fig.3). Although
GPS444Wikipedia: http://en.wikipedia.org/wiki/Global_Positioning_System was
originally launched as a military service, nowadays millions of people use it
for navigation and other tasks such as geo-caching or Augmented Reality. A GPS
receiver calculates its position by carefully timing the signals sent by the
constellation of GPS satellites. The accuracy of civilian GPS receivers is
typically in the range of 15 meter. More accuracy can be gained by using
Differential GPS (DGPS) that uses correction signals from fixed, ground-based
reference stations.

The Apple Newton Message Pad 100 was one of the earliest commercial personal
digital assistant (PDA)555Wikipedia: http://en.wikipedia.org/wiki/MessagePad.
Equipped with a stylus and handwritten recognition, and feature a screen in
black and white of 336x240 pixels (see Fig. 3).

## 1994

Steve Mann starts wearing a webcam for almost 2 years. From 1994-1996 Mann
wore a mobile camera plus display for almost every waking minute. Both devices
were connected to his website allowing online visitors to see what Steve was
seeing and to send him messages that would show up on his mobile display666S.
Mann, “Wearable Wireless Webcam,” personal WWW page. wearcam.org

Figure 4: (a): Milgram Continuum [41]. (b): Rekimotos NaviCam system [58].
(c): Rekimoto’s matrix marker [57]. (d) and (e): Touring Machine by Feiner _et
al_. [15].

Paul Milgram and Fumio Kishino write their seminal paper ”Taxonomy of Mixed
Reality Visual Displays” in which they define the Reality-Virtuality Continuum
[41] (see Fig.4). Milgram and Kishino describe a continuum that spans from the
real environment to the virtual environment. In between there are Augmented
Reality, closer to the real environment and Augmented Virtuality, which is
closer to the virtual environment. Today Milgram’s Continuum and Azuma’s
definition (1997) are commonly accepted as defining Augmented Reality.

## 1995

Jun Rekimoto and Katashi Nagao create the NaviCam, a tethered setup, similar
to Fitzmaurice’s Chameleon [58] (see Fig.4). The NaviCam also uses a nearby
powerful workstation, but has a camera mounted on the mobile screen that is
used for optical tracking. The computer detects color-coded markers in the
live camera image and displays context sensitive information directly on top
of the video feed in a see-through manner.

Benjamin Bederson introduced the term Audio Augmented Reality by presenting a
system that demonstrated an augmentation of the audition modality [5]. The
developed prototype uses a MD-player which plays audio information based on
the tracked position of the user as part of a museum guide.

## 1996

Jun Rekimoto presents 2D matrix
markers777http://www.sonycsl.co.jp/person/rekimoto/matrix/Matrix.html (square-
shaped barcodes), one of the first marker systems to allow camera tracking
with six degrees of freedom [57] (see Fig.4).

## 1997

Ronald Azuma presents the first survey on Augmented Reality. In his
publication, Azuma provides a widely acknowledged definition for AR [4], as
identified by three characteristics:

  * •

it combines real and virtual

  * •

it is interactive in real time

  * •

it is registered in 3D.

Steve Feiner _et al_. present the Touring Machine, the first mobile augmented
reality system (MARS) [15] (see Fig.4 and Fig. 4). It uses a see-through head-
worn display with integral orientation tracker; a backpack holding a computer,
differential GPS, and digital radio for wireless web access; and a hand-held
computer with stylus and touchpad interface888MARS:
http://graphics.cs.columbia.edu/projects/mars/mars.html.

Thad Starner _et al_. explore possible applications of mobile augmented
reality, creating a small community of users equipped with wearable computers
interconnected over a network [66]. The explored applications include an
information system for offices, people recognition and coarse localization
with infrared beacons.

Philippe Kahn invents the camera phone999Wikipedia Camera Phone:
http://en.wikipedia.org/wiki/Camera_phone, a mobile phone which is able to
capture still photographs (see Fig.5). Back in 1997, Kahn used his invention
to share a picture of his newborn daughter with more than 2000 relatives and
friends, spread around the world. Today more than half of all mobile phones in
use are camera phones.

Sony releases the Glasstron, a series of optical HMD (optionally see-through)
for the general public. Adoption was rather small, but the affordable price of
the HMD made it really popular in AR research labs and for the development of
wearable AR prototype (see Fig. 5).

## 1998

Figure 5: (a): Camera Phone Development by Kahn. (b): Sony Glasstron optical
HMD in 1997. (c): Thomas _et al_.’s Tinmith system [71]. (d): ARToolKit for
pose tracking in 6DOF [31]. (e): Palm VII, the first consumer LBS device.

Bruce Thomas et al. present ”Map-in-the-hat”, a backpack-based wearable
computer that includes GPS, electronic compass and a head-mounted display [71]
(see Fig.5). At this stage the system was utilized for navigation guidance,
but it later evolved into Tinmith, an AR platform used for several other AR
projects101010Tinmith webpage: http://www.tinmith.net/.

## 1999

Hirokazu Kato and Mark Billinghurst present ARToolKit, a pose tracking library
with six degrees of freedom, using square fiducials and a template-based
approach for recognition [31]. ARToolKit is available as open source under the
GPL license and is still very popular in the AR community (see Fig. 5).

Tobias Höllerer _et al_. develop a mobile AR system that allows the user to
explore hypermedia news stories that are located at the places to which they
refer and to receive a guided campus tour that overlays models of earlier
buildings [26] (see Fig. 6). This was the first mobile AR system to use RTK
GPS and an inertial-magnetic orientation tracker.

Tobias Höllerer _et al_. present a mobile augmented reality system that
includes indoor user interfaces (desktop, AR tabletop, and head-worn VR) to
interact with the outdoor user [27] (see Fig. 6). While outdoor users
experience a first-person spatialized multimedia presentation via a head-
mounted display, indoor users can get an overview of the outdoor scene.

Figure 6: (a): Höllerer _et al_.’s MARS system [26]. (b): Höllerer _et al_.’s
user interface [27]. (c) Benfon Esc! NT2002, the first GSM phone with a built-
in GPS sensor.

Jim Spohrer publishes the Worldboard concept, a scalable infrastructure to
support mobile applications that span from low-end location-based services, up
to high-end mobile AR [65]. In his paper, Spohrer also envisions possible
application cases for mobile AR, and social implications.

The first consumer LBS device was the Palm VII, only supporting zip code based
location services (see Fig.5). 2 years later, different mobile operators
provided different location based services using private network
technology111111Wikipedia: http://en.wikipedia.org/wiki/Palm_VII.

Benefon Esc! NT2002121212http://www.benefon.de/products/esc/, the first GSM
phone with a built-in GPS receiver is released in late 1999 (see Fig. 6). It
had a black and white screen with a resolution of 100x160 pixels. Due to
limited storage, the phone downloaded maps on demand. The phone also included
a friend finder that exchanged GPS positions with other Esc! devices via SMS.

The wireless network protocols 802.11a/802.11b131313Wikipedia:
http://en.wikipedia.org/wiki/802.11 \- commonly known as WiFi \- are defined.
The original version - obsolete - specifies bitrates of 1 or 2 megabits per
second (Mbit/s), plus forward error correction code.

## 2000

Bruce Thomas _et al_. present AR-Quake, an extension to the popular desktop
game Quake [70] (see Fig. 7). ARQuake is a first-person perspective
application which is based on a 6DOF tracking system using GPS, a digital
compass and vision-based tracking of fiducial markers. Users are equipped with
a wearable computer system in a backpack, an HMD and a simple two-button input
device. The game can be played in- or outdoors where the usual keyboard and
mouse commands for movement and actions are performed by movements of the user
in the real environment and using the simple input interface.

Regenbrecht and Specht present mPARD, using analogue wireless video
transmission to a host computer which is taking the burden of computation off
the mobile hardware platform [54] (see Fig. 7). The rendered and augmented
images are sent back to the visualization device over a separate analog
channel. The system can operate within 300m outdoors and 30m indoors, and the
batteries allow for an uninterrupted operation of 5 hours at max.

Fritsch _et al_. introduce a general architecture for large scale AR system as
part of the NEXUS project. The NEXUS model introduces the notion of augmented
world using distributed data management and a variety of sensor system [17].

Simon Julier _et al_. present BARS, the Battlefield Augmented Reality System
[29] (see Fig. 7). The system consists of a wearable computer, a wireless
network system and a see-through HMD. The system targets the augmentation of a
battlefield scene with additional information about environmental
infrastructure, but also about possible enemy ambushes.

Figure 7: (a): ARQuake by Thomas _et al_. [70]. (b): mPARD system by
Regenbrecht and Specht [54]. (c): BARS system by Julier _et al_. [29]. (d):
First commercial camera phone in 2000.

Sharp corporation releases the first commercial camera phone to public (see
Fig. 7). The official name of the phone is
J-SH04141414http://k-tai.impress.co.jp/cda/article/showcase_top/3913.html. The
phones’ camera has a resolution of 0.1 megapixels.

At ISAR, Julier _et al_. described the problem of information overload and
visual clutter within mobile Augmented Reality [28]. They proposed information
filtering for mobile AR based on techniques such as physically-based methods,
methods using the spatial model of interaction, rule-based filtering, and a
combination of these methods to reduce the information overload in mobile AR
scenarios.

Figure 8: (a): BatPortal by Newman _et al_. [49]. (b): TOWNWEAR system by Hara
_et al_. [62]. (c): Wireless AR setup concept by Fruend _et al_. [18]. (d):
Multi-user AR system by Reitmayr and Schmalstieg [55]. (e): ARCHEOGUIDE by
Flahakis _et al_. [73]. (f): Mobile AR restaurant guide by Bell _et al_. [6].
(g): First AR browser by Kooper and MacIntyre [35].

## 2001

Joseph Newman _et al_. present the BatPortal [49], a PDA-based, wireless AR
system (see Fig.8). Localization is performed by measuring the travel time of
ultra-sonic pulses between specially built devices worn by the user, so-called
Bats, and fixed installed receivers deployed in the floors ceilings building-
wide. The system can support an HMD-based system, but also the more well known
BatPortal using a handheld device. Based on a fixed configuration of the PDA
carried and the personal Bat worn, the direction of the users view is
estimated, and a model of the scene with additional information about the
scene is rendered onto the PDA screen.

Hara _et al_. introduce TOWNWEAR, an outdoor system that uses a fiber optic
gyroscope for orientation tracking [62] (see Fig.8). The high precision
gyroscope is used to measure the 3DOF head direction accurately with minimal
drift, which is then compensated by tracking natural features.

Jürgen Fruend _et al_. present AR-PDA, a concept for building a wireless AR
system and a special prototype of palm-sized hardware [18] (see Fig.8). Basic
design ideas include the augmentation of real camera images with additional
virtual objects, for example for illustration of functionality and interaction
with commonly used household equipment.

Reitmayr and Schmalstieg present a mobile, multi-user AR system [55] (see
Fig.8). The ideas of mobile augmented reality and collaboration between users
in augmented shared space are combined and merged into a hybrid system.
Communication is performed using LAN and wireless LAN, where mobile users and
stationary users are acting in a common augmented space.

Vlahakis et al. present Archeoguide, a mobile AR system for cultural heritage
sites [73] (see Fig.8). The system is built around the historical site of
Olympia, Greece. The system contains a navigation interface, 3D models of
ancient temples and statues, and avatars which are competing for the win in
the historical run in the ancient Stadium. While communication is based on
WLAN, accurate localization is performed using GPS. Within the system a
scalable setup of mobile units can be used, starting with a notebook sized
system with HMD, down to palmtop computers and Pocket PCs.

Kretschmer et al. present the GEIST system, a system for interactive story-
telling within urban and/or historical environments [36]. A complex database
setup provides information queues for the appearance of buildings in ancient
times or historical facts and events. Complex queries can be formulated and
stories can be told by fictional avatars or historical persons.

Columbia’s Computer Graphics and User Interfaces Lab does an outdoor
demonstration of their mobile AR restaurant guide at ISAR 2001, running on
their Touring Machine [6] (see Fig.8). Pop-up information sheets for nearby
restaurants are overlaid on the user’s view, and linked to reviews, menus,
photos, and restaurant URLs.

Kooper and MacIntyre create the RWWW Browser, a mobile AR application that
acts as an interface to the World Wide Web [35] (see Fig.8). It is the first
AR browser. This early system suffers from the cumbersome AR hardware of that
time, requiring a head mounted display and complicated tracking
infrastructure. In 2008 Wikitude implements a similar idea on a mobile phone.

Figure 9: (a): Navigation system by Kalkusch _et al_. [30]. (b): ARPad by
Mogilev _et al_. [43]. (c): Human Pacman by Cheok _et al_. [9]. (d): iLamps
system by Raskar _et al_. [53]. (e): Indoor AR guidance system by Wagner and
Schmalstieg [77]. (f) Siemens SX1 AR game ”Mozzies”. (g): Mobile Authoring
system by Guven and Feiner [22].

## 2002

Michael Kalkusch _et al_. present a mobile augmented reality system to guide a
user through an unfamiliar building to a destination room [30] (see Fig. 9).
The system presents a world-registered wire frame model of the building
labeled with directional information in a see-through heads-up display, and a
three-dimensional world-in-miniature (WIM) map on a wrist-worn pad that also
acts as an input device. Tracking is done using a combination of wall-mounted
ARToolkit markers observed by a head-mounted camera, and an inertial tracker.

Leonid Naimark and Eric Foxlin present a wearable low-power hybrid visual and
inertial tracker [46]. This tracker, later to be known as InterSense’s
IS-1200, can be used for tracking in large scale, such as a complete building.
This is achieved by tracking a newly designed 2-D barcode with thousands of
different codes and combining the result with an inertial sensor.

Mogilev _et al_. introduce the AR Pad, an ad-hoc mobile AR device equipped
with a spaceball controller [43] (see Fig 9).

## 2003

Adrian David Cheok _et al_. present the Human Pacman [9] (see Fig. 9). Human
Pacman is an interactive ubiquitous and mobile entertainment system that is
built upon position and perspective sensing via Global Positioning System and
inertia sensors; and tangible human-computer interfacing with the use of
Bluetooth and capacitive sensors. Pacmen and Ghosts are now real human players
in the real world experiencing mixed computer graphics fantasy-reality
provided by using wearable computers that are equipped with GPS and inertia
sensors for players’ position and perspective tracking. Virtual cookies and
actual tangible physical objects with Bluetooth devices and capacitive sensors
are incorporated into the game play to provide novel experiences of seamless
transitions between real and virtual worlds.

Ramesh Raskar _et al_. present iLamps [53] (see Fig. 9). This work created a
first prototype for object augmentation with a hand-held projector-camera
system. An enhanced projector can determine and respond to the geometry of the
display surface, and can be used in an ad-hoc cluster to create a self-
configuring display. Furthermore interaction techniques and co-operation
between multiple units are discussed.

Daniel Wagner and Dieter Schmalstieg present an indoor AR guidance system
running autonomously on a PDA [77] (see Fig. 9). They exploit the wide
availability of consumer devices with a minimal need for infrastructure. The
application provides the user with a three-dimensional augmented view of the
environment by using a Windows Mobile port of ARToolKit for tracking and runs
directly on the PDA.

The Siemens SX1 is released, coming with the first commercial mobile phone AR
camera game called Mozzies (also known as Mosquito Hunt) (see Fig. 9). The
mosquitoes are superimposed on the live video feed from the camera. Aiming is
done by moving the phone around so that the cross hair points at the
mosquitoes. Mozzies was awarded the title of best mobile game in 2003.

Sinem Guven presents a mobile AR authoring system for creating and editing 3D
hypermedia narratives that are interwoven with a wearable computer user’s
surrounding
environment151515http://graphics.cs.columbia.edu/projects/mars/Authoring.html
[22] (see Fig. 9). Their system was designed for authors who are not
programmers and used a combination of 3D drag-and-drop for positioning media
and a timeline for synchronization. It allowed authors to preview their
results on a desktop workstation, as well as with a wearable AR or VR system.

|  |
---|---|---
|

Figure 10: (a): Tracking 3D markers by Möhring _et al_. [44]. (b): Visual
Codes by Rohs and Gfeller [59]. (c): OSGAR system by Coelho _et al_. [10].
(d): The Invisible Train [75].

## 2004

Mathias Möhring _et al_. present a system for tracking 3D markers on a mobile
phone [44] (see Fig.10). This work showed a first video see-through augmented
reality system on a consumer cell-phone. It supports the detection and
differentiation of different 3D markers, and correct integration of rendered
3D graphics into the live video stream.

Michael Rohs and Beat Gfeller present Visual Codes, a 2D marker system for
mobile phones [59] (see Fig.10). These codes can be attached to physical
objects in order to retrieve object-related information and functionality.
They are also suitable for display on electronic screens.

Enylton Machado Coelho _et al_. presents OSGAR, a scene graph with uncertain
transformations [10] (see Fig.10). In their work they target the problem of
registration error, which is especially important for mobile scenarios when
high quality tracking is not available and overlay graphics will not align
perfectly with the real environment. OSGAR dynamically adapts the display to
mitigate the effects of registration errors.

The Invisible Train, is shown at SIGGRAPH 2004 Emerging Technologies161616The
Invisible Train: http://studierstube.icg.tugraz.at/invisible_train/ (see
Fig.10). The Invisible Train is the first multi-user Augmented Reality
application for handheld devices [75].

## 2005

Anders Henrysson ports ARToolKit to Symbian [23] (see Fig.11). Based on this
technology he presents the famous AR-Tennis game, the first collaborative AR
application running on a mobile phone. ARTennis was awarded the Indepdent
Mobile Gaming best game award for 2005, and the technical achievement award.

Project ULTRA shows how to use non-realtime natural feature tracking on PDAs
to support people in multiple domains such as the maintenance and support of
complex machines, construction and production, and edutainment and cultural
heritage [40]. Furthermore an authoring environment is developed to create the
AR scenes for the maintenance tasks.

The first mobile phones equipped with three-axis accelerometers were the Sharp
V603SH and the Samsung SCH-S310 both sold in Asia in 2005.

Figure 11: (a): AR-Tennis by Henrysson _et al_. [23]. (b): Going Out by
Reitmayr and Drummond [56]. (c): Mara system by Nokia in 2006.

## 2006

Reitmayr and Drummond present a model-based hybrid tracking system for outdoor
augmented reality in urban environments enabling accurate, real-time overlays
on a handheld device [56] (see Fig.11). The system combines an edge-based
tracker for accurate localization, gyroscope measurements to deal with fast
motions, measurements of gravity and magnetic field to avoid drift, and a back
store of reference frames with online frame selection to re-initialize
automatically after dynamic occlusions or failures.

Nokia presents Mara, a multi-sensor mobile phone AR guidance application for
mobile phones171717Mara: http://research.nokia.com/page/219. The prototype
application overlays the continuous viewfinder image stream captured by the
camera with graphics and text in real time, annotating the user’s surroundings
(see Fig.11).

## 2007

Klein and Murray present a system capable of robust real-time tracking and
mapping in parallel with a monocular camera in small workspaces [33] (see Fig.
12). It is an adaption of a SLAM approach which processes the tracking and
mapping task on two separated threads.

DiVerdi and Höllerer present the GroundCam, a system combining a camera and an
orientation tracker [12] (see Fig. 12). The camera points at the ground behind
the user and provides 2D tracking information. The method is similar to that
of an optical desktop mouse.

Figure 12: (a): PTAM by Klein and Murray [33]. (b): Groundcam by DiVerdi and
Höllerer [12]. (c): Map Navigation with mobile devices by Rohs _et al_. [60].
(d): Apple iPhone 2G. (e): AR advertising app by HIT Lab NZ and Saatchi.

Rohs _et al_. compare the performance of the following navigation methods for
map navigation on mobile devices: joystick navigation, the dynamic peephole
method without visual context, and the magic lens paradigm using external
visual context [60] (see Fig. 12). In their user study they demonstrate the
advantage of dynamic peephole and magic lens interaction over joystick
interaction in terms of search time and degree of exploration of the search
space.

The first multi-touch screen mobile phone, famously known as iPhone sold by
Apple, leverages a new way to interact on mobile devices (see Fig. 12).

HIT Lab NZ and Saatchi deliver the world’s first mobile phone based AR
advertising application for the Wellington Zoo HIT08 (see Fig. 12).

Figure 13: (a): Real-time natural feature tracking on mobile phones by Wagner
_et al_. [76]. (b): Commercial AR museum guide by METAIO [42]. (c): Wikitude
AR Browser.

## 2008

Wagner _et al_. present the first 6DOF implementation of natural feature
tracking in real-time on mobile phones achieving interactive frame rates of up
to 20 Hz [76] (see Fig. 13). They heavily modify the well known SIFT and Ferns
methods in order to gain more speed and reduce memory requirements.

METAIO presents a commercial mobile AR museum guide using natural feature
tracking or a six-month exhibition on Islamic art [42] (see Fig. 13). In their
paper they describe the experiences made in this project.

With Augmented Reality 2.0, Schmalstieg _et al_. presented at the Dagstuhl
seminar in 2008 for the first time a concept that combined ideas of the Web
2.0 such as social media, crowd sourcing through public participation, and an
open architecture for content markup and distribution, and applied it to
mobile Augmented Reality to create a scalable AR experience [63].

Mobilizy launches Wikitude181818Wikitude:
http://www.mobilizy.com/wikitude.php?lang=en, an application that combines GPS
and compass data with Wikipedia entries. The Wikitude World Browser overlays
information on the real-time camera view of an Android smartphone (see Fig.
13).

## 2009

Morrison _et al_. present MapLens which is a mobile augmented reality (AR) map
using a magic lens over a paper map [45] (see Fig. 14). They conduct a broad
user study in form of an outdoor location-based game. Their main finding is
that AR features facilitate place-making by creating a constant need for
referencing to the physical. The field trials show that the main potential of
AR maps lies in their use as a collaborative tool.

Figure 14: (a): MapLens by Morrison _et al_. [45]. (b) Hagbi’s pose tracking
using shape [7]. (c): SiteLens by White and Feiner [78]. (d): LayAR AR
browser. (e): ARhrrrr! Zombie game by Spreen _et al_. from Georgia Tech. (f):
Klein’s PTAM system running on an iPhone [34].

Hagbi _et al_. presented an approach allowing to track the pose of the mobile
device by pointing it to fiducials [7] (see Fig. 14). Unlike existing systems
the approach allows to track a wide set of planar shapes while the user can
teach the system new shapes at runtime by showing them to the camera. The
learned shapes are then maintained by the system in a shape library enabling
new AR application scenarios in terms of interaction with the scene but also
in terms of fiducial design.

Sean White introduces SiteLens (see Fig. 14), a hand-held mobile AR system for
urban design and urban planning site visits [78]. SiteLens creates ”situated
visualizations” that are related to and displayed in their environment. For
example, representations of geocoded carbon monoxide concentration data are
overlaid at the sites at which the data was recorded.

SPRXmobile launches Layar191919LayAR: http://layar.eu/, an advanced variant of
Wikitude (see Fig. 14). Layar uses the same registration mechanism as Wikitude
(GPS + compass), and incoperates this into an open client-server platform.
Content layers are the equivalent of web pages in normal browsers. Existing
layers include Wikipedia, Twitter and Brightkite to local services like Yelp,
Trulia, store locators, nearby bus stops, mobile coupons, Mazda dealers and
tourist, nature and cultural guides. On August 17th Layar went global serving
almost 100 content layers.

Kimberly Spreen _et al_. develop ARhrrrr!, the first mobile AR game with high
quality content at the level of commercial gamesGT09 (see Fig. 14). They use
an NVIDIA Tegra developer kit (”Concorde”) with a fast GPU. All processing
except for tracking are running on the GPU, making the whole application run
at high frame rates on a mobile phone class device despite the highly detailed
content and natural feature tracking.

Georg Klein presents a video showing his SLAM system running in real-time on
an iPhone [34] (see Fig. 14) and later presents this at ISMAR 2009 in Orlando,
Florida. Even though it has constrains in terms of working area it is the
first time a 6DoF SLAM system is known to run on mobile phones in sufficient
speed.

### Update April 2015:

The following parts of the document until beginning of 2015 cover the years
since the last homepage update, following the same categorization and scheme
as before. From end of 2009 onwards, AR research and development is generally
driven by high expectations and huge investments from world-leading companies
such as Microsoft, Google, Facebook, Qualcomm and others. At the same time,
the landscape of mobile phone manufacturers started to change radically. In
general the advances in mobile device capabilities introduce a strong drive
towards mobile computing, and the availability of cloud processing further
supports the proposal and development of server-client solutions for AR
purposes. One major trend starting around 2010, originating by the work of
Davison in 2003 [11] and later further explored by Klein and Murray [33, 34],
is the heavy use of SLAM in AR, which still continues to dominate a major part
of AR research and development as of beginning of 2015.

Microsoft presents ”Project Natal” at the game exhibition E3. It is the first
version of a new hardware interface, consisting of motion detection
technology, microphone, color camera and software, to be integrated into the
game console Xbox 360.

At ISMAR 2009, Clemens Arth _et al_. present a system for large-scale
localization and subsequent 6DOF tracking on mobile phones [3]. The system
uses sparse point clouds of city areas and FAST corners and SURF-like
descriptors that can be used on memory-limited devices (see Fig. 15).

Qualcomm Inc. acquires the mobile AR IP from Imagination Computer Services
GmbH., Vienna, and takes over the funding of the Christian Doppler Laboratory
for Handheld AR at Graz University of Technology. A research center to focus
on AR is opened later in 2010 in Vienna QC10.

Figure 15: (a): City reconstruction as used by Arth _et al_. [3]. (b):
Panoramic image captured on a mobile phone using the approach of Wagner _et
al_. [74]. (c) Apple iPad. (d): City-of-Sights paperboard models by Gruber _et
al_. [20]. (e) In-situ information creation by Langlotz _et al_. [37].

## 2010

A real-time panoramic mapping and tracking system for mobile phones is
presented by Wagner _et al_. at VR, which performs 3DOF tracking in cylindric
space and supports the use of panoramic imagery for improved usability and
experience in AR [74] (see Fig. 15).

KHARMA is a lightweight and open architecture for referencing and delivering
content explicitly aiming for mobile AR applications running on a global
scale. It uses KML for describing the geospatial or relative relation of
content while utilizing on HTML, JavaScript and CSS technologies for content
development and delivery [25].

Microsoft announces a close cooperation with Primesense MS10, an Israeli
company working on structured-light based 3D sensors, to supply their
technology to ”Project Natal”, now coined Kinect. The Kinect becomes
commercially available in November 2010.

Apple releases the iPad202020Wikipedia: http://en.wikipedia.org/wiki/IPad on
April 2010, which becomes the first tablet computer to be adopted by the large
public. The iPad featured an assisted GPS, accelerometers, magnetometers,
advanced graphics chipset (PowerVR SGX535), enabling the possibilities to
create efficient AR application on tablet computer (see Fig. 15).

At ISMAR Lukas Gruber _et al_. present the ”City of Sights”, a collection of
datasets and paperboard
models212121http://studierstube.icg.tugraz.at/handheld_ar/cityofsights.php to
evaluate the tracking and reconstruction performance of algorithms used in AR
[20] (see Fig. 15).

After several delays, Microsoft releases Windows Phone in October 2010, to
become the third major mobile phone operating system to challenge iOS and
Android.

Existing mobile AR applications where exclusively used to browser and consume
digital information. Langlotz et al. presented an new approach aiming for AR
browsers that also supported creation of digital information in-situ. The
information is registered with pixel-precision by utilizing a panorama of the
environment that is created in the background [37] (see Fig. 15).

## 2011

Qualcomm announces the release of its AR platform SDK to the public in April.
At that time it is called QCAR QC11, which will later be called Vuforia.

In August, Google announces the acquisition of Motorola Mobility for about
$12.5 billion GO11. A major asset of Motorola is a large patent portfolio,
which Google needs to secure the further Android platform development.

At ICCV 2011, Newcombe presents DTAM, a dense real-time tracking and mapping
algorithm [48]. Later at ISMAR 2011, Richard Newcombe presents the
KinectFusion work [47], in which depth images from the Kinect sensor are fused
to create a single implicit surface model. KinectFusion becomes publicly
available within the Kinect SDK later KF11 (see Fig. 16).

Figure 16:  (a): KinectFusion system presented by Newcombe _et al_. at ISMAR
2011 [47]. (b): Mobile phone scene reconstruction by Pan _et al_. [50].

Qi Pan presents his work on reconstructing scenes on mobile phones using
panoramic images. By using FAST corners and a SURF-like descriptor, multiple
panoramas are registered and a triangulated model is created after voxel
carving [50] (see Fig. 16).

Following the still challenging problem of running SLAM in real-time on
mobiles, Pirchheim presents an approach using planarity assumptions, and
demonstrates his approach on a Nokia N900 smartphone [51].

Grubert _et al_. publish a technical report about the plausibility of using AR
browsers [21], which becomes a survey about the pros and cons of AR browser
technology at that point in time.

## 2012

Smart watches are broadly introduced as a new generation of mobile wearables.
_Pebble_ and the _Sony SmartWatch_ are built to connect to a personal
smartphone and to provide simple functionality, such as notifications or call
answering.

Google Glass (also known as Google _Project Glass_) is firstly presented to
the public222222Google Glass project page on Google+:
https://plus.google.com/+GoogleGlass (see Fig.17). Goggle Glass is is an
optical HMD that can be controlled with an integrated touch-sensitive sensor
or natural language commands. After it’s public announcement Google Glass had
a major impact on research but even more on the public perception of mixed
reality technology.

NVidia is demonstrating at Siggraph Emerging Technologies their prototype of a
head mounted display supporting accurate accommodation, convergence, and
binocular-disparity depth cues (see Fig. 17). The prototype introduces a
light-field-based approach to near-eye displays and can be seen as a next
generation wearable display technology for AR as existing hardware can’t
provide accurate acommodation NV12.

13th lab released the first commercial mobile SLAM (Simultaneous localization
and mapping) system coined Pointcloud232323Pointcloud homepage:
http://pointcloud.io/ to the public, marking a major milestone for app
developers who want to integrate SLAM-based tracking into their
application242424Pointcloud video: http://www.youtube.com/watch?v=K5OKaK3Ay8U.

PrimeSense, the creator of the Microsoft Kinect, introduced a smaller version
of a 3D sensing device called Capri PS12 that is small enough to be integrated
into mobile devices such as tablets or smartphones252525Capri Video:
http://www.youtube.com/watch?v=ELTETXO02zE.

At ISMAR 2012, Steffen Gauglitz _et al_. present their approach on tracking
and mapping from both general and rotation-only camera motion [19].

In August, Oculus VR announces the Oculus Rift dev kit, a virtual reality
head-mounted display. This initiated a new hype in Virtual Reality and in the
development of more head-mounted displays for gaming purposes mainly (see
Fig.17).

## 2013

As opposed to previous work from Gauglitz _et al_., Pirchheim _et al_. present
an approach to handle pure camera rotation running on a mobile phone at ISMAR
[52].

Figure 17:  (a): _Oculus Rift_ developer edition. (b): Google Glass. (c):
Near-eye light field project by NVidia.

Google Glass, which was already announced as _Project Glass_ in 2012, becomes
available through the explorer program in late 2013. and raises positive and
negative attention, as well as concerns about privacy and ethical aspects (see
Fig.17).

At ICRA, Li _et al_. present an amazing approach for motion tracking with
inertial sensors and a rolling-shutter camera running in real-time on a mobile
phone [38].

Tan _et al_. propose an approach to SLAM working in dynamic environments,
allowing parts in the scene to be dynamic without breaking the mapping and
tracking [68].

On November 24, 2013, Apple Inc. confirms the purchase of PrimeSense for about
$350 million AP13. Primesense was working on shrinking their sensors to fit
into mobiles at that point in time.

Taskanen _et al_. propose an approach to perform full 3D reconstruction on a
mobile monocular smartphone and creating a dense 3D model with known absolute
scale [69].

## 2014

Three years after the acquisition, in January Google sells Motorola Mobility
to Lenovo for $2.91 million, however, keeping most of the patent portfolio
GO14.

Figure 18:  (a): SLAM map localization by Ventura _et al_. [72]. (b): _LSD-
SLAM_ reconstruction by Engel _et al_. [13].

Also in January, Qualcomm acquires Kooaba QC14, a Swiss ETH-spin-off founded
in 2007, built around image recognition using SURF features. Kooaba’s
technology is integrated into the services provided through the Vuforia SDK.

In February, Google announces Project Tango GO142, which is an Android
smartphone equipped with a full Kinect-like 3D sensor and hands out a few
hundred units to developers and companies.

In March, Facebook acquires Oculus VR for $2 billion, although Oculus does not
make any consumer products at that point in time yet FB14. This strengthens
the hype in upcoming VR interfaces.

At VR, Ventura _et al_. present an approach to localize SLAM maps built on a
mobile phone accurately wrt. a sparse 3D reconstruction of urban environments
[72] (see Fig.18).

In April, Microsoft announces the acquisition of Nokia’s Devices and Services
unit for $7.2 billion MS14, as Nokia is the primary vendor for Windows devices
devices, especially the Lumia phones.

Following up on previous work at ICCV 2013 [14], at ECCV Engel _et al_.
present LSD-SLAM, a feature-less monocular SLAM algorithm using keyframes and
semi-dense depth maps, and release the code to the public [13] (see Fig.18).
At ISMAR, a mobile version is presented as well [64].

At 3DV, Herrera _et al_. present DT-SLAM [24]. The key idea behind the
approach is to defer the triangulation step of 2D features matched across
keyframes until those have undergone a certain baseline, improving the overall
robustness of SLAM.

At ISMAR, Salas-Moreno _et al_. present Dense Planar SLAM, leveraging the
assumption that many man-made surfaces are planar [61].

## 2015

In January, Microsoft announces the _Hololens_ , a headset to fuse AR and VR
MS15 to be made available later in 2015. The device is a complete computer
with a see-through display and several sensors.

In May, DAQRI, a company working on AR helmets, acquires ARToolworks ART15.
Oculus VR announced the acquisition of Surreal Vision, bringing the company’s
expertise on recreating real-time 3D representations of the outside world into
virtualized environments OC15. A few days later, the acquisition of German AR
company Metaio by Apple is announced AP15. Metaio was a major player in
distribution of AR technology to developers through their SDK, which ends
abruptly now.

In June, Magic Leap announces that they will release its AR platform SDK to
the public soon. It is expected to support Unity and the Unreal engine ML15.

HTC and Valve start shipping their developer hardware in very limited
quantities VA15, including a Vive headset and two Lighthouse base stations as
passive components together with the two wireless Steam VR controllers.

In October 2015, Qualcomm sells its business unit in Vienna responsible for
the development of the Augmented Reality SDK _Vuforia_ to PTC PT15 for 65 Mio.
USD.

At ISMAR 2015, the group of Graz University of Technology wins the best paper
award for their work ”Instant Outdoor Localization and SLAM Initialization
from 2.5D Maps” [2]. The algorithm uses OpenStreetMap data and calculates the
pose for an image based on building edges and semantic image information.

## Acknowledgements

Thanks go to the ISMAR09 mobile committee and all others for their valuable
suggestions.

## References

  * [1] Radio Telephone System, U.S. Patent Application U.S. 3,906,166, September 16, 1975\.
  * [2] C. Arth, C. Pirchheim, J. Ventura, D. Schmalstieg, and V. Lepetit, “Instant outdoor localization and SLAM initialization from 2.5d maps,” IEEE Trans. Vis. Comput. Graph., vol. 21, no. 11, pp. 1309–1318, 2015.
  * [3] C. Arth, D. Wagner, M. Klopschitz, A. Irschara, and D. Schmalstieg, “Wide area localization on mobile phones,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 73–82, IEEE, 2009.
  * [4] R. T. Azuma, “A survey of augmented reality,” Presence: Teleoperators and Virtual Environments, vol. 6, pp. 355–385, August 1997.
  * [5] B. B. Bederson, “Audio augmented reality: A prototype automated tour guide,” in Conference Companion on Human Factors in Computing Systems, CHI ’95, (New York, NY, USA), pp. 210–211, ACM, 1995.
  * [6] B. Bell, S. Feiner, and T. Höllerer, “View management for virtual and augmented reality,” in Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, UIST ’01, (New York, NY, USA), pp. 101–110, ACM, 2001.
  * [7] O. Bergig, N. Hagbi, J. El-Sana, and M. Billinghurst, “In-place 3d sketching for authoring and augmenting mechanical systems,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 87–94, 2009.
  * [8] T. Caudell and D. Mizell, “Augmented reality: an application of heads-up display technology to manual manufacturing processes,” in System Sciences, 1992. Proceedings of the Twenty-Fifth Hawaii International Conference on, vol. ii, pp. 659–669 vol.2, Jan 1992.
  * [9] A. D. Cheok, S. W. Fong, K. H. Goh, X. Yang, W. Liu, and F. Farzbiz, “Human pacman: A sensing-based mobile entertainment system with ubiquitous computing and tangible interaction,” in Proceedings of the 2Nd Workshop on Network and System Support for Games, NetGames ’03, (New York, NY, USA), pp. 106–117, ACM, 2003.
  * [10] E. Coelho, S. Julier, and B. Maclntyre, “Osgar: a scene graph with uncertain transformations,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 6–15, Nov 2004.
  * [11] A. J. Davison, “Real-time simultaneous localisation and mapping with a single camera,” in 9th IEEE International Conference on Computer Vision (ICCV 2003), 14-17 October 2003, Nice, France, pp. 1403–1410, 2003.
  * [12] S. DiVerdi and T. Hollerer, “Groundcam: A tracking modality for mobile mixed reality,” in IEEE Virtual Reality Conference (VR), pp. 75–82, March 2007\.
  * [13] J. Engel, T. Schöps, and D. Cremers, “LSD-SLAM: Large-scale direct monocular SLAM,” in European Conference on Computer Vision (ECCV), September 2014.
  * [14] J. Engel, J. Sturm, and D. Cremers, “Semi-dense visual odometry for a monocular camera,” in IEEE International Conference on Computer Vision (ICCV), (Sydney, Australia), December 2013.
  * [15] S. Feiner, B. MacIntyre, T. Höllerer, and A. Webster, “A touring machine: prototyping 3d mobile augmented reality systems for exploring the urban environment,” in IEEE International Symposium on Wearable Computers, pp. 74–81, Oct 1997.
  * [16] G. W. Fitzmaurice, “Situated information spaces and spatially aware palmtop computers,” Commun. ACM, vol. 36, pp. 39–49, July 1993.
  * [17] D. Fritsch, D. Klinec, and S. Volz, “Nexus — positioning and data management concepts for location-aware applications,” Computers, Environment and Urban Systems, vol. 25, no. 3, pp. 279 – 291, 2001.
  * [18] J. Fruend, C. Geiger, M. Grafe, and B. Kleinjohann, “The augmented reality personal digital assistant,” in IEEE and ACM International Symposium on Augmented Reality (ISAR), 2001.
  * [19] S. Gauglitz, C. Sweeney, J. Ventura, M. Turk, and T. Hollerer, “Live tracking and mapping from both general and rotation-only camera motion,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 13–22, Nov 2012.
  * [20] L. Gruber, S. Gauglitz, J. Ventura, S. Zollmann, M. Huber, M. Schlegel, G. Klinker, D. Schmalstieg, and T. Höllerer, “The city of sights: Design, construction, and measurement of an augmented reality stage set,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 157–163, 2010.
  * [21] J. Grubert, T. Langlotz, and R. Grasset, “Augmented reality browser survey,” Tech. Rep. 1101, Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria, 2011.
  * [22] S. Guven and S. Feiner, “Authoring 3d hypermedia for wearable augmented and virtual reality,” in IEEE International Symposium on Wearable Computers, pp. 118–126, Oct 2003.
  * [23] A. Henrysson, M. Billinghurst, and M. Ollila, “Face to face collaborative ar on mobile phones,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 80–89, Oct 2005.
  * [24] D. Herrera C, K. Kim, J. Kannala, K. Pulli, and J. Heikkila, “Dt-slam: Deferred triangulation for robust slam,” in 3D Vision (3DV), 2014 2nd International Conference on, vol. 1, pp. 609–616, Dec 2014.
  * [25] A. Hill, B. MacIntyre, M. Gandy, B. Davidson, and H. Rouzati, “Kharma: An open kml/html architecture for mobile augmented reality applications,” in Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on, pp. 233–234, Oct 2010.
  * [26] T. Höllerer, S. Feiner, and J. Pavlik, “Situated documentaries: embedding multimedia presentations in the real world,” in IEEE International Symposium on Wearable Computers, pp. 79–86, Oct 1999.
  * [27] T. Höllerer, S. Feiner, T. Terauchi, G. Rashid, and D. Hallaway, “Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system,” Computers & Graphics, vol. 23, no. 6, pp. 779–785, 1999.
  * [28] S. Julier, Y. Baillot, D. Brown, and M. Lanzagorta, “Information filtering for mobile augmented reality,” Computer Graphics and Applications, IEEE, vol. 22, pp. 12–15, Sep 2002.
  * [29] S. Julier, Y. Baillot, M. Lanzagorta, D. Brown, and L. Rosenblum, “Bars: Battlefield augmented reality system,” in In NATO Symposium on Information Processing Techniques for Military Systems, pp. 9–11, 2000.
  * [30] M. Kalkusch, T. Lidy, M. Knapp, G. Reitmayr, H. Kaufmann, and D. Schmalstieg, “Structured visual markers for indoor pathfinding,” in Augmented Reality Toolkit, The First IEEE International Workshop, pp. 8 pp.–, 2002.
  * [31] H. Kato and M. Billinghurst, “Marker tracking and hmd calibration for a video-based augmented reality conferencing system,” in Augmented Reality, 1999. (IWAR ’99) Proceedings. 2nd IEEE and ACM International Workshop on, pp. 85–94, 1999.
  * [32] A. C. Kay, “A personal computer for children of all ages,” August 1972.
  * [33] G. Klein and D. Murray, “Parallel tracking and mapping for small ar workspaces,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), (Washington, DC, USA), pp. 1–10, IEEE Computer Society, 2007\.
  * [34] G. Klein and D. Murray, “Parallel tracking and mapping on a camera phone,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), (Orlando), IEEE Computer Society, October 2009.
  * [35] R. Kooper and B. MacIntyre, “Browsing the real-world wide web: Maintaining awareness of virtual information in an ar information space.,” Int. J. Hum. Comput. Interaction, vol. 16, no. 3, pp. 425–446, 2003.
  * [36] U. Kretschmer, V. Coors, U. Spierling, D. Grasbon, K. Schneider, I. Rojas, and R. Malaka, “Meeting the spirit of history,” in Proceedings of the 2001 Conference on Virtual Reality, Archeology, and Cultural Heritage, VAST ’01, (New York, NY, USA), pp. 141–152, ACM, 2001.
  * [37] T. Langlotz, D. Wagner, A. Mulloni, and D. Schmalstieg, “Online creation of panoramic augmented reality annotations on mobile phones,” Pervasive Computing, IEEE, vol. 11, pp. 56–63, Feb 2012.
  * [38] M. Li, B. H. Kim, and A. Mourikis, “Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera,” in Robotics and Automation (ICRA), 2013 IEEE International Conference on, pp. 4712–4719, May 2013.
  * [39] J. M. Loomis, R. G. Golledge, and R. L. Klatzky, “Personal guidance system for the visually impaired using gps, gis, and vr technologies,” in Proceedings of the First Annual International Conference, Virtual Reality and Persons with Disabilities, pp. 71–74, June 1993.
  * [40] A. Makri, D. Arsenijevic, J. Weidenhausen, P. Eschler, D. Stricker, O. Machui, C. Fernandes, S. Maria, G. Voss, and N. Ioannidis, “Ultra: An augmented reality system for handheld platforms, targeting industrial maintenance applications,” in Proceedings of 11th International Conference on Virtual Systems and Multimedia (VSMM’05), 2005.
  * [41] P. Milgram and F. Kishino, “Taxonomy of mixed reality visual displays,” IEICE Transactions on Information and Systems, vol. E77-D, no. 12, pp. 1321 – 1329, 1994.
  * [42] T. Miyashita, P. Meier, T. Tachikawa, S. Orlic, T. Eble, V. Scholz, A. Gapel, O. Gerl, S. Arnaudov, and S. Lieberknecht, “An augmented reality museum guide,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 103–106, Sept 2008.
  * [43] D. Mogilev, K. Kiyokawa, and M. Billinghurst, “Ar pad: An interface for face-to-face ar collaboration,” in In CHI 2002 Conference Proceedings, pp. 654–655, 2002.
  * [44] M. Möhring, C. Lessig, and O. Bimber, “Video see-through ar on consumer cell-phones,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 252–253, Nov 2004.
  * [45] A. Morrison, A. Oulasvirta, P. Peltonen, S. Lemmela, G. Jacucci, G. Reitmayr, J. Näsänen, and A. Juustila, “Like bees around the hive: A comparative study of a mobile augmented reality map,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, (New York, NY, USA), pp. 1889–1898, ACM, 2009.
  * [46] L. Naimark and E. Foxlin, “Circular data matrix fiducial system and robust image processing for a wearable vision-inertial self-tracker,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 27–36, 2002\.
  * [47] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. W. Fitzgibbon, “Kinectfusion: Real-time dense surface mapping and tracking,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136, 2011.
  * [48] R. A. Newcombe, S. Lovegrove, and A. J. Davison, “DTAM: dense tracking and mapping in real-time,” in IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327, 2011.
  * [49] J. Newman, D. Ingram, and A. Hopper, “Augmented reality in a wide area sentient environment,” in IEEE and ACM International Symposium on Augmented Reality (ISAR), pp. 77–86, 2001.
  * [50] Q. Pan, C. Arth, E. Rosten, G. Reitmayr, and T. Drummond, “Rapid scene reconstruction on mobile phones from panoramic images,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 55–64, 2011\.
  * [51] C. Pirchheim and G. Reitmayr, “Homography-based planar mapping and tracking for mobile phones,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 27–36, 2011.
  * [52] C. Pirchheim, D. Schmalstieg, and G. Reitmayr, “Handling pure camera rotation in keyframe-based SLAM,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 229–238, 2013.
  * [53] R. Raskar, J. van Baar, P. Beardsley, T. Willwacher, S. Rao, and C. Forlines, “ilamps: Geometrically aware and self-configuring projectors,” in ACM SIGGRAPH 2003 Papers, SIGGRAPH ’03, (New York, NY, USA), pp. 809–818, ACM, 2003\.
  * [54] H. Regenbrecht and R. Specht, “A mobile passive augmented reality device - mpard,” in IEEE and ACM International Symposium on Augmented Reality (ISAR), pp. 81–84, 2000.
  * [55] G. Reitmayr and D. Schmalstieg, “Mobile collaborative augmented reality,” in IEEE and ACM International Symposium on Augmented Reality (ISAR), pp. 114–123, 2001.
  * [56] G. Reitmayr and T. Drummond, “Going out: robust model-based tracking for outdoor augmented reality,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 109–118, Oct 2006.
  * [57] J. Rekimoto, “Augmented reality using the 2d matrix code,” in Proceedings of the Workshop on Interactive Systems and Software (WISS’96), 1996\.
  * [58] J. Rekimoto and K. Nagao, “The world through the computer: Computer augmented interaction with real world environments,” in Proceedings of the 8th Annual ACM Symposium on User Interface and Software Technology, UIST ’95, (New York, NY, USA), pp. 29–36, ACM, 1995.
  * [59] M. Rohs and B. Gfeller, “Using camera-equipped mobile phones for interacting with real-world objects,” in Advances in Pervasive Computing, pp. 265–271, 2004.
  * [60] M. Rohs, J. Schöning, M. Raubal, G. Essl, and A. Krüger, “Map navigation with mobile devices: Virtual versus physical movement with and without visual context,” in Proceedings of the 9th International Conference on Multimodal Interfaces, ICMI ’07, (New York, NY, USA), pp. 146–153, ACM, 2007.
  * [61] R. Salas-Moreno, B. Glocken, P. Kelly, and A. Davison, “Dense planar slam,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 157–164, Sept 2014.
  * [62] K. Satoh, M. Anabuki, H. Yamamoto, and H. Tamura, “A hybrid registration method for outdoor augmented reality,” in IEEE and ACM International Symposium on Augmented Reality (ISAR), vol. 29-30, pp. 67–76, Oct. 2001.
  * [63] D. Schmalstieg, T. Langlotz, and M. Billinghurst, “Augmented reality 2.0,” in Virtual Realities (G. Brunnett, S. Coquillart, and G. Welch, eds.), pp. 13–37, Springer Vienna, 2011.
  * [64] T. Schöps, J. Engel, and D. Cremers, “Semi-dense visual odometry for AR on a smartphone,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), September 2014.
  * [65] J. Spohrer, “Information in places,” IBM Systems Journal, vol. 38, no. 4, pp. 602–628, 1999.
  * [66] T. Starner, S. Mann, B. J. Rhodes, J. Levine, J. Healey, D. Kirsch, R. W. Picard, and A. Pentland, “Augmented reality through wearable computing,” Presence, vol. 6, no. 4, pp. 386–398, 1997.
  * [67] I. E. Sutherland, “A head-mounted three dimensional display,” in Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I, AFIPS ’68 (Fall, part I), (New York, NY, USA), pp. 757–764, ACM, 1968.
  * [68] W. Tan, H. Liu, Z. Dong, G. Zhang, and H. Bao, “Robust monocular slam in dynamic environments,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 209–218, Oct 2013.
  * [69] P. Tanskanen, K. Kolev, L. Meier, F. Camposeco, O. Saurer, and M. Pollefeys, “Live metric 3d reconstruction on mobile phones,” in IEEE International Conference on Computer Vision (ICCV), pp. 65–72, Dec 2013.
  * [70] B. Thomas, B. Close, J. Donoghue, J. Squires, P. De Bondi, M. Morris, and W. Piekarski, “Arquake: an outdoor/indoor augmented reality first person application,” in IEEE International Symposium on Wearable Computers, pp. 139–146, Oct 2000.
  * [71] B. Thomas, V. Demczuk, W. Piekarski, D. Hepworth, and B. Gunther, “A wearable computer system with augmented reality to support terrestrial navigation,” in IEEE International Symposium on Wearable Computers, pp. 168–171, Oct 1998.
  * [72] J. Ventura, C. Arth, G. Reitmayr, and D. Schmalstieg, “Global localization from monocular slam on a mobile phone,” in IEEE Virtual Reality, 03/2014 2014.
  * [73] V. Vlahakis, J. Karigiannis, M. Tsotros, M. Gounaris, L. Almeida, D. Stricker, T. Gleue, I. T. Christou, R. Carlucci, and N. Ioannidis, “Archeoguide: First results of an augmented reality, mobile computing system in cultural heritage sites,” in Proceedings of the 2001 Conference on Virtual Reality, Archeology, and Cultural Heritage, VAST ’01, (New York, NY, USA), pp. 131–140, ACM, 2001.
  * [74] D. Wagner, A. Mulloni, T. Langlotz, and D. Schmalstieg, “Real-time panoramic mapping and tracking on mobile phones,” in IEEE Virtual Reality Conference (VR), pp. 211–218, March 2010.
  * [75] D. Wagner, T. Pintaric, F. Ledermann, and D. Schmalstieg, “Towards massively multi-user augmented reality on handheld devices,” in Pervasive Computing (H.-W. Gellersen, R. Want, and A. Schmidt, eds.), vol. 3468 of Lecture Notes in Computer Science, pp. 208–219, Springer Berlin Heidelberg, 2005.
  * [76] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmalstieg, “Pose tracking from natural features on mobile phones,” in IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 125–134, Sept 2008.
  * [77] D. Wagner and D. Schmalstieg, “First steps towards handheld augmented reality,” in IEEE International Symposium on Wearable Computers, pp. 127–135, Oct 2003.
  * [78] S. White and S. Feiner, “Sitelens: Situated visualization techniques for urban site visits,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09, (New York, NY, USA), pp. 1117–1120, ACM, 2009\.

plain links


G. Snaauw et al., ”Mutual Information Neural Estimation for Unsupervised
Multi-Modal Registration of Brain Images,” 2022 44th Annual International
Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2022,
pp. 3510-3513, doi: 10.1109/EMBC48229.2022.9871220.

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

# Mutual Information Neural Estimation for
Unsupervised Multi-Modal Registration of Brain Images

Gerard Snaauw1, Michele Sasdelli1, Gabriel Maicas1, Stephan Lau1,2,
Johan Verjans1,2, Mark Jenkinson1,2,∗, and Gustavo Carneiro1 1Australian
Institute for Machine Learning (AIML), University of Adelaide, Adelaide,
Australia, 2South Australian Health and Medical Research Institute (SAHMRI),
Adelaide, Australia, *Correspondence may be directed to Prof. Mark Jenkinson:
<EMAIL_ADDRESS>
###### Abstract

Many applications in image-guided surgery and therapy require fast and
reliable non-linear, multi-modal image registration. Recently proposed
unsupervised deep learning-based registration methods have demonstrated
superior performance compared to iterative methods in just a fraction of the
time. Most of the learning-based methods have focused on mono-modal image
registration. The extension to multi-modal registration depends on the use of
an appropriate similarity function, such as the mutual information (MI). We
propose guiding the training of a deep learning-based registration method with
MI estimation between an image-pair in an end-to-end trainable network. Our
results show that a small, 2-layer network produces competitive results in
both mono- and multi-modal registration, with sub-second run-times.
Comparisons to both iterative and deep learning-based methods show that our
MI-based method produces topologically and qualitatively superior results with
an extremely low rate of non-diffeomorphic transformations. Real-time clinical
application will benefit from a better visual matching of anatomical
structures and less registration failures/outliers.

##  I Introduction

Multi-modal registration has many use cases, such as in image-guided therapy
where pre-therapeutic imaging is aligned with live data to guide treatment
(e.g., tumor localization in surgery or radiotherapy). The large computational
run-time of conventional iterative methods mean that registration methods in
these applications are limited to inaccurate linear transforms [1]. Recent
advances in unsupervised deep learning-based registration have demonstrated
sub-second run-times in accurate mono-modal deformable registration [2]. A
range of auxiliary approaches have since been investigated to improve
registration performance, specifically segmentation maps, multi-resolution,
and locally adaptive regularisation [2, 3, 4, 5, 6]. The extension of deep
learning methods to multi-modal registration depends on an appropriate
similarity function, such as mutual information (MI) [7], which is one of the
most studied and successful similarity metrics in multi-modal iterative
registration approaches [1]. Application of MI in a learning-based setting is
not trivial due to the non-differential quantization step (i.e., binning of
intensity values). One solution is to use a continuous quantization function
[6].

In this paper, we propose estimating MI using a 2-layer convolutional neural
network (CNN) to extend deep learning-based frameworks to multi-modal
registration. More specifically, we estimate MI with mutual information neural
estimation (MINE) that eliminates the cumbersome discretization of the density
estimation step and allows end-to-end training as it is fully differentiable
[8].

##  II Methods

###  II-A Registration framework

Image registration estimates a transform
$\phi_{\mathbf{z}}:\Omega\rightarrow\mathbb{R}^{3}$ (parameterized by
$\mathbf{z}$) that aligns a moving image
$\mathbf{m}:\Omega\rightarrow\mathbb{R}$ to a fixed image
$\mathbf{f}:\Omega\rightarrow\mathbb{R}$, such that
$\mathbf{m}(\phi_{\mathbf{z}}(\omega))$ becomes similar to
$\mathbf{f}(\omega)$, where $\omega$ are points on the image lattice $\Omega$.
For brevity we drop $\omega$ and write transformation as
$\mathbf{m}\circ\phi_{\mathbf{z}}$.

For our registration framework we use the diffeomorphic variant of VoxelMorph
(VM) [3], see Fig. 1, but our method is not specific to this framework. VM
takes the input images $\mathbf{f},\mathbf{m}$ and passes them through a Unet-
like structure [9] to output a posterior distribution from which a sample is
taken that is a stationary velocity field [3]. This is then integrated to form
a diffeomorphic transform $\phi_{\mathbf{z}}$ using a scaling and squaring
approach [10], and applied to the moving image. VM applies the mean square
error (MSE), a mono-modal similarity metric, between
$\mathbf{m}\circ\phi_{\mathbf{z}}$ and $\mathbf{f}$ and a regularization
function that contains one hyper-parameter $\lambda$ to control the scale of
the velocity field.

Figure 1: Overview of the unsupervised registration approach using MINE. Left
illustrates a generic unsupervised registration framework that produces the
transform to align an image pair. Right shows our proposed method for training
the framework by estimating MI with MINE. The MINE network forms a single end-
to-end differentiable network with the registration framework during training,
and can be removed hereafter.

###  II-B Novel image matching loss function

We replace the image matching term in the loss function of VM [3] with a loss
function based on MI that has been demonstrated to work effectively for multi-
modal problems in traditional (non-machine-learning) registration methods [7].
The design of our image matching loss function is based on MINE [8] and
replaces the cumbersome approximation of MI through a binning approach by
training of a network that estimates MI and works on multi-dimensional
continuous-valued data.

The MI-based registration of $\mathbf{m}$ to $\mathbf{f}$ is defined by

$I(\mathbf{f},\mathbf{m})=\int_{\Omega}p(\mathbf{f},\mathbf{m}\circ\phi_{\mathbf{z}})\>\log\left(\frac{p(\mathbf{f},\mathbf{m}\circ\phi_{\mathbf{z}})}{p(\mathbf{f})p(\mathbf{m}\circ\phi_{\mathbf{z}})}\right)d\omega,$
(1)

where $p(\mathbf{f},\mathbf{m}\circ\phi_{\mathbf{z}})$ measures the joint
probability between the gray values of the fixed and moving images, and
$p(\mathbf{f})$ and $p(\mathbf{m}\circ\phi_{\mathbf{z}})$ are the marginal
probabilities of $\mathbf{f}$ and $\mathbf{m}\circ\phi_{\mathbf{z}}$ – note
that the MI in (1) is equivalent to the Kullback-Leibler (KL) divergence
between the joint-probability density
$\mathbb{P}=p(\mathbf{f},\mathbf{m}\circ\phi_{\mathbf{z}})$ and product of
marginals $\mathbb{Q}=p(\mathbf{f})p(\mathbf{m}\circ\phi_{\mathbf{z}})$,
defined by $D_{KL}\left(\ \mathbb{P}\ ||\ \mathbb{Q}\ \right)$. We extend MINE
[8] by training the image matching network with a lower-bound estimate on the
MI in (1) using the following inequality [11]:

$D_{KL}\left(\>\mathbb{P}\>||\>\mathbb{Q}\>\right)\geq\sup_{F_{\theta}\in\mathcal{F}}\mathbb{E}_{\mathbb{P}}\left[F_{\theta}(\mathbf{f},\mathbf{\hat{m}})\right]-\log\left(\mathbb{E}_{\mathbb{Q}}\left[e^{F_{\theta}(\mathbf{f},\mathbf{\hat{m}})}\right]\right)$
(2)

where ${F}_{\theta}:\Omega\to\mathbb{R}$ (with ${F}_{\theta}\in\mathcal{F}$)
represents any class of functions (parameterized by $\theta$) that satisfies
the integrability constraints [11], and
$\mathbf{\hat{m}}=\mathbf{m}\circ\phi_{\mathbf{s}}\circ\phi_{\mathbf{z}}$ with
$\phi_{\mathbf{s}}(\omega)=\omega$ when the expectation is over $\mathbb{P}$
(i.e., $\phi_{\mathbf{s}}$ is the identify function, so $\mathbf{f}$ and
$\mathbf{m}$ match according to $\phi_{\mathbf{z}}$), and
$\phi_{\mathbf{s}}(\omega)\sim U_{\Omega}(\omega)$ (with $U_{\Omega}(\omega)$
being a flat distribution over $\Omega$, such as a random permutation of
locations) when the expectation is over $\mathbb{Q}$ – this means that for the
product of marginals $\mathbb{Q}$, $\mathbf{f}$ and $\mathbf{m}$ are unlikely
to match according to $\phi_{\mathbf{z}}$.

The learning function that we use to optimise our model is the following:

$\begin{split}\theta^{*},\psi^{*}=\operatorname*{argmin}_{\theta,\psi}&-\alpha\left(\mathbb{E}_{\mathbb{P}}\left[F_{\theta}(\mathbf{f},\mathbf{\hat{m}})\right]-\log\left(\mathbb{E}_{\mathbb{Q}}\left[e^{F_{\theta}(\mathbf{f},\mathbf{\hat{m}})}\right]\right)\right)\\\
&+R(\phi_{\mathbf{z}},\lambda),\end{split}$ (3)

where $R(.)$ is the regularization function to approximate the variational
posterior $q_{\psi}(\mathbf{z}|\mathbf{f},\mathbf{m})$ and the prior
$p(\mathbf{z})$ [3], $\phi_{\mathbf{z}}$ is the diffeomorphic transform
computed from the network (see left hand side of Fig. 1), and
$\phi_{\mathbf{s}}$ is defined in Eq. (2).

In (3)$,F_{\theta}$ is modeled as a simple 2-layer CNN with 1x1x1 filters to
preserve the independent voxel assumption in MI. The first layer combines $L$
linear transformed versions of the input to create $L$ features per voxel-
pair. Then, a second layer maps these features back to a single value per
voxel that are used to generate representations for $\mathbb{P}$ and
$\mathbb{Q}$. During inference, we are given two test images
$\tilde{\mathbf{m}},\tilde{\mathbf{f}}$, and the registration is obtained by
the diffeomorphic transform $\phi_{\mathbf{z}}$.

For the function $U_{\Omega}(\omega)$ in (2), we evaluate global and local
shuffling strategies referred to as MINEglobal and MINElocal. In MINEglobal,
$\phi_{s}(\omega)\sim{U}_{\Omega}(\Omega)$, which means that it randomly
shuffles all voxel coordinates (using a uniform distribution over the whole
input lattice $\Omega$) to estimate the distribution over voxel pairs for
unaligned images. However, since a large fraction of voxels in our data belong
to background, we also evaluate a local shuffling strategy to estimate the
distribution of voxel pairs in their expected neighborhood, as would be
expected after sensible pre-processing and initialisation. This is denoted by
MINElocal, $\phi_{s}(\omega)\sim{U}_{\Omega}([\omega-N,\omega+N])$. MINElocal
returns a new input address that is within $2\times N$ (in each direction) of
the original address $\omega$.

###  II-C Data and preprocessing

This research study was conducted retrospectively using human subject data
made available in open access [12, 13]. All ethical standards defined in the
data use agreement have been implemented. For our experiments we use the
T1-weighted scans from the OASIS-2 dataset [12], and a subset of the ADNI
dataset [13], where we included all subjects that had a T1-weighted and FLAIR
scan with high spatial resolution acquired in the initial session.
Preprocessing was performed using FreeSurfer [14] on all T1-weighted scans and
consisted of resampling to 1 mm isotropic voxels, affine spatial
normalization, brain extraction, and automated generation of segmentation
maps. Since both modalities of the ADNI dataset are acquired in the same
session we assume that they are well aligned and apply the preprocessing
parameters (including segmentation maps) found for T1 to the FLAIR volumes.
Anatomical regions were excluded if they had less than 100 voxels in any of
the subjects, resulting in 30 regions that will be used for evaluation (see
later). Volumes are cropped to $160\times 192\times 224$ voxels and, after
removal of subjects that failed pre-processing, split into
training:validation:test sets with 230:24:118 subjects for OASIS and
293:24:118 for ADNI.

###  II-D Training process

We evaluate our method on two tasks. First, we replicate the setup from [3]
and evaluate mono-modal registration to an atlas. Second, we evaluate the
performance of our method on multi-modal inter-patient registration by
aligning T1-weighted and FLAIR volumes, which we regard as an example test for
the non-linear registration required in image-guided procedures. For the
training of the MINE-network $F_{\theta(.)}$ in (3), we set the feature vector
length $L=30$, where the neighbouring region for MINElocal is set at $N=8$
voxels. Evaluation of these values was limited by available memory and
prolonged training and chosen on the basis of capturing a reasonable level of
anatomical context (for $N$) and achieving fast and stable convergence (for
$L$). In (3), we set the regularization parameter $\lambda=10$ for all
experiments and training was performed using the ADAM optimizer with
$\beta_{1}=0.99$, $\beta_{2}=0.999$ and learning rate $10^{-4}$ for 1500
epochs with a batch size of 1 and Kaiming initialization.

###  II-E Comparison with alternative methods

The baseline methods that we include are ANTs [15] and the diffeomorphic
variant of VoxelMorph [3]. ANTs registration uses the default settings that
combines SyN [15] with cross correlation (CC) and MI to produce the method’s
optimal results for both registration tasks. The training of VM is performed
using local normalized cross-correlation (VM-CC) with neighborhood $9^{3}$
[2]. We use CC rather than MSE as CC is also capable of multi-modal
registration.

Evaluation is separated into a quantitative and qualitative part. The
quantitative part evaluates $1)$ Dice similarity coefficient (DSC) as a
measure of registration accuracy, $2)$ incidence of non-positive determinants
for Jacobian values $|J_{\phi}(\omega)|\leq 0$ as a measure of the
diffeomorphic performance, and $3)$ run-time at test-time to access
feasibility for image guided therapies. The qualitative part compares visual
registration results and shows the response for the joint distribution
$\mathbb{P}$ in (2).

##  III Results

Figure 2: Performance for different values of similarity weight $\alpha$ (3)
for mono-modal registration (solid curves) and multi-modal registration
(dashed curves). Left: average Dice score. Right: number of non-diffeomorphic
voxels.

###  III-A Quantitative results

Tuning results for the similarity weight $\alpha$ (in Eq. 3), for all VM
methods, are shown in Fig. 2 with the best results summarized in Table I. Note
that the DSC is not very sensitive to the choice of the similarity weight for
MINElocal, and the only notable sensitivity for non-positive Jacobian
determinants occurs when $\alpha>5$. On mono- and multi-modal tasks, all
methods achieve comparable results at peak performance with a $\sim 20\%$
increase for mono-modal and $\sim 14\%$ increase for multi-modal DSC compared
to the affine results. (Side-by-side comparison of all methods in terms of
dice performance per area are available on request.)

[b]  Mono-modality Multi-modality Framework Mean Dice $|J_{\phi}|\leq 0$ Mean
Dice $|J_{\phi}|\leq 0$ Affine $57.6\>(10.8)$ - $54.2\>(11.9)$ - ANTs
$77.7\>(3.6)$ $4.0\>(32.3)$ $68.7\>(10.7)$ $2.1\>(9.7)$ VM-CC $77.3\>(3.7)$
$0.3\>(1.4)$ $67.7\>(8.5)$ $0\>(0)$ VM-MINEg $76.7\>(3.6)$ $0\>(0)$
$67.4\>(8.5)$ $0\>(0)$ VM-MINEl $76.7\>(3.6)$ $0\>(0)$ $68.2\>(8.2)$ $0\>(0)$

TABLE I: Best registration results, per task and method.

Even though ANTs combines multiple similarity metrics and SyN, it does not
substantially outperform the VM-based methods and has more outliers in failed
registrations, making the deep-learning-based approaches more reliable for
clinical applications. Comparing the VM-based methods for multi-modal
registration in terms of DSC, MINElocal performs better than CC, with
MINEglobal a close third place. For mono-modal registration, CC is slightly
better than MINElocal and MINEglobal. However, there are some limits to how
strongly we can extrapolate from these findings, as there can be imperfections
in automated segmentation, pre-processing performance across multiple
modalities, and observed run-to-run variations in mean DSC differences (of up
to $0.5\%$). Table I and Fig. 2 show that the diffeomorphic properties of VM
are unaffected by the switch to multi-modal registration and choice of
similarity metric. The number of non-diffeomorphic transforms are negligible
in all cases up to the peak-performance point, based on DSC, though they
rapidly increase thereafter.

Since the similarity metric does not need to be evaluated at test-time, run-
time of VM is the same for all methods at $0.46\>(\pm\>0.02)$ seconds in the
mono-modal tasks, increasing to $0.60\>(\pm\>0.02)$ seconds for multi-modal
registration on a Titan Xp GPU. In both tasks, deep-learning-based methods are
orders of magnitudes faster than ANTs (approx. $10^{4}\>(\pm\>10^{3})$
seconds), and are fast enough for most real-time applications.

The low sensitivity to $\alpha$, together with the similar DSC performance,
lower outliers, and faster execution, makes MINElocal the overall best, or at
least equally favourable to VM-CC, in both mono- and multi-modal tasks on the
basis of the quantitative results.

Figure 3: Top: average results for the mono-modal registration after aligning
all images to the atlas and absolute difference with the atlas. Bottom:
Representative samples of multi-modal registration.

###  III-B Qualitative results

Representative registration results are shown in Fig. 3. All methods produce
accurate results on both tasks with minor differences in varying locations.
Ignoring the extracortical differences resulting from background masking in
the atlas, the most prominent differences for all methods are in the cortex
where the largest inter-patient anatomical variation occurs, though
differences in the lateral ventricles are also visible. In the mono-modal task
these differences are consistent throughout the image for ANTs and MINE while
VM-CC shows larger average misalignment in the frontal lobe, and especially
the anterior horn of the lateral ventricles, compared to the rest of the
image. This is consistent for all values of $\alpha$ in all similarity metrics
evaluated. We hypothesize that intensity normalization in small neighborhoods
for CC, while producing excellent overall results, can have a negative effect
on learning features for registration where there is large anatomical
variation. Since MINE is a global metric, even in our local sampling strategy,
it does not suffer from this variation. The difference images also show that
MINE behaves consistently everywhere with minimal differences on sub-structure
boundaries where voxel intensities change.

To further investigate the MINE-network we took two artificial inputs that
cover the range of possible intensity-pair combinations present in the
training data to visualize the network’s joint-distribution prediction map
$\mathbb{P}$, see Fig. 4 for multi-modal results. The local shuffling strategy
has removed less informative background-brain intensity-pairs compared to
MINEglobal, instead using local brain-brain-pairs, as seen in unaligned
inputs. This is shown by the more complex joint-distribution for MINElocal,
using a 2-layer CNN, creating better features for registration.

In clinical applications it is important to avoid non-diffeomorphic
transformations as these represent cases where anatomically implausible
results are generated (e.g. structures disappear or break apart). For real-
time applications it is very important to maintain good regional performance
(measured by DSC) and anatomical fidelity (measured by Jacobian determinants)
and having fast, sub-second run-times. The local form of the MINE-network
allows us to satisfy all three of these criteria, and without difficulty in
tuning hyper-parameters. This makes MINE-local an excellent candidate for
real-time clinical applications, from both a theoretical and practical
standpoint.

Figure 4: Multi-modal joint distribution responses ($\mathbb{P}$ in Eq. 2) to
artificial inputs over the data range. Compared to the $MINE_{global}$, the
sampling strategy of MINElocal results in a more complex response that is a
closer approximation of, and guides learning towards, the true joint
distribution of an aligned multi-modal image pair.

##  IV Conclusion

We have shown that a small 2-layer CNN MINE-network is capable of producing
state-of-the-art mono- and multi-modal registration results on continuous data
when smart sampling strategies are used. This eliminates the cumbersome
quantization step of conventional MI-based registration, and has relatively
low sensitivity to the only key hyper-parameter – the image similarity
training weight. This makes the similarity metric easy to use, which provides
accurate results to tune training, with an extremely low rate of producing
non-diffeomorphic transformations or failures/outliers. Overall, it produces
accurate and anatomically correct results that are fast enough for most real-
time applications of multi-modal deformable registration in image-guided
therapies.

## References

  * [1] A. Sotiras, C. Davatzikos, and N. Paragios,  “Deformable medical image registration: A survey,”  IEEE Trans. on Medical Imaging, vol. 32, pp. 1153–1190, 2013\.
  * [2] G. Balakrishnan, A. Zhao, M.R. Sabuncu, J. Guttag, and A.V. Dalca,  “Voxelmorph: a learning framework for deformable medical image registration,”  IEEE Trans. on Medical Imaging, vol. 38, pp. 1788–1800, 2019\.
  * [3] A.V. Dalca, G. Balakrishnan, J. Guttag, and M.R. Sabuncu,  “Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces,”  Medical Image Analysis, vol. 57, pp. 226–236, 2019.
  * [4] A. Nan, M. Tennant, U. Rubin, and N. Ray,  “Drmime: Differentiable mutual information and matrix exponential for multi-resolution image registration,”  Proceedings of Machine Learning Research, vol. 121, pp. 527–543, 2020.
  * [5] Y. Zhu, Z. Zhou, G. Liao, and K. Yuan,  “A novel unsupervised learning model for diffeomorphic image registration,”  in Proc. SPIE Medical Imaging 2021, 2021, p. 115960M.
  * [6] Huaqi Qiu, Chen Qin, Andreas Schuh, Kerstin Hammernik, and Daniel Rueckert,  “Learning diffeomorphic and modality-invariant registration using b-splines,”  in Medical Imaging with Deep Learning, 2021.
  * [7] W.M. Wells, P. Viola, H. Atsumi, S. Nakajima, and R. Kikinis,  “Multi-modal volume registration by maximization of mutual information,”  Medical Image Analysis, vol. 1, pp. 35–51, 1996.
  * [8] M.I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm,  “Mutual information neural estimation,”  in Int. Conf. on Machine Learning, 2018, pp. 531–540.
  * [9] O. Ronneberger, P. Fischer, and T. Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. Springer, 2015, pp. 234–241.
  * [10] V. Arsigny, O. Commowick, X. Pennec, and N. Ayache,  “A log-euclidean framework for statistics on diffeomorphisms,”  in Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. Springer, 2006, pp. 924–931.
  * [11] M.D. Donsker and S.R.S. Varadhan,  “Asymptotic evaluation of certain markov process expectations for large time. iv,”  Communications on Pure and Applied Mathematics, vol. 36, pp. 183–212, 1983.
  * [12] D.S. Marcus, A.F. Fotenos, J.G. Csernansky, J.C. Morris, and R.L. Buckner,  “Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults,”  Journal of Cognitive Neuroscience, vol. 22, pp. 2677–2684, 2010\.
  * [13] P.S. Petersen, R.C.and Aisen, L.A. Beckett, M.C. Donohue, A.C. Gamst, D.J. Harvey, C.R. Jack, W.J. Jagust, L.M. Shaw, A.W. Toga, et al.,  “Alzheimer’s disease neuroimaging initiative (adni): clinical characterization,”  Neurology, vol. 74, pp. 201–209, 2010.
  * [14] B. Fischl,  “Freesurfer,”  Neuroimage, vol. 62, pp. 774–781, 2012.
  * [15] B.B. Avants, C.L. Epstein, M. Grossman, and J.C. Gee,  “Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain,”  Medical Image Analysis, vol. 12, pp. 26–41, 2008.