#  TOM: A Development Platform For Wearable Intelligent Assistants

Nuwan Janaka<EMAIL_ADDRESS>0000-0003-2983-6808 Synteraction Lab,Smart
Systems Institute, National University of SingaporeSingapore ,  Shengdong Zhao
<EMAIL_ADDRESS>0000-0001-7971-3107 Synteraction Lab,School of
Creative Media & Department of Computer Science, City University of Hong
KongHong KongChina ,  David Hsu<EMAIL_ADDRESS>0000-0002-2309-4535
School of Computing,Smart Systems Institute, National University of
SingaporeSingapore ,  Sherisse Tan Jing Wen<EMAIL_ADDRESS>0009-0003-7078-2642 School of Computing, National University of
SingaporeSingapore and  Koh Chun Keat<EMAIL_ADDRESS>0000-0001-9879-5419
Smart Systems Institute, National University of SingaporeSingapore

###### Abstract.

Advanced digital assistants can significantly enhance task performance, reduce
user burden, and provide personalized guidance to improve users’ abilities.
However, the development of such intelligent digital assistants presents a
formidable challenge. To address this, we introduce TOM, a conceptual
architecture and software platform (https://github.com/TOM-Platform) designed
to support the development of intelligent wearable assistants that are
contextually aware of both the user and the environment. This system was
developed collaboratively with AR/MR researchers, HCI researchers, AI/Robotic
researchers, and software developers, and it continues to evolve to meet the
diverse requirements of these stakeholders. TOM facilitates the creation of
intelligent assistive AR applications for daily activities and supports the
recording and analysis of user interactions, integration of new devices, and
the provision of assistance for various activities. Additionally, we showcase
several proof-of-concept assistive services and discuss the challenges
involved in developing such services.

context-aware system, wearable, AI assistance, smart glasses, HMD,
interactions

††ccs: Human-centered computing Ubiquitous and mobile computing systems and
tools††ccs: Human-centered computing Mobile devices††ccs: Computing
methodologies Artificial intelligence

##  1\. Introduction

With advancements in Machine Learning (ML) and Artificial Intelligence (AI)
technologies, digital assistants have become integral to everyday life. These
include voice assistants like Siri, Alexa, or Google, and some even support
visual modality (e.g., (noa, 2024; rabbit research team, 2023)), enabling rich
inputs and outputs. Similar to science fiction, advanced digital assistants
can practically aid users in performing both familiar and new tasks, reduce
task load and errors, and enhance task performance (Castelo et al., 2023).
Moreover, these assistants offer personalization, optimizing support for
individual needs and broadening accessibility.

Despite existing interaction paradigms such as Heads-Up Computing (Zhao et
al., 2023) and Dynamicland (Victor, 2018) aiming to realize digital assistance
in daily activities, several challenges persist. These include a lack of
understanding of the required system capabilities, development guidance, and a
platform for rapid development of such assistance (see Sec 2-3 for details).

To tackle these challenges, we introduce TOM, an intelligent wearable
assistive system developed by identifying user, researcher, and developer
needs. TOM facilitates the creation and analysis of assistive applications,
enabling context and user understanding and supporting multimodal interactions
with AR/MR devices and ML/AI technologies. Through developing several proof-
of-concept services (e.g., running coach assistance, translation, and querying
assistance), we have identified challenges in different daily activities and
the necessary future improvements for the TOM system to support envisioned
usage scenarios.

The contributions of this paper are twofold: 1) Identifying the required
capabilities for an intelligent wearable assistive system and developing a
conceptual architecture; 2) Creating the TOM system platform to enable the
development of assistive services for various activities and demonstrating its
application in several use cases.

##  2\. Related Work

Our work, which envisions an intelligent wearable assistive system, is related
to context-aware and assistive Augmented Reality (AR) systems.

###  2.1. Context-Aware Systems

In general, context can be defined as “any information used to characterize
the situation of an entity. An entity is a person, place, or object that is
considered relevant to the interaction between a user and an application,
including the user and the application themselves, and, by extension, the
environment in which the user and applications are embedded. A system is
context-aware if it uses context to provide relevant information and/or
services to the user, where relevancy depends on the user’s task.” (Dey et
al., 2001; Abowd et al., 1999). In the development of context awareness in
augmented reality (AR) or pervasive augmented reality, Grubert et al. (Grubert
et al., 2017) propose a taxonomy considering context sources (i.e., context
factors to which AR systems can adapt), context targets (i.e., content that
should be adapted), and context controllers (i.e., how the adaptation should
be made). Although they outline high-level categories of context sources
(e.g., human, environmental, system factors) and context targets (e.g., system
input, system output, system configuration), they do not offer specific
guidance for potential capabilities or implementations for context-aware AR
systems. While Dey et al. (Dey et al., 2001) provide guidelines and an
architecture for developing context-aware applications through the Context
Toolkit, they do not address the various stakeholders’ requirements for such
systems. Therefore, we first attempt to identify the capabilities for an
intelligent wearable context-aware AR system and then adapt the factors and
guidelines provided by Dey et al. (Dey et al., 2001) and Grubert et al.
(Grubert et al., 2017) to develop such a system.

The Platform for Situated Intelligence (\psi), developed by Bohus, Andrist, et
al. (Bohus et al., 2021; Andrist et al., 2022), shares certain high-level
objectives and architectural similarities with TOM; both systems utilize
timestamped multimodal streaming data from various sensors to facilitate rapid
development and research within interactive systems. However, TOM specifically
focuses on wearable, user-centered AR applications, necessitating distinct
requirements and system capabilities such as user sensing, compared to
traditional situated interactive systems targeted by \psi. Moreover, the
interactions and interfaces of TOM are optimized to provide intelligent
assistance during daily activities while minimizing interference with daily
tasks, with the intention of enabling dynamic context- and user-dependent
adaptive interfaces.

###  2.2. Assistive AR Systems

Augmented and Mixed Reality (AR/MR) have been used to develop assistive
systems that augment users’ perception with digitally superimposed content in
the physical world (Azuma, 1997; Speicher et al., 2019). These systems have
been applied in various domains such as education, training, repair and
maintenance, healthcare and medicine, and education due to their advantages in
minimizing errors and reducing cognitive load (Billinghurst et al., 2015; Wang
et al., 2016; Barsom et al., 2016). However, most of these systems are
tailored to specific tasks and lack adaptability for various daily activities.

Several toolkits have been proposed for developing assistive AR systems, yet
they are often designed for specialized purposes. For instance, RagRug (Fleck
et al., 2023) and DXR (Data visualizations in eXtended Reality) (Sicat et al.,
2019) are designed for situated and immersive analytics, enabling the
authoring of context-aware visualizations. Project Aria (Engel et al., 2023)
facilitates the collection of egocentric multi-modal data for training ML/AI
models in context-aware assistive AR systems but does not offer implementation
guidance. In contrast, ARGUS (Augmented Reality Guidance and User-modeling
System) (Castelo et al., 2023), a visual analytics tool, specializes in
multimodal data collection and modeling environmental and performer behaviors.
It supports retrospective data analysis through AR sensors and ML models to
enhance task guidance. ARGUS aligns partially with our envisioned system,
facilitating the generation of large amounts of annotated data for training
and developing ML/AI models. These models are designed to enhance decision-
making in assistive AR systems or to improve existing systems by visualizing
and understanding behaviors. However, as an analytics tool, ARGUS lacks in-
built development support for comprehensive end-to-end assistive AR systems
(e.g., sensor data integration, context understanding, decision-making for
assistance, dynamic feedback delivery via actuators) across various
application contexts. This is a gap that our system, TOM, attempts to address.
The very recently introduced SIGMA (Situated Interactive Guidance, Monitoring,
and Assistance) (Bohus et al., 2024), extends \psi, enabling a mature system
for (linear) procedural task guidance using MR. While it has advanced 3D scene
understanding and visualization capabilities (due to \psi), it lacks system-
level capabilities to understand users and primarily supports specific
hardware (i.e., HoloLens2), and does not support general-purpose daily tasks
by default, which TOM attempts to address.

With the advancement of AI technologies, new commercial wearable assistive AI
systems like Humane Ai Pin (noa, 2024) and Rabbit R1 (rabbit research team,
2023) have emerged, supporting specific activities (e.g., booking taxis,
querying objects). However, these systems display a limited understanding of
users, often restricted to speech interactions, and they do not provide
adequate support for developers and researchers in analyzing and visualizing
data to enhance assistance or support new activities. In contrast, our system,
TOM, aims to understand users’ multimodal interactions in context and provide
multimodal assistance using AI technologies. Additionally, it attempts to
empower developers and researchers to create assistance for various
activities, offering robust support for data recording capabilities.

##  3\. TOM: The Other Me

###  3.1. Envisioned Usage Scenarios

Consider Jane, who regularly uses Jerry, a digital assistant developed using
TOM, in her daily life. Jerry sees what Jane sees, hears what Jane hears,
knows her preferences, and understands her emotional and physical conditions.

##### Scenario

Unable to decide on a dish to prepare for herself and her toddler and wishing
to try something new, she opens the refrigerator and asks, “Hey, Jerry. Can
you suggest a new dish for us?” Jerry scans and identifies the ingredients in
the refrigerator, finds possible dishes, and renders three new dish
suggestions with their images. Jane finds the second suggestion appealing and
inquires about the preparation process. Jerry then guides her through
preparing the new dish, providing real-time, step-by-step feedback
superimposed in real-world objects.

Later, Jane receives a delivery of a play table set for her toddler, ordered
through Jerry during an online browsing session. She notices her toddler’s
eagerness to assist in assembling the set. Examining the package, she asks,
“Hey Tom, can you help me build this?” By identifying the play table set and
retrieving instructions, Jerry displays step-by-step virtual instructions
superimposed on the physical parts, which Jane follows while involving her
toddler. Suddenly, her toddler accidentally drops a piece of the set, striking
his leg and causing him to cry. Jane becomes panicked. Sensing the situation,
Jerry instructs her to remain calm and inspect her toddler’s leg. As Jane
consoles her child, Jerry assesses the situation and provides first-aid
instructions. During the first aid, Jerry asks whether to contact her husband,
family doctor, or hospital for further treatment. Upon request, Jerry connects
with the family doctor via video call to further observe the toddler’s leg.

###  3.2. System Capabilities

In our quest for an envisioned intelligent wearable assistant, Jerry, we
observed that while certain capabilities are supported by existing context-
aware and assistive AR/MR systems, a complete integration of these
capabilities into a single system is lacking. The Heads-Up Computing Paradigm
(Zhao et al., 2023), while theoretically supporting our envisioned use cases,
does not provide guidance on implementing such a system or the capabilities
required to further research optimal Human-AI interactions during daily
activities. Drawing from literature, our experience working with AR/MR and AI
researchers, and testing assistive Human-AI interfaces (including early
prototypes of TOM) with participants and their feedback, we have formulated
the following system capabilities. These are categorized based on three major
stakeholders’ requirements, which, though distinct, have overlapping
capabilities.

##### Just-in-time Assistance for Users

Users should be able to interact ($\textbf{C1}_{a}$) with the system (i.e.,
provide input and receive feedback) naturally and optimally to obtain the
desired assistance (Hornbæk and Oulasvirta, 2017; Zhao et al., 2023). Such
assistance should be delivered just in time to match the user’s current needs
or proactively when users have limited knowledge of system capabilities
(Rhodes and Maes, 2000; Yang et al., 2020; Amershi et al., 2019), with minimal
interference in the user’s ongoing activities while accommodating the user’s
cognitive capabilities (Anderson et al., 2018; McCrickard and Chewar, 2003).
To achieve this, the system should understand the user ($\textbf{C1}_{b}$) and
context ($\textbf{C1}_{c}$) to provide the most appropriate feedback to
support the user’s ongoing activities (Dey et al., 2001; Zhao et al., 2023).
Such understanding aids in modeling the human and the world to minimize
awareness mismatch between user expectations and system feedback and
maintaining profiles (Schmidt, 2014).

##### Data Recording and Analysis for Researchers

To understand user interactions with such a system and to design optimal
interactions, researchers need to record ($\textbf{C2}_{a}$), visualize
($\textbf{C2}_{b}$), and analyze ($\textbf{C2}_{c}$) the data and develop
models (Castelo et al., 2023; Murray-Smith et al., 2022; Engel et al., 2023).
This involves collecting data to support real-time and retrospective
observations, training models to predict optimal feedback and analyzing their
performance, and understanding the underlying reasons for user and system
behaviors (Engel et al., 2023; Oulasvirta et al., 2022; Castelo et al., 2023).

##### Ease of Development for Developers

Considering the variety of activities users may engage in and their unique
assistance requirements, the system should enable developers to create
different assistive features easily. This requires that developers can
integrate new devices ($\textbf{C3}_{a}$) easily (e.g., sensors to understand
new contexts or actuators to provide optimal feedback), deploy new assistance
and models ($\textbf{C3}_{b}$) (e.g., to predict optimal feedback), and access
and control current data ($\textbf{C3}_{c}$) (e.g., from existing devices or
models).

###  3.3. Conceptual Architecture

(a) Conceptual entities related to TOM.

(b) High-level conceptual architecture of TOM.

Figure 1. Conceptual entities and high-level modules associated with TOM.
Arrow directions represent the communication (e.g., data/interaction) flow.

The figure consists of two subfigures: (a) and (b). Subfigure (a) presents the
conceptual entities related to TOM, including the Context and the User.
Subfigure (b) illustrates the high-level conceptual modules of TOM,
encompassing sensing and understanding the user, device, context, human
interactions, activity-oriented services for reasoning, planning, and
coordination to provide optimal feedback, as well as data recording for
analysis.

To support the above requirements, we consider three main entities: user
(i.e., the individual receiving assistance), context (i.e., the user’s
perceptual space and associated tasks), and the system, TOM, as shown in
Figure 1(a), following the high-level context sources (Grubert et al., 2017).

Separating the user from the context enables us to develop user interaction
models (Zhao et al., 2023). These models sense and understand the user
($\textbf{C1}_{b}$, e.g., cognitive states, affective states, physical states
(Grubert et al., 2017)) to provide personalized feedback. Thus, TOM maintains
user profiles to cater to individual preferences and capabilities.

Given that daily activities, such as cooking, typically involve both digital
(e.g., viewing a recipe) and physical tasks (e.g., selecting the proper
portion), TOM offers system-level support to connect the digital world with
the physical world by understanding the context ($\textbf{C1}_{c}$, e.g.,
physical environment) and utilizing pervasive augmented reality (Grubert et
al., 2017). This involves a multi-modal and multifaceted understanding of the
environment (e.g., visual and auditory scene understanding, understanding the
ongoing activities and associated objects, understanding the relationship
between the user and environment, understanding the social context) as well as
understanding the devices that facilitate interactions (e.g., device resource
availability).

In terms of input, TOM supports the user’s explicit multi-modal inputs
($\textbf{C1}_{a}$, such as voice and gesture) as well as implicit inputs
($\textbf{C1}_{a}$, like gaze and physiological data), in addition to
processing multi-modal context information.

After understanding the context (e.g., ongoing activity) and user (e.g.,
intention), TOM activates a context-aware service, employing domain knowledge
to generate real-time proactive suggestions through reasoning and planning.
These suggestions are conveyed to users as multi-sensory feedback
($\textbf{C1}_{a}$), tailored to their cognitive capacity, including visual,
auditory, and/or haptic modalities. The feedback is dynamically updated based
on the user’s actions; for instance, if the user does not follow a given
suggestion, TOM formulates the next appropriate suggestion, considering the
user’s current status and context, facilitating a closed-loop control system
(See Appendix A-Figure 6).

When multiple TOM users are involved in a collaborative activity (e.g., group
discussion on an artifact), each TOM system enables multi-agent coordination
to complete the collaborative activity optimally.

###  3.4. System Architecture

We develop the following system/physical architecture based on the
requirements and envisioned use cases.

####  3.4.1. Devices and Technologies

Following a user-centric approach (Zhao et al., 2023), TOM uses wearable
devices that align with human input-output channels (e.g., eyes, hands), such
as Optical See-Through Head-Mounted Displays (OHMD, Augmented Reality Smart
Glasses, e.g., HoloLens2, Nreal Light) and ring-mouses, to support multi-modal
interactions ($\textbf{C1}_{a}$). It also includes everyday wearable devices
like smartwatches (e.g., Samsung Galaxy/WearOS, Fitbit) to understand users,
smartphones (e.g., Android) to provide familiar interactions, and web browsers
for visualizations.

To understand users, devices, and the environment, and to facilitate
reasoning, planning, and coordination, TOM employs AI (Rzepka and Berger,
2018) technologies. These include scene understanding, speech recognition,
object recognition/tracking, natural language processing, and large
language/multimodal models (LLM, LMM). For user feedback, TOM utilizes AR/MR
technologies (e.g., OHMD). TOM uses databases (e.g., PostgreSQL, Milvus) for
data recording ($\textbf{C2}_{a}$), training ($\textbf{C3}_{b}$), and
visualization ($\textbf{C2}_{b}$). Communication between devices and with
external APIs (e.g., ChatGPT) is handled using data communication protocols
(e.g., WebSocket, WebRTC, REST API) and wireless mediums (e.g., WiFi, BLE).

(a) System (physical) architecture.

(b) Server architecture.

(c) Client architecture.

Figure 2. System architecture of TOM. Arrow directions represent the data
flow. (a) Client-Server architecture with multiple clients (b) Server
architecture with multiple layers (c) Client architecture with multiple
layers. Dashed boxes indicate the communication between the Server and
Clients.

The figure comprises three subfigures: (a), (b), and (c). Subfigure (a)
displays the overall Client-Server architecture, showing multiple clients such
as Smart Glasses, Phones, Watches, and Web clients, communicating with the
Server. It also illustrates the technology stack, including Python, Unity,
WearOS, and HTML/JS. Subfigure (b) details the Server Architecture,
highlighting Widgets that receive Sensor data or Inputs, (Pre-) Processors,
Services that process data, and (Post-) Processors that send data to Actuators
or Outputs. Subfigure (c) shows the Client Architecture, demonstrating how
Sensors connect to the Server via Widgets or Processors, and how Server
outputs are transmitted to the Actuators of the Client.

###  3.5. Client-Server Architecture

Given the limited computational resources of wearable devices (Castelo et al.,
2023), TOM is implemented as a client-server architecture, as illustrated in
Figure 2(c). The server hosts services for processing and orchestrating data,
supports ML/AI inferences, and provides real-time feedback to clients.
Clients, such as OHMDs and smartwatches, send sensor data to the server and
display feedback to the user. This separation also allows TOM to be device-
agnostic, supporting various OHMDs through the same server
($\textbf{C3}_{a}$).

##### Server Architecture

Designed for flexibility and simplicity, the server acts as a one-stop
platform for deploying various services optimized for different activities and
switching between them as needed. Adapting the architecture of the Context
Toolkit (Dey et al., 2001), the TOM server (Figure 2(b)) is implemented with
independent components under three distinct layers: Widgets (i.e., components
that listen for sensors and receive input data), Processors (i.e., stateless
components that process and transform input data from Widgets or output data
from Services), and Services ($\textbf{C3}_{b}$, i.e., stateful components
that process data from Widgets and/or Processors to produce desired
outcomes)111TOM does not incorporate Interpreters and Aggregators, unlike the
Context Toolkit; their functions are managed by either Processors or Services
in TOM, minimizing stateful components to enhance testability.. These layers
are interconnected with Sensors/Inputs and Actuators/Outputs linked to
Clients. This setup supports the separation of concerns, distributed
communication, context storage, and resource discovery. A specialized service,
the Context Service, determines the most suitable Service based on current
input data (e.g., through explicit user interactions or automatically
determined by context data), switching services to support ongoing user
activities.

##### Client Architecture

As shown in Figure 2(c), Clients mirror the Server’s architecture. However,
instead of Services, they interact with the Server to stream sensor data and
receive real-time feedback for actuators. Time-critical processing can also be
implemented in clients (i.e., on-device) to overcome potential latency issues
between the client and server.

##### Data Flow and Communication

Data or messages, tagged with source and time, are transferred between
different layers (e.g., Input -¿ Widget -¿ Processor -¿ Service -¿ Output,
Figure 2(b), Appendix A-Figure6) via message channels controlled by
configuration files ($\textbf{C3}_{c}$, e.g., Figure3(b)). This arrangement
allows for the reuse of various components across multiple Services and
supports distributed communication, thus reducing development efforts.
Moreover, each component can store the data it handles in a local database for
post-analysis ($\textbf{C2}_{c}$, e.g., visualization, aggregation) or ML
model training ($\textbf{C3}_{b}$). The WebSocket protocol is employed for
real-time bidirectional communication between Clients and the Server. REST
APIs are used for communication with external APIs, while WebRTC and the Real-
Time Streaming Protocol (RTSP) facilitate the streaming of real-time video
data.

###  3.6. Implementation

The Server (Figure 2(a)-Y) is implemented in Python (3.9), chosen for its
extensive user base and support of numerous ML/AI libraries with
multiprocessing capabilities. The built-in library, SQLAlchemy, also supports
Data Storage.

The OHMD or mobile phone clients (Figure 2(a)-X1) are developed using Unity3D
(2021.3) and MRTK222Mixed Reality Toolkit:
https://github.com/Microsoft/MixedRealityToolkit-Unity (2.8) to provide AR/MR
content. This setup accommodates various devices (e.g., HoloLens2 with UWP,
Nreal with Android) and enables mixed reality capabilities with OpenXR
support. Other clients, such as the Smartwatch (Figure 2(a)-X2), designed to
sense the user, are implemented using WearOS due to its widespread use.
Additionally, a web client (Figure 2(a)-X3) is used for visualization,
utilizing HTML/VueJS and NodeJS.

For more details, please refer to https://github.com/TOM-Platform and Appendix
A, which details the supported capabilities, devices, technologies, and data.

(a)

(b)

Figure 3. Running assistance implemented in TOM. (a) System components that
enable the running assistance, $\textbf{C3}_{b}$. Dashed-line boxes indicate
implemented Client components, solid lines represent implemented Server
components, and dotted lines denote Server components under development. (b)
The configuration file that controls the data flow, $\textbf{C3}_{c}$. Data is
received in one or more components in the Input Layer (e.g., ‘camera’
component) and is sent to the next component as specified in the next key
(e.g., ‘yolov8’ component in the Processing Layer). This process occurs
similarly for all components regardless of the layer, with the entry point
dictating the method in each component that receives the data from the
previous component. The exit point then dictates the method for each
component, which is called when they should be stopped (e.g., when the context
switch indicates the component is no longer required).

The figure consists of two subfigures: (a) and (b). Subfigure (a) depicts the
high-level components of the running assistance, including widgets like
WebSocket and Camera; processors such as YoloDetector; profiles like Running
Profile; domain knowledge like Route Info; services including Running Service
and Layout Management Service; and output components like WebSocket. Subfigure
(b) presents the configuration file for the Running Service, detailing the
data flow, which includes inputs, processors, services, and outputs
components, each with a name, entry point, exit point, and next location for
data passage.

##  4\. Demonstration During Daily Activities

We have implemented several proof-of-concept services to support daily
activities, realizing our vision of an intelligent, wearable, proactive
assistant.

###  4.1. During Exercise: Running Assistance

##### Scenario

Jack uses Jerry (implemented using TOM) to assist with his running exercises.
He wears an OHMD and a smartwatch (connected to a Server operating on a
laptop333In the future, we plan to run the Server in the cloud). He initiates
his running (speed or distance) training using voice interactions. Jerry
provides route options, and he selects one using either voice commands or mid-
air gestures. During his run, Jerry provides personalized running coach
instructions (e.g., speeding up or slowing down based on his current speed,
training plan, and user profile) and proactive suggestions (e.g., encouraging
feedback based on duration, alerting about potential dangers like traffic
lights based on environment sensing, giving direction cues based on location,
indicating waterpoints based on location) using either visual or auditory
modality when required (i.e., by default, Jerry will provide only essential
details, such as the time, to reduce the display clutter and information
overload). At the end of his run, he receives a summary of the exercise.

##### System

As shown in Figure 3(b), the current running assistance is implemented as a
Running Service. This service processes sensor data from the smartwatch, user
interactions, and the egocentric camera view from the OHMD, and route
information from external API to determine the next running coach instruction
and provide feedback. Then, the service sends the feedback to the OHMD using a
pre-configured display layout (Figure 4) designed based on user testing. If
the user does not specify certain details required for running (e.g., expected
speed), the system uses the user’s profile to determine them.

##### Limitations

During preliminary user testing, we identified several device and technical
limitations. These include impaired visibility of the OHMD’s visual feedback
in outdoor environments, misrecognition of voice commands due to background
noise and user fatigue during running, the OHMD’s weight affecting the
exercise experience, and instability of visual feedback from frequent head
movements (content jumps) (Itoh et al., 2021; Janaka et al., 2023).
Additionally, participants requested adaptive user interfaces tailored to
their preferences and environment (Orlosky et al., 2013) for enhanced
visibility.

Figure 4. The running assistance UI supports voice and mid-air gesture input
interactions. (a) The user starts the running assistance and is prompted to
select a route. (b) Jerry provides personalized training guidance, proactive
feedback on potential dangers or encouragement, and details about water points
while running. (c) In the end, Jerry presents the user with a run summary.

Three-panel illustration of the running assistance UI. Panel A shows the route
selection interface, Panel B displays the in-progress running interface with
guidance and feedback, and Panel C showcases the post-run summary screen.

###  4.2. During Dining and Shopping: Translation and Querying Assistance

##### Scenario

Jack visits a new restaurant and discovers that the menu is only in Mandarin,
which he cannot understand (See Figure 5 for details). He verbally requests
Jerry’s assistance, and it displays the translated menu in English,
superimposed on the original menu. When he shows prolonged gaze duration on a
particular menu item, Jerry automatically displays an image with a brief
description. Upon inquiring whether it is vegetarian, Jerry informs him that
it contains some non-vegetarian ingredients. Consequently, Jack orders the
recommended dish.

Figure 5. The translation assistance UI supports voice, mid-air gesture, and
gaze input interactions. (a) The user is prompted for the action they wish to
take and chooses the ‘Translate Text’ option. (b) Jerry translates the
Mandarin text on the screen into English and overlays the translated
information onto the location of the Mandarin text. (c) The user shows
interest in ‘Herbal Jelly’ and seeks more general information. (d) The user
verbally inquires, “Ok Jerry, is this vegetarian?”

Four-panel illustration of the translation assistance UI. Panel A shows the
action selection interface. Panel B displays the screen with Mandarin text
being translated to English and overlaid. Panel C focuses on ’Herbal Jelly’
with additional information being displayed. Panel D depicts the user asking
Jerry about the vegetarian status of the item.

After dining, Jack goes to a supermarket. He notices some medical supplements
that are different from his usual purchases. Jack asks Jerry about them while
looking at (or pointing to) them, and Jerry provides detailed information
about them and their usage, assisting him in making an informed decision
before purchasing.

##### System

The current translation assistance system uses an OHMD camera to scan and
recognize texts, translates them, and employs a Large Language Model (LLM) to
adapt them to the local context. Subsequently, TOM displays the translations
aligned with the original menu. Based on the user’s explicit (e.g., voice) or
implicit (e.g., gaze) inputs, TOM presents the menu images and detailed
descriptions and verbally answers questions. Similarly, the querying
assistance system scans and recognizes objects and texts using the OHMD
camera, based on the user’s explicit interactions (e.g., gaze plus voice,
gesture plus voice). It then interacts with an LLM to provide responses to the
user’s verbal queries.

##### Limitations

Similar to the running assistance, additional limitations were noted in the
translation and querying assistance. These include the inability to recognize
small text in dim lighting, occasional retrieval of incorrect images for
specific dishes, and delays (2-8 seconds) in displaying feedback, which is
attributable to the response times of external APIs (e.g., ChatGPT, Bing Image
Search).

##  5\. Limitations and Future Improvements

In addition to the identified technical limitations from specific
demonstrations, interaction design challenges surfaced in daily activities.
Situational impairments, especially in dynamic environments, restrict certain
user interactions (e.g., diminished voice command accuracy in outdoor wind),
underscoring the need for methods to support seamless input modality
transitions (Janaka et al., 2023). Inaccuracies in AI-generated suggestions
also contribute to user mistrust, necessitating more transparent AI
explanations (Carroll, 2022; Xu et al., 2023; Eiband et al., 2018).

Moreover, TOM’s current implementation exhibits limitations. A notable area is
the automatic switching of services based on user inputs to optimize service
execution for ongoing activities, when multiple services match expected
assistance, requiring further research in this area. Despite TOM’s support for
the Large Language Model (LLM) in facilitating human-like conversations and
tasks (Bubeck et al., 2023; Yang et al., 2023; Brown et al., 2020; Radford et
al., 2019), integrating Large Action Models (LAM) (rabbit research team, 2023)
could enhance interaction efficiency with external applications and improve
user action understanding. Effective live monitoring is available, but TOM
needs better visualizations for comprehensive retrospective analysis of long-
term user behaviors. Aggregated visualization techniques, similar to ARGUS
(Castelo et al., 2023), and retrospective analysis such as PilotAR (Janaka et
al., 2024) could aid in this. Additionally, advancing the system’s grasp of
users’ cognitive states and their activity correlations requires sophisticated
modeling and simulation (Murray-Smith et al., 2022; Oulasvirta et al., 2022).
Similarly, facilitating effective multi-agent collaboration while preserving
user autonomy in multi-user TOM scenarios remains a significant research
challenge.

Finally, developing such systems implicates privacy, security, safety, and
ethical challenges. Despite Institutional Review Board (IRB) approval for user
studies, real-world deployment raises critical privacy, safety, and social
acceptability concerns, considering both users and bystanders (Slater et al.,
2020; Alallah et al., 2018; Denning et al., 2014). Issues include monitoring
and recording users’ physical and cognitive states, capturing bystanders’
behaviors without consent, securely handling sensitive data, and anonymizing
data for aggregate analysis. Although on-device/edge computing provides
partial solutions (Zhang et al., 2018), the limitations of current devices
necessitate further advancements.

##  6\. Conclusion

We have presented the anticipated capabilities of developing an intelligent
wearable assistive system and introduced TOM, an architecture and open-source
implementation (https://github.com/TOM-Platform) that enables researchers and
developers to create and analyze assistive applications for supporting daily
activities. We welcome contributions from the community to expand its
supported devices and usage scenarios. We envision that TOM will serve as a
software platform for researchers and developers to develop innovative,
intelligent assistance in various tasks, facilitating human-computer, human-
AI, and human-robot interactions. Our future plans include extending TOM’s
capabilities to enable remote robot interactions, where humans can share
information (e.g., intentions) with a remote robot to execute tasks.

###### Acknowledgements.

We would like to express our gratitude to the volunteers who participated in
our studies and to the interns of TOM project who developed certain system
components, including Teo Yun Yew Jarrett. This research is supported by the
National Research Foundation, Singapore, under its AI Singapore Programme
(AISG Award No: AISG2-RP-2020-016). The CityU Start-up Grant 9610677 also
provides partial support. Any opinions, findings, conclusions, or
recommendations expressed in this material are those of the author(s) and do
not reflect the views of the National Research Foundation, Singapore.

## References

  * (1)
  * noa (2024) 2024\.  Ai Pin Overview.  https://hu.ma.ne/aipin
  * Abowd et al. (1999) Gregory D. Abowd, Anind K. Dey, Peter J. Brown, Nigel Davies, Mark Smith, and Pete Steggles. 1999.  Towards a Better Understanding of Context and Context-Awareness. In _Handheld and Ubiquitous Computing_ _(Lecture Notes in Computer Science)_ , Hans-W. Gellersen (Ed.). Springer, Berlin, Heidelberg, 304–307.  https://doi.org/10.1007/3-540-48157-5_29
  * Alallah et al. (2018) Fouad Alallah, Ali Neshati, Yumiko Sakamoto, Khalad Hasan, Edward Lank, Andrea Bunt, and Pourang Irani. 2018.  Performer vs. observer: whose comfort level should we consider when examining the social acceptability of input modalities for head-worn display?. In _Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology_ _(VRST ’18)_. Association for Computing Machinery, New York, NY, USA, 1–9.  https://doi.org/10.1145/3281505.3281541
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Anderson et al. (2018) Christoph Anderson, Isabel Hübener, Ann-Kathrin Seipp, Sandra Ohly, Klaus David, and Veljko Pejovic. 2018.  A Survey of Attention Management Systems in Ubiquitous Computing Environments.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 2, 2 (July 2018), 1–27.  https://doi.org/10.1145/3214261
  * Andrist et al. (2022) Sean Andrist, Dan Bohus, Ashley Feniello, and Nick Saw. 2022.  Developing Mixed Reality Applications with Platform for Situated Intelligence. In _2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_. 48–50.  https://doi.org/10.1109/VRW55335.2022.00018
  * Azuma (1997) Ronald T Azuma. 1997.  A Survey of Augmented Reality.  _Presence: Teleoperators and Virtual Environments_ 6, 4 (Aug. 1997), 355–385.  https://doi.org/10.1162/pres.1997.6.4.355
  * Barsom et al. (2016) E. Z. Barsom, M. Graafland, and M. P. Schijven. 2016.  Systematic review on the effectiveness of augmented reality applications in medical training.  _Surgical Endoscopy_ 30, 10 (Oct. 2016), 4174–4183.  https://doi.org/10.1007/s00464-016-4800-6
  * Billinghurst et al. (2015) Mark Billinghurst, Adrian Clark, and Gun Lee. 2015.  A Survey of Augmented Reality.  _Foundations and Trends® in Human–Computer Interaction_ 8, 2-3 (March 2015), 73–272.  https://doi.org/10.1561/1100000049
  * Bohus et al. (2021) Dan Bohus, Sean Andrist, Ashley Feniello, Nick Saw, Mihai Jalobeanu, Patrick Sweeney, Anne Loomis Thompson, and Eric Horvitz. 2021.  Platform for Situated Intelligence.  https://doi.org/10.48550/arXiv.2103.15975 arXiv:2103.15975 [cs].
  * Bohus et al. (2024) Dan Bohus, Sean Andrist, Nick Saw, Ann Paradiso, Ishani Chakraborty, and Mahdi Rad. 2024.  SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research.  https://arxiv.org/abs/2405.13035v1
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  https://doi.org/10.48550/arXiv.2005.14165
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  https://doi.org/10.48550/arXiv.2303.12712
  * Carroll (2022) John M. Carroll. 2022.  Why should humans trust AI?  _Interactions_ 29, 4 (June 2022), 73–77.  https://doi.org/10.1145/3538392
  * Castelo et al. (2023) Sonia Castelo, Joao Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, Jing Qian, Kyunghyun Cho, He He, Qi Sun, Huy Vo, Juan Bello, Michael Krone, and Claudio Silva. 2023.  ARGUS: Visualization of AI-Assisted Task Guidance in AR.  https://doi.org/10.48550/arXiv.2308.06246
  * Denning et al. (2014) Tamara Denning, Zakariya Dehlawi, and Tadayoshi Kohno. 2014.  In situ with bystanders of augmented reality glasses: perspectives on recording and privacy-mediating technologies. In _Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14_. ACM Press, Toronto, Ontario, Canada, 2377–2386.  https://doi.org/10.1145/2556288.2557352
  * Dey et al. (2001) Anind K. Dey, Gregory D. Abowd, and Daniel Salber. 2001.  A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications.  _Human–Computer Interaction_ 16, 2-4 (Dec. 2001), 97–166.  https://doi.org/10.1207/S15327051HCI16234_02
  * Eiband et al. (2018) Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike Haug, and Heinrich Hussmann. 2018.  Bringing Transparency Design into Practice. In _23rd International Conference on Intelligent User Interfaces_ _(IUI ’18)_. Association for Computing Machinery, New York, NY, USA, 211–223.  https://doi.org/10.1145/3172944.3172961
  * Engel et al. (2023) Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan Frost, Edward Miller, Elias Mueggler, Evgeniy Oleinik, Fan Zhang, Guruprasad Somasundaram, Gustavo Solaira, Harry Lanaras, Henry Howard-Jenkins, Huixuan Tang, Hyo Jin Kim, Jaime Rivera, Ji Luo, Jing Dong, Julian Straub, Kevin Bailey, Kevin Eckenhoff, Lingni Ma, Luis Pesqueira, Mark Schwesinger, Maurizio Monge, Nan Yang, Nick Charron, Nikhil Raina, Omkar Parkhi, Peter Borschowa, Pierre Moulon, Prince Gupta, Raul Mur-Artal, Robbie Pennington, Sachin Kulkarni, Sagar Miglani, Santosh Gondi, Saransh Solanki, Sean Diener, Shangyi Cheng, Simon Green, Steve Saarinen, Suvam Patra, Tassos Mourikis, Thomas Whelan, Tripti Singh, Vasileios Balntas, Vijay Baiyya, Wilson Dreewes, Xiaqing Pan, Yang Lou, Yipu Zhao, Yusuf Mansour, Yuyang Zou, Zhaoyang Lv, Zijian Wang, Mingfei Yan, Carl Ren, Renzo De Nardi, and Richard Newcombe. 2023.  Project Aria: A New Tool for Egocentric Multi-Modal AI Research.  https://doi.org/10.48550/arXiv.2308.13561
  * Fleck et al. (2023) Philipp Fleck, Aimée Sousa Calepso, Sebastian Hubenschmid, Michael Sedlmair, and Dieter Schmalstieg. 2023.  RagRug: A Toolkit for Situated Analytics.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 7 (July 2023), 3281–3297.  https://doi.org/10.1109/TVCG.2022.3157058 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
  * Grubert et al. (2017) Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. 2017.  Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality.  _IEEE Transactions on Visualization and Computer Graphics_ 23, 6 (June 2017), 1706–1724.  https://doi.org/10.1109/TVCG.2016.2543720
  * Hornbæk and Oulasvirta (2017) Kasper Hornbæk and Antti Oulasvirta. 2017.  What Is Interaction?. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, Denver Colorado USA, 5040–5052.  https://doi.org/10.1145/3025453.3025765
  * Itoh et al. (2021) Yuta Itoh, Tobias Langlotz, Jonathan Sutton, and Alexander Plopski. 2021.  Towards Indistinguishable Augmented Reality: A Survey on Optical See-through Head-mounted Displays.  _Comput. Surveys_ 54, 6 (July 2021), 120:1–120:36.  https://doi.org/10.1145/3453157
  * Janaka et al. (2024) Nuwan Janaka, Runze Cai, Ashwin Ram, Lin Zhu, Shengdong Zhao, and Yong Kai Qi. 2024.  PilotAR: Streamlining Pilot Studies with OHMDs from Concept to Insight.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ (Sept. 2024).  https://doi.org/10.1145/3678576
  * Janaka et al. (2023) Nuwan Janaka, Jie Gao, Lin Zhu, Shengdong Zhao, Lan Lyu, Peisen Xu, Maximilian Nabokow, Silang Wang, and Yanch Ong. 2023.  GlassMessaging: Towards Ubiquitous Messaging Using OHMDs.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 3 (Sept. 2023), 100:1–100:32.  https://doi.org/10.1145/3610931
  * McCrickard and Chewar (2003) D. Scott McCrickard and C. M. Chewar. 2003.  Attuning notification design to user goals and attention costs.  _Commun. ACM_ 46, 3 (March 2003), 67.  https://doi.org/10.1145/636772.636800
  * Murray-Smith et al. (2022) Roderick Murray-Smith, Antti Oulasvirta, Andrew Howes, Jörg Müller, Aleksi Ikkala, Miroslav Bachinski, Arthur Fleig, Florian Fischer, and Markus Klar. 2022.  What simulation can do for HCI research.  _Interactions_ 29, 6 (Nov. 2022), 48–53.  https://doi.org/10.1145/3564038
  * Orlosky et al. (2013) Jason Orlosky, Kiyoshi Kiyokawa, and Haruo Takemura. 2013.  Dynamic text management for see-through wearable and heads-up display systems. In _Proceedings of the 2013 international conference on Intelligent user interfaces - IUI ’13_. ACM Press, Santa Monica, California, USA, 363.  https://doi.org/10.1145/2449396.2449443
  * Oulasvirta et al. (2022) Antti Oulasvirta, Jussi P. P. Jokinen, and Andrew Howes. 2022.  Computational Rationality as a Theory of Interaction. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3491102.3517739
  * rabbit research team (2023) rabbit research team. 2023.  Learning human actions on computer applications.  https://rabbit.tech/research
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, D. Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  https://api.semanticscholar.org/CorpusID:160025533
  * Rhodes and Maes (2000) B. J. Rhodes and P. Maes. 2000.  Just-in-time information retrieval agents.  _IBM Systems Journal_ 39, 3.4 (2000), 685–704.  https://doi.org/10.1147/sj.393.0685
  * Rzepka and Berger (2018) Christine Rzepka and Benedikt Berger. 2018.  User Interaction with AI-enabled Systems: A Systematic Review of IS Research.  _ICIS 2018 Proceedings_ (Dec. 2018).  https://aisel.aisnet.org/icis2018/general/Presentations/7
  * Schmidt (2014) Albrecht Schmidt. 2014.  _Context-Aware Computing_.  https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/context-aware-computing-context-awareness-context-aware-user-interfaces-and-implicit-interaction
  * Sicat et al. (2019) Ronell Sicat, Jiabao Li, Junyoung Choi, Maxime Cordeil, Won-Ki Jeong, Benjamin Bach, and Hanspeter Pfister. 2019.  DXR: A Toolkit for Building Immersive Data Visualizations.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 1 (Jan. 2019), 715–725.  https://doi.org/10.1109/TVCG.2018.2865152 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
  * Slater et al. (2020) Mel Slater, Cristina Gonzalez-Liencres, Patrick Haggard, Charlotte Vinkers, Rebecca Gregory-Clarke, Steve Jelley, Zillah Watson, Graham Breen, Raz Schwarz, William Steptoe, Dalila Szostak, Shivashankar Halan, Deborah Fox, and Jeremy Silver. 2020.  The Ethics of Realism in Virtual and Augmented Reality.  _Frontiers in Virtual Reality_ 1 (2020).  https://doi.org/10.3389/frvir.2020.00001
  * Speicher et al. (2019) Maximilian Speicher, Brian D. Hall, and Michael Nebeling. 2019.  What is Mixed Reality?. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3290605.3300767
  * Victor (2018) Bret Victor. 2018.  Dynamicland.  https://dynamicland.org/
  * Wang et al. (2016) X. Wang, S. K. Ong, and A. Y. C. Nee. 2016.  A comprehensive survey of augmented reality assembly research.  _Advances in Manufacturing_ 4, 1 (March 2016), 1–22.  https://doi.org/10.1007/s40436-015-0131-4
  * Xu et al. (2023) Xuhai Xu, Anna Yu, Tanya R. Jonker, Kashyap Todi, Feiyu Lu, Xun Qian, João Marcelo Evangelista Belo, Tianyi Wang, Michelle Li, Aran Mun, Te-Yen Wu, Junxiao Shen, Ting Zhang, Narine Kokhlikyan, Fulton Wang, Paul Sorenson, Sophie Kim, and Hrvoje Benko. 2023.  XAIR: A Framework of Explainable AI in Augmented Reality. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, 1–30.  https://doi.org/10.1145/3544548.3581500
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376301
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023.  The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision).  https://doi.org/10.48550/arXiv.2309.17421
  * Zhang et al. (2018) Jiale Zhang, Bing Chen, Yanchao Zhao, Xiang Cheng, and Feng Hu. 2018.  Data Security and Privacy-Preserving in Edge Computing Paradigm: Survey and Open Issues.  _IEEE Access_ 6 (2018), 18209–18237.  https://doi.org/10.1109/ACCESS.2018.2820162
  * Zhao et al. (2023) Shengdong Zhao, Felicia Tan, and Katherine Fennedy. 2023.  Heads-Up Computing Moving Beyond the Device-Centered Paradigm.  _Commun. ACM_ 66, 9 (Aug. 2023), 56–63.  https://doi.org/10.1145/3571722

##  Appendix A Detailed System Architecture

Figure 6 depicts the high-level components of the system architecture. The
implemented Server currently supports three distinct types of Services:
running assistance, providing speed/distance training with directional
support; learning assistance, enabling inquiries about objects in the frame
selected through gestures and/or gaze; and translation assistance, which
facilitates text translation within the frame.

Figure 6. A snapshot of the system architecture and the data flow between
high-level components. Solid-lined boxes indicate implemented components, and
dotted lines represent components in development.

The figure illustrates the detailed system architecture and the data flow
among high-level components. It starts with Sensors/Widgets capturing sensor
data, which is then processed by (Pre-)Processors. This data flows into
Service Inputs, leading to various Services that generate Service Outputs.
These outputs are further processed by (Post-)Processors, eventually reaching
Actuators. The diagram visually represents a closed-loop control system,
achieved by continuously sensing the user and context, and adjusting system
feedback based on user actions. The flow is depicted through a series of
interconnected boxes and arrows, with solid lines for existing components and
dotted lines for those under development.

Table 1 provides an overview of the technologies that support the system’s
current capabilities and Table 2 illustrates the various client data
supporting the system’s capabilities. These data sources include Ring Mouse
Controllers connected to OHMDs, Gesture and Gaze Detection through OHMDs, and
User Input via the Touch Screen on Android Phones.

Table 1. Technologies used within the current TOM. For the latest supported
technologies, refer to the TOM-Platform

Capability | Type | Server | Client
---|---|---|---
Python Server (Laptop) | HoloLens 2 | XReal Light | Web Client | Android Phone
Context Understanding | Visual | GoogleCloudVision, Yolov8 | - | - | - | MLKit
Audio | MediaPipe | - | - | - | -
Spatial | GoogleCloudVision | SLAM (inbuilt) | SLAM (inbuilt) | - | -
User Understanding | Physiological | Fitbit API | - | - | - | WearOS Watch
Interactions | Visual | Bing Image Search API | - | - | Graphs, Charts | -
Audio |  | Speech-to-Text (Whisper),
---
Text-to-Speech
| Speech-to-Text (Azura),
---
Text-to-Speech
- | - |  | Speech-to-Text (Android),
---
Text-to-Speech
Other | - |  | Hand Tracking (inbuilt),
---
Gaze Tracking (inbuilt)
Hand Tracking (inbuilt) | Visualizations | Touch
Communication | APIs | REST | - | - | - | -
Data Stream | Websocket | Websocket | Websocket, TCPSocket | Websocket, TCPSocket | TCPSocket
Video Stream | WebRTC, RTSP | - | WebRTC | WebRTC | WebRTC
Assistance |  | Queries,
---
Memory
| ChatGPT, Claude,
---
Local LLM (TinyLlama)
Encoding (Clip, ImageBind)
Vector Database (Milvus)
- | - | - | -

This table outlines the technologies utilized in the system. It comprises five
rows, each labeled with a different aspect of the system: Context
Understanding, User Understanding, Interactions, Communication, and
Assistance. The table also includes column headers for Server (Python) and
five Clients (HoloLens2, XReal Light, Web Client, Android Phone), indicating
the technologies implemented in each. The arrangement of the table provides a
clear depiction of how various technologies are applied across different
components of the system.

Table 2. Data from currently supported Clients. For the latest supported
devices/capabilities, refer to the https://github.com/TOM-PlatformTOM-Platform

Data | Type | Client
---|---|---
HoloLens2 | XReal Light | WearOS Watch | Web Client | Android Phone
Context Understanding | Visual | Video | Video | - | - | Video
Auditory | Audio | Audio | - | - | Audio
Spatial | WorldMesh | WorldMesh | Location (GPS) | - | Location (GPS)
User Understanding | Physiological | - | - | Heart Rate | - | -
Physical | - | - | Speed, Calories | - | -
Interactions (Output) | Visual |  | Text, Image,
---
Video, 3D Object
| Text, Image,
---
Video, 3D Object
| Text, Image,
---
2D Object
| Text, Images,
---
Video
| Text, Image,
---
Video, 3D Object
Auditory | Audio, Text | Audio, Text | Audio | Audio | Audio, Text
Interactions (Input) | Voice | Audio (Speech) | Audio (Speech) | - | - | Audio (Speech)
Gesture | Hand, Finger | Hand, Finger | - | - | -
Gaze | 3D Gaze, Gaze Collision | - | - | - | -
Controller | 2D Press | 3D Press | - | - | 2D Touch

This table details the types of data related to Context Understanding, User
Understanding, Interaction Outputs, and Interaction Inputs used in the system.
It is structured into categories across rows and specifies the corresponding
client technologies across columns, including HoloLens2, XReal Light, WearOS
Watch, Web Client, and Android Phone. The table enumerates various data types,
such as Visual, Auditory, and Spatial for context understanding, and different
modes of interaction like Voice, Gesture, and Gaze.