































Telepresence: Harnessing the
Human-Computer-Machine Interface

Editor’s note: This is the second in a series of five articles on themes for future logistics innovation identified by the Army Logistics Innovation Agency’s Futures Group. Each of the articles describes plausible future advances in technology and business processes that may significantly improve logistics effectiveness and efficiency. Together, they bring an advanced look at some amazing possibilities for Army logistics.

Telepresence can be defined as a human-computer-machine condition in which a user (a Soldier) receives sufficient information about a remote, real-world site (a battlefield) through a machine (a robot) so that the user feels physically present at the remote, real-world site. With telepresence, a user’s physical presence (body) exists at one location (the home site) while his ability to act and interact with a remote, real-world environment occurs at one or more locations (the remote site or sites). Telepresence is achieved by using advanced sensors, communications, remote action, and feedback stimuli to allow personnel to project their presence around the world through remote sites, giving a user the ability to see, hear, touch, taste, and smell those sites. Soldiers using telepresence capabilities will be able to act and receive stimuli just as if they were present at the remote, real-world location.

Logistics Implications of Telepresence

Telepresence will enable sustained combat power in the theater of operations while minimizing human resource requirements. It is truly “the next best thing to being there.” The vision for telepresence is: Fewer humans vulnerable to asymmetric attack, reduced requirements for consumables, and greater availability of continental United States (CONUS)-based expertise. Possible second-tier effects of telepresence include—

  • Reduced sustainment requirements (food and water; clothing and individual equipment; personal items; religious, legal, and financial support; and medical services, including medical materiel) as a result of fewer personnel in the battlespace.
  • Reduced requirements for airlift, sealift, billeting, and force protection.
  • Leap-ahead capabilities in the areas of maintenance, depot operations, force reception, and theater distribution.
  • Improved nuclear-biological-chemical detection and decontamination and explosive ordnance disposal.
  • Enhanced telemedicine and emergency medical services, such as telesurgery for wounded Soldiers.
  • Better mission rehearsal, improved route reconnaissance for convoys by remotely controlled systems, and more comprehensive collaborative planning.
  • Improved training and education because of advances in virtual reality environments and perception capabilities.

Robots have been used for years by commercial industry to accomplish repetitive manufacturing tasks. With the use of advanced effectors (the equivalents of human limbs and hands), the complete telepresence-capable robotic system truly will become an enabler of logistics operations. Telepresence will allow certain logistics functions formerly performed by a human to be completed in a much more adaptive manner.

Some near-term telepresence efforts include allowing medical providers to see, hear, and touch patients in real time with the necessary visual, auditory, and tactile perception in order to conduct or assist in remote surgery (telemedicine). In many respects, telemedicine leads the way in the advancement of telepresence capabilities. While the technology still falls short of supporting telesurgery for Soldiers fighting in Iraq and Afghanistan, the Army Medical Command's Telemedicine and Advanced Technology Research Center successfully conducted a robotic telesurgery over the Internet at a conference in April 2005.

Effectively using technologies and advances from the telemedicine community could extend the use of telepresence to mechanics so that they could use robotic platforms to conduct high-risk battlefield recovery and maintenance. Telepresence also could be used to operate materials-handling equipment and conduct port discharge and depot repair operations. In those cases, operators would have greater control over processes without incurring additional safety or manpower burdens within the battlespace.

Telemaintenance, as envisioned for the deep future, may allow the Army’s most experienced maintenance technician to participate from a CONUS-based depot in the repair of equipment on the battlefield. Conceivably, he would work as if truly present with a team of Soldiers on the ground in the theater of operations. Likewise, those who designed the system—the scientist or engineer team from an Army research, development, and engineering center or contractor support team—could participate in the maintenance activity.

Achieving Full Telepresence Immersion

A telepresence system is required to accomplish telepresence. This system is composed of three essential subsystems (and their related technologies): the home site, the communications link, and the remote site. Telepresence technologies associated with these three subsystems are similar to technologies found in virtual reality except that, in telepresence, a user must have feedback stimuli from, and the ability to exercise control over, the remote site. The ultimate goal of a telepresence system is to produce a transparent link from human to machine that passes information so naturally between user and environment that the user achieves a complete sense of immersion in the remote-site environment. To accomplish this, sensory impressions obtained from the remote site and delivered to the home site must engage the human senses fully with sufficient breadth, volume, and quality. As an example, consider that you are at a home site viewing a video of a remote-site environment on a theater-sized screen. The video provides only a minor feature of the much larger real-world environment present at the remote site. If a majority of your total vision is subjected to the video image on a large, curved screen and the video image depicts natural human motions, then the visual element becomes perceptually real world. Engaging the other senses similarly and in a synchronous manner will enable you to progress toward the feeling of being fully immersed in the remote, real-world environment. The ideal situation occurs when high-quality, high-resolution, and consistent information is presented to all of your senses.

Telepresence requires a complete human-computer-machine interface that incorporates audio, visual, haptic (touch), olfactic (smell), and gustatic (taste) technologies with home site elements perceptually identical to remote, real-world elements. To date, the senses of sight and hearing have been the focus of intense research and have formed the core of virtual reality systems because these senses are the most important senses by which we receive information related to our surroundings. The contributions of smell, touch, and taste are not as great, and current technologies for reproducing smell and taste are difficult to implement. One might argue that, compared with the other senses, taste (and perhaps smell) play marginal roles in creating a full immersion experience. Nevertheless, technology progression eventually will enable all human senses to be engaged in the telepresence experience.

Aside from the human-computer-machine sensory interface, communication between the home and remote sites must be “real time” so the user feels that he is indeed in the remote, real-world environment. Home or remote site latency detracts from the realism that telepresence systems seek to achieve. Any communications link may be used by a telepresence system. The specific type of link depends on distance, bandwidth requirements, latency tolerance, availability of services between sites, and so on. To achieve high fidelity immersion in military applications, direct, dedicated umbilical links between home and remote sites are desirable. Latency in communication between the various sensory elements also will erode the feeling of being fully immersed in the remote real-world environment. Current and future advances in processing power will reduce latency, which will provide users with more accurate and readily available telepresence systems for myriad applications.

Enabling Technologies for Telepresence Immersion

A number of enabling home-site interface technologies are required to realize telepresence fully. In some cases, these technologies are under study; in other cases, commercial off-the-shelf technologies are already available.

Visual technologies. Humans mainly interact with other humans through vision. Therefore, visual sensory elements must be mature. The typical binocular field of view (FOV) of a human is approximately 180 degrees horizontal (with approximately 120 degrees of binocular overlap) and 150 degrees vertical (limited by facial features such as cheeks, nose, and forehead). Current prototype telepresence systems use head-mounted displays (HMDs) to provide visual information to the user. HMDs are stereoscopic devices that can convert two-dimensional video images of remote-site environments into three-dimensional (3–D) visual images. Typical FOVs for HMDs are quite narrow—approximately 100 degrees horizontal and 60 degrees vertical—although full-immersion HMDs with FOVs of more than 180 degrees horizontal and 80 degrees vertical do exist. Full-immersion HMDs have been demonstrated in military environments, but they exhibited poor display resolution, limited FOV, and visual, position, and tracking latencies. Latency greatly affects the illusion of full immersion, and visual, position, and tracking latencies create visually induced motion sickness because the motions perceived in the real, remote-world environment are not reflected in the user’s body. Achieving near-zero visual latency is quite challenging because of the complexity of the real-world environment but may become possible with advances in quantum computation and quantum communication. (See “Quantum Computation and Communication” in the September–October issue of Army Logistician.) An alternative 3–D vision technology (holography) provides a way to create images without using lenses. Although this technology is very promising, moving holographic images are currently difficult to provide and therefore probably will not be developed until the 2010–2020 timeframe. Additional vision technologies, such as 3–D computer displays and virtual retinal displays, are under study and have great potential.

Auditory technologies. In addition to the visual sensory element, full telepresence immersion requires authentic auditory reproduction of sound. The original sound field recorded at the remote site must be identical to the sound field reproduced at the home site. Today, audio reproduction is engineered with fidelity that exceeds the limits of human perception. Hearing, however, is inherently a spatial perception. The human auditory system detects sound waves with two ears (binaural hearing) to determine information about the 3–D location, distance, and size of sound sources. Therefore, the ultimate goal is to reproduce the spatial properties of sound as accurately as possible. Current prototype telepresence systems achieve accurate spatial reproduction through the use of high-quality stereo headphones. Multichannel audio systems such as 5.1 (6-channel) and 10.2 (14-channel) surround-sound systems improve spatial reproduction by increasing the number of channels around a user. Future telepresence systems will include headphones that incorporate virtual sound or multichannel surround-sound headphones, the capabilities of which exceed current 5.1 surround-sound headphones. Additional auditory technologies under study include sound transmission through the skull and HyperSonic Sound (HSS) technology. (HSS is intense focusing and channeling of sound over great distances without dispersing its quality.) These are not yet mainstream technologies, but they are on the horizon and are very promising.

Tactile technologies. The human sense of touch is conveyed to the human brain through the haptic sensory system. Haptic technologies seek to apply tactile sensations to a human’s interaction with a computer using a haptic device such as a data glove equipped with sensors to sense the bending of the fingers and movements of the hand. The goal is not only to allow the user to feed information into a computer but to permit the user to receive information through a haptic interface. Using a data glove in virtual reality, for example, a user can pick up a virtual object such as a cup. A computer then senses the movement of the user’s hand and moves the virtual cup on a display. This provides the user with the feel of the cup in his hand through tactile sensations sent by the computer.

Olfactory and gustatory technologies. Technologies to reproduce the human senses of smell (olfactic) and taste (gustatic) have, for the most part, been ignored compared with visual, audio, and haptic technologies. To appreciate the complexity of reproducing these sensory elements, keep in mind that humans have approximately 10 million sensory neurons for smell and approximately 10,000 taste buds that contain between 50 and 100 taste cells representing sweet, sour, bitter, salty, and umami (the flavor that is characteristic of glutamates such as monosodium glutamate). Current electronic noses can recognize certain odors. These electronic noses, which are composed of arrays of electronic chemical sensors and pattern-recognition capabilities, are much simpler than their biological counterparts. Therefore, users are limited to a predetermined set of odors. Unlike electronic noses, gustatic technologies are quite complex. Currently, users experience taste through biological lipid and polymer membrane sensors. Full telepresence immersion will require maturation of olfactic and gustatic technologies.

Achieving Telepresence

In addition to advances in enabling technologies, telepresence requires advances in remote-site technology (robots) and communications link technology (to achieve near-zero communication latency).

Robots. Sensory elements received from the remote site will be obtained by effectors. An example of an effector is a human-like robot. Robots controlled by users from remote locations will carry out operations required in the field. Robots must be able to perform myriad tasks with the versatility of humans and, in some cases, with strength exceeding that of humans.

Advances in robot technologies have increased over the years, but robots remain very primitive. Research, for the most part, is confined to universities. Robots have the ability to see and hear but lack extensive haptic, olfactic, and gustatic sensory elements and are primarily purpose-built for specific tasks. Advances in telepresence will require advanced robots that can perform multiple tasks and have the ability to adapt.

Near-zero communication latency.Communication latency is perhaps one of the biggest detractors from the feeling of “being there here,” or fully immersed in a remote environment. To demonstrate the effects of communication latency, consider the following example. Assume that you want to send a simple communications signal, such as a pulse of light, around the globe. Without the use of any communications networks, it would take 0.13 milliseconds for the pulse to make the trip. This time delay is referred to as a distance-induced latency. Humans can detect time delays of approximately 16 milliseconds and greater. Therefore, in this simple example, the communication latency would be negligible. Typically, however, communication between a source and a receiver involves the transfer of a large amount of data. As a result, distance-induced latencies must be coupled with latencies in processing speed (transmission-induced latencies, or throughput). Today, packet switching is the dominant means of transmitting communications. Packets of information are sent individually between nodes of a network in a way that optimizes bandwidth and minimizes latency.

Near-zero communication latency requires that, for the user to achieve full telepresence immersion, communication between a robot and the user occur in less than 16 milliseconds. Unfortunately, current high-bandwidth data transmissions (full motion video) have significant latency. This latency is both distance-induced and transmission-induced. This is sometimes evident, for example, when a news reporter presents a video report from a remote location halfway around the world. Latencies of a second are quite common and very noticeable. Although the distance-induced latency is very small, transmission-induced latencies built into the equipment are significant.

Future processor technologies undoubtedly will be faster than those today, and new communications paradigms will be developed. Advances in quantum computation could benefit true telepresence. In the near term, however, steps can be taken to decrease bandwidth requirements and reduce communication latency so that certain elements of telepresence can be realized. For instance, greater levels of interaction between unmanned systems can be facilitated by creating environment or terrain models in advance of operations. Representations of the physical environment can be mapped ahead of mission execution to help reduce bandwidth requirements. The remaining bandwidth then can be dedicated to representing dynamic features that will allow new levels of human-machine interface that otherwise would be infeasible. Enabling technologies for decreasing latency, such as data compression, increased digital modulation, and subdivided optical nodes, also can be implemented.

Certainly, near-zero communication latency affects global military logistics beyond its impact on telepresence. As Dr. Theodore Bially, the Director of the Defense Advanced Research Projects Agency Information Exploitation Office, said in 2004, “The fog of war will plague us as long as the information provided to any level of command is incomplete, inconsistent, delayed in time, difficult to manipulate or hard to visualize. To lift that fog we must provide each of our warfighters with total, accurate and up-to-the-minute battlefield situational information . . . .” It is easy to see how telepresence could facilitate this task.

What can we expect in the deep future (2030)? In the deep future, sensory stimulation will completely bypass the human sensory organs, and the perceptual neurons in the brain will be stimulated directly. Sensory threshold filters will prevent overload of the human perceptual neurons. Full-immersion telepresence will be realized with completely noninvasive sensory stimulation directly to the brain. Sensory element information sent from a user to a robot will be accomplished using noninvasive brain-computer interfaces. Recent advances in brain-computer interfaces have demonstrated that noninvasive readings of brain activity can be harnessed to perform primitive robotic motions. In the future, users will have the ability to control robots just by thinking. Finally, configurable, remotely assembling components (robot swarms) will have the ability to adapt to remote, real-world environments, thereby enhancing telepresence capabilities and applications.

Telepresence will become possible following the development and improvement of robots, near-zero latency communication capabilities, multisensor integration and fusion, and multirobot systems. Telepresence may be feasible in limited applications by 2015.

Research in various fields, such as computer graphics, computer vision, human-computer interaction, brain-computer-machine interaction, acoustics, networking, and databases, all support the telepresence theme. Work is underway in the areas of telepresence systems design and the architecture of physical spaces, multimodal sensing (including camera and computer vision, microphone arrays and acoustics, haptic sensors, and active badges), multimodal presentation and display systems, virtual and augmented reality, ambient intelligence, network infrastructure, and spatio-temporal databases. Current telepresence design and studies in the United States, the Netherlands, India, Japan, and the United Kingdom should mature significantly in the next 10 to 15 years.

Some network and communications hurdles exist in the use of telepresence for military applications. Telepresence devices use high-bandwidth fiber optics, which currently are not available to forward-deployed units. Until they are, short-range wireless and existing fiber optic networks could be used for communications in logistics applications such as materials-handling and depot operations. Full battlefield utility of telepresence may require quantum computation and communication advances or other paradigm-shifts to support and overcome current data ransmission and bandwidth limitations. However, it is plausible to expect that prototype telepresence systems for logistics applications could be fielded in the near term.

