the visual environment provides a vast assortment of information to the sighted individual during the acts of daily living . for the visually - impaired individual ,
a need exists to compensate for the lack of this information , which has motivated the introduction of a wide variety of devices that transfer at least some visual information to another sense .
however , the primary assistive technologies widely used today by the visually - impaired to navigate through the environment are essentially unchanged from those of twenty years ago , namely , white canes and guide dogs .
although these two methods can facilitate the ability to travel safely in a variety of indoor and outdoor environments , neither provide the kind of assistance needed to straighten a picture frame on the wall or find a can of soup on a counter - top .
electronic navigation aids are finding some acceptance , for example , ultrasonic canes ( ultracane from sound foresight technology , ltd . ) that provide tactile cues to objects beyond the tip of the cane , as well as portable computers with global positioning systems ( gps ) and electronic braille or speech interfaces .
however , replacing the more general capabilities of vision to provide detailed information about objects in the environment has proven more difficult .
there exists a need for a device that allows active interrogation and sensing of the 3d visual environment surrounding the operator while moving through everyday environments .
there also exists a need for a device that not only allows the user to sense the environment but also provides control of specific aspects of the environment that have been detected .
these abilities to interrogate and control the environment should not be limited to a specific , predetermined environment , such as those that already contain infrared ( ir ) transmitters .
devices using a concept we call fingersight are intended to allow visually impaired users to remotely identify objects of interest in , and navigate through , natural environments , by acting as a sensory substitution system for the user 's sense of sight .
there are already a number of technologies that monitor hand , eye , and body motion from a fixed or mobile camera to interpret gestures , commands , gaze direction , e.g. , .
some of these may be used to track the operator 's interrogation of the environment , as with the commercially available eye - tracking glasses ( e.g. , tobii , tokyo ) , but more often they are used to interpret the user 's motions as commands .
our original goal was , and largely remains , to serve the population who are visually impaired .
most devices designed with this goal make use of sensory substitution systems , which transmit stimuli normally interpreted by one sense through another sense .
sensory substitution systems generally consist of three components : a sensor that collects a signal , a coupling system that processes the signal , and finally an output transducer , which transmits the signal to a sense organ not normally used for that signal , .
we specifically consider substitution for the visual system by the haptic system ( i.e. , one 's sense of touch ) .
although hearing offers a potentially higher bandwidth than touch , it is crucial not to impede the existing use of hearing , which can be acutely well developed by those who lack normal vision , providing essential acoustic cues about the environment . as for touch , the hands offer the greatest versatility and sensitivity , but still we must not completely usurp their use , since they are essential for so many tasks in daily living .
some employ tactile pin arrays , which feature a grid of small pins with adjustable heights .
tactile pin arrays can be used for either passive stimulation of the fingers or during active exploration to simulate contact with surfaces and objects .
existing systems capture an image and then display that image on a tactile screen that can be worn on a belt , for example .
the fingers can then be used to interrogate the image depicted on the tactile screen in an effort to  visualize  the image , .
some devices have been developed to directly convey a predetermined map from coordinates of physical space to a tactile spatial layout .
an example is bach - y - rita 's brainport device , which uses an array of electrodes placed on the tongue to relay camera information .
sensory substitution systems also exist that provide an audio output , such as text - to - speech devices that employ optical character recognition , canes that provide audio feedback to enhance the natural tactile forces detected at the tip itself , and the voice system , which maps image pixels into frequency and time .
vibrotactile sensitivity is found in mechanoreceptors lying within the subcutaneous tissue , which respond maximally to frequencies of 250300 hz , as well as kinesthetic receptors . the cybertouch glove ( immersion corp .
, san jose , ca ) uses optical tracking of each finger to determine when vibratory stimulators on that finger should be activated . some sensory aids , such as the optacon ( telesensory corp . ,
defunct ) , aim to simulate tactile exploration of virtual objects and have been developed specifically for reading the printed page .
our proposed device uses a vibrotactile stimulator to provide visual information gathered from a color camera system .
the information from the camera system can be processed into a single feature whose presence or absence is communicated through the stimulator .
for example , the camera may provide a positive signal when an edge is detected in the field of view .
our system is similar in some ways to that of lenay ,   who attached a photocell to the tip of a finger and used it to activate a vibrator held in the other hand in an all - or - nothing manner .
another system , developed by burch and pawluk , is mounted on the same finger as the sensor , for interacting with graphical objects on a flat screen .
this system senses a single rgb pixel at a short distance ( i.e. , to the screen that the subject is touching ) , and vibrates based upon the color sensed .
our system goes beyond this basic idea by detecting not just the general directional presence of light or contact with a single light source , but rather actual images ( or a series of images over time ) using a miniature camera . like burch and pawluk
we believe that this placement promotes an intuitive coupling between the scene pointed to by the camera mounted on the fingertip and the feedback felt by that finger . in general , it is imperative for designers of any sensory substitution system to consider not only what is technologically feasible , but also what is functional in the context of the sensory and cognitive limitations of the user . in a review of sensory substitution for the blind , loomis and klatzky   pointed to the need to couple device design to the capabilities of the user for a given task .
low bandwidth , for example , need not be a negative feature of a device if a task can competently be performed on the basis of the information provided .
high bandwidth communication to the user is not an advantage if the target sensory channel can not process the information provided .
furthermore , it should not be assumed that information - processing capacities of blind and sighted are equivalent ; for example , superior tactile sensory systems may be preserved in older braille readers relative to sighted populations of the same age , whereas tasks that draw on visual imagery may be impeded in a blind population without any experience of sight .
we have developed a series of working prototypes , leading up to the one used in the experiments described in the present paper . as the progression is informative
, we review them briefly here . our initial system permitted active interrogation of the visual surroundings with a miniature red laser attached to the fingertip ( see fig .  1 ) .
the laser was modulated at 10 khz , allowing its reflection by objects in the environment to be detected by a non - directional phototransistor in the midst of other sources of light .
thus , like the burch device , it sensed the light ( in this case reflected ) from a single point in space , but at a distance .
moreover , it differed from that device by actively interrogating across the detected point for an edge .
this was accomplished by means of a solenoid constructed from tiny magnets and a coil .
regenerative feedback between the amplitude of the detected signal from the reflected laser and the solenoid caused the laser to vibrate vertically when the laser spot was located on any edge between light and dark objects .
thus the system created a haptic stimulus ( vibration ) whenever a properly oriented visual edge was encountered by the laser . 

the laser - based system was limited to simple edges and had the disadvantage that the active visible light source could be disturbing to others .
more importantly , limiting fingersight to what amounts to a single scanning line makes it extremely difficult to integrate more than the most rudimentary features in the visual field .
a full video image captured at each pose of the finger offers many advantages , including immediate extraction of more complex features , identification of entire objects , and even determination of camera motion directly from the changing image .
the recent availability of very inexpensive and small cameras , brought about largely by their use in cell phones , has led us to adopt a camera - based approach , the first version of which is shown in fig .  2 
 . a small black - and white camera ( supercircuits pc206xp 510492 ) was mounted on the subject 's finger along with a cell - phone vibrator .
real - time analysis of the video signal from a miniature camera was used to control the cell - phone vibrator .
the cell - phone vibrator has the advantages of low cost and size , but presents problems in that amplitude and frequency are inextricably linked together by the operating voltage .
also , the frequency is low enough to cause noticeable vibration in the camera image , and the time for the motor to come up to speed is relatively long . 

two small speakers ( gui , inc . # gc0251k - nd , 1w , \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ 8\omega$\end{document } ) were converted into haptic vibratory stimulators , or tactors , by cementing a short wooden dowel onto the central cones of each speaker .
the converted speakers were mounted on either side of the finger and held there , along with the camera , by a springed clip , leaving the palmar aspect of the fingertip bare to use for touching and grasping objects .
we hoped , by including two such tactors , to be able to use the relative strength of the vibration from each tactor to convey a parameter such as location of an image feature along the horizontal axis . 

this model did not perform well for identifying the location of visual targets , primarily due to mechanical design issues .
it proved problematic to mount the device securely on the distal phalanx of the finger . furthermore , given the influence of flexion / extension at the distal inter - phalangeal joint on the camera orientation , it was difficult for users to judge where they were aiming the device .
in addition , there was low sensitivity to vibration the speakers mounted on the lateral aspect of the finger , as compared to the speaker on the medial aspect .
collectively , these issues made any asymmetry in signal strength from the two tactors of little value to the user 's perception .
the current version of fingersight uses the same speaker - based tactor shown in fig .  3 , but with only one tactor on the ( more sensitive ) medial aspect of the finger , along with a miniature color camera ( supercircuits pc208 , 510492 ) .
the camera extended approximately 8 mm from the tip of the finger , and the speaker was attached to the side of the splint .
the splint weighed 18.8 g and caused minimal interference with movement of the finger as a whole , though it did restrict flexion between the phalanges , thereby reducing uncertainty in camera orientation . covering the palmer aspect of the finger made it difficult to use the finger for other purposes , but we were willing to accept this for the present experiment .
the camera and the speaker were connected to a computer system , which performed spatiotemporal edge detection ( described below ) and controlled vibrator output .
the haptic feedback was provided by a low - frequency ( 20 hz ) audio signal sent to the speaker , the amplitude of which was adjusted to produce detectable vibrations . 
 fig . 
4.current fingersight device apparatus with camera and stimulator attached . 
  for the purposes of experimental testing
the area of the circle was divided across its center , one semicircle containing red and one containing green pixels ( see fig .  5 ) .
the line across the center of the circle separating red from green pixels could be set to any desired orientation .
the circular area was surrounded by white pixels and the images were projected on a screen , where the user could point the finger - mounted camera at it . 

5.examples of edge stimuli , at 12 ( a ) and 4 ( b ) o'clock , respectively . 
 
software was developed to analyze the video image from the finger - mounted camera in real time and to control the vibratory feedback to that finger .
the goal was to generate vibrations whenever the user 's finger was pointing at , or had just passed over , the boundary between the red and green areas , irrespective of the edge 's orientation .
optical edge detection was performed with a novel spatiotemporal kernel , designed to detect edges both in the spatial domain ( when the camera was resting on an edge ) and the temporal domain ( when the camera had moved completely past an edge between one video frame and the next ) .
this design accommodated fast hand movements , which might cause the edge to never actually have been within the kernel in any given frame , as well as slower continual tracking along the edge .
the spatiotemporal kernel accomplished both of these for edges of any orientation by comparing each pixel in one time frame with the corresponding pixel directly across the center of a circular kernel in the next time frame .
if the two pixels differed , an edge was present or had just been crossed .
it is important to note that unlike the general usage of the term  kernel  in image processing , especially for convolution , our spatiotemporal kernel was applied not at every pixel , but only at one location , the center of the image .
just as the fovea of the eye ( the high resolution central region ) is constantly moving for human vision to function , our kernel was physically moved by the user 's finger along with the entire image , to interrogate the visual environment .
the optimal size of the circular kernel varied depending on the scale of the edge features to be considered .
for the particular experiment described here , the optimal radius of the kernel was determined experimentally to be 50 pixels .
a detailed formulation of the algorithm follows : we define k as the set of pixels within the circular kernel at the center of the image . based on the particular red - green - blue ( rgb ) video image coming from the camera in real time , a color label \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 

\usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ c(x , y)$\end{document } was created for each pixel in k , denoting whether the pixel at location \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ ( x , y)$\end{document } was predominantly red , green , or ambiguous.\documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ $ c\left(x , y\right)=\cases{+1/2,&$ if~predominantly~red$\cr-1/2,&$ if~predominantly~green$\cr 0,&$if~ambiguous$\cr}\eqno{\hbox{(1)}}$$\end{document } for our purposes ,  predominantly  indicates that the red ( r ) or blue ( b ) color value for the pixel in question was above a specific threshold , while the other two color values in the rgb pixel were both below their corresponding thresholds .
appropriate thresholds for red , green , and blue were determined experimentally , so that red , green , and white pixels ( which registered as ambiguous ) responded appropriately .
we also created a binary mask value \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ w(x , y)$\end{document } for each pixel that had the value of 1 wherever c was non - zero , so that boundaries including the white area surrounding the circle could be excluded from generating edges.\documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ $ w\left(x , y\right)=\cases{1&$if~c\left(x , y\right)\ne 0$\cr 0&$otherwise$\cr}\eqno{\hbox{(2)}}$$\end{document } for each pixel in k , we also considered another pixel in k directly across the origin ( center of the kernel ) at location \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ ( -x ,- y)$\end{document}. our measure of boundary strength was calculated by comparing the value \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ c_{t}$\end{document } for the pixel at \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ ( x , y)$\end{document } in the current image at time \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ t$\end{document } to value of \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ c_{t-1}$\end{document } for the pixel at \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ ( -x ,- y)$\end{document } in the previous image at time \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ t-1$\end{document}. the mask values \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ w_{t}$\end{document } and \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ w_{t-1}$\end{document } were used to ensure that no comparisons were made using  ambiguous  pixels .
thus a measure of edge strength \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ d$\end{document } was computed as \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ $ d\!=\!\!\!\sum_{\left(x,\!y\right)\in k}\!\!{\left\vert c_{t}\left(x , y\right)\!-\!c_{t-1}\left(-x,\!-y\right)\right\vert w_{t}\left(x , y\right)w_{t-1}\!\left(-x,\!-y\right)}\eqno{\hbox{(3)}}$$\end{document } if the edge measure \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ d$\end{document } was above some threshold ( determined experimentally ) , the software reported that an edge had been detected , and triggered the haptic feedback in the form of vibration at the fingertip .
the algorithm has the advantage that it measures equally well the conditions of sitting on an edge ( of any orientation ) for two successive timeframes and having passed completely over the edge from one timeframe to the next .
for example , in 3d one would simply compare pixels directly across the center of a spherical rather than a circular kernel .
twelve sighted subjects ( 10 males and 2 female ) were recruited from the university population , with an average age of \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ 29.4+/-{9.8}~{\rm years}$\end{document}. all gave informed consent .
subjects took part in a series of trials , in which they identified the angle of an edge stimulus according to a closed set of alternatives . each edge stimulus , as described above , was projected on a display screen and consisted of a circular area 115 cm in diameter on a white background bisected with one half colored red and the other green .
the circle was surrounded by white , so that the perimeter of the circle would not itself be interpreted as an edge ( as described above ) .
subjects were fitted with earplugs , blindfolded , and seated with the right arm resting on a foam support 79 cm above the floor .
the circles were projected ( epson powerlite 70c ) on a 154115 cm screen positioned 168 cm in front of the subject 's fingertip .
the arm support was arranged so that the subject 's arm and finger were initially pointing perpendicular to the center of the projected image . during each trial ,
the subject freely explored the display edge with the fingersight device as long as desired .
generally , subjects maintained their arm on the rest and moved their forearm and wrist .
angle set 1 consisted of six angles equally spaced over a 180 range in 30 steps ( from positive vertical , 90 , to just short of negative vertical , \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ { -}{60}^{\circ}$\end{document } ) , corresponding to clock - hours from 12 to 5 o'clock .
angle set 2 consisted of seven angles equally spaced over a 90 range in 15 steps ( from positive vertical , 90 , to horizontal , 0 ) corresponding to the hours and half hours between 12 to 3 o'clock .
subjects reported their responses using these  clock face  labels . within each set ,
tables  1 and 2 show the confusion matrices for all subjects for angle sets 1 and 2 , respectively , plotting reported angle vs. stimulus angle .
a confusion matrix shows correct measurements along the diagonal , with increasing error as one moves off the diagonal .
overall , subjects had an accuracy rate ( proportion correct ) of \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ 0.77\pm 0.05$\end{document } on angle set 1 and \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ 0.52\pm 0.09$\end{document } on angle set 2 ( 95% confidence intervals ) .
these levels are well above chance in this six - alternative choice task \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ ( { \rm chance}=0.16)$\end{document}. 
 table iconfusion matrix for set 1reported angle ( deg)90603003060stimulus angle ( deg)9071131011060168252003008843100101081313000068196040003260confusion matrix for angle set 1 .
each entry is the number of occurrences of the associated stimulus / response pair . 

table iiconfusion matrix for set 2reported angle ( deg)0153045607590stimulus angle ( deg)0642532200151949242101301205519100450836391210600115234692750011533389900011112558confusion matrix for angle set 2 .
each entry is the number of occurrences of the associated stimulus / response pair . 
 
the data were also analyzed by measuring information transfer \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ i$\end{document } , in order to determine the number of  bits  that can be transmitted by the device under the experimental conditions .\documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ $ i=\sum_{j=1}^{k}\sum_{i=1}^{k}{p\left(s_{i},r_{j}\right)\log_{2}\left({{p\left(s_{i}\vert r_{j}\right)}\over{p\left(s_{i}\right)}}\right)}.\eqno{\hbox{(4)}}$$\end{document } here , \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ k$\end{document } is the number of possible stimuli , and \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ s_{i}$\end{document } and \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ r_{j}$\end{document } represent a specific stimulus - response pair .
this statistic can be estimated from the confusion matrices according to equation 6.\documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ $ i_{est}=\sum_{j=1}^{k}\sum_{i=1}^{k}\left({{n_{ij}}\over{n}}\right)\log_{2}\left({{n_{ij}\times n}\over{n_{i}\times n_{j}}}\right)\!.\eqno{\hbox{(5)}}$$\end{document } here , \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ n_{ij}$\end{document } is the number of joint occurrences of stimulus \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ i$\end{document } and response \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ j , n_{i}$\end{document } is the overall occurrence of stimulus \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ i$\end{document } , \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ n_{j}$\end{document } is the overall occurrence of stimulus \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ j$\end{document } , and \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ n$\end{document } is the total number of trials .
relative to the 2.58 bits of information in the six - alternative choice task , angle set 1 ( angles with 30 increments ) had an information transfer of 1.62 bits , while angle set 2 ( angles with 15 increments ) had an information transfer of 1.12 bits .
we have developed a new method of providing the finger with haptic feedback about visual targets in the general environment , by permitting the finger to scan the environment directly to locate those targets .
we have developed this idea , which we call fingersight , through a series of prototypes , exploring the particular application of detecting edges . in the process , we have developed a new spatiotemporal \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ n$\end{document}-dimensional edge detection algorithm of potentially general interest , which is simultaneously capable of detecting the presence of a stationary edge as well as having passed over an edge between successive timeframes .
we have conducted proof - of - concept experiments showing that under active exploration the fingersight device is capable of transmitting visual spatial information through haptic channels . sighted , blindfolded subjects were able to use the device to differentiate between angles separated by as little as 15. performance with the coarser angle set , where responses were separated by 30 showed that fingersight transmitted close to 2/3 of the available information in the stimulus , with most errors being near misses .
the information transmission of the fingersight device is intrinsically constrained by basic sensorimotor abilities of motor control , kinesthetic sensing , and spatial cognitive processing .
was found that when blindfolded , sighted adults tried to report the orientation of a raised line that was easily tracked by touch on the plane of a tabletop , their responses pulled the true value towards the sagittal axis by about 25% ( plus some constant error ) .
since motor control was minimized in the task in , errors can be entirely attributed to the processes of kinesthetic sensing and building a spatial representation .
the observed level of distortion is by itself substantial enough to lead to confusion errors in the present task .
these observations suggest that performance with fingersight could be improved by features that support motor control and augment kinesthetic feedback .
the present experiments measure only the performance of blindfolded sighted subjects who are novices using the device ; training would be expected to considerably improve performance .
siegle and warren have shown that distal attribution , i.e. , the direct perception of objects in external space , can occur with similar finger - mounted sensory substitution systems after a period of training .
however the device used in their experiments had the vibrotactile stimulator mounted on the subject 's back .
it is possible that training time may be decreased with a device such as fingersight , where the stimulus is more strongly tied to the subject 's proprioceptive knowledge of the hand 's location in space .
it is interesting to compare the accuracy of angular measurements by fingersight in 3d space with other researchers who restricted interrogation to a fixed plane .
, explored the ability to recognize 90- or 180-degree rotation on raised - outline drawings directly touched by the blind or sighted - blindfolded subjects , but did not explore finer angular resolution .
postma , et al . , demonstrated the role of visual experience in such haptic spatial tasks , including the description of angles between bars on a table , showing that blindfolded sighted subjects outperformed late blind subjects , who outperformed early blind subjects .
they found that having experienced vision , even to a limited extent , helps in the interpretation of angle by touch alone .
the worst performance in verbally judging the angle of the bars ( demonstrated by the early blind ) was 7.2 degrees , a good deal better than our errors , possibly because they were constrained to a tabletop .
it would also be interesting to see if sighted individuals also have a similar advantage in learning to use fingersight over blind individuals , due to previous visual experience . also relevant to the question of angle and touch
, with haptic computer mice , which are modified to have tactile pin arrays on their upper surface .
significant lack of accuracy in the haptic position information , which is critical for individuals to haptically piece together a 2-d graphic . 
the inaccuracy is due to the fact that the tactile mouse ( or any normal mouse ) is a relative positioning device , dependent upon the speed of motion and orientation of the mouse to determine total displacement on the screen .
in contrast , fingersight is inherently an absolute positioning device , given a stationary environment , and as such , fingersight may have an advantage .
the system of burch and pawluk previously mentioned   uses a fingertip photosensor and piezoelectric stimulator to scan specially created graphical displays , in which texture is added to enhance perception of edges and orientations .
a single photosensor thus suffices for this purpose , only because preprocessing is performed to populate regions on either side of boundaries with differing textures .
multiple photosensors on different fingers were found to improve results with this system , because the operator 's knowledge of the spatial relationship between the fingertips could be used to integrate the inputs .
but still the approach relies on preprocessing the image to create textures , after which individual photosensors can be effective . with fingersight , we are exploring the unadulterated 3d environment , where depending on only a few photosensors is not sufficient .
we benefit greatly by having a multi - pixel image at each timeframe . by further processing of the camera images ,
the fingersight device could be adapted to identify more complicated features than simple visual edges , perhaps even constituting an object recognition system for a blind operator .
for example , it could be used to find a particular person in a crowd , or identify a door displaying a particular sign . the rapid advancement in computer vision algorithms , driven in part by the security and social networking industries , will provide ever more sophisticated capabilities .
for example , determining camera motion and target depth from a image sequence could permit greater 3d integration of multiple perspectives and facilitate providing navigational cues to the blind operator , such as where the curbside is , whether the approaching stairwell goes up or down , or whether one is moving towards or away from the elevator .
such analyses are not feasible for a single photosensor , but rather , they require an entire image .
a common problem in any real - world computer vision application is the variability of lighting , and there are established techniques for solving this problem using such constructs as the  illumination cone  .
for example , a recently developed time - of - flight ( tof ) 3d camera , the swiss ranger sr4000 ( mesa imaging , zuerich ) , can deliver a 176144 pixel image with each pixel reporting range up to 10 m with 1 cm accuracy .
the present camera is roughly 7 \documentclass[12pt]{minimal } 
 \usepackage{amsmath } 
 \usepackage{wasysym } 
 \usepackage{amsfonts } 
 \usepackage{amssymb } 
 \usepackage{amsbsy } 
 \usepackage{upgreek } 
 \usepackage{mathrsfs } 
 \setlength{\oddsidemargin}{-69pt } 
 \begin{document } 
 } { } $ { \rm cm}^{3}$\end{document } and is being integrated by at least one research group into portable devices for the blind .
for some of these more sophisticated systems , it may become problematic to rely solely on vibrotactile feedback to the finger .
the bandwidth of audio , especially combined with language , makes it an appealing option , though as noted above , one does not want to impede the natural use of auditory cues that are especially crucial for the vision impaired . however , intermittent use of verbal output by fingersight , and for that matter , verbal commands by the operator to the device , could prove extremely useful , while not impeding auditory cues form the environment . as noted above , it is imperative to test whether features added to the technology ultimately add to the functional capability of the user , given intrinsic limitations on human information processing .
that said , one promising avenue is the incorporation of control capabilities into the fingersight device .
one of our previous systems included the capability to control graphical objects on a screen . in this implementation
, the location of a small white square on a screen is controlled by motion of the finger .
the algorithm detects the square in the field of view of the camera , moving it on the screen to keep it constantly in the center of the camera image , while providing haptic feedback about whether the tracking system has locked onto the target .
a variation on this system constrains the square to move along a straight line , simulating the action of a slide pot under the operator 's control .
a further variation uses a small white triangle to simulate a knob , whose orientation is determined using standard computer vision techniques and subsequently controlled by rotation of the finger .
clearly , such systems are not limited to actively controlling graphical objects on a screen , but could also identify inanimate objects such as a light switch or door latch .
in such cases , remote control could still be achieved by motion of the finger once the target has been identified , using a separate control channel to turn on the light or lock the door . continued work on the control aspects of fingersight
a final note is merited on the eventual miniaturization of a fingersight device such that it might be small enough to be worn like an artificial fingernail .
the cameras themselves are almost small enough already , and the main considerations are power and communications .
one can envision radio communication between a fingertip device and a pocket unit , much as wireless earphones and microphones communicate with cell phones now .
such devices might be fully integrated into the everyday activities of the vision - impaired .
 we present a novel device mounted on the fingertip for acquiring and transmitting visual information through haptic channels . 
 in contrast to previous systems in which the user interrogates an intermediate representation of visual information , such as a tactile display representing a camera generated image , our device uses a fingertip - mounted camera and haptic stimulator to allow the user to feel visual features directly from the environment . visual features ranging from simple intensity or oriented edges to more complex information identified automatically about objects in the environment may be translated in this manner into haptic stimulation of the finger . 
 experiments using an initial prototype to trace a continuous straight edge have quantified the user 's ability to discriminate the angle of the edge , a potentially useful feature for higher levels analysis of the visual scene . 
Introduction
Methods
Results
Conclusion