# Two-Disk Compound Symmetry Groups

Robert A. Hearn1, William Kretschmer2,
Tomas Rokicki3, Benjamin Streeter4, and Eric Vergo5
1Gathering4Gardner<EMAIL_ADDRESS>2UT Austin<EMAIL_ADDRESS>
3Radical Eye Software<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Symmetry is at the heart of much of mathematics, physics, and art. Traditional
geometric symmetry groups are defined in terms of isometries of the ambient
space of a shape or pattern. If we slightly generalize this notion to allow
the isometries to operate on overlapping but non-identical metric spaces, we
obtain what we call _compound symmetry groups_. A natural example is that of
the groups generated by discrete rotations of overlapping disks in the plane.
Investigation of these groups reveals a new family of fractals, as well as a
rich structure that is intriguing both mathematically and artistically. We
report on our initial investigations.

## Introduction

Symmetry is of fundamental importance in many disciplines of mathematics, from
the fields of Galois theory, to the automorphisms of abstract algebra, to the
isometries of wallpaper patterns and crystals. In physics, Noether’s theorem
connects the symmetries of space and time with conservation laws. In art,
subtleties of degrees and kinds of symmetry arguably lie at the heart of what
constitutes beauty.

Here we expand on the traditional notion of geometric symmetry, with
mathematical and artistic consequences, at least. The _symmetry group_ of a
shape or pattern is defined as the group of all isometries of the ambient
space that preserve that shape or pattern. Isometries may only be combined in
so many ways in any given metric space: there are (up to isomorphism) only 17
wallpaper groups, 7 frieze groups, 230 3-dimensional space groups, etc. We
cannot have, for example, five-fold rotational symmetry in any repeating
pattern in the plane—though quasicrystals can approximate this [6].

(a)

(b)

Figure 1: Images of compound symmetry groups: (a) $GG_{3,5}(2.42,2.41)$, (b)
the $n=5$ fractal.

By considering groups generated by isometries of metric subspaces that are not
identical, but instead overlap, we can probe some of these “forbidden”
symmetries in new ways. A portion of one of these “compound symmetry groups”
is shown in Figure 1. Locally, this image contains regions of three-fold and
five-fold symmetry—necessarily broken on larger scales—as well as combinations
such as 15-fold. A new family of fractals lies embedded within these groups at
critical parameter values, as shown in Figure 1.

In this paper we explore the characteristics of these new kinds of symmetry
group. Of particular importance will be determining the parameter values at
which the groups become infinite, exploring the underlying dynamics, and also
understanding the characteristic fractals, or sometimes pseudofractals, that
appear precisely at these transitions.

Figure 2: The Gizmo Gears puzzle.

This work began with the study of the behavior of certain “circle puzzles” [1,
2]—especially _Gizmo Gears_ , shown in Figure 2, designed by Doug Engel.111The
specific question was this: can Gizmo Gears be finitely “unbandaged”, or
chopped into pieces so that all natural turns ($30^{\circ}$ in this case) are
unblocked, regardless of configuration? If not, a puzzle is said to “jumble”.
Against all intuition at the time, Gizmo Gears does in fact jumble—it is past
the critical radius (barely) for the compound symmetry group family $GG_{12}$.
The original discussion can be found here:
https://twistypuzzles.com/forum/viewtopic.php?t=25752. A two-disk compound
symmetry group is the mathematical generalization of a circle puzzle. The
study of the group structure of circle puzzles seems to have been initiated in
[5].

## Definitions and Basic Properties of Two-Disk Systems

A _compound symmetry group_ is a group generated by a set of isometries of
subspaces of a metric space.

Here we will primarily be concerned with compound symmetry groups generated by
discrete rotations of two overlapping closed disks in the Euclidean plane. We
sometimes call these _two-disk systems_. Without loss of generality, let the
two disks be centered at $(-1,0)$ and $(1,0)$. Denote the left disk’s radius
as $r_{1}$, and the right disk’s as $r_{2}$. The generators ${a,b}$ are
rotation of the left disk by $-2\pi/n_{1}$, and of the right disk by
$-2\pi/n_{2}$. The group operation is function composition on the left222We
choose this convention so that move sequences can be read left to right.
Similarly, the generators are defined to be clockwise rotations for
compatibility with normal twisty puzzle notation.: $ab(x)=b(a(x))$. We denote
the group with these properties333$GG$ was chosen in honor of Gizmo Gears, and
also, conveniently, to indicate the interaction of two groups. as
$GG_{n_{1},n_{2}}(r_{1},r_{2})$. If $n_{1}=n_{2}$ we use a single subscript,
and similarly for $r_{1}$ and $r_{2}$. We can also omit the radius
specification to indicate a family of groups with unspecified but equal radii.
For example, one very important family is $GG_{5}$—the groups generated by
five-fold rotation of two equal disks.

To build some intuition about how two-disk systems work, consider Figure 3.
This figure shows the action of the group elements on points in the plane, for
$GG_{5}$ at various $r$. Regions that remain connected under all elements
(_pieces_) are colored identically; the color is a function of the size of the
orbit. In Figure 3, $r<1$, and the two rotations do not interact—the group is
isomorphic to $C_{5}\times C_{5}$. In Figure 3, the disks overlap, so there is
some interaction. Viewed as a circle puzzle, we have added 9 pieces to the
puzzle. This group is isomorphic to $C_{5}\times C_{5}\times A_{9}$ (we have
added the even permutations of the wedge pieces). In Figure 3, the overlap has
increased, and many more small pieces are created. Observe that in all cases,
we do have five-fold rotational symmetry about two different points—but only
within a fixed radius.

(a)

(b)

(c)

Figure 3: Different cases of $GG_{5}$. (a) is isomorphic to $C_{5}\times
C_{5}$; (b) is isomorphic to $C_{5}\times C_{5}\times A_{9}$;
(c) is more complicated.

It is important to note that while regular symmetry groups are generated by
and consist of isometries, compound symmetry groups are likewise generated by
isometries, but the general group element is _not_ an isometry: If we perform
$ab$, different regions have been rotated by different amounts about different
centers. This is called a _piecewise isometry_ [3].

### Infinite Groups

A key question about any two-disk group is whether it is finite or infinite.
If a family $GG$ has some infinite member, it will have a _critical radius_ ,
$r_{c}(GG)$, such that $GG(r)$ is finite when $r<r_{c}(GG)$, and infinite when
$r>r_{c}(GG)$.444We believe, but have not proved, that it will also be
infinite exactly at the critical radius. We know this because the size of the
orbit of any given point cannot decrease as $r$ increases—all group elements
that affect it are still available—so $GG$ can never go from infinite to
finite as $r$ increases. We can also speak of the critical radius when
$r_{1}\neq r_{2}$ if we fix one radius.

We can precisely characterize which $GG_{n_{1},n_{2}}$ have infinite members:

######  Theorem 1.

There exists some $r$ for which $GG_{n_{1},n_{2}}(r)$ is infinite if and only
if $\operatorname{lcm}(n_{1},n_{2})\not\in\\{2,3,4,6\\}$.555This fact is
closely related to the crystallographic restriction theorem.

###### Proof.

We omit the “only if” proof in this paper, and prove the more interesting
direction.

First, assume that $n_{1}=n_{2}=n$. Observe that $a^{-1}b$ is a translation of
all points moved by both rotations: The point at for example $(-1,0)$ is moved
(if $r\geq 2$), but the net rotation is 0. In particular, $a^{-1}b$ represents
one side of a regular $n$-gon of circumradius 2, as shown in Figure 4.
$a^{-2}b^{2}$ is another translation, and so on. We can generate translations
from any one vertex of this $n$-gon to any other, as long as $r$ allows the
point to remain within both disks, by composing these sequences and their
inverses.

Figure 4: Constructible translations can shrink arbitrarily.

There are two cases. If $n>5$, we take all the translations from one polygon
vertex to an adjacent one, and observe the images we get of the origin under
all of them. The resulting $n$ points form another $n$-gon, but smaller than
the original. But again, we can form translations between any of these new
points by taking differences of the appropriate translations, and then repeat
the process, resulting in a yet smaller polygon. Thus, we can generate
arbitrarily small translations, and the group must be infinite.

If $n=5$, we start at the origin and apply every other pentagon edge
translation, resulting in a pentagram shape whose vertices again form a
pentagon smaller than the original. Again, we can iterate.

We can do this with bounded $r$—inspection shows that $r\geq 4$ is sufficient;
no moves described ever move a relevant point more than a distance of 4 from
either disk center.

If $n_{1}\neq n_{2}$, then it is easy to show that $(a^{-1}b)^{\alpha}$ and
$(ba^{-1})^{\alpha}$, for some $\alpha$, are rotations by
$2\pi/\operatorname{lcm}(n_{1},n_{2})$ about two different centers. We can use
these rotations in place of $a$ and $b$ in the proof above (with $r\geq 8$). ∎

## Geometric Constructions

For some $n$, we have geometric constructions showing that $GG_{n}$ is
infinite at a value of $r$ matching our numerical estimates for critical
radius. For other $n$ we have plausible geometric constructions which agree
well with our numerical estimates. But for most $n$, all we have is our
numerical estimates.

The simplest case is $n=5$. Figure 5(a) shows the relevant geometry. Using the
shown relationships, simple trigonometry yields $r=\sqrt{3+\varphi}\approx
2.149,$ where $\varphi$ is the golden ratio. A similar analysis of the
geometry in Figure 5(b), where $n=10$, gives $r=\sqrt{4-\varphi}\approx
1.543$.

The dynamical processes behind the transition from finite to infinite at the
critical radius remain mysterious in most cases. However, for $n=5$, we can
see what is going on. (We omit due to space a proof that the single generator
$ab^{-1}$ produces the same behavior.)

######  Theorem 2.

$GG_{5}$ is infinite at $r=\sqrt{3+\varphi}$.

###### Proof.

Referring to Figure 5(a), interpreted now as the complex plane, let
$\zeta_{n}=e^{2\pi i/n}$, and the point $E=\zeta_{5}-\zeta_{5}^{2}$. Note that
$|E+1|=r$. We focus on how the line segment $E^{\prime}E$ moves under specific
sequences. The point $F=1-\zeta_{5}+\zeta_{5}^{2}-\zeta_{5}^{3}$ lies on
$E^{\prime}E$, as does the point $G=2F-E$. We have three cases:

  1. 1.

Line segment $E^{\prime}F^{\prime}$ is transformed by
$a^{-2}b^{-1}a^{-1}b^{-1}$ to line segment $GF$.

  2. 2.

Line segment $F^{\prime}G^{\prime}$ is transformed by $abab^{2}$ to line
segment $FE$.

  3. 3.

Line segment $G^{\prime}E$ is transformed by $abab^{-1}a^{-1}b^{-1}$ to line
segment $E^{\prime}G$.

Together, these three operations can translate any portion of the line segment
$E^{\prime}E$ piecewise onto itself. At no time does any point leave the
intersection of the two disks during these transformations. The first two
cases are translations of length $|F-F^{\prime}|$, and the third case is a
translation of length $|E-G|$. These two values are not rationally related to
the total length $|E-E^{\prime}|$, since
$|E-E^{\prime}|/|F-F^{\prime}|=\varphi$. We can thus map the origin to
successive points along $E^{\prime}E$, by repeatedly choosing the
transformation matching the region the point is in, indefinitely; it has an
infinite image. ∎

(a)

(b)

(c)

(d)

Figure 5: Geometric constructions for critical radius: (a) $n=5$, (b) $n=10$,
(c) $n=8$, (d) $n=12$.

For the cases of $n=8$ (Figure 5(c)) and $n=12$ (Figure 5(d)), their
characteristic fractals (see below) can provide insight. A path of consecutive
line segments that follows the fractal structure can be realized, starting
from the center of one disk and approaching the disk boundary from the
interior. For $n=8$, consecutive segments scale down by a factor of
$\sqrt{2}-1$ and traverse angles of $\pi/8$. We can calculate the
corresponding limit point from the path to yield
$r=\sqrt{5(2-\sqrt{2})}\approx 1.711.$ For $n=12$, an analogous construction
gives a scale factor of $2-\sqrt{3}$, and $r=\sqrt{2(20-11\sqrt{3})}\approx
1.377.$ These closed-form radii rely on the limit point lying on the disk
boundary at the critical value, which has not been proven.

## Critical Transitions and Fractals

Precisely at any group’s critical radius, we always observe a distinct fractal
embedded in the image. It seems remarkable that these fractals have gone
unnoticed for so long; they seem as natural as, e.g., the Mandelbrot set. We
can define the _characteristic fractal_ for $GG$ to be the set of points with
infinite orbits at $GG(r_{c}(GG))$. In particular, this associates a unique
fractal with every $n\not\in\\{2,3,4,6\\}$, and similarly when $n_{1}\neq
n_{2}$.666We also have fractals when $r_{1}\neq r_{2}$, but in that case the
critical radius becomes a two-parameter family. The canonical example is the
fractal for $n=5$, shown in Figure 1. We also include in this paper the
fractals for $n=8$ (Figure 5(c)) and $n=12$ (Figure 5(d)). In Appendix B, we
include higher-resolution images of the characteristic fractals for all $n$ up
to 20.

The appearance of fractals here seems somewhat mysterious. The hallmark of a
fractal is the repeating of a pattern on successively smaller scales. But
unlike fractals constructed with an explicit recursive rule, there are no
scaling operations in these systems, only rotations. Where do they come from?
Why do they look so different for different $n$? Some insight may be gained by
considering Figure 4, which shows that in fact, rotations can combine to
shrink patterns.

A natural question for any characteristic fractal is whether it is the closure
of the orbit of a single point, or whether it consists of finitely or
infinitely many disjoint closures of orbits. In most cases the fractals seem
to be closures of the orbit of a single point. For $n=23$, it appears that the
fractal consists of two distinct pieces, symmetric about the $x$-axis.
However, it is possible that these pieces are connected, and we have not been
able to reach this connection numerically.

An especially interesting case is $n=7$. Here, there seems to be a clear
fractal structure. We can zoom in several times and see the same structure
repeating. But then at a certain point, after zooming in by a factor of
500,000, the pattern suddenly changes. This was not apparent until we imaged
the fractal to over 2 trillion points. It is possible that this broken scale
symmetry is a numerical artifact, but evidence argues against this. If it is
real, this is perhaps the most mysterious process related to these systems we
have yet observed. A video showing this transition may be found at
https://www.youtube.com/watch?v=FFeSlh0ifYE. A portion of $GG_{7}$ just short
of the critical radius, shown in Figure 6, reveals the characteristic $n=7$
fractal motif.

In Figure 6, we see that embedded within $GG_{12}$ lies a Koch-snowflake-like
fractal seemingly based on four-fold symmetry, rather than the traditional
three-fold.

(e) A portion of $GG_{7}(1.6233)$.

(f) A portion of the generator $ab^{-1}$ for $GG_{12}(\sqrt{2})$.

Figure 6:

## Numerical Models and Algorithms

All of the images in this paper were generated by programs which simulate
compound symmetry groups. Conceptually, for each point we want to image, we
consider all possible rotation sequences applied to it, and plot the
results—i.e., we plot its orbit. Figure 1, for example, was generated this
way. When we want to show the action of the group on the entire space, it is
more efficient to only image the disk boundaries. We image a boundary by
discretizing it into small segments, imaging those segments, drawing the
images into a high-resolution bitmap, then filling the resulting spaces within
the bounded regions with appropriate colors, according to some coloring rule.
This is how Figures 1, 6, 7, and 7 were generated.

(a)

(b)

Figure 7: Filled images: (a) a portion of $GG_{9}(1.408)$, (b) a portion of
$GG_{5}(2.144)$.

The use of frontier search [4] when generating orbits helps enormously,
allowing us to not keep the entire orbit in memory. It has enabled some points
to be imaged to over 10 trillion distinct targets, helping us refine our
critical radius estimates, and exposing the curious behaviors of some fractals
discussed above.

## Single-Generator Images

Additional structure can be revealed by imaging orbits under a single
generator, rather than all group elements. In the simplest case, we have
$a^{\alpha}b^{\beta}$. Two examples are shown in Figure 8. This defines a
discrete dynamical system which is an iterated piecewise isometry. Some work
has been done to characterize these systems, but many questions remain. [7, 8]

(a) ${GG_{21}}(5.82)$ using $a^{2}b^{-1}$

(b) ${GG_{8}}(2.12)$ using $a^{2}b^{5}$

Figure 8: (a) The orbit of the upper intersection point is plotted and colored
according to its local density. (b) The full boundary is plotted, with spaces
colored according to the order of their orbit.

## Critical Radii

Table 1 summarizes our knowledge of critical radii for $GG_{n}$ with $n<20$. A
more complete table, up to $n=100$, may be found in Appendix A. In all cases,
points were found with a minimum of 10 billion images, with some up to 10
trillion. These estimates may be too high because points with infinite image
were missed, or too low because the points searched had very large but finite
images. However, there is good agreement with the geometrically derived
values, and we have confidence they are good to about 5 decimal places.

Table 1: Critical radii for $GG_{n}$.

$n$ | Numerical estimate | Algebraic expression | Minimum Polynomial
---|---|---|---
5 | 2.148961 | $\sqrt{3+\varphi}\approx 2.148961$ | $x^{4}-7x^{2}+11$
7 | 1.623574 |  |
8 | 1.711411 | $\sqrt{5(2-\sqrt{2})}\approx 1.711412$ | $x^{4}-20x^{2}+50$
9 | 1.408482 |  |
10 | 1.543357 | $\sqrt{4-\varphi}\approx 1.543362$ | $x^{4}-7x^{2}+11$
11 | 1.290582 |  |
12 | 1.376547 | $\sqrt{2(20-11\sqrt{3})}\approx 1.376547$ | $x^{4}-80x^{2}+148$

$n$ | Numerical estimate
---|---
13 | 1.213594
14 | 1.196554
15 | 1.163276
16 | 1.148470
17 | 1.127509
18 | 1.121505
19 | 1.104246

## Summary and Conclusions

The concept of compound symmetry group opens up a new frontier in mathematics.
Here we have just begun this exploration, by considering the two-disk compound
symmetry groups. This investigation has revealed a new family of fractals, and
a rich new source of spaces that combine symmetries in interesting and
unexpected ways. These are new “places” to explore, similar for example to
“Seahorse Valley” in the Mandelbrot set, that have until now remain unseen. We
must omit due to space many additional topics we would like to cover, such as
the appearance of quasicrystals when we move beyond the critical radius, and
other observations of aperiodic behavior.

While we have made significant progress in understanding two-disk compound
symmetry groups, yet more work can still be done. In many cases, we lack even
a basic theoretical understanding of the behaviors that cause the transition
from finite to infinite size in these groups. For example, does every infinite
two-disk group contain a point whose image is infinite, or a generator of
infinite order? Is the critical radius of a two-disk system always an
algebraic number, and is there a general formula for the critical radius? What
dynamics are responsible for the creation of the fractals?

Similar questions can be raised more generally for multi-disk systems or
arbitrary compound symmetry groups, where the behavior governing infinite size
groups is more complicated. For example, we observe that Theorem 1 fails to
generalize to three-disk systems: consider a set of three disks centered at,
say, $(0,0)$, $(1,0)$, and $(\sqrt{2},0)$, and consider the compound symmetry
group obtained by taking rotation increments $n_{1}=n_{2}=n_{3}=2$. Then if we
choose disk radii to be sufficiently large, the resulting compound symmetry
group will be infinite, because the three corresponding rotations of the plane
generate a pair of translations along the $x$-axis whose ratio is irrational.
Another question that presents itself is whether it is even decidable whether
a given multi-disk compound symmetry group is finite.

Some movies of systems with varying zooms and radii may be found at
https://tinyurl.com/yz9pntwy.

## Acknowledgments

We thank Doug Engel for Gizmo Gears, and his two definitive books on circle
puzzles. We thank Brandon Enright for his significant contributions, insights,
and ideas. We thank Oskar van Deventer and Carl Hoff, who first introduced
Hearn to this problem, and contributed insights. We also thank Bram Cohen,
Scott Elliott, Landon Kryger, Andreas Nortmann, Jason Smith, Nathaniel Virgo,
and all others who shared interest and insights in the twistypuzzles.com
forum.

## References

  * [1] D. Engel. Circle Puzzler’s Manual. 1986. https://www.jaapsch.net/puzzles/circleman.htm.
  * [2] D. Engel. Circle Puzzler’s Manual Part 2. 2019. https://www.puzzleatomic.com/circlemanpart2.pdf.
  * [3] A. Goetz. Piecewise isometries—an emerging area of dynamical systems. Springer, 2003.
  * [4] R. Korf, W. Zhang, I. Thayer, and H. Hohwald. “Frontier search”. Journal of the ACM (JACM), vol. 52, no. 5, 2005, pp 715–748.
  * [5] W. Kretschmer. “Groups in Circle Puzzles”. Game & Puzzle Design, vol. 3, no. 2, 2017, pp 15–26.
  * [6] M. Senechal. Quasicrystals and geometry. CUP Archive, 1996.
  * [7] L. Smith, P. Umbanhowar, R. Lueptow, J. Ottino. “The geometry of cutting and shuffling: An outline of possibilities for piecewise isometries”. Physics Reports, vol. 802, 2019, pp 1–22
  * [8] M. Gassow. Generalisations of the Cohan Circle Puzzle. 2020. https://vrs.amsi.org.au/wp-content/uploads/sites/92/2020/01/gossow˙martin˙vrs-report.pdf.

## Appendix A: Critical Radii up to $n=100$

Table 2: Critical radii for $GG_{n}$.

$n$ | Numerical estimate
---|---
5 | 2.148961
7 | 1.623574
8 | 1.711411
9 | 1.408482
10 | 1.543357
11 | 1.290582
12 | 1.376547
13 | 1.213594
14 | 1.196554
15 | 1.163276
16 | 1.148470
17 | 1.127509
18 | 1.121505
19 | 1.104246
20 | 1.100581
21 | 1.086016
22 | 1.078162
23 | 1.072011
24 | 1.071404
25 | 1.061321
26 | 1.056958
27 | 1.052668
28 | 1.049731
29 | 1.045731

$n$ | Numerical estimate
---|---
30 | 1.043758
31 | 1.040158
32 | 1.037741
33 | 1.035512
34 | 1.033498
35 | 1.031647
36 | 1.030127
37 | 1.028368
38 | 1.026958
39 | 1.025591
40 | 1.024352
41 | 1.023174
42 | 1.022142
43 | 1.021105
44 | 1.020197
45 | 1.019279
46 | 1.018459
47 | 1.017692
48 | 1.016968
49 | 1.016287
50 | 1.015651
51 | 1.015053
52 | 1.014492
53 | 1.013942

$n$ | Numerical estimate
---|---
54 | 1.013437
55 | 1.012954
56 | 1.012498
57 | 1.012067
58 | 1.011657
59 | 1.011267
60 | 1.010908
61 | 1.010545
62 | 1.010210
63 | 1.009890
64 | 1.009585
65 | 1.009294
66 | 1.009018
67 | 1.008750
68 | 1.008499
69 | 1.008253
70 | 1.008020
71 | 1.007797
72 | 1.007582
73 | 1.007377
74 | 1.007180
75 | 1.006990
76 | 1.006809
77 | 1.006633

$n$ | Numerical estimate
---|---
78 | 1.006465
79 | 1.006302
80 | 1.006146
81 | 1.005996
82 | 1.005851
83 | 1.005712
84 | 1.005578
85 | 1.005447
86 | 1.005321
87 | 1.005200
88 | 1.005083
89 | 1.004970
90 | 1.004860
91 | 1.004754
92 | 1.004652
93 | 1.004552
94 | 1.004456
95 | 1.004363
96 | 1.004273
97 | 1.004186
98 | 1.004101
99 | 1.004019
100 | 1.003939

## Appendix B: Characteristic Fractals up to $n=20$

Here we show only the set of points with infinite orbit, at the critical
radius, for each $n$.

Figure 9: $GG_{5}(2.148958)$. Figure 10: $GG_{7}(1.623574)$. Figure 11:
$GG_{8}(1.711405)$. Figure 12: $GG_{9}(1.408482)$. Figure 13:
$GG_{10}(1.543357)$. Figure 14: $GG_{11}(1.290582)$. Figure 15:
$GG_{12}(1.376547)$. Figure 16: $GG_{13}(1.213594)$. Figure 17:
$GG_{14}(1.196554)$. Figure 18: $GG_{15}(1.163275)$. Figure 19:
$GG_{16}(1.1484703)$. Figure 20: $GG_{17}(1.127509)$. Figure 21:
$GG_{18}(1.121504)$. Figure 22: $GG_{19}(1.104246)$. Figure 23:
$GG_{20}(1.100581)$.


# VisPercep: A Vision-Language Approach to Enhance Visual Perception for
People with Blindness and Low Vision

Yu Hao, Fan Yang, Hao Huang, Shuaihang Yuan,
Sundeep Rangan, John-Ross Rizzo, Yao Wang, and Yi Fang
New York University
USA

###### Abstract

People with blindness and low vision (pBLV) encounter substantial challenges
when it comes to comprehensive scene recognition and precise object
identification in unfamiliar environments. Additionally, due to the vision
loss, pBLV have difficulty in accessing and identifying potential tripping
hazards on their own. In this paper, we present a pioneering approach that
leverages a large vision-language model to enhance visual perception for pBLV,
offering detailed and comprehensive descriptions of the surrounding
environments and providing warnings about the potential risks. Our method
begins by leveraging a large image tagging model (i.e., Recognize Anything
(RAM)) to identify all common objects present in the captured images. The
recognition results and user query are then integrated into a prompt, tailored
specifically for pBLV using prompt engineering. By combining the prompt and
input image, a large vision-language model (i.e., InstructBLIP) generates
detailed and comprehensive descriptions of the environment and identifies
potential risks in the environment by analyzing the environmental objects and
scenes, relevant to the prompt. We evaluate our approach through experiments
conducted on both indoor and outdoor datasets. Our results demonstrate that
our method is able to recognize objects accurately and provide insightful
descriptions and analysis of the environment for pBLV.

##  1 Introduction

The prevalence of visual impairment has reached alarming levels, affecting
millions of individuals worldwide, as highlighted by recent estimates from the
World Health Organization (WHO) [24, 11]. The number of people experiencing
moderate to severe visual impairment or complete blindness continues to rise
steadily, with projections indicating a further surge in these numbers by 2050
[23]. Visual impairment, whether partial or complete, presents significant
challenges that profoundly impact various aspects of the daily life of the
pBLV [20]. Among the critical tasks that pose difficulties for pVLB is visual
search, which involves actively scanning the environment and locating a
specific target among distracting elements [30]. Even for individuals with
normal vision, visual search can be demanding, especially in complex
environments. However, for individuals with blindness or low vision, these
challenges are further compounded [19]. Those with peripheral vision loss,
central vision loss, or hemi-field vision loss struggle to pinpoint a
particular location or search for objects due to the reduced field of view.
They often require assistance to accurately identify the environment or locate
objects of interest. Similarly, individuals experiencing blurred vision or
nearsightedness encounter difficulties in identifying objects at varying
distances. Color-deficient vision and low-contrast vision further exacerbate
the challenges of distinguishing objects from the background when they share
similar colors. In addition to understanding their surroundings and locating
objects of interest, assessing potential risks and hazards within the visual
environment becomes an intricate task, demanding a comprehensive analysis to
ensure personal safety [6]. The magnitude of these challenges emphasizes the
need for innovative solutions to enhance visual perception and empower
visually impaired individuals in their daily lives.

Figure 1: Our method employs a large vision-language model to provide visually
impaired individuals a comprehensive and detailed description of object
composition in the camera view, and a risk assessment based on the user query.
Our system utilizes a smartphone to capture images and record user questions
(left). Based on the input image and user question, VisPercep generates
detailed and comprehensive scene descriptions and risk assessments (right).
[The camera input image is from Visual7W dataset [34]]

Current assistive technologies [13, 29, 18] driven by computer vision
approaches and wearable devices have led to the development of assistive
systems that utilize object recognition [27], GPS navigation [8], and text-to-
speech tools [16]. While these technologies have provided valuable assistance
to visually impaired individuals [22], they still face certain challenges and
limitations. One of the primary challenges with existing assistive
technologies is their limited ability to provide comprehensive scene
understanding and guidance to address the specific needs of visually impaired
individuals. For instance, while many tools focus on specific functionalities,
such as obstacle detection or route planning, they often fall short in
delivering detailed descriptions and guidance based on user questions. The
current solutions lack the capability to generate contextually relevant
information about objects, scenes, and potential risks in the environment,
limiting an in-depth understanding of the environment for visually impaired
individuals. This limitation hinders their ability to fully perceive and
understand their surroundings, resulting in reduced independence and increased
reliance on external assistance.

In this work, as shown in Figure 1, we present a novel approach named
VisPercep that leverages the advanced large vision-language model to enhance
visual perception for individuals with blindness and low vision including
scene understanding, object localization, and risk assessment. Our work
addresses the challenges faced by pBLV by providing them with detailed and
comprehensive scene descriptions and risk guidance based on user questions,
enabling an in-depth understanding of their surroundings, locating objects of
interest, and identifying potential risks.

Our system includes three main modules, as illustrated in Figure 2: image
tagging module, prompt engineering module, and vision-language module. The
image tagging module, implemented using Recognize Anything Model (RAM) [33],
recognizes all objects in the captured image. We then integrate the recognized
objects and user questions into a customized prompt designed for visually
impaired individuals through prompt engineering. Finally, the vision-language
model utilizes InstructBLIP [15] to generate detailed and contextually
relevant text, facilitating comprehensive scene understanding, object
recognition, and risk assessment for visually impaired individuals. Our
experiments demonstrate that our system is able to recognize objects of
interest and provide detailed answers to user questions, significantly
enhancing the visual understanding of visually impaired individuals about
their surroundings.

Figure 2:  Overview. Our method includes three key modules: image tagging
module, prompt engineering module and vision-language module. Firstly, the
image tagging module identifies all common objects present in the captured
image. Secondly, using prompt engineering, we integrate the recognized objects
and user queries to create customized prompts tailored for visually impaired
individuals. Lastly, the vision-language module generates detailed and
contextually relevant output text, enabling comprehensive and precise scene
understanding, object localization, and risk assessment for visually impaired
individuals. [The input image is from Visual7W dataset [34]].

##  2 Related Works

Several assistive technologies and applications have been developed to support
individuals with visual impairments in understanding their environments and
enhancing their visual perception [20, 7, 32]. Traditional tools such as white
canes [21] and guide dogs [31] have long been used to aid in mobility and
spatial awareness. Additionally, advancements in technology have led to the
development of various assistive devices, including wearable cameras [10], GPS
navigation systems, and object recognition technologies.

Wearable camera systems, such as OrCam MyEye and Seeing AI [9], offer real-
time text reading and text-to-speech capabilities to provide auditory feedback
to visually impaired individuals. These systems assist in object
identification, text reading, and facial recognition, enhancing their ability
to interact with their surroundings. GPS navigation systems, such as
BlindSquare [14] and Lazarillo [3], utilize location-based services to provide
audio instructions and guidance for navigation in both indoor and outdoor
environments.

Computer vision-based technologies have also been explored for visual
perception enhancement. These include object detection systems using deep
learning models like YOLO [26] and Faster R-CNN [28], which provide real-time
identification of objects in the environment. Detect and Approach [12]
proposes a real-time monocular-based navigation solution based on YOLO for
pBLV. Additionally, vision-language models like VizWiz [2] and SoundScape [1]
incorporate natural language processing to describe visual scenes, answer
questions, and provide context-aware information.

While these existing assistive technologies have made significant
advancements, they still face limitations. Many systems provide partial
solutions focused on specific functionalities such as object recognition or
detection, but often fall short in delivering comprehensive scene
understanding and detailed descriptions. Moreover, these technologies may lack
the ability to provide guidance based on user questions, limiting their
effectiveness in addressing the specific needs and queries of visually
impaired individuals [29]. Furthermore, these technologies often require
multiple devices or interfaces, leading to complexity and decreased usability
for visually impaired individuals [7]. In contrast to these existing
approaches, our proposed method offers a comprehensive and integrated
solution. By combining advanced vision-language models, image tagging, and
prompt engineering, our approach enhances visual perception, provides real-
time guidance, and offers context-aware prompts tailored specifically for
visually impaired individuals.

##  3 Method

Our method aims to overcome the limitations of existing assistive technologies
and empower visually impaired individuals with improved guidance. In section
3.1, we introduce our image tagging module. Section 3.2 illustrates the prompt
engineering tailored specifically for visually impaired individuals. We
explain the large vision-language module in section 3.3.

###  3.1 Image Tagging Module

As shown in the yellow box of Figure 2, the image tagging module is utilized
to generate tags for each oject present in the captured images, which is
crucial as it provides a comprehensive understanding of the visual scene by
accurately recognizing various objects. By incorporating the image tagging
module, we obtain a catalog of objects present in the environment,
facilitating a more precise and comprehensive environment description. We
employ the Recognize Anything Model (RAM) [33] as our image tagging module,
which has demonstrated the zero-shot ability to recognize any common category
with high accuracy.

Specifically, The image tagging module begins with a pre-trained image
encoder, which processes an input image and extracts high-level visual
features. These features capture important characteristics and representations
of the objects in the image. After the initial feature extraction stage, an
attention mechanism [17] is employed to focus on the most salient regions
within the image. This attention mechanism allows the model to pay more
attention to relevant objects and suppress irrelevant ones. Thus, the image
tagging module can generate accurate and informative tags for the recognized
objects. The final stage involves mapping the extracted features to a set of
object categories or tags by the image-tag recognition decoder. This mapping
is learned through a training process that leverages large-scale annotated
datasets, ensuring the model’s ability to generalize to various objects and
scenes. The trained RAM model can then be applied to new images, accurately
recognizing and generating tags for the objects present in the environment.

###  3.2 Prompt Engineering for pBLV

We incorporate prompt engineering, as shown in the green box of Figure 2, to
create customized prompts tailored specifically for visually impaired
individuals. This involves integrating the output of the image tagging module
with user questions to form contextually relevant and informative prompts.
Moreover, the use of prompt engineering eliminates the need for traditional
machine learning approaches that require training models on labeled datasets,
as prompt engineering focuses on generating effective prompts rather than
optimizing model’s parameters.

The RAM generates a set of tags that represent the recognized objects within
the captured images. We utilize these object {tags} to enhance the final
prompt. We include the prompt “The image may contain elements of {tags}” to
seamlessly integrate the object recognition results into an prompt. By
incorporating these recognized object tags into the prompt, we ensure that the
vision-language module receives specific and accurate information about the
objects in their surroundings. This approach significantly enhances the
understanding and awareness of the visual environment for the users.

Furthermore, we consider user questions as a vital input for prompt
engineering. By incorporating user questions into the prompts, we address the
individual’s specific needs for environment understanding and ensure that the
prompts are highly relevant to their current situation. This personalized
approach allows visually impaired individuals to obtain the targeted
information about their environment and the objects of interest. For example,
in the case of risk assessment, we employ a specific prompt that guides the
model to act as an assistant for visually impaired individuals, providing
comprehensive analysis. The prompt we use is “I am visually impaired. You are
the assistant for visually impaired individuals. Your role is to provide
helpful information and assistance based on my query. Your task is to provide
a clear and concise response that addresses my needs effectively. Don’t
mention that I am visually impaired to offend me. Now, please answer my
questions: [user_query]. Your answer should be like a daily conversation with
me.” where [user_query] is the user question. This prompt enables the model to
deliver detailed and accurate explanations regarding potential risks, ensuring
that the information is communicated in a respectful and informative manner.

Input: Image: The captured image

UserQuery: The user question

Output: OutputText: The generated output text

Step 1: Predict Tags

Image $\longrightarrow$ Image Tagging Module $\longrightarrow$ Tags

Step 2: Prompt Engineering for pBLV

Tags + UserQuery $\longrightarrow$ Prompt Engineering for pBLV
$\longrightarrow$ Prompt

Step 3: Generate OutputText

Image + Prompt $\longrightarrow$ Vision-Language Module $\longrightarrow$
OutputText

Algorithm 1 Algorithm of VisPercept  Figure 3: Client-server architecture.

###  3.3 Vision-Language Module

To generate descriptive output text based on the prompts obtained by the
prompt engineering module, we employ InstructBLIP [15], a powerful large
vision language model for comprehensive scene understanding and analysis as
shown in the right blue box of Figure 2.

Specifically, InstructBLIP begins by encoding the input image using the frozen
Vision Transformer (VIT) [5], which captures high-level visual representations
of the image. The input prompt is also encoded as the token by the tokenizer.
Then, the encoded prompt tokens and visual features are fed into the Q-Former
[15] together to generate contextualized image tokens through cross-attention.
Then, a linear projection layer is employed to convert the image tokens into
the tokens that Large Language Model (LLM) can understand. We further utilize
a LLM, i.e., Vicuna-13B [4], to generate the final output text. The LLM
incorporates both the generated image token and the text token from the user
question to generate rich and comprehensive textual descriptions. We
demonstrate the algorithm of our VisPercep in Algorithm 1.

Figure 4: Ablation study with different model settings on Visual7W dataset.

##  4 Experiment

###  4.1 Implementation Details

Our system leverages the capabilities of a smartphone, employing a monocular
phone camera to capture images and the phone’s microphone to receive user
voice questions, creating a seamless interaction between the user and the
system as shown in Figure 3. The image and voice input are then transferred to
our server, where the processing and generation of comprehensive descriptions
take place. To convert the user’s voice question into text for further
processing, we employ Whisper [25], a powerful speech recognition system. This
technology accurately transcribes the user’s voice question into a textual
form, enabling seamless integration with our vision-language model. After the
input text is obtained, our system processes the image and text to generate
detailed and contextually relevant output descriptions. The system selects the
corresponding image frame once the user question is detected, ensuring
accurate and timely responses. The output text is then transformed into audio
format to provide a more accessible experience for visually impaired
individuals. For text-to-speech conversion, we utilize the robust system Azure
[16]. This allows us to transform the output text into clear and natural-
sounding audio. The synthesized audio is then sent from the server to the
user’s phone, enabling real-time delivery of the enhanced visual perception
information. By implementing this client-server architecture and incorporating
speech recognition and synthesis technologies, our system facilitates seamless
interaction between the user and our system.

###  4.2 Tests on Visual7W dataset

####  4.2.1 Dataset Preparation

We evaluate our proposed approach on the Visual7W dataset [34]. Compared with
previous studies that solely relied on textual answers, Visual7W introduces a
novel form of question answering that includes visual answers [34]. This is
achieved by establishing a semantic connection through object-level grounding
between textual descriptions and image regions [34]

We notice that there are strong connections between objects in images, both in
terms of spatial location and meaning of existence. To test our model in
assisting people with visual impairment, we selected some images from specific
perspectives in this dataset. From these perspectives, blind people often
require additional assistance to better understand the current environment.

####  4.2.2 Ablation Study

We conduct ablation studies to verify the effectiveness of the individual
module in our model. The experimental settings are listed in Figure 4 where
“✓” denotes the module is enabled. In the first experimental setting, we only
utilize the vision-language module, which directly sends user questions and
images to InstructBLIP. In the second experimental setting, we employ the
image tagging module to generate tags for the input image, which are then
integrated into the user question. Then, both the modified question and the
input image are fed into the vision-language module. In the third experimental
setting, we employ prompt engineering specifically designed for visually
impaired individuals to further refine the prompt by incorporating the
generated tags and user questions.

As shown in Figure 4, for the left scene, when only using the vision-language
module, the model provides some answers that do not match the facts, such as
“cat sitting inside an open suitcase”, “possibly a table or a shelf” and “such
as a pillow”. These are more likely to be inferred from a large language model
due to model’s bias (learnt from the training data) than what actually exists
in the image. After combining with the image tagging module, the model dropped
answers that does not match the facts in the image and the generated answer
correctly describes the current scenario. You should describe what is the
modified user questions after integrating with the tags. Furthermore, if
prompt engineering for pBLV is applied, answers become more precise e.g., it
also accurately describes the location of the cat. Here you should describe
what is the final prompt generated.

In the case of the right scene, the model that only uses the vision-language
module does provide a detailed description of the scene, but there are still
errors. The description “There are several cars parked on the street, and a
traffic light can be seen in the background.” is inconsistent with the fact
shown in the image, as there is no traffic light and only a white street light
and an orange fire hydrant. When adding the image tagging module, the model
gives a more factual description but lacks details. Again you should provide
the new user question based on the tags In contrast, Prompt Engineering for
pBLV makes the answer more precise and detailed. Again you should provide the
new user question after prompt engineering.

Figure 5: Random selected results of image tagging module on Visual7W dataset.

The example in Figure 4 demonstrates that the integration of the vision-
language module, image tagging module, and prompt engineering yield the most
accurate and detailed descriptions. In Figure 5, we further present some
randomly selected results of the image tagging module. As depicted in the
figure, the module successfully recognizes common objects within the images,
demonstrating its ability to provide a comprehensive understanding of the
visual scene by accurately identifying various objects.

####  4.2.3 Scene Understanding

In this section, we evaluate the effectiveness of our approach on outdoor and
indoor scene understanding. Sample results are shown at the top of Figure 6.
In our experiment, the user’s input is “Can you describe the environment
around?”. For both indoor and outdoor examples, it is evident that the model’s
output provides a comprehensive and accurate description of the object
composition in the environment depicted in the image.

####  4.2.4 Object Localization

Figure 6: Examples of scene understanding (top), object localization (middle),
and risk assessment (bottom) on Visual7W dataset. Figure 7: Examples of scene
understanding, object detection and localization, and risk assessment under
real-world settings.

In this section, we evaluate the effectiveness of our approach in addressing
object recognition challenges, as demonstrated in the middle of Figure 6. The
user question text for this task is “Where is the {giraffe, sheep, bookshelf,
rubbish bin} in the image?”, where “{ }” is what the user wants to find out.

In the outdoor scene, the left image is focused on the giraffe. From the
answer, we can see that the results are very detailed, not only describing the
location of the giraffe on the grass and under the trees, but also providing
information “The giraffes appear to be enjoying the shade provided by the tree
and the lush green environment around them.” for users to better understand
the capture environmental images.

####  4.2.5 Risk Assessment

For risk assessment, as shown at the bottom of Figure 6, our model provides
safety tips for people with visual impairment to help them identify and deal
with potential risks according to the current environment. The question is “Is
there a risk for me to continue moving forward?”.

The first picture depicts a scene where a pedestrian crossing has a red light.
The model can provide feedback to the user regarding the risk of crossing the
street when the traffic signal is red. In the second scene, a train is
approaching, which can be extremely dangerous if proper precautions are not
taken. The model can send an alert that it is risky to cross the railway at
the current time. It demonstrates that our model can effectively analyze risks
and provide necessary alerts for visually impaired individuals.

###  4.3 Real-world Tests

We also conducted experiments to evaluate the proposed system in real-world
situations as shown in Figure 7. Specifically, we simulate the walking process
of visually impaired people. The main content is a real video of a person
walking on the street entering and then exiting a subway station in New York.
Even though this route is easy for ordinary people, it may engender many risks
for people with visual impairment. We captured several characteristic images
from this video and passed them into our model for evaluation. These scenes
are on the street, before entering the station, in the subway station, and
after exiting the station.

The first scene shows a street with a crowd. Moreover, there is a shop in the
left of the image. Our model returns the answer “A crowded shopping street is
filled with people walking and strolling along the pavement.”, which is
consistent with the image content. For the second scene, the user is walking
on a straight and empty street, when the user asks “Is there a risk for me to
continue moving forward?” The model answers that “No, it is not risky for you
to go ahead.”, which is also in line with the actual situation. In the third
scene, when the user asks where the subway gate is, the model provides a very
detailed explanation of the location of the subway gate with directional
adjectives, such as front, back, left, and right. From the answer, it is clear
that there are two gates and they are on left and right sides. In the last
scenario where there is a staircase, and the model reminds the user that there
currently exists a certain level of risk. Due to the presence of stairs, the
ground becomes more difficult for the user to move forward. Therefore, the
model provides this answer “be careful about going up the stairs because they
are narrow”.

##  5 Conclusion

In this paper, we present a pioneering approach that addresses the challenges
faced by people with blindness and low vision (pBLV) in comprehensive scene
understanding, precise object localization, and risk assessment in unfamiliar
environments. By leveraging the large vision language model and integrating it
with an image tagging module, our method provides visually impaired
individuals detailed and comprehensive descriptions and guidance to address
their specific needs based on the user’s questions. [I think you need to say
some thing about Prompt engineering.] We evaluate our approach through
experiments conducted on both indoor and outdoor datasets. Our results
demonstrate that our method is able to recognize objects accurately and
provides insightful descriptions and analysis for visually impaired
individuals.

## References

  * [1] Francesco Aletta, Jian Kang, and Östen Axelsson.  Soundscape descriptors and a conceptual framework for developing predictive soundscape models.  Landscape and Urban Planning, 149:65–74, 2016.
  * [2] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al.  Vizwiz: nearly real-time answers to visual questions.  In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342, 2010.
  * [3] Queiroz Cardoso, Aislan Vieira de Melo, Alline Lam Orué, et al.  Accessibility analysis for the visually impaired using lazarilloapp.
  * [4] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [6] Hugo Fernandes, Paulo Costa, Vitor Filipe, Hugo Paredes, and João Barroso.  A review of assistive spatial orientation and navigation technologies for the visually impaired.  Universal Access in the Information Society, 18(1):155–168, 2019\.
  * [7] Nicholas A Giudice and Gordon E Legge.  Blind navigation and the role of technology.  The engineering handbook of smart technology for aging, disability, and independence, pages 479–500, 2008.
  * [8] GPS.gov.  Gps accuracy.  Official U.S. government information about the Global Positioning System (GPS) and related topics.
  * [9] Christina Granquist, Susan Y Sun, Sandra R Montezuma, Tu M Tran, Rachel Gage, and Gordon E Legge.  Evaluation and comparison of artificial intelligence vision aids: Orcam myeye 1 and seeing ai.  Journal of Visual Impairment & Blindness, 115(4):277–285, 2021\.
  * [10] Tanishq Gupta and Holden Li.  Indoor mapping for smart cities—an affordable approach: Using kinect sensor and zed stereo camera.  In 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pages 1–8. IEEE, 2017.
  * [11] Lilit Hakobyan, Jo Lumsden, Dympna O’Sullivan, and Hannah Bartlett.  Mobile assistive technologies for the visually impaired.  Survey of ophthalmology, 58(6):513–528, 2013.
  * [12] Yu Hao, Junchi Feng, John-Ross Rizzo, Yao Wang, and Yi Fang.  Detect and approach: Close-range navigation support for people with blindness and low vision.  In European Conference on Computer Vision, pages 607–622. Springer, 2022.
  * [13] Vaishnav Kameswaran, Jatin Gupta, Joyojeet Pal, Sile O’Modhrain, Tiffany C Veinot, Robin Brewer, Aakanksha Parameshwar, and Jacki O’Neill.  ’we can go anywhere’ understanding independence through a case study of ride-hailing use by people with visual impairments in metropolitan india.  Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–24, 2018.
  * [14] PV Arul Kumar, J Vivek, N Senniangiri, S Nagarajan, and K Chandrasekaran.  A study of added sic powder in kerosene for the blind square hole machining of cfrp using electrical discharge machining.  Silicon, 14(4):1831–1849, 2022.
  * [15] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [16] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu.  Neural speech synthesis with transformer network.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6706–6713, 2019.
  * [17] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [18] Jack M Loomis, Reginald G Golledge, Roberta L Klatzky, Jon M Speigle, and Jerome Tietz.  Personal guidance system for the visually impaired.  In Proceedings of the first annual ACM conference on Assistive technologies, pages 85–91, 1994.
  * [19] Manfred MacKeben and Donald C Fletcher.  Target search and identification performance in low vision patients.  Investigative Ophthalmology & Visual Science, 52(10):7603–7609, 2011.
  * [20] Daniela Massiceti, Stephen Lloyd Hicks, and Joram Jacob van Rheede.  Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm.  PloS one, 13(7):e0199389, 2018.
  * [21] Troy McDaniel, Sreekar Krishna, Vineeth Balasubramanian, Dirk Colbry, and Sethuraman Panchanathan.  Using a haptic belt to convey non-verbal communication cues during social interactions to individuals who are blind.  In 2008 IEEE international workshop on haptic audio visual environments and games, pages 13–18. IEEE, 2008.
  * [22] Daniel R Montello.  Cognitive research in giscience: Recent achievements and future prospects.  Geography Compass, 3(5):1824–1840, 2009.
  * [23] World Health Organization et al.  Visual impairment and blindness fact sheet n 282.  World Health Organization, 2014.
  * [24] Donatella Pascolini and Silvio Paolo Mariotti.  Global estimates of visual impairment: 2010.  British Journal of Ophthalmology, 96(5):614–618, 2012.
  * [25] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  In International Conference on Machine Learning, pages 28492–28518. PMLR, 2023.
  * [26] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [27] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [29] Uta R Roentgen, Gert Jan Gelderblom, Mathijs Soede, and Luc P De Witte.  Inventory of electronic mobility aids for persons with visual impairments: A literature review.  Journal of Visual Impairment & Blindness, 102(11):702–724, 2008\.
  * [30] Anne M Treisman and Garry Gelade.  A feature-integration theory of attention.  Cognitive psychology, 12(1):97–136, 1980.
  * [31] Lorraine Whitmarsh.  The benefits of guide dog ownership.  Visual impairment research, 7(1):27–42, 2005.
  * [32] Zhongzheng Yuan, Tommy Azzino, Yu Hao, Yixuan Lyu, Haoyang Pei, Alain Boldini, Marco Mezzavilla, Mahya Beheshti, Maurizio Porfiri, Todd E Hudson, et al.  Network-aware 5g edge computing for object detection: Augmenting wearables to “see” more, farther and faster.  IEEE Access, 10:29612–29632, 2022.
  * [33] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al.  Recognize anything: A strong image tagging model.  arXiv preprint arXiv:2306.03514, 2023.
  * [34] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei.  Visual7W: Grounded Question Answering in Images.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016.