Some 2-adic conjectures concerning
polyomino tilings of Aztec diamonds

James Propp, UMass Lowell

August 10, 2022

Dedicated to Michael Larsen on the occasion of his 60th birthday

###### Abstract

For various sets of tiles, we count the ways to tile an Aztec diamond of order
$n$ using tiles from that set. The resulting function $f(n)$ often has
interesting behavior when one looks at $n$ and $f(n)$ modulo powers of 2.

##  1 Introduction

I had a great time working on domino tilings of Aztec diamonds with Noam
Elkies, Greg Kuperberg, and Michael Larsen back in the late 1980s, and the
paper we wrote together [EKLP] had a huge impact on my career. So I’d like to
honor Michael by proposing some new problems about tilings of Aztec diamonds
(and other regions), many of which are more challenging than the one I shared
with him thirty-something years ago and have a number-theoretic slant that I
think he will enjoy. Ideally the solutions to these problems will involve
interesting applications of algebra to combinatorics.

Figure 1: A domino tiling of the Aztec diamond of order 4.

Here is some general background.

The main result of [EKLP] was that the number of domino-tilings of an Aztec
diamond of order $n$ is $2^{n(n+1)/2}$ (A006125), where a domino is a
rectangle in $\mathds{R}^{2}$ of the form $[i,i+1]\times[j,j+2]$ or
$[i,i+2]\times[j,j+1]$ (with $i,j\in\mathds{Z}$) and the Aztec diamond of
order $n$ is the union of the squares $[i,i+1]\times[j,j+1]$ contained within
the region $\\{(x,y):\ |x|+|y|\leq n+1\\}$. Figure 1 shows one of the
$2^{(4)(5)/2}$ domino tilings of the Aztec diamond of order 4.

Mihai Ciucu [Ci] proved combinatorially that the number of domino tilings of
the $2n$-by-$2n$ square (A004003) can be written in the form $2^{n}f(n)^{2}$
where $f(n)$ is the number of domino tilings of the region exemplified for
$n=4$ in Figure 2 (A065072).

Figure 2: Ciucu’s way of halving the 8-by-8 square.

Henry Cohn [Co] proved that the function sending $n$ to $f(n)$ is uniformly
continuous under the 2-adic metric and thus extends to a function defined on
all of $\mathds{Z}$ and indeed all of $\mathds{Z}_{2}$; moreover, he showed
that this extension satisfies

$f(-1-n)=\left\\{\begin{array}[]{rl}f(n)&\mbox{when $n$ is congruent to 0 or 3
(mod 4)},\\\ -f(n)&\mbox{when $n$ is congruent to 1 or 2 (mod
4)}.\end{array}\right.$ (1)

Barkley and Liu [BL] have recently proved results about 2-divisibility for the
number of perfect matchings of a graph, including as a special case the number
of domino tilings of a rectangle, but there is more refined work still to be
done along the lines of Cohn’s paper. For instance, the mod 8 residue of the
number of domino tilings of the $2n$-by-$(2n+2)$ rectangle appears to depend
only on the mod 4 residue of $n$; the same goes for the number of domino
tilings of the $2n$-by-$4n$ rectangle.

In this article we extend the discussion to other sorts of tiles,
specifically, tetrominos. A tetromino is a connected subset of the grid that
is a union of four grid-squares, just as a domino is a union of two grid-
squares. Up to symmetry, there are five kinds of tetrominos: straight
tetrominos, skew tetrominos, L-tetrominos, square tetrominos, and
T-tetrominos. They are shown in Figure 3, preceded by the domino. These six
tiles can be placed on a square grid in 2, 2, 4, 8, 1, and 4 translationally-
inequivalent ways, respectively (where rotations and reflections are
permitted). These are the sorts of tiles considered in this article. (Trominos
– unions of three grid-squares – will be considered elsewhere.)

Figure 3: A domino, a straight tetromino, a skew tetromino, an L-tetromino, a
square tetromino, and a T-tetromino.

##  2 Skew and straight tetrominos

Figure 4: Tiling the Aztec diamond of order 3 with skew and straight
tetrominos.

I’ll start with a warm-up puzzle that’s roughly at the level of a math
olympiad: Prove that an Aztec diamond of order $n$ can be tiled by skew and
straight tetrominos (as shown in Figure 4 for $n=3$) only if $n$ is congruent
to 0 or 3 (mod 4).

The puzzle can be solved using a valuation argument (sometimes called a
generalized coloring argument): one can construct a mapping from the grid-
cells to an appropriate abelian group (a “weight function”) and show that when
$n$ is 1 or 2 (mod 4), the sum of the weights of the tiles can’t equal the sum
of the weights of the region being tiled, where the weight of a tile or a
region being tiled is the sum of the weights of the constituent cells. Readers
who are already familiar with this technique might enjoy the challenge of
attempting to solve the problem purely mentally.

##  3 Dominos and square tetrominos

In this section we use dominos and square tetrominos. Thus an Aztec diamond of
order 1 (better known as the 2-by-2 square) can be tiled in 3 ways: with two
horizontal dominos, two vertical dominos, or a single square tetromino. The
Aztec diamond of order 2 can be tiled in $2^{(2)(3)/2}=8$ ways using dominos,
and can be tiled in an additional 11 ways if one or more square tetrominos are
included, as shown in Figure 5. Thus there are a total of $8+11=19$ ways to
tile an Aztec diamond of order 2 using dominos and square tetrominos.

Figure 5: Tiling the Aztec diamond of order 2 with dominos and at least one
square tetromino.

Define $M(n)$ (with $n\geq 0$) as the number of tilings of the Aztec diamond
of order $n$ using dominos and square tetrominos. This is A356512. Trivially
we have $M(0)=1$ (since the Aztec diamond of order 0 is empty) and we have
already seen that $M(1)=3$ and $M(2)=19$. Figure 6 shows the terms of the
sequence $M(n)$ for $n$ ranging from 0 to 12, computed using a program written
by David desJardins.

The reader may wish to pause here to consider the problem of showing that
$M(n)$ is always odd; a solution will be given in section 5.

$\begin{array}[]{r|r}n&M(n)\\\ \hline\cr 0&1\\\ 1&3\\\ 2&19\\\ 3&293\\\
4&10917\\\ 5&996599\\\ 6&222222039\\\ 7&121552500713\\\ 8&162860556763865\\\
9&535527565429290907\\\ 10&4318205059450240425083\\\
11&85475498697714319842817853\\\
12&4151186175463797888945512144221\end{array}$ Figure 6: Enumeration of
tilings of Aztec diamonds using dominos and square tetrominos.

These numbers grow quadratic-exponentially as a function of $n$, and I have no
conjectural formula for the $n$th term, nor a conjectural recurrence relation
for the sequence, nor any efficient method of computing terms. Nonetheless,
something systematic is going on. I have already mentioned that all the terms
are odd. Taking this observation further, one notices that the numbers’
residues mod 4 are

$1,3,3,1,1,3,3,1,1,3,3,1,1,$

the residues mod 8 are

$1,3,3,5,5,7,7,1,1,3,3,5,5,$

and the residues mod 16 are

$1,3,3,5,5,7,7,9,9,11,11,13,13.$

Conjecture 1: For all $k\geq 1$, the mod $2^{k}$ residue of $M(n)$ is periodic
with period dividing $2^{k}$. That is, $2^{k}$ divides $M(n+2^{k})-M(n)$ for
all $k,n$.

I tried to prove this conjecture by reducing it to an assertion about
alternating-sign matrices but I was unsuccessful.

Note that if the conjecture is true then $M(n)\equiv n+1+(1+(-1)^{n+1})/2$
(mod 8). This congruence might also hold mod 16 but it certainly cannot hold
mod $2^{k}$ for all $k$, since that would require that $M(n)$ actually equals
$n+1+(1+(-1)^{n+1})/2$, which is clearly not the case for $n\geq 2$. And
indeed $M(2)=19\not\equiv 3$ (mod 32).

A deeper consequence of Conjecture 1 is that the function sending $n$ to
$M(n)$ is 2-adically continuous. Moreover, the function appears to satisfy a
kind of symmetry analogous to the functional equation (1) mentioned at the end
of section 1.

Conjecture 2: For all $k\geq 1$, if $n+n^{\prime}\equiv-3$ (mod $2^{k}$) then
$M(n)+M(n^{\prime})\equiv 0$ (mod $2^{k}$).

That is, if one extends $M:\mathds{N}\rightarrow\mathds{N}$ to the 2-adic
function $\widehat{M}:\mathds{Z}_{2}\rightarrow\mathds{Z}_{2}$, one has
$\widehat{M}(-3-n)=-\widehat{M}(n)$.

Although in this article I am limiting myself to discussion of tilings of
Aztec diamonds, I have also looked at tilings of other regions using dominos
and square tetrominos, and the same phenomenon of 2-adic continuity arises
fairly broadly there. For instance, for the $2n$-by-$2n$ square, the
$2n$-by-$(2n+2)$ rectangle, and the $2n$-by-$4n$ rectangle, the number of
tilings with dominos and square tetrominos always seems to be congruent to
$2n+1$ mod 8.

##  4 Skew tetrominos and square tetrominos

In [Pr] I considered tilings of Aztec diamonds by skew tetrominos and square
tetrominos. If we require that all skew tetrominos be horizontal, interesting
numerical patterns appear. (Of course we would get the same result if we
required that all skew tetrominos be vertical.) In this section we allow
horizontal skew tetrominos and square tetrominos as seen in Figure 7, which
depicts all six tilings of the Aztec diamond of order 3 using square
tetrominos and horizontal skew tetrominos.

Figure 7: Tiling the Aztec diamond of order 3 with horizontal skew tetrominos
and square tetrominos. $\begin{array}[]{r|r}n&L(n)\\\ \hline\cr 0&1\\\ 1&1\\\
2&2\\\ 3&6\\\ 4&40\\\ 5&364\\\ 6&7904\\\ 7&226152\\\ 8&15835008\\\
9&1439900880\\\ 10&324189571584\\\ 11&94080051207136\\\
12&68041472016287744\\\ 13&63145927127133361600\\\
14&146637148542938673930240\\\ 15&435697213021432661980535936\end{array}$
Figure 8: Enumeration of tilings of Aztec diamonds using horizontal skew
tetrominos and square tetrominos.

Define $L(n)$ (with $n\geq 0$) as the number of tilings of the Aztec diamond
of order $n$ using horizontal skew tetrominos and square tetrominos. This is
A356513. Trivially we have $L(0)=1$ and $L(1)=1$. Figure 8 shows the terms of
the sequence $L(n)$ for $n$ ranging from 0 to 15, again computed using the
program written by David desJardins.

The sequence grows quadratic-exponentially, and once again, I have no
conjectural formula, but as before there are patterns that call out for
explanation. Noticing that all but the first two terms are even, one might
naturally think to look at the multiplicity of 2 in the prime factorization of
$L(n)$, obtaining the sequence 0,0,1,1,3,2,5,3,7,4,9,5,11,6,13,7,…, which
(once we throw out the initial 0) we recognize as an interspersal of the
arithmetic progressions 0,1,2,3,4,5,6,7,…and 1,3,5,7,9,11,13,….

Conjecture 3: For $n\geq 1$, the multiplicity of 2 in the prime factorization
of $L(n)$ is $n-1$ if $n$ is even and $(n-1)/2$ if $n$ is odd.

Going further, let $L_{0}(m)=L(2m)/2^{2m-1}$ and $L_{1}(m)=L(2m-1)/2^{m-1}$,
so that (if Conjecture 3 holds) $L_{0}(m)$ and $L_{1}(m)$ are odd integers for
all $m$. These two new sequences are shown in Figure 9.

$\begin{array}[]{r|r}m&L_{0}(m)\\\ \hline\cr 1&1\\\ 2&5\\\ 3&247\\\
4&123711\\\ 5&633182757\\\ 6&33223375007953\\\ 7&17900042546745443595\\\
&{}\hfil\end{array}\ \ \ \begin{array}[]{r|r}m&L_{1}(m)\\\ \hline\cr 1&1\\\
2&3\\\ 3&91\\\ 4&28269\\\ 5&89993805\\\ 6&2940001600223\\\
7&986655111361458775\\\ 8&3403884476729942671722937\end{array}$ Figure 9:
Values of $L_{0}(m)$ and $L_{1}(m)$.

The mod 4 residues of the $L_{0}$ sequence go $1,1,3,3,1,1,3$ while those of
the $L_{1}$ sequence go $1,3,3,1,1,3,3,1$. That’s not much evidence to go on,
so perhaps it would be prudent not to make a conjecture, but I choose to be
hopeful.

Conjecture 4: For all $k\geq 0$, the mod $2^{k}$ residue of $L_{0}(m)$ is
periodic with period dividing $2^{k}$. Likewise for $L_{1}(m)$.

We do not observe such patterns in the numbers of tilings when both horizontal
and vertical skew tetrominos are allowed along with square tetrominos as in
[Pr]. More specifically, if we count tilings of Aztec diamonds in which we are
permitted to use all four kinds of skew tetrominos as well as square
tetrominos, the resulting sequence, taken mod 4, goes 1, 0, 0, 0, 0, 0, 2, 2,
0, 0, 0, 0, …; if there is a period here, and it is a power of 2, it must be
at least 16.

The prime $p=2$ appears to be special for the enumerative problems I described
above; looking at the $M$ and $L$ sequences mod 3 or mod 5 yields no
discernible patterns.

##  5 Assorted congruential problems

For each of the sixty-three nonempty subsets of the set of six tiles shown in
Figure 3, we can ask in how many ways it is possible to tile the Aztec diamond
of order $n$ using only tiles from that set, allowing translations, rotations,
and reflections of tiles. These are the enumerative problems considered in
this section.

(One could expand the set of tiling problems by distinguishing between
different orientations of the tiles, as was done in the preceding section
where we permitted horizontal skew tetrominos but forbade vertical skew
tetrominos; since there are $2+2+4+8+1+4=21$ different tiles up to
translation, we would obtain over two million different problems, and even if
we mod out the $2^{21}-1$ problems by a dihedral action of order 8, that is
still too many problems to consider exhaustively. One that appears to be
interesting is discussed at the end of this section.)

In each of the sixty-three cases, I used the aforementioned program to count
the tilings of the Aztec diamond of order $n$, with $n$ going from 1 to 8,
using the allowed tiles. Although no 2-adic continuity phenomena arose from
these experiments, there were definite patterns in the parity, and in a few
cases there were congruence patterns modulo higher powers of 2. Here I will
adopt a six-bit code to represent the sixty-three tiling problems, in which
the six successive bits (from left to right) equal 1 or 0 according to whether
or not dominos, straight tetrominos, skew tetrominos, L-tetrominos, square
tetrominos, and T-tetrominos are allowed. For instance, the case treated in
section 2, in which only straight tetrominos and skew tetrominos are allowed
(see Figure 4), would be assigned the code 011000; the case treated in section
3, in which only dominos and square tetrominos are allowed (see Figure 5),
would be assigned the code 100010; and the case of unconstrained skew and
square tetrominos (briefly discussed in section 4) would be assigned the code
001010.

In one-third of the 63 cases, I observed that for all $n$ between 1 and 8, the
number of tilings of the Aztec diamond of order $n$ is even. These were the
cases associated with the six-bit codes 001001, 001100, 001101, 011001,
011100, 011101, 100001, 100100, 100101, 101000, 101001, 101100, 101101,
110000, 110001, 110100, 110101, 111000, 111001, 111100, and 111101.

Presumably some (perhaps all) of these examples can be resolved by showing
that there are no tilings that are invariant under the full dihedral group,
since in that case all orbits would contain an even number of tilings.

Three of the 21 cases were especially interesting. In case 011100, all terms
were divisible by 8; in case 100001, all terms after the first were divisible
by 8; and in case 110001, all terms were congruent to 2 (mod 4).

There were also four cases in which I observed that the number of tilings of
the Aztec diamond of order $n$ is even for all $n$ between 2 and 8 (with the
number of tilings being the odd number 1 in the case $n=1$). These were the
cases associated with the six-bit codes 001010, 001110, 011010, and 011110.

In the cases 001101, 100001, 100011, and 111000 it appears that the exponent
of 2 in the number of tilings may be going to infinity with $n$, though with
such scant evidence it would be rash to place too much faith in this guess.

Additionally, there is one case in which the number of tilings of the Aztec
diamond of order $n$ is always odd, namely, tilings using only dominos and
square tetrominos. Indeed, if we assign each tiling weight $(-1)^{s}$ where
$s$ is the number of square tetrominos, I claim that the sum of the weights is
1. We can prove this using a sign-reversing involution that scans through the
tiling in some fashion in search of a 2-by-2 block that is tiled either with a
square tetromino or with two vertical dominos and switches between the two
possibilities. The fixed points of this involution are tilings that use only
horizontal dominos, and there is just one of those.

Finally, leaving the small world of the $2^{6}-1$ problems and dipping our toe
into the big world of the $2^{21}-1$ problems, we consider tilings of the
Aztec diamond of order $n$ by dominos and horizontal straight tetrominos. This
is A356523, and begins 1, 2, 11, 209, 12748, 2432209, 1473519065, …. It
appears that the number of tilings is even when $n\equiv 1$ (mod 3) and odd
otherwise; this has been verified for $1\leq n\leq 16$.

##  6 Reduction to perfect matchings

The $L$ sequence from section 4 has an interpretation in terms of perfect
matchings. To see why, suppose we have a tiling of the Aztec diamond of order
$n$ using horizontal skew tetrominos and square tetrominos. Dividing each
tetromino into two horizontal dominos gives us a tiling of the Aztec diamond
by horizontal dominos, but it is easy to see that there is exactly one such
tiling (call it $T$). Hence each tetromino is obtained by gluing together two
dominos in $T$. That is, the tetromino tilings correspond to perfect matchings
in the graph whose vertices correspond to the dominos in $T$ with an edge
joining two vertices if the corresponding dominos form a horizontal skew
tetromino or square tetromino. It is not hard to see that this graph is
similar to the $n$-by-$n$ square except that the diagonal has been “doubled”;
for instance, the right panel of Figure 10 shows the graph for $n=4$.

Figure 10: Deriving the square graph with doubled diagonal.

A similar analysis can be applied to tilings of Aztec diamonds using
horizontal skew tetrominos and horizontal straight tetrominos. In this case
the Aztec diamond splits into two non-interacting halves (top half and bottom
half), each of which can be tiled independently of the other, and the tilings
of either half correspond to perfect matchings of a triangle graph as shown in
Figure 11. Thus the number of such tetromino tilings of the Aztec diamond of
order $n$ is equal to the square of the $n$th term of sequence A071093.
Studying the first 25 terms, I find that the sequence seems to have 2-adic
properties of its own. The largest power of 2 dividing the $n$th term of the
sequence A071093 appears to be $\lfloor n/2\rfloor$, and the 2-free part
appears to satisfies 2-adic continuity: for instance, its value mod 16 seems
to be determined by $n$ mod 16.

Figure 11: Deriving the double triangle graph.

What if we superimpose the two graphs, obtaining the graph shown at the right
half of Figure 12? This is equivalent to tiling an Aztec diamond using
horizontal skew tetrominos, horizontal straight tetrominos, and square
tetrominos. Then, counting the tilings, we obtain the integer sequence 1, 2,
10, 116, 3212, 209152, 32133552, 11631456480, 9922509270288,
19946786274879008, 94492874103638971552, 1054865198752147761744448, …. This is
A356514. It appears that the number of tilings is divisible by $2^{\lfloor
n/2\rfloor}$.

Figure 12: Superimposing Figures 10 and 11.

##  7 Some thoughts

The articles of Lovasz [Lo], Ciucu [Ci], Pachter [Pa], and Barkley and Liu
[BL] give ways to find the largest power of 2 that divides the number of
perfect matchings of a graph. This should provide traction for Conjecture 3,
since we saw in section 6 that the $L$ sequence has an interpretation in terms
of perfect matchings of certain graphs. Graphs of this kind appear in the
paper of Ciucu [Ci]; in particular, his Lemma 1.1 shows that the number of
perfect matchings is divisible by $2^{\lfloor n/2\rfloor}$. By bringing ideas
from Pachter [Pa], one might be able to prove Conjecture 3, as well as some of
the other 2-divisibility conjectures from this article.

The only work I know of that provides detailed 2-adic information about the
2-free part of numbers that count tilings is the work of Cohn [Co]. Cohn’s
approach presupposes the existence of an exact formula (in Cohn’s case, an
explicit product of algebraic integers); perhaps something similar can be done
for perfect matchings of the square graph with doubled diagonal, yielding a
proof of Conjecture 4.

Conjectures 1 and 2 seem harder. The product formula exploited by Cohn was
discovered by Temperley and Fischer [TF] and independently by Kasteleyn [Ka]
at about the same time; those researchers made use of the fact that, just as
determinants and Pfaffians of matrices can be expressed as sums of terms
associated with perfect matchings of the set of rows and columns, one can
conversely express the number of perfect matchings of a planar graph in terms
of the determinant or Pfaffian of an associated matrix. I know of no way of
recast the $M$ sequence as enumerating perfect matchings of graphs. However,
it is easy to recast the $M$ sequence as enumerating perfect matchings of
certain hypergraphs. Can any of the existing notions of hyperdeterminants be
brought to bear? Perhaps a reading of [GKZ] would suggest possible approaches.

Kuperberg’s elegant solution [Ku] to the alternating sign matrix conjecture
exploits the power of the Yang-Baxter equation in statistical mechanics. It’s
possible that tools for analyzing the new problems described in this article
will be found in the existing literature at the interface between algebra and
statistical mechanics.

In any case, inasmuch as Conjectures 1 and 2 are reminiscent of Cohn’s work,
and inasmuch as Cohn’s argument hinges on an exact product formula, one might
hope that an exact formula of some kind can be found for the $M$ sequence.
Such an exact formula would have other uses. In [CEP] and [CLP], Henry Cohn,
Noam Elkies, Michael Larsen and I used exact enumeration results to prove
concentration theorems for random tilings. One might hope that the curious
2-adic phenomena discussed in this article hint at the existence of algebraic
machinery that could be applied to the task of showing us what random tilings
associated with Conjecture 1 look like in the limit as size goes to infinity.
Preliminary experiments suggest that there is a “frozen region” near the
boundary, but I have no idea how far into the interior it extends.

I thank David desJardins for the software that made this research possible,
and Noam Elkies for helpful comments. Above all I thank Michael Larsen for his
many years of friendship and mathematical camaraderie.

## References

  * [BL] Grant T. Barkley and Ricky Ini Liu, _Channels, billiards, and perfect matching 2-divisibility_ , Electronic Journal of Combinatorics 28(2) (2021), #P2.51.
  * [Ci] Mihai Ciucu, _Enumeration of perfect matchings in graphs with reflective symmetry_ , Journal of Combinatorial Theory Series A 77 (1997), 67–97.
  * [Co] Henry Cohn, _2-adic behavior of numbers of domino tilings_ , Electronic Journal of Combinatorics 6 (1999), #R14.
  * [CEP] Henry Cohn, Noam Elkies, and James Propp, _Local statistics for random domino tilings of the Aztec diamond_ , Duke Mathematical Journal 85 (1996), 117–166.
  * [CLP] Henry Cohn, Michael Larsen, and James Propp, _The shape of a typical boxed plane partition_ , New York Journal of Mathematics 4 (1998), 137–165.
  * [EKLP] Noam Elkies, Greg Kuperberg, Michael Larsen, and James Propp, _Alternating sign matrices and domino tilings_ , Journal of Algebraic Combinatorics 1 (1992), 111–132, 219–234.
  * [GKZ] I. M. Gelfand, M. M. Kapranov, and A. V. Zelevinsky, Discriminants, Resultants and Multidimensional Determinants, 1994\.
  * [Ka] P. W. Kasteleyn, _The statistics of dimers on a lattice, I. The number of dimer arrangements on a quadratic lattice_ , Physica 27 (1961), 1209–1225.
  * [Ku] Greg Kuperberg, _Another proof of the alternating sign matrix conjecture_ , International Mathematics Research Notices, 1996(3) (1996), 139–150.
  * [Lo] László Lova’sz, Combinatorial Problems and Exercises, North-Holland Publishing Company, Amsterdam, 1979.
  * [Pa] Lior Pachter, _Combinatorial approaches and conjectures for 2-divisibility problems concerning domino tilings of polyominoes_ , Electronic Journal of Combinatorics 4 (1997), #R29.
  * [Pr] James Propp, _A Pedestrian Approach To a Method of Conway, or, A Tale of Two Cities_ , Mathematics Magazine 70 (1997), 327–340.
  * [TF] H. N. V. Temperley and Michael E. Fisher, _Dimer problem in statistical mechanics-an exact result_ , Philosophical Magazine 6:68 (1961), 1061–1063.


# Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-
Specific Imaging

Tzofi Klinghoffer${}^{\textrm{*}}$, Siddharth Somasundaram${}^{\textrm{*}}$,
Kushagra Tiwary${}^{\textrm{*}}$, Ramesh Raskar ${}^{\textrm{*}}$ Equal
contribution (alphabetical order)Each author is with Massachusetts Institute
of Technology

###### Abstract

Cameras were originally designed using physics-based heuristics to capture
aesthetic images. In recent years, there has been a transformation in camera
design from being purely physics-driven to increasingly data-driven and task-
specific. In this paper, we present a framework to understand the building
blocks of this nascent field of _end-to-end design_ of camera hardware and
algorithms. As part of this framework, we show how methods that exploit both
physics and data have become prevalent in imaging and computer vision,
underscoring a key trend that will continue to dominate the future of task-
specific camera design. Finally, we share current barriers to progress in end-
to-end design, and hypothesize how these barriers can be overcome.

###### Index Terms:

Camera Design, Computational Imaging, Perception, Computer Vision, Machine
Learning

##  1 Introduction

Early advances in imaging were inspired by human vision. For example, the
Bayer filter was inspired by human retinal sensitivity to red, green, and blue
light. Such insights from human vision have led to camera designs that have
yielded remarkable digital photography capabilities and have widespread
consumer applications.

The goals of _imaging_ have since shifted from photography to solving tasks
such as 3D shape reconstruction, phase estimation, and material estimation.
These tasks rely on information beyond what the human eye can directly
measure, so it no longer makes sense to design imaging systems based on the
eye. Much like the evolution of animal vision, camera design has evolved to
adapt to the needs of the task and environment [1]. By using known physics of
light-matter interactions, physical cues such as polarization, interference,
and spectrum are exploited to encode task-relevant information. Measurements
of these cues can then be decoded into the scene parameter of interest by
solving a model inversion problem. This idea of jointly exploiting physical
cues and computation is the premise of the field of _computational imaging_.

Whereas imaging deals with capturing image representations of the world,
_computer vision_ extracts meaningful information from these images for high-
level tasks, such as classification, detection, and segmentation. The modern
era of computer vision was ushered in by advances in sensors, computing, and
algorithms. High-resolution sensors paved the way to megapixel resolution,
computing systems provided the bandwidth needed to process high-dimensional
data, and deep learning provided a framework to learn from large amounts of
data.

Figure 1: We map inverse problems in visual perception by how they parametrize
$\mathcal{F}$. Deep learning has focused on learning priors through data-
driven methods, whereas classical computer vision, optics, and computational
imaging have relied on physics. We anticipate future imaging systems will use
physics and data for joint optimization (green box).

Together, these two steps, imaging and vision, form the visual perception
pipeline. The imaging step can be interpreted as a physical encoder,
transforming the 3D scene into a lower-dimensional representation (i.e.
image), and the vision step as a digital decoder, using the representation to
perform high-level tasks, as shown in Figure 4. While perception is more
closely linked with vision, imaging has a downstream effect on perception that
has historically been ignored. Recent works in _end-to-end design_ have
brought together the design of the physical encoder (i.e. camera) and digital
decoder (i.e. neural network) for task-specific perception.

We envision design of future imaging systems to be done in an _end-to-end_
manner. Consider the problem of face recognition. A computational imaging
approach would enable acquisition of a robust representation of the face (e.g.
3D shape and color from stereo), but inherently cannot perform face
recognition. Vision approaches can learn from data to recognize faces, but
will perform poorly in edge cases (e.g. low light). Intelligent and integrated
design of both the imaging and vision steps can enable acquisition of robust
image representations that lead to high-accuracy face recognition.

We discuss two major trends in this paper:

  1. 1.

Visual perception is being solved in an _end-to-end_ , task-specific manner,
with the imaging hardware jointly optimized with the vision algorithm.

  2. 2.

Imaging, while traditionally physics-based, has recently become increasingly
data-driven to perform optimally on high-level vision tasks. Similarly, deep
learning, while typically data-driven, has begun to incorporate physical
models to drive advances in areas such as neural rendering [2].

Tasks-specific cameras that are jointly designed with vision algorithms will
continue to be a theme in the next generation of visual perception systems.

###  1.1 Contributions

  * •

We formulate a modular framework to understand the foundations of _end-to-end
design_.

  * •

A toolkit to design end-to-end imaging systems based on constraints on how
much physics and data are known or available.

  * •

We define five ingredients for designing end-to-end imaging systems. We also
identify open challenges and hypothesize solutions along each axis.

##  2 Optical Camera Design

The most primitive camera is arguably the pinhole camera. A pinhole camera
consists of a tiny aperture placed in front of the image plane to map a single
scene point to a single image point, based on ray optics. In practice,
diffraction effects and SNR limitations of a tiny aperture cause us to use
lenses to focus light. More complex multi-lens systems enable extended depth
of field and correction of optical aberrations. These imaging systems were
derived from fundamental ideas in optics. In this section, we discuss
mathematical- and simulation-based methods for designing optical imaging
systems.

###  2.1 Inverse Optics

The initial goal of camera design was to efficiently map light coming from one
point in a scene to one point on the image sensor using compound optics.
Engineers design optical systems with a desired input-output pair by combining
a sequence of optical components. They may leverage insights from first order
optics using approximations such as thin lenses, paraxial approximations, and
plane/spherical wave models. They then verify the design by using commercial
raytracing software such as Zemax [3] or Code V [4]. However, such design is
often a process of trial and error.

###  2.2 PSF Engineering

Figure 2: Ingredients and open challenges for End-to-End Methods. We present
five ingredients that we consider most important when designing end-to-end
imaging systems, and evaluate how each field (computer vision, computational
imaging, optics, current end-to-end methods) takes advantage of these
ingredients. We anticipate future imaging systems to make more effective use
of the five ingredients from the fields to span the entire radar chart. We
discuss their progress and challenges in Section 6.1.

The point spread function (PSF) of an imaging system is its response to a
point source of light, or the 2D spatial impulse response. A measured image
$y$ can be expressed as

$y=f(x*h+\eta),$ (1)

Figure 3: 4f system: a simple example of optical signal processing. The mask
at the Fourier plane is a pinhole here, which means the system behaves as a
low-pass filter.

where $x$ is the actual scene, $f(\cdot)$ is the camera response function, $*$
is the convolution operator, and $\eta$ is read noise. $x$ can be estimated by
solving the deconvolution problem in the Fourier domain. However, under
certain conditions, the deconvolution problem is ill-posed and certain
assumptions must be employed. Carefully designing the PSF (i.e. PSF
engineering) can numerically guarantee more robust reconstruction of $x$. PSF
engineering is the process of wavefront coding, in which the electromagnetic
field distribution at the aperture plane is modified in a known way to encode
information about the scene. $h$ is physically constructed by refractive
elements, phase masks, and amplitude masks.

PSF engineering has applications in microscopy for single-particle tracking
[5, 6, 7]. It has also been employed for extended depth of field [8]. We will
discuss approaches that enable optimization of the PSF. There are works that
optimize the PSF using deep learning [9], but we restrict our discussion to
analytical methods in this section.

####  2.2.1 Fourier Domain Approaches

One of the simplest examples of PSF engineering is the manipulation the
optical signal in the 2D spatial frequency domain. This manipulation can be
done using a 4f optical system, as shown in Fig. 3. A lens takes a 2D Fourier
transform of the incident beam. At the Fourier plane, an amplitude (or phase)
mask can be placed to manipulate the relative intensities (or phases) of
different frequency components. Another lens takes the inverse Fourier
transform of the beam before it is fed to the sensor. In this case, the mask
at the Fourier plane determines the optical transfer function (OTF), which is
the Fourier Transform of the PSF. The 4f system is frequently used for optical
signal filtering.

####  2.2.2 Information Theory Approaches

The goal of imaging is to obtain some scene property, $x$, from a set of
observations, $y$. Doing so requires us to design an ”optically informative”
PSF. Fisher information and statistical information theory give us a method of
doing so. Fisher information gives us a way to quantify the information
content of a PSF by measuring the sensitivity of $y$ to changes in $x$, given
a PSF. The Fisher information matrix can be used to compute Cramér Rao lower
bounds on the variance of the estimation. This method has been applied in
microscopy for single particle tracking [10], fluorescence imaging beyond the
diffraction limit [11], and depth estimation [12].

####  2.2.3 Wigner Distribution and Ambiguity Function

The space-bandwidth product theorem states that the product of the moduli of a
function and its Fourier transform must be greater than or equal to some
constant. In ray optics, this theorem translates to the idea that there will
exist a tradeoff between the accuracy in measuring the position and direction
of a ray. The Wigner distribution enables a visualization of the Fourier
conjugate variables: complex electric field amplitude and spatial frequency
($k$ vector). The ambiguity function is the Fourier dual of the Wigner
distribution. For a more detailed explanation of the Wigner distribution, the
interested reader is directed to [13, 14]. Horstmeyer _et al._ use the
ambiguity function to design an optimal depth-dependent PSF [15].

##  3 Computational Imaging

[RGB]0,0,0

|  [RGB]250,175,175Space |  [RGB]250,175,175Time |  [RGB]250,175,175Angle |  [RGB]250,175,175Spectrum |  [RGB]250,175,175Polarization
---|---|---|---|---|---
[RGB]175,250,175Coding with |  |  |  |  |
Illumination |  [RGB]250,250,175 Direct-Global |  |  |  |
Separation [16] |  |  |  |  |
Photometric Stereo [17] |  [RGB]250,250,175 Light Transport |  |  |  |
[18, 19] |  |  |  |  |
Multipath |  |  |  |  |
Interference [20] |  |  |  |  |
Fluorescence |  |  |  |  |
Imaging [21] |  [RGB]250,250,175 Refraction |  |  |  |
Measurement [22] |  [RGB]250,250,175 Multispectral |  |  |  |
Imaging[23] |  |  |  |  |
Low-Light |  |  |  |  |
[RGB]250,250,175 Photography [24] |  [RGB]250,250,175 Descattering [25] |  |  |  |
[RGB]175,250,175Coding with |  |  |  |  |
Optics |  [RGB]250,250,175 HDR Imaging [26] |  |  |  |
[RGB]250,250,175 Single-Shot Depth |  |  |  |  |
[RGB]250,250,175 [27, 28, 29] |  [RGB]250,250,175 Time Stretch |  |  |  |
Imaging |  |  |  |  |
Motion |  |  |  |  |
Deblurring [30] |  [RGB]250,250,175 Digital |  |  |  |
Refocusing [31] |  |  |  |  |
Novel View |  |  |  |  |
Estimation [32] |  [RGB]250,250,175 Multispectral |  |  |  |
Imaging |  |  |  |  |
[33, 34, 35, 36, 37] |  [RGB]250,250,175 Shape Estimation |  |  |  |
[38, 39] |  |  |  |  |
Stokes Imaging [40] |  |  |  |  |
Light Transport |  |  |  |  |
Decomposition [41, 42] |  |  |  |  |
[RGB]175,250,175Coding with |  |  |  |  |
Sensors |  [RGB]250,250,175 Stereo Vision [43] |  |  |  |
HDR Imaging [44] |  |  |  |  |
Gradient Camera [45] |  [RGB]250,250,175 Depth Estimation (ToF) |  |  |  |
NLOS Imaging [46] |  |  |  |  |
Imaging Through |  |  |  |  |
Scattering [47] |  [RGB]250,250,175 [RGB]250,250,175 Wavefront |  |  |  |
Sensing [48] |  [RGB]250,250,175 Spatio-Spectral |  |  |  |
Superresolution [49] |  |  |  |  |
Seeing Occluded |  |  |  |  |
Objects [50, 51] |  [RGB]250,250,175 3D Imaging [39] |  |  |  |

Table I: A classification of computational imaging problems based on how
physics is encoded. Note that some works can fall into multiple categories.
This is not an exhaustive list, but is meant to illustrate some key examples.

Light is well described by the _plenoptic dimensions_ : space, time, angle,
and spectrum. The goal of plenoptic imaging is to capture information about
each of these dimensions, but in practice, most cameras capture only a subset
of this information for a given application [52]. For example, the intensity
measured by a standard intensity camera can be expressed by the following
integral

$I(\mathbf{r})=\int
I(\mathbf{r},\mathbf{\Theta},t,\lambda)\,d\mathbf{\Theta}\,dt\,d\lambda,$ (2)

where $\mathbf{r}=(x,y)$ is the spatial position,
$\mathbf{\Theta}=(\theta,\phi)$ is the angle of incident light (i.e. light
field), $t$ is the time of arrival of the light, and $\lambda$ is the
wavelength. The plenoptic dimensions are commonly used in graphics because
they enable a convenient representation of light in ray space. Additional wave
properties, such as polarization, coherence, and phase, are also frequently
used outside of graphics.

Each dimension of light (space, time, angle, spectrum, polarization, phase,
etc.) provides different information about the scene. For example, the time of
flight (ToF) can be used to estimate depth, spectrum is strongly tied to
material properties, polarization has fundamental relationships to shape and
texture, and light field probing is tied to image focusing and depth of field.

_Computational imaging_ is the co-design of imaging hardware and
reconstruction algorithms for a vision application. The hardware is used to
encode information about the scene in an image measurement. The algorithm is
then designed to decode this scene information. We direct the interested
reader to Bhandari _et al._ for a more comprehensive treatment of
computational imaging [53].

A general forward imaging model is described in Fig. 4. A light source first
illuminates the scene. The scene scatters light back to the imaging system,
where the light is optically transformed by a configuration of optical
elements. Then, light is measured by a sensor, which digitizes analog values
into a digital image. Finally, the image is processed by an algorithm to
provide perceptual information about the scene. From this, we see there are
four degrees of freedom when designing computational imaging systems:
illumination, optics, sensor, and algorithms. We discuss some examples of each
degree of freedom in this section. We categorize each example in Table 1.

###  3.1 Coding with Illumination

Active illumination is the process of illuminating the scene with a known and
controlled pattern of light, as opposed to passive illumination (i.e. ambient
light). Active illumination enables more robust reconstruction of scene
properties and, in some cases, enables faster image acquisition because light
sources can be dynamically manipulated more easily than optical elements in
front of the sensor.

Nayar _et al._ use a high-frequency checkerboard pattern to decompose the
light transport of a scene into direct and global components [16]. Park _et
al._ acquire multispectral images by capturing multiple images with different
multiplexed spectral sources [23]. Krishnan and Fergus use flash at near-
infrared wavelengths for low-light photography and obtain reconstructions at
visible wavelengths using spatio-spectral correlations [24]. Wetzstein _et
al._ measure the position and angle of light rays when probed with a light
field to measure materials with varying refractive fields [22]. Kadambi _et
al._ use custom temporal codes for robust depth measurements in the presence
of multi-path interference [20]. Bhandari _et al._ use time-coded illumination
to estimate fluorescence lifetime with continuous wave ToF sensors [21].

_Photometric stereo_ is a technique to obtain the surface normals of an object
under controlled lighting [17]. The object is illuminated sequentially with
point sources from multiple lighting directions and the camera in a fixed
position. Using shading cues and a diffuse surface approximation, photometric
stereo fits a Lambertian model to intensity measurements. Tanaka _et al._ use
transient profiles of thermal light to extract diffuse reflectance of objects
and perform photometric stereo, leveraging that the propagation speed of heat
is resolvable at video frame rates [18].

One way to image through _scattering media_ is by illuminating with modulated
light. Light that returns to the sensor in the same modulated state is
considered unscattered. For example, if light is linearly polarized and
emitted at a scene immersed in a scattering media, the sensor would only
measure light arriving at the linear polarization angle with strongest
intensity, and reject all other light (i.e. polarization gating) [25]. The
same principle can also be applied with coherent light through scattering
media, where only coherent light is measured (i.e. coherence gating). These
principles are based on the idea that scattered light reduces degree of
polarization and coherence. ToF imaging can also be used (i.e. time gating)
since unscattered light (i.e. ballistic photons) has the shortest pathlength,
meaning it arrives at the sensor first [19].

###  3.2 Coding with Optics

Optical coding refers to optically transforming light arriving from the scene
before it reaches the sensor. This transformation can be performed by optical
components such as refractive elements (e.g. lenses), phase and amplitude
masks, and color filter arrays.

A classic example of optical coding is the Bayer filter. By creating a mosaic
of red, green, and blue pixels, an RGB image can be estimated by interpolation
(i.e. demosaicking). Similarly, a multispectral image can be obtained by
placing a multispectral filter array in front of the sensor, instead of a
color filter array [33, 34]. A multispectral image can also be obtained by
placing diffraction gratings [35], prisms [36], or compound optics [37] in
front of the sensor. In a similar spirit to the Bayer filter, full Stokes
imaging can be performed by using a micropolarimeter, where micropolarizers at
different linear polarization orientations are placed in front of each pixel
[40]. Ng _et al._ place a microlens array in front of a camera to capture a 4D
light field, which enables digital refocusing of an image [31]. It is also
possible to generate novel views using a 4D light field [32].

As discussed earlier, a well-chosen PSF can make the deconvolution problem
well-posed. Raskar _et al._ solve the problem of motion blurring in video
cameras by using a coded exposure [30]. A coded aperture is an array of
spatially varying opaque and transparent materials that modulate the phase and
amplitude of incident light. Coded apertures were first deployed in X-ray
imaging to provide a computational solution to forming X-rays, since X-rays
cannot be focused with lenses like light [54]. Since then, coded apertures
have been applied to solve problems, such as single-shot depth [27, 28, 29],
high dynamic range (HDR) imaging [26], and 4D light field acquisition [55].
Some methods place a diffuser in front of a sensor for 3D reconstruction [56],
hyperspectral imaging [57], and microscopy [58].

The light transport of specular vs. diffuse light [41] and real vs. virtual
images (caused by transparent materials) [42] can be decomposed using
polarization filters. Intuitively, this decomposition is based on the idea of
cross-polarization, where the undesirable reflectance component (e.g. specular
reflection) is highly polarized and can be filtered out by placing a linear
polarizer in front of the sensor at an orientation orthogonal to that
polarization state. Polarization filters can also be used for measuring shape
of objects, using known models of Fresnel reflection [59, 39, 38].

###  3.3 Coding with Sensors

The choice of sensor dictates what type of information can be captured in a
scene. For example, a single photon avalanche detector (SPAD) or streak camera
can be used to capture transients on the order of picoseconds. These time
profiles can be used to capture light-in-flight videos, measure ToF, and
analyze transient decays of fluorescent samples. Velten _et al._ use ToF
information and tomographic reconstruction for non-line-of-sight (NLOS)
imaging. Other approaches to NLOS use thermal light emanating from occluded
humans [51] and RF antennas to image through the occluder [50]. Satat _et al._
use ToF information to image through volumetric scattering by modeling
scattering as a probabilistic spatio-temporal convolution [47]. A Shack-
Hartmann sensor can be used to measure the shape (i.e. phase distribution) of
a wavefront [48].

Combining measurements from different sensing modalities, or _sensor fusion_ ,
has also shown great promise for different applications. Stereo vision uses
two or more spatially offset RGB cameras that observe the same scene to obtain
binocular cues, which are used for depth estimation [43]. Kadambi _et al._
combine depth maps from a Kinect sensor with polarization images to obtain
textures at micron scale [39]. The idea is that depth maps are noisy, but
polarization images alone can result in ambiguous geometries. Combining both
modalities overcomes the individual weaknesses of each sensing modality.
Similarly, Cao _et al._ combine a high spatial resolution RGB image with a low
spatial resolution multispectral image to obtain high spatial resolution
multispectral images [49]. Ferstl _et al._ use high spatial resolution
intensity images to upsample low spatial resolution depth maps obtained from a
Kinect sensor to estimate high resolution depth maps [60].

Manipulating backend electronics of the sensor also has interesting
applications. Bhandari _et al._ show that modulo sampling at the sensor level
enables reconstruction of HDR images [44]. Tumblin _et al._ manipulate the
sensor to send the logarithm of the gradients of intensity to the analog-to-
digital (ADC) converter (instead of intensity), which has applications for
downstream tasks and HDR imaging [45].

Figure 4: Modular formulation for end-to-end optimization of the imaging and
vision pipeline. Components can be selected for each numbered module and then
jointly optimized. Existing end-to-end methods optimize modules 2-4 with 5
(described in Section 5). The early parts of the image formation pipeline are
implemented in hardware and encode the scene into RAW measurements (_physical
encoder_). Then, the _digital decoder_ produces the desired output, e.g. HDR
image, classification, etc. With this proposed formulation, end-to-end systems
can be created that use novel, non-obvious combinations of computer vision,
computational imaging, and optics that are jointly optimized to solve
downstream tasks.

###  3.4 Forward Model Inversion

An imaging system is typically modeled by a forward operator
$\mathcal{F}(\cdot)$, such that the measured image, $y$, can be related to a
scene parameter, $x$, by $y=\mathcal{F}(x)+\eta$. Solutions for $x$ can be
found by solving an optimization problem of the form

$\hat{x}=\operatorname*{arg\,min}_{x}\\{||y-\mathcal{F}(x)||_{2}^{2}+\alpha\phi(x)\\},$
(3)

where $\phi(x)$ is a regularization term and $\alpha$ is the weight. If
$\mathcal{F}$ is differentiable, we can solve for $x$ using gradient descent.
This is the premise of _differentiable optics_ and _differentiable rendering_.
For a tutorial on classic optimization methods, we refer the reader to Boyd
and Vandenberghe [61]. Other methods use domain-specific algorithms or neural
networks. We discuss these ideas in the following sections.

##  4 Physics-Based Computer Vision

Physics-based vision incorporates physical priors into otherwise data-driven
deep learning methods, often by solving problems similar to forward model
inversion, but through the use of a neural network. Deep learning approaches
have largely dominated the computer vision community over the last decade, and
involve the use of large amounts of data to iteratively learn an algorithm
with backpropogation and gradient descent. In recent years, the incorporation
of more _physics_ in deep learning vision models has unleashed new
breakthroughs, such as in the area of neural rendering. Physics can be
incorporated into deep learning vision models in several ways, as discussed in
the rest of this section.

###  4.1 Physics-Based Learning

Physics-based learning (PBL) is a group of methods that combine the use of
physical priors with data-driven neural networks. We adopt the categorization
of PBL methods, as proposed by [62], into physical fusion, residual physics,
embedded physics, and physical regularization.

_Physical fusion_ models use the output of physical models as input to neural
networks, which are then optimized for a downstream task, and have been
employed in applications such as shape from polarization [59]. _Residual
physics_ , on the other hand, uses the output of a physical model as ground
truth to guide the training of a neural network. Applications of residual
physics include training a neural network to create CT reconstructions from
limited views, using filtered back projection (FBP) to generate ground truth
as done by Jin et al. [63]. _Embedded physics_ deals with methods in which the
neural network is tasked with learning the parameters of a physical model,
such as in algorithm unrolling, where each layer corresponds with an iteration
of an iterative model [64]. Lastly, we explore _physics regularization_ in
more depth in Section 4.1.1, as it is used for a variety of vision tasks.

Additional PBL methods have also been proposed that incorporate physics into
deep learning into other ways, such as using simulated data from a physics
model to pre-train a neural network for better weight initialization [65, 66].

####  4.1.1 Physics Regularization

A loss function consists of a primary term and regularization term(s). Physics
regularization uses known physics of the scene parameter $x$ and enforces this
as an additional term in the loss function. In such a setup, a neural network
is often used to predict physical parameters from input data, such as images.
A loss can then be applied on the predicted parameters, or they can be used by
the physical model to create additional output(s). For example, in the work of
Che et al. [67], the physical scene parameters necessary to reconstruct an
image with a renderer, such as shape, material, and illumination, are
predicted and compared to known ground truth parameter values. In general,
access to ground truth parameters of a physical model may not be available,
motivating methods that predict parameters of a physical model and enforce
their accuracy only through the final output, such as image reconstruction.
These methods are commonly used for inverse and neural rendering, discussed
further in Section 4.3. Physics regularization has also been used for areas
such as image dehazing by estimating parameters of the atmospheric scattering
model [68], deblurring [69], and depth prediction [70], to name a few.

###  4.2 Differentiable Rendering

Rendering in graphics uses known models of the world, including scene
geometry, illumination models, material properties, etc., to render the scene
from the camera’s perspective. The field of inverse graphics aims to recover
geometry, reflectance, material properties and illumination from images of the
scene. _Differentiable rendering_ stems from the field of inverse graphics,
where the forward model is made differentiable, thus enabling the scene
parameters to be recovered by computing gradients and performing gradient
descent. OpenDR [71] is one of the earliest implementations of this idea.
Although it assumes a simple lighting model and cannot render complex effects
such as inter-reflections, OpenDR paved the way for an entire class of
differentiable renderers, such as SoftRas [72], Mitsuba2 [73], PyRedner [74],
among others [75] [76], which can handle ever more complex forward light
simulations. While differentiable rendering can be used in isolation, the
framework can also work alongside neural networks. The machine learning
component typically predicts the scene parameters such as geometry [77],
textures, or diffuse and specular reflections. More recently, neural networks
have been used to encode the scene directly. We discuss this transition to
using more data and machine learning in the following section.

###  4.3 Neural Rendering

Neural rendering refers to the field that emerges from the use of neural
networks to learn scene parameters, which can then be used generate photo-
realistic imagery in a controllable way [78]. 2D neural rendering often relies
on statistical approaches for image synthesis, such as deep generative models,
in contrast to graphics frameworks that traditionally rely on light transport
and the rendering equation to render images [2] [79]. Approaches that rely on
deep generative models typically use generative adversarial networks (GANs)
[80], variational auto-encoders (VAEs) [81], or diffusion models [82] to learn
a distribution by training for image-to-image translation. Therefore, these
methods are data-driven and do not exploit physics. Recent 2D neural rendering
techniques have merge physically inspired modules into data-driven methods by
using techniques from computer graphics, such as illumination models [83],
textures [84] and multi-view constraints [85] [86], among others. Exploiting
physics-based priors gives greater control over image synthesis over purely
data-driven methods, by allowing light, geometry, and camera to be
disentangled. For a comprehensive review for 2D neural rendering, we refer the
reader to [2].

The field of 3D neural rendering has also seen tremendous growth primarily due
to the use of differentiable renderers discussed in Section 4.2. 3D neural
rendering aims to learn and render a 3D neural scene representation from real-
world imagery and relies on the image formation model by leveraging techniques
from computer graphics. State-of-the-art techniques parameterize the scene
with a neural network using a differentiable renderer for novel-view
synthesis. Typically, volumetric rendering is used for the differentiable
rendering component as its continuous representation has been shown to work
well with gradient descent [76] [87], although there are a plethora of
techniques that use different scene representations and rendering techniques
[78, 2]. This physics and machine learning framework has been highly effective
and subsequent works have added additional physics-based priors such as
reflectance models [88], [89], normal estimation [90], and shadow models [91]
to enable better novel-view synthesis and 3D reconstruction. Moreover, these
physics-based priors are now also used to train on classical computer vision
tasks, such as object classification and segmentation, and show improved
performance over purely data-driven techniques [92] [93]. Neural de-rendering
has also been used for unsupervised representation learning, and has led to
improved downstream accuracy over purely data-driven methods [94].

##  5 Joint Optimization of Optics & Algorithm

Imaging systems have typically been designed independently of the downstream
perception task that they seek to solve. _Joint optimization_ , also known as
_end-to-end optimization_ or _deep optics_ , seeks to address this by jointly
optimizing optics and image processing together for either low-level imaging
or high-level vision tasks. Deep optics has been applied to low-level
problems, such as color imaging and demosaicking [95], extended depth of field
and superresolution imaging [96], high dynamic range (HDR) imaging [97], and
depth estimation [98, 99, 100], and high level problems, such as
classification [101] and object detection [102]. Deep optics has also been
used in time of flight imaging [103, 104, 105], computational microscopy [106,
107, 108], and imaging through scattering [109]. Existing work primarily
focuses on optimizing the parameters of the optical element, sensor, and image
signal processor (ISP), each of which can be considered a core block of the
imaging pipeline and can be filled with different components in a plug-and-
play manner. Optimization of each block is described in the following
sections.

###  5.1 Optical Element Optimization

The first component of the imaging pipeline that is considered for end-to-end
optimization is the optical element. Advances in optimization and
autodifferentiation have enabled the optics parameters of imaging systems to
be optimized for downstream imaging and perception tasks. Sitzmann _et al._
[96] propose the joint optimization of the parameters of an imaging system’s
optical element and reconstruction algorithm by defining a fully
differentiable wave optics image formation model. They experiment with
optimizing a Zernike parameterization and Fourier coefficient parameterization
of a lens using gradient-based optimization in simulation. Once optimized in
simulation, the optical element is manufactured and results are validated in
the real world. While the work of Sitzmann _et al._ [96] is applied to
learning optimal parameters for the tasks of achromatic extended depth of
field and super-resolution imaging, the proposed framework is task-agnostic
and can be extended to optimize for other downstream tasks. [97] extend this
method to optimize optics for high dynamic range (HDR) imaging, while [98]
extend this method to optimize optics for depth estimation. Ikoma et al. [110]
optimize the aperture design for monocular depth estimation. Tseng et al.
[111] optimize the compound optics, i.e. parameters of multiple lenses, along
with the ISP and downstream neural network, in a fully end-to-end pipeline.
After optimal lens parameters are learned in simulation, the lens can be
manufactured and used in production systems.

Rather than optimize an optical element, other methods replace the optical
elements altogether and instead optimize a phase mask, such as in the work of
Wu _et al._ [28]. The phase mask is manufactured with photolithography, and is
used to change the phase of light passing through it. Wu _et al._ [28]
introduce a framework for jointly learning the optimal phase mask and
reconstruction algorithm for depth estimation.

###  5.2 Sensor Optimization

Recent advances in image sensor technology have resulted in the capability for
per-pixel sensing and processing, unleashing new opportunities for sensor
optimization [112]. Martel et al. [26] propose a method to learn spatially
varying pixel exposures for the tasks of HDR and high-speed imaging. Rather
than use expensive spatial light modulators (SLMs) to implement optical
coding, exposure times are directly learned and then programmed onto a focal-
plane sensor–processor (FPSP), which can run both sensing and processing for
each pixel. In their work, a per-pixel shutter function is parameterized by a
neural network and learned with stochastic gradient descent in simulation
before being programmed onto the sensor. The shutter function encodes the
exposure time of each pixel. Nguyen et al. [113] adopt a similar method for
motion deblurring through spatially varying pixel exposures. Li et al. [114]
propose a different way to optimize the sensor for compressive sensing (CS).
In their work, a CS mask is directly learned by a neural network to encode the
scene with fewer bits and then programmed on a coded two-bucket (C2B) camera
[115].

###  5.3 Image Signal Processor (ISP) Optimization

Image signal processors (ISPs) produce color images from RAW pixel
measurements from the sensor through a pipeline of demosaicking, denoising,
deblurring, tone mapping, etc. [116]. ISPs can either be implemented on
hardware, as traditionally done, or in software, which, while easier to
optimize, is often slower. Hardware ISPs contain hundreds of categorical and
continuous parameters that can be tuned based on application. However, because
the inner workings of hardware ISPs are usually not revealed to users, imaging
experts are needed to manually configure the parameters, which can be time-
intensive. As a result, methods that can automatically identify the optimal
values of ISP parameters for a given downstream task, such as object
detection, are of interest. While grid search can be used to optimize the
parameters of a hardware ISP when there is a small number of parameters, the
search space grows exponentially as more parameters are added. More
sophisticated optimizations, such as the artificial bee colony algorithm have
also been used for hardware ISP optimization as done by Nishimura et al.
[117], but were demonstrated on only 3-4 continuous parameters. Because the
ISP is implemented in hardware and is non-differentiable, gradient-based
approaches cannot be directly applied. Tseng et al. [118] observe that ISPs
can instead be made differentiable by modeling them with neural networks.
Thus, since the inner workings of hardware ISPs are hidden to protect trade
secrets, the hardware ISP can be treated as a black box and optimized using a
differentiable proxy function, i.e. neural network. Tseng et al. use hardware
in the training loop to acquire RAW pixel measurements of projected images and
randomly sample hardware ISP parameters. The RAW pixel measurements and
sampled hardware ISP parameters are used to train a neural network to generate
the same output as the hardware ISP. Once trained, the parameters of the
hardware ISP can be directly optimized with gradient descent for a given
downstream task.

Robidoux et al. [102] propose an alternative method for hardware ISP
optimization in the framework of end-to-end optimization of multi-exposure
high dynamic range (HDR) camera systems. In their work, optimization
alternates between 0th-order evolutionary search over sensor and ISP
parameters and 1st-order gradient descent on the neural network weights, where
the neural network is trained for a perception task, such as automotive object
detection, using images captured by the camera system. An evolutionary search
algorithm similar to CMA-ES (Covariance Matrix Adaptation Evolution Strategy)
[119] is used to optimize the hardware ISP parameters. To optimize the model,
raw data is collected with 254 different ISP parameters and simulated SNR drop
artifacts are added.

A variety of other methods attempt to replace the image processing steps done
by the ISP with a single optimized or learned operation [120, 121, 122, 123].
Diamond et al. [123] propose Anscombe networks as neural ISPs that transform
RAW pixel measurements to an output image, and jointly train the neural ISP
with a network for the downstream task. They find that directly learning the
downstream task from RAW pixel measurements leads to worse performance than
when either a hardware or neural ISP is used, and that neural ISPs are
especially helpful in low-light conditions where traditional ISPs produce
worse output.

###  5.4 Optical Neural Networks

Optical neural networks (ONNs) are physical implementations of neural networks
using optical components. Because operations are done with photons rather than
electrons, ONNs may bring major increases in bandwidth. Early research found
that fully connected layers can be cheaply implemented with optics components
[124], followed by more recent advances in implementing convolutional layers
in optics [101]. However, because optics operations are typically linear,
implementing nonlinearity, an essential component to the success of deep
neural networks, with optics is an active area of research. For more
information on ONNs, we refer the reader to [125] and [126].

Figure 5: Future of Camera System Design. Progress in joint optimization of
camera hardware and software has enabled task-specific cameras. In the future,
this trend will continue and extend to fully end-to-end optimizable imaging
_systems_ that choose which modalities to use to achieve a task.

##  6 Ingredients for Joint Optimization

We present five ingredients that we anticipate will drive progress in end-to-
end methods and rank relevant fields for each ingredient in Fig. 2. These
ingredients are optical simulators, differentiability, interpretability,
learned priors, and exploitation of physics. While there has been significant
progress in advancing each tool, we identify open challenges and present some
insights into overcoming them.

###  6.1 Progress and Open Challenges

Optical simulators. Optical simulators like Zemax and techniques like PSF
engineering with deep learning have enabled data- and task-driven optical
design. Yet, optical simulation remains a core challenge for end-to-end
methods. Simulating complex optics of a camera is difficult and often
proprietary. One solution to this challenge is to create open source
simulators that can drive research and lower the barrier of entry for new
researchers. Data-driven simulators, such as [127], can help in overcoming
this challenge.

Differentiability: Although differentiability is not imperative for
optimization, recent progress in computer vision, as discussed in Section 4.2,
has shown that differentiable systems converge faster and are more optimal.
Advances in autodifferentiation have enabled progress in end-to-end design.
Nevertheless, the problem of how much differentiability and how to make
complex simulators differentiable is an open question for these systems.

Interpretability: As task-specific imaging systems that learn over data and
physics become ubiquitous, interpretability of the systems decrease. Many end-
to-end methods analyze the learned PSFs of the system or perform analysis on
intermediate outputs. Nevertheless, while the physical encoder remains more
interpretable than pure deep learning systems, interpretability will remain a
challenge as more learned priors are used. We can begin to overcome this by
leveraging known physics and inputs, and comparing against expected outputs.

Learned Priors: Ideally, the end-to-end imaging system will continually learn
and self-optimize to generalize to new environments that it may not have
encountered in simulated training data. The key challenge is the sim-to-real
gap and labeling real-world data on-the-fly. Existing methods using static or
simulated scenes to avoid expensive relabeling of large datasets. Therefore,
we anticipate the need for multi-step optimization, where training first
happens in simulation and is then evaluated on real-world data, using the
evaluation score to inform continued training in simulation.

Exploiting Physics: Physics can be exploited in hardware and algorithms. End-
to-end frameworks exploit physics at the hardware level, but do not fully
exploit physics at the algorithm level. We anticipate future work will use
differentiable, physics-based forward models that encode properties, such as
reflectance and geometry, for better convergence.

##  7 Discussion

Camera design has typically been rooted in the physics of light transport, but
there has been progress recently towards combining physics-based design with
_data-driven_ design. We claim that both are essential tools for task-specific
camera design. Simultaneously, modern computer vision, which typically relies
on data-driven deep learning, has begun to incorporate physics into deep
learning methods. We propose a toolkit that classifies state-of-art methods
based on their use of physics and data/learned priors. We also identify five
key ingredients for end-to-end design, and show how challenges in these five
areas can be addressed. We envision a modular end-to-end pipeline to drive
non-intuitive camera design that employs principles from computational
imaging, physics-based computer vision, and optics.

## References

  * [1] T. W. Cronin, S. Johnsen, N. J. Marshall, and E. J. Warrant, “Visual ecology,” in _Visual Ecology_. Princeton University Press, 2014.
  * [2] A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B. Goldman, and M. Zollhöfer, “State of the art on neural rendering,” 2020. [Online]. Available: https://arxiv.org/abs/2004.03805
  * [3] J. M. Geary, _Introduction to lens design: with practical ZEMAX examples_. Willmann-Bell Richmond, VA, USA:, 2002.
  * [4] B. H. Walker, _Optical engineering fundamentals_. Spie Press Bellingham, 2008, vol. 82.
  * [5] D. M. Wulstein and R. McGorty, “Point-spread function engineering enhances digital fourier microscopy,” _Optics letters_ , vol. 42, no. 22, pp. 4603–4606, 2017.
  * [6] N. Opatovski, Y. Shalev Ezra, L. E. Weiss, B. Ferdman, R. Orange-Kedem, and Y. Shechtman, “Multiplexed psf engineering for three-dimensional multicolor particle tracking,” _Nano Letters_ , vol. 21, no. 13, pp. 5888–5895, 2021\.
  * [7] Y. Shechtman, L. E. Weiss, A. S. Backer, M. Y. Lee, and W. Moerner, “Multicolour localization microscopy by point-spread-function engineering,” _Nature photonics_ , vol. 10, no. 9, pp. 590–594, 2016.
  * [8] E. R. Dowski and W. T. Cathey, “Extended depth of field through wave-front coding,” _Applied optics_ , vol. 34, no. 11, pp. 1859–1866, 1995.
  * [9] E. Nehme, D. Freedman, R. Gordon, B. Ferdman, L. E. Weiss, O. Alalouf, T. Naor, R. Orange, T. Michaeli, and Y. Shechtman, “Deepstorm3d: dense 3d localization microscopy and psf design by deep learning,” _Nature methods_ , vol. 17, no. 7, pp. 734–740, 2020.
  * [10] Y. Shechtman, S. J. Sahl, A. S. Backer, and W. E. Moerner, “Optimal point spread function design for 3d imaging,” _Physical review letters_ , vol. 113, no. 13, p. 133902, 2014.
  * [11] S. R. P. Pavani, M. A. Thompson, J. S. Biteen, S. J. Lord, N. Liu, R. J. Twieg, R. Piestun, and W. E. Moerner, “Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function,” _Proceedings of the National Academy of Sciences_ , vol. 106, no. 9, pp. 2995–2999, 2009.
  * [12] A. Greengard and R. Piestun, “Fisher information of 3d rotating point spread functions,” in _Computational Optical Sensing and Imaging_. Optical Society of America, 2005, p. CMA1.
  * [13] M. A. Alonso, “Wigner functions in optics: describing beams as ray bundles and pulses as particle ensembles,” _Advances in Optics and Photonics_ , vol. 3, no. 4, pp. 272–365, 2011.
  * [14] M. Testorf, B. Hennelly, and J. Ojeda-Castañeda, _Phase-space optics: fundamentals and applications_. McGraw-Hill Education, 2010.
  * [15] R. Horstmeyer, S. B. Oh, O. Gupta, and R. Raskar, “Partially coherent ambiguity functions for depth-variant point spread function design,” 2011.
  * [16] S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” in _ACM SIGGRAPH 2006 Papers_ , 2006, pp. 935–944.
  * [17] R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” _Optical engineering_ , vol. 19, no. 1, pp. 139–144, 1980\.
  * [18] K. Tanaka, N. Ikeya, T. Takatani, H. Kubo, T. Funatomi, and Y. Mukaigawa, “Time-resolved light transport decomposition for thermal photometric stereo,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4804–4813.
  * [19] J. Martin, Y. Lecarpentier, A. Antonetti, and G. Grillon, “Picosecond laser stereometry light scattering measurements on biological material,” _Medical and Biological Engineering and Computing_ , vol. 18, no. 2, pp. 250–252, 1980.
  * [20] A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” _ACM Transactions on Graphics (ToG)_ , vol. 32, no. 6, pp. 1–10, 2013.
  * [21] A. Bhandari, C. Barsi, R. Whyte, A. Kadambi, A. Das, A. Dorrington, and R. Raskar, “Coded time-of-flight imaging for calibration free fluorescence lifetime estimation,” in _Imaging Systems and Applications_. Optical Society of America, 2014, pp. IW2C–5.
  * [22] G. Wetzstein, R. Raskar, and W. Heidrich, “Hand-held schlieren photography with light field probes,” in _2011 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2011, pp. 1–8.
  * [23] J.-I. Park, M.-H. Lee, M. D. Grossberg, and S. K. Nayar, “Multispectral imaging using multiplexed illumination,” in _2007 IEEE 11th International Conference on Computer Vision_. IEEE, 2007, pp. 1–8.
  * [24] D. Krishnan and R. Fergus, “Dark flash photography,” _ACM Trans. Graph._ , vol. 28, no. 3, p. 96, 2009.
  * [25] J. Schmitt, A. Gandjbakhche, and R. Bonner, “Use of polarized light to discriminate short-path photons in a multiply scattering medium,” _Applied optics_ , vol. 31, no. 30, pp. 6535–6546, 1992.
  * [26] J. N. Martel, L. K. Mueller, S. J. Carey, P. Dudek, and G. Wetzstein, “Neural sensors: Learning pixel exposures for hdr imaging and video compressive sensing with programmable sensors,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 7, pp. 1642–1653, 2020.
  * [27] A. Levin, R. Fergus, F. Durand, and W. T. Freeman, “Image and depth from a conventional camera with a coded aperture,” _ACM transactions on graphics (TOG)_ , vol. 26, no. 3, pp. 70–es, 2007.
  * [28] Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, and A. Veeraraghavan, “Phasecam3d—learning phase masks for passive single view depth estimation,” in _2019 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2019, pp. 1–12.
  * [29] K. Yanny, N. Antipa, W. Liberti, S. Dehaeck, K. Monakhova, F. L. Liu, K. Shen, R. Ng, and L. Waller, “Miniscope3d: optimized single-shot miniature 3d fluorescence microscopy,” _Light: Science & Applications_, vol. 9, no. 1, pp. 1–13, 2020.
  * [30] R. Raskar, A. Agrawal, and J. Tumblin, “Coded exposure photography: motion deblurring using fluttered shutter,” in _ACM SIGGRAPH 2006 Papers_ , 2006, pp. 795–804.
  * [31] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Ph.D. dissertation, Stanford University, 2005.
  * [32] M. Levoy, “Light fields and computational imaging,” _Computer_ , vol. 39, no. 8, pp. 46–55, 2006.
  * [33] G. Themelis, J. S. Yoo, and V. Ntziachristos, “Multispectral imaging using multiple-bandpass filters,” _Optics letters_ , vol. 33, no. 9, pp. 1023–1025, 2008.
  * [34] Y. Monno, D. Kiku, M. Tanaka, and M. Okutomi, “Adaptive residual interpolation for color and multispectral image demosaicking,” _Sensors_ , vol. 17, no. 12, p. 2787, 2017.
  * [35] S. Alvarez-Cortes, T. Kunkel, and B. Masia, “Practical low-cost recovery of spectral power distributions,” in _Computer Graphics Forum_ , vol. 35, no. 1. Wiley Online Library, 2016, pp. 166–178.
  * [36] S.-H. Baek, I. Kim, D. Gutierrez, and M. H. Kim, “Compact single-shot hyperspectral imaging using a prism,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 6, pp. 1–12, 2017.
  * [37] R. Shogenji, Y. Kitamura, K. Yamada, S. Miyatake, and J. Tanida, “Multispectral imaging using compact compound optics,” _Optics Express_ , vol. 12, no. 8, pp. 1643–1655, 2004.
  * [38] V. Taamazyan, A. Kadambi, and R. Raskar, “Shape from mixed polarization,” _arXiv preprint arXiv:1605.02066_ , 2016.
  * [39] A. Kadambi, V. Taamazyan, B. Shi, and R. Raskar, “Polarized 3d: High-quality depth sensing with polarization cues,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 3370–3378.
  * [40] X. Zhao, A. Bermak, F. Boussaid, and V. G. Chigrinov, “Liquid-crystal micropolarimeter array for full stokes polarization imaging in visible spectrum,” _Optics express_ , vol. 18, no. 17, pp. 17 776–17 787, 2010\.
  * [41] S. K. Nayar, X.-S. Fang, and T. Boult, “Separation of reflection components using color and polarization,” _International Journal of Computer Vision_ , vol. 21, no. 3, pp. 163–186, 1997.
  * [42] Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface,” in _Proceedings of the seventh IEEE international conference on computer vision_ , vol. 2. IEEE, 1999, pp. 814–819.
  * [43] R. Hartley and A. Zisserman, _Multiple view geometry in computer vision_. Cambridge university press, 2003\.
  * [44] A. Bhandari, F. Krahmer, and R. Raskar, “On unlimited sampling,” in _2017 International Conference on Sampling Theory and Applications (SampTA)_. IEEE, 2017, pp. 31–35.
  * [45] J. Tumblin, A. Agrawal, and R. Raskar, “Why i want a gradient camera,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 103–110.
  * [46] A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” _Nature communications_ , vol. 3, no. 1, pp. 1–8, 2012.
  * [47] G. Satat, B. Heshmat, D. Raviv, and R. Raskar, “All photons imaging through volumetric scattering,” _Scientific reports_ , vol. 6, no. 1, pp. 1–8, 2016\.
  * [48] B. C. Platt and R. Shack, “History and principles of shack-hartmann wavefront sensing,” pp. S573–S577, 2001.
  * [49] X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in _CVPR 2011_. IEEE, 2011, pp. 297–304.
  * [50] F. Adib and D. Katabi, “See through walls with wifi!” in _Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM_ , 2013, pp. 75–86.
  * [51] T. Maeda, Y. Wang, R. Raskar, and A. Kadambi, “Thermal non-line-of-sight imaging,” in _2019 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2019, pp. 1–11.
  * [52] J. R. Bergen and E. H. Adelson, “The plenoptic function and the elements of early vision,” _Computational models of visual processing_ , vol. 1, p. 8, 1991.
  * [53] A. Bhandari, A. Kadambi, and R. Raskar, _Computational Imaging_ , 2022.
  * [54] E. Caroli, J. Stephen, G. Di Cocco, L. Natalucci, and A. Spizzichino, “Coded aperture imaging in x-and gamma-ray astronomy,” _Space Science Reviews_ , vol. 45, no. 3, pp. 349–403, 1987.
  * [55] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, “Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” _ACM Trans. Graph._ , vol. 26, no. 3, p. 69, 2007.
  * [56] N. Antipa, G. Kuo, R. Heckel, B. Mildenhall, E. Bostan, R. Ng, and L. Waller, “Diffusercam: lensless single-exposure 3d imaging,” _Optica_ , vol. 5, no. 1, pp. 1–9, 2018.
  * [57] K. Monakhova, K. Yanny, N. Aggarwal, and L. Waller, “Spectral diffusercam: Lensless snapshot hyperspectral imaging with a spectral filter array,” _Optica_ , vol. 7, no. 10, pp. 1298–1307, 2020.
  * [58] F. L. Liu, G. Kuo, N. Antipa, K. Yanny, and L. Waller, “Fourier diffuserscope: single-shot 3d fourier light field microscopy with a diffuser,” _Optics Express_ , vol. 28, no. 20, pp. 28 969–28 986, 2020.
  * [59] Y. Ba, A. Gilbert, F. Wang, J. Yang, R. Chen, Y. Wang, L. Yan, B. Shi, and A. Kadambi, “Deep shape from polarization,” in _European Conference on Computer Vision_. Springer, 2020, pp. 554–571.
  * [60] D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof, “Image guided depth upsampling using anisotropic total generalized variation,” in _Proceedings of the IEEE international conference on computer vision_ , 2013, pp. 993–1000.
  * [61] S. Boyd, S. P. Boyd, and L. Vandenberghe, _Convex optimization_. Cambridge university press, 2004.
  * [62] Y. Ba, G. Zhao, and A. Kadambi, “Blending diverse physical priors with neural networks,” _arXiv preprint arXiv:1910.00201_ , 2019.
  * [63] K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convolutional neural network for inverse problems in imaging,” _IEEE Transactions on Image Processing_ , vol. 26, no. 9, pp. 4509–4522, 2017.
  * [64] K. Monakhova, J. Yurtsever, G. Kuo, N. Antipa, K. Yanny, and L. Waller, “Learned reconstructions for practical mask-based lensless imaging,” _Optics express_ , vol. 27, no. 20, pp. 28 075–28 090, 2019.
  * [65] X. Jia, J. Willard, A. Karpatne, J. S. Read, J. A. Zwart, M. Steinbach, and V. Kumar, “Physics-guided machine learning for scientific discovery: An application in simulating lake temperature profiles,” _ACM/IMS Transactions on Data Science_ , vol. 2, no. 3, pp. 1–26, 2021.
  * [66] J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar, “Integrating physics-based modeling with machine learning: A survey,” _arXiv preprint arXiv:2003.04919_ , vol. 1, no. 1, pp. 1–34, 2020.
  * [67] C. Che, F. Luan, S. Zhao, K. Bala, and I. Gkioulekas, “Inverse transport networks,” _arXiv preprint arXiv:1809.10820_ , 2018.
  * [68] P. Morales, T. Klinghoffer, and S. Jae Lee, “Feature forwarding for efficient single image dehazing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2019.
  * [69] H. Chen, J. Gu, O. Gallo, M.-Y. Liu, A. Veeraraghavan, and J. Kautz, “Reblur2deblur: Deblurring videos via self-supervised learning,” in _2018 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2018, pp. 1–9.
  * [70] X. Fei, A. Wong, and S. Soatto, “Geo-supervised visual depth prediction,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1661–1668, 2019\.
  * [71] M. M. Loper and M. J. Black, “Opendr: An approximate differentiable renderer,” in _European Conference on Computer Vision_. Springer, 2014, pp. 154–169.
  * [72] S. Liu, T. Li, W. Chen, and H. Li, “Soft rasterizer: A differentiable renderer for image-based 3d reasoning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7708–7717.
  * [73] M. Nimier-David, D. Vicini, T. Zeltner, and W. Jakob, “Mitsuba 2: A retargetable forward and inverse renderer,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 6, pp. 1–17, 2019.
  * [74] T.-M. Li, M. Aittala, F. Durand, and J. Lehtinen, “Differentiable monte carlo ray tracing through edge sampling,” _ACM Trans. Graph. (Proc. SIGGRAPH Asia)_ , vol. 37, no. 6, pp. 222:1–222:11, 2018.
  * [75] C. Zhang, B. Miller, K. Yan, I. Gkioulekas, and S. Zhao, “Path-space differentiable rendering,” _ACM Trans. Graph._ , vol. 39, no. 4, pp. 143:1–143:19, 2020.
  * [76] J. Weiss and N. Navab, “Deep direct volume rendering: Learning visual feature mappings from exemplary images,” _arXiv preprint arXiv:2106.05429_ , 2021\.
  * [77] Y. Ye, S. Tulsiani, and A. Gupta, “Shelf-supervised mesh prediction in the wild,” 2021. [Online]. Available: https://arxiv.org/abs/2102.06195
  * [78] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, Y. Wang, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi, T. Simon, C. Theobalt, M. Niessner, J. T. Barron, G. Wetzstein, M. Zollhoefer, and V. Golyanik, “Advances in neural rendering,” 2021. [Online]. Available: https://arxiv.org/abs/2111.05849
  * [79] J. T. Kajiya, “The rendering equation,” _SIGGRAPH Comput. Graph._ , vol. 20, no. 4, p. 143–150, aug 1986. [Online]. Available: https://doi.org/10.1145/15886.15902
  * [80] L. Wang, W. Chen, W. Yang, F. Bi, and F. R. Yu, “A state-of-the-art review on image synthesis with generative adversarial networks,” _IEEE Access_ , vol. 8, pp. 63 514–63 537, 2020.
  * [81] H. Huang, Z. Li, R. He, Z. Sun, and T. Tan, “Introvae: Introspective variational autoencoders for photographic image synthesis,” 2018. [Online]. Available: https://arxiv.org/abs/1807.06358
  * [82] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” 2021\. [Online]. Available: https://arxiv.org/abs/2105.05233
  * [83] A. Meka, C. Haene, R. Pandey, M. Zollhoefer, S. Fanello, G. Fyffe, A. Kowdle, X. Yu, J. Busch, J. Dourgarian, P. Denny, S. Bouaziz, P. Lincoln, M. Whalen, G. Harvey, J. Taylor, S. Izadi, A. Tagliasacchi, P. Debevec, C. Theobalt, J. Valentin, and C. Rhemann, “Deep reflectance fields - high-quality facial reflectance field inference from color gradient illumination,” vol. 38, no. 4, July 2019. [Online]. Available: http://gvv.mpi-inf.mpg.de/projects/DeepReflectanceFields/
  * [84] J. Thies, M. Zollhöfer, and M. Nießner, “Deferred neural rendering: Image synthesis using neural textures,” _ACM Trans. Graph._ , vol. 38, no. 4, jul 2019. [Online]. Available: https://doi.org/10.1145/3306346.3323035
  * [85] S. Lombardi, J. Saragih, T. Simon, and Y. Sheikh, “Deep appearance models for face rendering,” _ACM Trans. Graph._ , vol. 37, no. 4, pp. 68:1–68:13, Jul. 2018.
  * [86] J. Philip, M. Gharbi, T. Zhou, A. Efros, and G. Drettakis, “Multi-view relighting using a geometry-aware network,” _ACM Transactions on Graphics (SIGGRAPH Conference Proceedings)_ , vol. 38, no. 4, July 2019. [Online]. Available: http://www-sop.inria.fr/reves/Basilic/2019/PGZED19
  * [87] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in _The European Conference on Computer Vision (ECCV)_ , 2020\.
  * [88] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron, “Nerv: Neural reflectance and visibility fields for relighting and view synthesis,” 2020.
  * [89] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. P. Lensch, “Nerd: Neural reflectance decomposition from image collections,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [90] Z. Kuang, K. Olszewski, M. Chai, Z. Huang, P. Achlioptas, and S. Tulyakov, “NeROIC: Neural object capture and rendering from online image collections,” _Computing Research Repository (CoRR)_ , vol. abs/2201.02533, 2022.
  * [91] K. Tiwary, T. Klinghoffer, and R. Raskar, “Towards learning neural representations from shadows,” 2022. [Online]. Available: https://arxiv.org/abs/2203.15946
  * [92] C. Xie, K. Park, R. Martin-Brualla, and M. Brown, “Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling,” 2021. [Online]. Available: https://arxiv.org/abs/2104.08418
  * [93] K. Stelzner, K. Kersting, and A. R. Kosiorek, “Decomposing 3d scenes into objects via unsupervised volume segmentation,” 2021. [Online]. Available: https://arxiv.org/abs/2104.01148
  * [94] T. Klinghoffer, K. Tiwary, A. Balata, V. Sharma, and R. Raskar, “Physically disentangled representations,” _arXiv preprint arXiv:2204.05281_ , 2022.
  * [95] A. Chakrabarti, “Learning sensor multiplexing design through back-propagation,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [96] V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein, “End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging,” _ACM Trans. Graph. (SIGGRAPH)_ , 2018.
  * [97] C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for single-shot high-dynamic-range imaging,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1375–1385.
  * [98] J. Chang and G. Wetzstein, “Deep optics for monocular depth estimation and 3d object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 10 193–10 202.
  * [99] H. Haim, S. Elmalem, R. Giryes, A. M. Bronstein, and E. Marom, “Depth estimation from a single image using deep learned phase coded mask,” _IEEE Transactions on Computational Imaging_ , vol. 4, no. 3, pp. 298–310, 2018.
  * [100] L. He, G. Wang, and Z. Hu, “Learning depth from single images with deep neural network embedding focal length,” _IEEE Transactions on Image Processing_ , vol. 27, no. 9, pp. 4676–4689, 2018.
  * [101] J. Chang, V. Sitzmann, X. Dun, W. Heidrich, and G. Wetzstein, “Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification,” _Scientific reports_ , vol. 8, no. 1, pp. 1–10, 2018.
  * [102] N. Robidoux, L. E. G. Capel, D.-e. Seo, A. Sharma, F. Ariza, and F. Heide, “End-to-end high dynamic range camera pipeline optimization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6297–6307.
  * [103] J. Marco, Q. Hernandez, A. Munoz, Y. Dong, A. Jarabo, M. H. Kim, X. Tong, and D. Gutierrez, “Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 6, pp. 1–12, 2017.
  * [104] S. Su, F. Heide, G. Wetzstein, and W. Heidrich, “Deep end-to-end time-of-flight imaging,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6383–6392.
  * [105] E. Nehme, D. Freedman, R. Gordon, B. Ferdman, T. Michaeli, and Y. Shechtman, “Dense three dimensional localization microscopy by deep learning,” _arXiv preprint arXiv:1906.09957_ , 2019.
  * [106] E. Hershko, L. E. Weiss, T. Michaeli, and Y. Shechtman, “Multicolor localization microscopy and point-spread-function engineering by deep learning,” _Optics express_ , vol. 27, no. 5, pp. 6158–6183, 2019.
  * [107] R. Horstmeyer, R. Y. Chen, B. Kappes, and B. Judkewitz, “Convolutional neural networks that teach microscopes how to image,” _arXiv preprint arXiv:1709.07223_ , 2017.
  * [108] M. Kellman, E. Bostan, M. Chen, and L. Waller, “Data-driven design for fourier ptychographic microscopy,” in _2019 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2019, pp. 1–8.
  * [109] A. Turpin, I. Vishniakou, and J. D. Seelig, “Light scattering control with neural networks in transmission and reflection,” _Arxiv: 180505602 [Cs]_ , 2018.
  * [110] H. Ikoma, C. M. Nguyen, C. A. Metzler, Y. Peng, and G. Wetzstein, “Depth from defocus with learned optics for imaging and occlusion-aware depth estimation,” in _2021 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2021, pp. 1–12.
  * [111] E. Tseng, A. Mosleh, F. Mannan, K. St-Arnaud, A. Sharma, Y. Peng, A. Braun, D. Nowrouzezahrai, J.-F. Lalonde, and F. Heide, “Differentiable compound optics and processing pipeline optimization for end-to-end camera design,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 2, pp. 1–19, 2021.
  * [112] J. Martel and G. Wetzstein, “Computational imaging with vision sensors embedding in-pixel processing,” in _2021 IEEE International Electron Devices Meeting (IEDM)_. IEEE, 2021, pp. 35–3.
  * [113] C. M. Nguyen, J. N. Martel, and G. Wetzstein, “Learning spatially varying pixel exposures for motion deblurring,” _arXiv_ , 2022.
  * [114] Y. Li, M. Qi, R. Gulve, M. Wei, R. Genov, K. N. Kutulakos, and W. Heidrich, “End-to-end video compressive sensing using anderson-accelerated unrolled networks,” in _2020 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 2020, pp. 1–12.
  * [115] M. Wei, N. Sarhangnejad, Z. Xia, N. Gusev, N. Katic, R. Genov, and K. N. Kutulakos, “Coded two-bucket cameras for computer vision,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 54–71.
  * [116] R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew, “Color image processing pipeline,” _IEEE Signal Processing Magazine_ , vol. 22, no. 1, pp. 34–43, 2005.
  * [117] J. Nishimura, T. Gerasimow, R. Sushma, A. Sutic, C.-T. Wu, and G. Michael, “Automatic isp image quality tuning using nonlinear optimization,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 2471–2475.
  * [118] E. Tseng, F. Yu, Y. Yang, F. Mannan, K. S. Arnaud, D. Nowrouzezahrai, J.-F. Lalonde, and F. Heide, “Hyperparameter optimization in black-box image processing using differentiable proxies.” _ACM Trans. Graph._ , vol. 38, no. 4, pp. 27–1, 2019.
  * [119] G. A. Jastrebski and D. V. Arnold, “Improving evolution strategies through active covariance matrix adaptation,” in _2006 IEEE international conference on evolutionary computation_. IEEE, 2006, pp. 2814–2821.
  * [120] F. Heide, M. Steinberger, Y.-T. Tsai, M. Rouf, D. Pajak, D. Reddy, O. Gallo, J. Liu, W. Heidrich, K. Egiazarian _et al._ , “Flexisp: A flexible camera image processing framework,” _ACM Transactions on Graphics (ToG)_ , vol. 33, no. 6, pp. 1–13, 2014.
  * [121] H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” _IEEE Transactions on Image Processing_ , vol. 26, no. 10, pp. 5032–5042, 2017.
  * [122] C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3291–3300.
  * [123] S. Diamond, V. Sitzmann, F. Julca-Aguilar, S. Boyd, G. Wetzstein, and F. Heide, “Dirty pixels: Towards end-to-end image processing and perception,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 3, pp. 1–15, 2021.
  * [124] N. H. Farhat, D. Psaltis, A. Prata, and E. Paek, “Optical implementation of the hopfield model,” _Applied optics_ , vol. 24, no. 10, pp. 1469–1475, 1985\.
  * [125] X. Sui, Q. Wu, J. Liu, Q. Chen, and G. Gu, “A review of optical neural networks,” _IEEE Access_ , vol. 8, pp. 70 773–70 783, 2020.
  * [126] G. Wetzstein, A. Ozcan, S. Gigan, S. Fan, D. Englund, M. Soljačić, C. Denz, D. A. Miller, and D. Psaltis, “Inference in artificial intelligence with deep optics and photonics,” _Nature_ , vol. 588, no. 7836, pp. 39–47, 2020.
  * [127] A. Chopra, E. Gel, J. Subramanian, B. Krishnamurthy, S. Romero-Brufau, K. S. Pasupathy, T. C. Kingsley, and R. Raskar, “Deepabm: Scalable, efficient and differentiable agent-based simulations via graph neural networks,” _arXiv preprint arXiv:2110.04421_ , 2021.