∎

11institutetext: S. Mottelet 22institutetext: Laboratoire de Mathématiques
Appliquées
Université de Technologie de Compiègne
60205 Compiègne France
22email<EMAIL_ADDRESS>33institutetext: L. de Saint Germain, O.
Mondin44institutetext: Luxilon
21, rue du Calvaire
92210 Saint-Cloud France
44email<EMAIL_ADDRESS>
# Smart depth of field optimization applied to a robotised view camera

Stéphane Mottelet  Luc de Saint Germain  Olivier Mondin

###### Abstract

The great flexibility of a view camera allows the acquisition of high quality
images that would not be possible any other way. Bringing a given object into
focus is however a long and tedious task, although the underlying optical laws
are known. A fundamental parameter is the aperture of the lens entrance pupil
because it directly affects the depth of field. The smaller the aperture, the
larger the depth of field. However a too small aperture destroys the sharpness
of the image because of diffraction on the pupil edges. Hence, the desired
optimal configuration of the camera is such that the object has the sharpest
image with the greatest possible lens aperture. In this paper, we show that
when the object is a convex polyhedron, an elegant solution to this problem
can be found. The problem takes the form of a constrained optimization
problem, for which theoretical and numerical results are given.

###### Keywords:

Large format photography Computational photography Scheimpflug principle

##  1 Introduction

---
(a)
(b)
Figure 1: The Sinar e (a) and its metering back (b).

Since the registration of Theodor Scheimpflug’s patent in 1904 (see
Scheimpflug ), and the book of Larmore in 1965 where a proof of the so-called
Scheimpflug principle can be found (see (larmore, , p. 171-173)), very little
has been written about the mathematical concepts used in modern view cameras,
until the development of the Sinar e in 1988 (see Figure 1). A short
description of this camera is given in (Tillmans, , p. 23):

The Sinar e features an integrated electronic computer, and in the studio
offers a maximum of convenience and optimum computerized image setting. The
user-friendly software guides the photographer through the shot without
technical confusion. The photographer selects the perspective (camera
viewpoint) and the lens, and chooses the areas in the subject that are to be
shown sharp with a probe. From these scattered points the Sinar e calculates
the optimum position of the plane of focus, the working aperture needed, and
informs the photographer of the settings needed

Sinar sold a few models of this camera and discontinued its development in the
early nineties. Surprisingly, there has been very little published user
feedback about the camera itself. However many authors started to study (in
fact, re-discover) the underlying mathematics (see e.g. Merklinger  and the
references therein). The most crucial aspect is the consideration of depth of
field and the mathematical aspects of this precise point are now well
understood. When the geometrical configuration of the view camera is precisely
known, then the depth of field region (the region of space where objects have
a sharp image) can be determined by using the laws of geometric optics.
Unfortunately, these laws can only be used as a rule of thumb when operating
by hand on a classical view camera. Moreover, the photographer is rather
interested in the inverse problem: given an object which has to be rendered
sharply, what is the optimal configuration of the view camera? A fundamental
parameter of this configuration is the aperture of the camera lens. Decreasing
the lens aperture diameter increases the depth of field but also increases the
diffraction of light by the lens entrance pupil. Since diffraction decreases
the sharpness of the image, the optimal configuration should be such that the
object fits the depth of field region with the greatest aperture.

This paper presents the mathematical tools used in the software of a computer
controlled view camera solving this problem. Thanks to the high precision
machining of its components, and to the known optical parameters of the lens
and digital sensor, a reliable mathematical model of the view camera has been
developed. This model allows the acquisition of 3D coordinates of the object
to be photographed, as explained in Section 2. In Section 3 we study the depth
of field optimization problem from a theoretical and numerical point of view.
We conclude and briefly describe the architecture of the software in Section
4.

##  2 Basic mathematical modeling

###  2.1 Geometrical model of the View Camera

---
(a)
(b)
Figure 2: Geometrical model (a) and robotised view camera (b).

We consider the robotised view camera depicted in Figure 2(a) and its
geometrical model in Figure 2(b). We use a global Euclidean coordinate system
$(\mathbf{O},X_{1},X_{2},X_{3})$ attached to the camera’s tripod. The front
standard, symbolized by its frame with center $L$ of global coordinates
$\mathbf{L}=(L_{1},L_{2},L_{3})^{\top}$, can rotate along its tilt and swing
axes with angles $\theta_{L}$ and $\phi_{L}$. Most camera lenses are in fact
thick lenses and nodal points $H^{\prime}$ and $H$ have to be considered (see
Ray  p. 43-46). The rear nodal plane, which is parallel and rigidly fixed to
the front standard, passes through the rear nodal point $H^{\prime}$. Since
$L$ and $H^{\prime}$ do not necessarily coincide, the translation between
these two points is denoted $\mathbf{t^{L}}$. The vector
$\overrightarrow{HH^{\prime}}$ is supposed to be orthogonal to the rear nodal
plane.

The rear standard is symbolized by its frame with center $S$, whose global
coordinates are given by $\mathbf{S}=(S_{1},S_{2},S_{3})^{\top}$. It can
rotate along its tilt and swing axes with angles $\theta_{S}$ and $\phi_{S}$.
The sensor plane is parallel and rigidly fixed to the rear standard. The
eventual translation between $S$ and the center of the sensor is denoted by
$\mathbf{t^{S}}$.

The rear standard center $S$ can move in the three $X_{1},X_{2}$ and $X_{3}$
directions but the front standard center $L$ is fixed. The rotation matrices
associated with the front and rear standard alt-azimuth mounts are
respectively given by $\mathbf{R^{L}}=\mathbf{R}(\theta_{L},\phi_{L})$ and
$\mathbf{R^{S}}=\mathbf{R}(\theta_{S},\phi_{S})$ where

$\mathbf{R}(\theta,\phi)=\left(\begin{array}[]{ccc}\cos\phi&-\sin\phi\sin\theta&-\sin\phi\cos\theta\\\
0&\cos\theta&-\sin\theta\\\
-\sin\phi&\cos\phi\sin\theta&\cos\phi\cos\theta\end{array}\right).$

The intrinsic parameters of the camera (focal length $f$, positions of the
nodal points $H$, $H^{\prime}$, translations $\mathbf{t^{S}}$,
$\mathbf{t^{L}}$, image sensor characteristics) are given by their respective
manufacturers data-sheets. The extrinsic parameters of the camera are
$\mathbf{S}$, $\mathbf{L}$, the global coordinate vectors of $S$ and $L$, and
the four rotation angles $\theta_{S}$, $\phi_{S}$, $\theta_{L}$, $\phi_{L}$.
The precise knowledge of the extrinsic parameters is possible thanks to the
computer-aided design model used for manufacturing the camera components. In
addition, translations and rotations of the rear and front standards are
controlled by stepper motors whose positions can be precisely known. In the
following, we will see that this precise geometrical model of the view camera
allows one to solve various photographic problems. The first problem is the
determination of coordinates of selected points of the object to be
photographed.

In the sequel, for sake of simplicity, we will give all algebraic details of
the computations for a thin lens, i.e. when the two nodal points $H$,
$H^{\prime}$ coincide. In this case, the nodal planes are coincident in a so-
called lens plane. We will also consider that $\mathbf{t^{L}}=(0,0,0)^{\top}$
so that $L$ is the optical center of the lens. Finally, we also consider that
$\mathbf{t}^{S}=(0,0,0)^{\top}$ so that $S$ coincides with the center of the
sensor.

###  2.2 Acquisition of object points coordinates

---
Figure 3: Graphical construction of the image $A$ of an object point $X$ in
the optical coordinate system. Green rays and blue rays respectively lie in
the $(L,x_{3},x_{2})$ and $(L,x_{3},x_{1})$ planes and $F$ is the focal point.
---
Figure 4: Image formation when considering a lens with a pupil.

Let us consider a point $X$ with global coordinates given by
$\mathbf{X}=(X_{1},X_{2},X_{3})^{\top}$. The geometrical construction of the
image $A$ of $X$ through the lens is depicted in Figure 3. We have considered
a local optical coordinate system attached to the lens plane with origin $L$.
The local coordinates of $X$ are given by
$\mathbf{x}=(\mathbf{R^{L}})^{-1}(\mathbf{X}-\mathbf{L})$ and the focal point
$F$ has local coordinates $(0,0,f)^{\top}$. Elementary geometrical optics (see
Ray  p. 35-42) allows one to conclude that if the local coordinates of $A$ are
given by $\mathbf{a}=(a_{1},a_{2},a_{3})^{\top}$, then $a_{3}$, $x_{3}$ and
$f$ are linked by the thin lens equation given in its Gaussian form by

$-\frac{1}{a_{3}}+\frac{1}{x_{3}}=\frac{1}{f}.$

Since $A$ lies on the $(XL)$ line, the other coordinates are obtained by
straightforward computations and we have the conjugate formulas

$\displaystyle\mathbf{a}$ $\displaystyle=\frac{f}{f-x_{3}}\mathbf{x},$ (1)
$\displaystyle\mathbf{x}$ $\displaystyle=\frac{f}{f+a_{3}}\mathbf{a}.$ (2)

Bringing an object into focus is one of the main tasks of a photographer but
it can also be used to calculate the coordinates of an object point. It is
important to remember that all light rays emanating from $X$ converge to $A$
but pass through a pupil (or diaphragm) assumed to be circular, as depicted in
Figure 4. Since all rays lie within the oblique circular cone of vertex $A$
and whose base is the pupil, the image of $X$ on the sensor will be in focus
only if the sensor plane passes through $A$, otherwise its extended image will
be a blur spot. By using the full aperture of the lens, the image will rapidly
go out of focus if the sensor plane is not correctly placed, e.g. by
translating $S$ into the $x_{3}$ direction. This is why auto-focus systems on
classical cameras only work at near full aperture: the distance to an object
is better determined when the depth of field is minimal.

The uncertainty on the position of $S$ giving the best focus is related to the
diameter of the so-called “circle of confusion”, i.e. the maximum diameter of
a blur spot that is indistinguishable from a point. Hence, everything depends
on the size of photosites on the sensor and on the precision of the focusing
system (either manual or automatic). This uncertainty is acceptable and should
be negligible compared to the uncertainty of intrinsic and extrinsic camera
parameters.

The previous analysis shows that the global coordinates of $X$ can be
computed, given the position $(u,v)^{\top}$ of its image $A$ on the sensor
plane. This idea has been already used on the Sinar e, where the acquisition
of $(u,v)^{\top}$ was done by using a mechanical metering unit (see Figure 1
(b)). In the system we have developed, a mouse click in the live video window
of the sensor is enough to indicate these coordinates. Once $(u,v)^{\top}$ is
known, the coordinates of $A$ in the global coordinate system are given by

$\mathbf{A}=\mathbf{S}+\mathbf{R^{S}}\left(\begin{array}[]{c}u\\\ v\\\
0\end{array}\right),$

and its coordinates in the optical system by

$\mathbf{a}=(\mathbf{R^{L}})^{-1}(\mathbf{A}-\mathbf{L}).$

Then the local coordinate vector $\mathbf{x}$ of the reciprocal image is
computed with (2) and the global coordinate vector $\mathbf{X}$ is obtained by

$\mathbf{X}=\mathbf{L}+\mathbf{R^{L}}\mathbf{x}.$

By iteratively focusing on different parts of the object, the photographer can
obtain a set of points $\mathcal{X}=\\{{X}^{1},\dots,{X}^{n}\\}$, with $n\geq
3$, which can be used to determine the best configuration of the view camera,
i.e. the positions of front and rear standards and their two rotations, in
order to satisfy focus requirements.

##  3 Focus and depth of field optimization

In classical digital single-lens reflex (DLSR) cameras, the sensor plane is
always parallel to the lens plane and to the plane of focus. For example,
bringing into focus a long and flat object which is not parallel to the sensor
needs to decrease the aperture of the lens in order to extend the depth of
field. On the contrary, view cameras with tilts and swings (or DLSR with a
tilt/shift lens) allow to skew away the plane of focus from the parallel in
any direction. Hence, bringing into focus the same long and flat object with a
view camera can be done at full aperture. This focusing process is
unfortunately very tedious. However, if a geometric model of the camera and
the object are available, the adequate rotations can be estimated precisely.
In the next sections, we will explain how to compute the rear standard
position and the tilt and swing angles of both standards to solve two
different problems:

  1. 1.

when the focus zone is roughly flat, and depth of field is not a critical
issue, then the object plane is computed from the set of object points
$\mathcal{X}$. If $n=3$ and the points are not aligned then this plane is
uniquely defined. If $n>3$ and at least $3$ points are not aligned, we compute
the best fitting plane minimizing the sum of squared orthogonal distances to
points of $\mathcal{X}$. Then, we are able to bring this plane into sharp
focus by acting on:

    1. (a)

the angles $\theta_{L}$ and $\phi_{L}$ of the front standard and the position
of the rear standard, for arbitrary rotation angles $\theta_{S}$, $\phi_{S}$.

    2. (b)

the angles $\theta_{S}$, $\phi_{S}$ and position of the rear standard, for
arbitrary rotation angles $\theta_{L}$, $\phi_{L}$ (in this case there is a
perspective distortion).

  2. 2.

when the focus zone is not flat, then the tridimensional shape of the object
has to be taken into account.

The computations in case 1a are detailed in Section 3.1. In Section 3.3 a
general algorithm is described that allows the computation of angles
$\theta_{L}$ and $\phi_{L}$ of the front standard and the position of the rear
standard such that all the object points are in the depth of field region with
a maximum aperture. We give a theoretical result showing that the
determination of the solution amounts to enumerate a finite number of
configurations.

###  3.1 Placement of the plane of sharp focus by using tilt and swing angles

Figure 5: Illustration of the Scheimpflug rule. Figure 6: Illustration of the
Hinge rule.

In this section we study the problem of computing the tilt and swing angles of
front standard and the position of the rear standard for a given sharp focus
plane. Although the underlying laws are well-known and are widely described
(see Merklinger ; Wheeler ; Evens ), the detail of the computations is always
done for the particular case where only the tilt angle $\theta$ is considered.
Since we aim to consider the more general case where tilt and swing angles are
used, we will describe the various objects (lines, planes) and the associated
computations by using linear algebra tools.

####  3.1.1 The Scheimpflug and the Hinge rules

In order to explain the Scheimpflug rule, we will refer to the diagram
depicted in Figure 5. The Plane of sharp focus (abbreviated $\mathrm{SFP}$) is
determined by a normal vector $\mathbf{n^{SF}}$ and a point $Y$. The position
of the optical center $L$ and a vector $\mathbf{n^{S}}$ normal to the sensor
plane (abbreviated $\mathrm{SP}$) are known. The unknowns are the position of
the sensor center $S$ and a vector $\mathbf{n^{L}}$ normal to the lens plane
(abbreviated $\mathrm{LP}$).

The Scheimplug rule stipulates that if $\mathrm{SFP}$ is into focus, then
$\mathrm{SP}$, $\mathrm{LP}$ and $\mathrm{SFP}$ necessarily intersect on a
common line called the ”Scheimpflug Line” (abbreviated $\mathrm{SL}$). The
diagram of Figure 5a should help the reader to see that this rule is not
sufficient to uniquely determine $\mathbf{n^{L}}$ and $\mathrm{SP}$, as this
plane can be translated toward $\mathbf{n^{S}}$ if $\mathbf{n^{L}}$ is changed
accordingly.

The missing constraints are provided by the Hinge rule, which is illustrated
in Figure 6. This rule considers two complimentary planes: the front focal
plane (abbreviated $\mathrm{FFP}$), which is parallel to $\mathrm{LP}$ and
passes through the focal point $F$, and the parallel to sensor lens plane
(abbreviated $\mathrm{PSLP}$), which is parallel to $\mathrm{SP}$ and passes
through the optical center $L$. The Hinge Rule stipulates that $\mathrm{FFP}$,
$\mathrm{PSLP}$ and $\mathrm{SFP}$ must intersect along a common line called
the Hinge Line (abbreviated $\mathrm{HL}$). Since $\mathrm{HL}$ is uniquely
determined as the intersection of $\mathrm{SFP}$ and $\mathrm{PSLP}$, this
allows one to determine $\mathbf{n^{L}}$, or equivalently the tilt and swing
angles, such that $\mathrm{FFP}$ passes through $\mathrm{HL}$ and $F$. Then
$\mathrm{SL}$ is uniquely defined as the intersection of $\mathrm{LP}$ and
$\mathrm{SFP}$ by the Scheimpflug rule (note that $\mathrm{SL}$ and
$\mathrm{HL}$ are parallel by construction). Since $\mathbf{n^{S}}$ is already
known, any point belonging to $\mathrm{SL}$ is sufficient to uniquely define
$\mathrm{SP}$. Hence, the determination of tilt and swing angles and position
of the rear standard can be summarized as follows:

  1. 1.

determination of $\mathrm{HL}$, intersection of $\mathrm{FFP}$ and
$\mathrm{SFP}$,

  2. 2.

determination of tilt and swing angles such that $\mathrm{HL}$ belongs to
$\mathrm{FFP}$,

  3. 3.

determination of $\mathrm{SL}$, intersection of $\mathrm{LP}$ and
$\mathrm{SFP}$,

  4. 4.

translation of $S$ such that $\mathrm{SL}$ belongs to $\mathrm{SP}$.

####  3.1.2 Algebraic details of the computations

In this section the origin of the coordinate system is the optical center $L$
and the inner product of two vectors $\mathbf{X}$ and $\mathbf{Y}$ is
expressed by using the matrix notation ${\mathbf{X}}^{\top}{\mathbf{Y}}$. All
planes are defined by a unit normal vector and a point in the plane as
follows:

$\displaystyle\mathrm{SP}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X-S})}^{\top}{\mathbf{n^{S}}}=0\right\\}$
$\displaystyle\mathrm{PSLP}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{X}}^{\top}{\mathbf{n^{S}}}=0\right\\},$
$\displaystyle\mathrm{LP}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{X}}^{\top}{\mathbf{n^{L}}}=0\right\\},$
$\displaystyle\mathrm{FFP}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{X}}^{\top}{\mathbf{n^{L}}}-f=0\right\\},$
$\displaystyle\mathrm{SFP}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{Y})}^{\top}{\mathbf{n^{SF}}}=0\right\\},$

where the equation of $\mathrm{FFP}$ takes this particular form because the
distance between $L$ and $F$ is equal to the focal length $f$ and we have
imposed that $n^{L}_{3}>0$. The computations are detailed in the following
algorithm:

###### Algorithm 1

  * Step 1 : compute the Hinge Line by considering its parametric equation

$\displaystyle\mathrm{HL}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;\exists\,t\in\mathbb{R},\;\mathbf{X}=\mathbf{W}+t\mathbf{V}\right\\},$

where $\mathbf{V}$ is a direction vector and $\mathbf{W}$ is the coordinate
vector of an arbitrary point of $\mathrm{HL}$. Since this line is the
intersection of $\mathrm{PSLP}$ and $\mathrm{SFP}$, $\mathbf{V}$ is orthogonal
to $\mathbf{n^{L}}$ and $\mathbf{n^{SF}}$. Hence, we can take $\mathbf{V}$ as
the cross product

$\mathbf{V}=\mathbf{n^{SF}}\times\mathbf{n^{S}}$

and $\mathbf{W}$ as a particular solution (e.g. the solution of minimum norm)
of the overdetermined system of equations

$\displaystyle{\mathbf{W}}^{\top}{\mathbf{n^{S}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{W}}^{\top}{\mathbf{n^{SF}}}$
$\displaystyle={\mathbf{Y}}^{\top}{\mathbf{n^{SF}}}.$

  * Step 2 : since $\mathrm{HL}$ belongs to $\mathrm{FFP}$ we have

${(\mathbf{W}+t\mathbf{V})}^{\top}{\mathbf{n^{L}}}-f=0,\quad\forall~{}t\in\mathbb{R},$

hence $\mathbf{n^{L}}$ verifies the overdetermined system of equations

$\displaystyle{\mathbf{W}}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=f,$ (3)
$\displaystyle{\mathbf{V}}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=0.$ (4)

with the constraint $\|\mathbf{n^{L}}\|^{2}=1$. The computation of
$\mathbf{n^{L}}$ can be done by the following two steps:

    1. 1.

compute $\widetilde{\mathbf{W}}=\mathbf{V}\times\mathbf{W}$ and
$\widetilde{\mathbf{V}}$ the minimum norm solution of system (3)-(4), which
gives a parametrization

$\mathbf{n^{L}}=\widetilde{\mathbf{V}}+t\widetilde{\mathbf{W}},$

of all its solutions, where $t$ is an arbitrary real.

    2. 2.

determination of $t$ such that $\|\mathbf{n^{L}}\|^{2}=1$: this is done by
taking the solution $t$ of the second degree equation

${\widetilde{\mathbf{W}}}^{\top}{\widetilde{\mathbf{W}}}t^{2}+2{\widetilde{\mathbf{W}}}^{\top}{\widetilde{\mathbf{V}}}t+{\widetilde{\mathbf{V}}}^{\top}{\widetilde{\mathbf{V}}}-1=0,$

such that $n^{L}_{3}>0$. The tilt and swing angles are then obtained as

$\theta_{L}=-\arcsin
n_{2}^{L},\quad\phi_{L}=-\arcsin\frac{n_{1}^{L}}{\cos\theta_{L}}.$

  * Step 3 : since $\mathrm{SL}$ is the intersection of $\mathrm{LP}$ and $\mathrm{SFP}$, the coordinate vector $\mathbf{U}$ of a particular point $U$ on $\mathrm{SL}$ is obtained as the minimum norm solution of the system

$\displaystyle{\mathbf{U}}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{U}}^{\top}{\mathbf{n^{SF}}}$
$\displaystyle={\mathbf{W}}^{\top}{\mathbf{n^{SF}}},$

where we have used the fact that $W\in\mathrm{SFP}$.

  * Step 4 : the translation of $S$ can be computed such that $U$ belongs to $\mathrm{SP}$, i.e.

${(\mathbf{U}-\mathbf{S})}^{\top}{\mathbf{n^{S}}}=0.$

If we only act on the third coordinate of $S$ and leave the two others
unchanged, then $S_{3}$ can be computed as

$S_{3}=\frac{{\mathbf{U}}^{\top}{\mathbf{n^{S}}}-S_{1}n^{S}_{1}-S_{2}n^{S}_{2}}{n^{S}_{3}}.$

###### Remark 1

When we consider a true camera lens, the nodal points $H,H^{\prime}$ and the
front standard center $L$ do not coincide. Hence, the tilt and swing rotations
of the front standard modify the actual position of the $\mathrm{PSLP}$ plane.
In this case, we use the following iterative fixed point scheme:

  1. 1.

The angles $\phi_{L}$ and $\theta_{L}$ are initialized with starting values
$\phi_{L}^{0}$ and $\theta_{L}^{0}$.

  2. 2.

At iteration $k$,

    1. (a)

the position of $\mathrm{PSLP}$ is computed considering $\phi_{L}^{k}$ and
$\theta_{L}^{k}$,

    2. (b)

the resulting Hinge Line is computed, then the position of $\mathrm{FFP}$ and
the new values $\phi_{L}^{k+1}$ and $\theta_{L}^{k+1}$ are computed.

Point 2 is repeated until convergence of $\phi_{L}^{k}$ and $\theta_{L}^{k}$
up to a given tolerance. Generally 3 iterations are sufficient to reach the
machine precision.

###  3.2 Characterization of the depth of field region

Figure 7: Position of planes $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$
delimiting the depth of field region.

As in the previous section, we consider that $L$ and the nodal points $H$ and
$H^{\prime}$ coincide. Moreover, $L$ will be the origin of the global
coordinates system.

We consider the configuration depicted in Figure 7 where the sharp focus plane
$\mathrm{SFP}$, the lens plane $\mathrm{LP}$ and the sensor plane
$\mathrm{SP}$ are tied by the Scheimpflug and the Hinge rule. The depth of
field can be defined as follows:

###### Definition 1

Let $X$ be a 3D point and $A$ its image through the lens. Let $\mathcal{C}$ be
the disk in $\mathrm{LP}$ of center $L$ and diameter $f/N$, where $N$ is
called the f-number. Let $K$ be the cone of base $\mathcal{C}$ and vertex $A$.
The point $X$ is said to lie in the depth of field region if the diameter of
the intersection of $\mathrm{SP}$ and $K$ is lower that $c$, the diameter of
the so-called circle of confusion.

The common values of $c$, which depend on the magnification from the sensor
image to the final image and on its viewing conditions, lie typically between
0.2 mm and 0.01 mm. In the following the value of $c$ is not a degree of
freedom but a given input.

If the ellipticity of extended images is neglected, the depth of field region
can be shown to be equal to the unbounded wedge delimited by
$\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ intersecting at $\mathrm{HL}$, where
the corresponding sensor planes $\mathrm{SP_{1}}$ and $\mathrm{SP_{2}}$ are
tied to $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ by the Scheimpflug rule. By
mentally rotating $\mathrm{SFP}$ around $\mathrm{HL}$, it is easy to see that
$\mathrm{SP}$ is translated through $\mathbf{n^{S}}$ and spans the region
between $\mathrm{SP_{1}}$ and $\mathrm{SP_{2}}$. The position of
$\mathrm{SP_{1}}$ and $\mathrm{SP_{2}}$, the f-number $N$ and the diameter of
the circle of confusion $c$ are related by the formula

$\frac{Nc}{f}=\frac{p_{1}-p_{2}}{p_{1}+p_{2}},$ (5)

where $p_{1}$, respectively $p_{2}$, are the distances between the optical
center $L$ and $\mathrm{SP_{1}}$, respectively $\mathrm{SP_{2}}$, both
measured orthogonally to the optical plane. The distance $p$ between
$\mathrm{SP}$ and L can be shown to be equal to

$p=\frac{2p_{1}p_{2}}{p_{1}+p_{2}},$ (6)

the harmonic mean of $p_{1}$ and $p_{2}$. This approximate definition of the
depth of field region has been proposed by various authors (see Wheeler ;
Bigler ) but when the ellipticity of images is taken into account a complete
study can be found in Evens . For sake of completeness, we give the
justification of formulas (5) and (6) in Appendix A.1. In most practical
situations the necessary angle between $\mathrm{SP}$ and $\mathrm{LP}$ is
small (less that 10 degrees), so that this approximation is correct.

###### Remark 2

The analysis in Appendix A.1 shows that the ratio $\frac{Nc}{f}$ in equation
(5) does not depend on the direction used for measuring the distance between
$\mathrm{SP}$, $\mathrm{SP_{1}}$, $\mathrm{SP_{2}}$ and $L$. The only
condition, in order to take into account the case where $\mathrm{SP}$ and
$\mathrm{LP}$ are parallel, is that this direction is not orthogonal to
$\mathbf{n^{S}}$. Hence, by taking the direction given by $\mathbf{n^{S}}$, we
can obtain an equivalent formula to (5). To compute the distances, we need the
coordinate vector of two points $U_{1}$ and $U_{2}$ on $\mathrm{SP_{1}}$ and
$\mathrm{SP_{2}}$ respectively. To this purpose we consider Step 3 of
Algorithm 1 in section 3.1: if $\mathbf{W}$ is the coordinate vector of any
point $W$ of $\mathrm{HL}$, each vector $\mathbf{U}^{i}$ can be obtained as a
particular solution of the system

$\displaystyle{\mathbf{U}^{i}}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{U}^{i}}^{\top}{\mathbf{n}^{i}}$
$\displaystyle={\mathbf{W}}^{\top}{\mathbf{n}^{i}}.$

Since $\mathrm{SP_{i}}$ can be defined as

$\mathrm{SP_{i}}=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{U}^{i})}^{\top}{\mathbf{n^{S}}}=0\right\\},$

and $\|\mathbf{n^{S}}\|=1$, the signed distance from $L$ to $\mathrm{SP_{i}}$
is equal to

$d(L,\mathrm{SP_{i}})={\mathbf{U_{i}}}^{\top}{\mathbf{n^{S}}}.$

So the equivalent formula giving the ratio $\frac{Nc}{f}$ is given by

$\displaystyle\frac{Nc}{f}$
$\displaystyle=\left|\frac{d(L,\mathrm{SP_{1}})-d(L,\mathrm{SP_{2}})}{d(L,\mathrm{SP_{1}})+d(L,\mathrm{SP_{2}})}\right|,$
$\displaystyle=\left|\frac{{(\mathbf{U_{1}}-\mathbf{U_{2}})}^{\top}{\mathbf{n^{S}}}}{{(\mathbf{U_{1}}+\mathbf{U_{2}})}^{\top}{\mathbf{n^{S}}}}\right|.$
(7)

The above considerations show that for a given orientation of the rear
standard given by $\mathbf{n^{S}}$, if the depth of field wedge is given, then
the needed f-number, the tilt and swing angles of the front standard and the
translation of the sensor plane, can be determined. The related question that
will be addressed in the following is the question: given a set of points
$\mathcal{X}=\\{{X}^{1},\dots,{X}^{n}\\}$, how can we minimize the f-number
such that all points of $\mathcal{X}$ lie in the depth of field region?

###  3.3 Depth of field optimization with respect to tilt angle

We first study the depth of field optimization in two dimensions, because in
this particular case all computations can be carried explicitly and a closed
form expression is obtained, giving the f-number as a function of front
standard tilt angle and of the slope of limiting planes. First, notice that
$N$ has a natural upper bound, since (5) implies that

$N\leq\frac{f}{c}.$

####  3.3.1 Computation of f-number with respect to tilt angle and limiting
planes

Figure 8: The depth of field region when only a tilt angle $\theta$ is used.

Without loss of generality, we consider that the sensor plane has the normal
$\mathbf{n^{S}}=(0,0,1)^{\top}$. Let us denote by $\theta$ the tilt angle of
the front standard and consider that the swing angle $\phi$ is zero. The lens
plane is given by

$\mathrm{LP}=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{X}}^{\top}{\mathbf{n^{L}}}=0\right\\},$

where

$\mathbf{n^{L}}=(0,-\sin\theta,\cos\theta)^{\top},$

and any collinear vector to $\mathbf{n^{L}}\times\mathbf{n^{S}}$ is a
direction vector of the Hinge Line. Hence we can take, independently of
$\theta$,

$\mathbf{V}=(1,0,0)^{\top}$

and a parametric equation of $\mathrm{HL}$ is thus given by

$\displaystyle\mathrm{HL}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;\exists\,t\in\mathbb{R},\;\mathbf{X}=\mathbf{W}(\theta)+t\mathbf{V}\right\\},$

where $\mathbf{W}(\theta)$ is the coordinate vector of a particular point
${W}(\theta)$ on $\mathrm{HL}$, obtained as the minimum norm solution of

$\displaystyle{\mathbf{W}(\theta)}^{\top}{\mathbf{n^{S}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{W}(\theta)}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=f.$

Straightforward computations show that for $\theta\neq 0$

$\mathbf{W}(\theta)=\left(\begin{array}[]{c}0\\\ -\frac{f}{\sin\theta}\\\
0\end{array}\right).$

Consider, as depicted in Figure 8, the two sharp focus planes
$\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ passing through $\mathrm{HL}$, with
normals $\mathbf{n^{1}}=(0,-1,a_{1})^{\top}$ and
$\mathbf{n^{2}}=(0,-1,a_{2})^{\top}$,

$\displaystyle\mathrm{SFP_{1}}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{1}}}=0\right\\},$
$\displaystyle\mathrm{SFP_{2}}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{2}}}=0\right\\}.$

The two corresponding sensor planes $\mathrm{SP_{i}}$ are given by following
Steps 3-4 in Algorithm 1 by

$\displaystyle\mathrm{SFP_{i}}=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;X_{3}=t_{i}\right\\},\;i=1,2,$

where

$t_{i}=\frac{f}{a_{i}\sin\theta-\cos\theta},\;i=1,2.$

Using equation (7) the corresponding f-number is equal to

$N(\theta,\mathbf{a})=\left|\frac{t_{1}-t_{2}}{t_{1}+t_{2}}\right|\left(\frac{f}{c}\right),$
(8)

where we have used the notation $\mathbf{a}=(a_{1},a_{2})$. Finally we have,
for $\theta\neq 0$

$N(\theta,\mathbf{a})=\operatorname{sign}\theta\frac{(a_{1}-a_{2})\sin\theta}{2\cos\theta-(a_{1}+a_{2})\sin\theta}\left(\frac{f}{c}\right).$
(9)

###### Remark 3

When $\theta=0$, then $\mathrm{SFP}$ does not intersect $\mathrm{PSLP}$ and
the depth of field region is included between two parallel planes
$\mathrm{SFP_{i}}$ given by

$\displaystyle\mathrm{SFP_{i}}=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;X_{3}=z_{i}\right\\},\;i=1,2,$

where $z_{1}$ and $z_{2}$ depend on the f-number and on the position of
$\mathrm{SFP}$. One can show by using the thin lens equation and equation (5)
that the corresponding f-number is equal to

$N_{0}(z_{1},z_{2})=\frac{\left|\frac{1}{z_{2}}-\frac{1}{z_{1}}\right|}{\frac{1}{z_{1}}+\frac{1}{z_{2}}-\frac{2}{f}}\left(\frac{f}{c}\right).$
(10)

####  3.3.2 Theoretical results for the optimization problem

Without loss of generality, we will consider a set of only three non-aligned
points $\mathcal{X}=\\{{X}^{1},{X}^{2},{X}^{3}\\}$, which have to be within
the depth of field region with minimal f-number and we denote by
$\mathbf{X}^{1},\mathbf{X}^{2},\mathbf{X}^{3}$ their respective coordinate
vectors.

The corresponding optimization problem can be stated as follows: find

$(\theta^{*},\mathbf{a}^{*})=\arg\min_{\begin{array}[]{c}\theta\in\mathbb{R}\\\
a\in\mathcal{A(\theta)}\end{array}}N(\theta,\mathbf{a}),$ (11)

where for a given $\theta$ the set $\mathcal{A(\theta)}$ is defined by the
inequalities

$\displaystyle{(\mathbf{X}^{i}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{1}}}$
$\displaystyle\geq 0,~{}i=1,2,3,$ (12)
$\displaystyle{(\mathbf{X}^{i}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{2}}}$
$\displaystyle\leq 0,~{}i=1,2,3,$ (13)

meaning that ${X}^{1},{X}^{2},{X}^{3}$ are respectively under
$\mathrm{SFP_{1}}$ and above $\mathrm{SFP_{2}}$, and by the inequalities

$\displaystyle-{(\mathbf{X}^{i}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{L}}}+f$
$\displaystyle\leq 0,~{}i=1,2,3,$ (14)

meaning that ${X}^{1},{X}^{2},{X}^{3}$ are in front of $\mathrm{FFP}$.

###### Remark 4

Points behind the focal plane cannot be in focus, and the constraints (14) are
just expressing this practical impossibility. However, we have to notice that
when one of these constraints is active, we can show that $a_{1}=\cot\theta$
or $a_{2}=\cot\theta$ so that $N(\mathbf{a},\theta)$ reaches its upper bound
$\frac{f}{c}$. Moreover, we have to eliminate the degenerate case where the
points ${X}^{i}$ are such that there exists two active constraints in (14): in
this case, there exists a unique admissible pair $(\mathbf{a},\theta)$ and the
problem has no practical interest. To this purpose, we can suppose that
$X^{i}_{3}>f$ for $i=1,2,3$.

For $\theta\neq 0$ the gradient of $N(\mathbf{a},\theta)$ is equal to

$\mathbf{\nabla}(\mathbf{a},\theta)=\frac{2\operatorname{sign}\theta}{(2\cot\theta-(a_{1}+a_{2}))^{2}}\left(\begin{array}[]{c}\cot\theta-
a_{2}\\\ -\cot\theta+a_{1}\\\
\frac{a_{1}-a_{2}}{\sin^{2}\theta}\end{array}\right)$

and cannot vanish since $a_{1}=a_{2}$ is not possible because it would mean
that ${X}^{1},{X}^{2},{X}^{3}$ are aligned. This implies that $\mathbf{a}$
lies on the boundary of $\mathcal{A}(\theta)$ and we have the following
intuitive result (the proof is given in Appendix A.2):

###### Proposition 1

Suppose that $X^{i}_{3}>f$, for $i=1,2,3$. Then when $N(\mathbf{a},\theta)$
reaches its minimum, there exists $i_{1}$, $i_{2}$ with $i_{1}\neq i_{2}$ such
that

$\displaystyle{(\mathbf{X}^{i_{1}}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{1}}}$
$\displaystyle=0,$
$\displaystyle{(\mathbf{X}^{i_{2}}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{2}}}$
$\displaystyle=0.$

###### Remark 5

The above result shows that at least two points touch the depth of field
limiting planes $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ when the f-number is
minimal. In the following, we will show that the three points
${X}^{1},{X}^{2}$ and ${X}^{3}$ are necessarily in contact with one of the
limiting planes (the proof is given in Appendix A.3):

###### Proposition 2

Suppose that the vertices $\\{{X}^{i}\\}_{i=1\dots 3}$ verify the condition

$\frac{\|\mathbf{X}^{i}\times\mathbf{X}^{j}\|}{\|\mathbf{X}^{i}-\mathbf{X}^{j}\|}>f,~{}~{}i\neq
j.$ (15)

Then $N(\mathbf{a},\theta)$ reaches its minimum when all vertices are in
contact with the limiting planes.

###### Remark 6

If $\theta$ is small, then $N(\theta,\mathbf{a})$ in (9) can be approximated
by

$\tilde{N}(\theta,\mathbf{a})=\operatorname{sign}\theta{(a_{1}-a_{2})\sin\theta}\left(\frac{f}{2c}\right),$
(16)

and the proof of Proposition 2 is considerably simplified: the same result
holds with the weaker condition

$\|\mathbf{X}^{i}\times\mathbf{X}^{j}\|>0.$ (17)

In fact, an approximate way of specifying the depth of field region using the
hyperfocal distance, proposed in Merklinger , leads to the same approximation
of $N(\theta,\mathbf{a})$, under the a priori hypothesis of small $\theta$ and
distant objects, i.e. $X^{i}_{3}\gg f$. This remark is clarified in Appendix
A.4.

We will illustrate the theoretical result by considering sets of 3 points. For
a set with more than 3 vertices (but being equal to the vertices of the convex
hull of $\mathcal{X}$), the determination of the optimal solution is purely
combinatorial, since it is enough to enumerate all admissible situations where
two points are in contact with one plane, and a third one with the other. The
value $\theta=0$ also has to be considered because it can be a critical value
if the object has a vertical edge. We will also give an Example which violates
condition (15) and where $N(\mathbf{a},\theta)$ reaches its minimum when only
two vertices are in contact with the limiting planes.

####  3.3.3 Numerical results

In this section, we will consider the following function, defined for
$\theta\neq 0$

$n(\theta)=\min_{a\in\mathcal{A}(\theta)}N(\mathbf{a},\theta).$

Finding the minimum of this function allows one to solve the original
constrained optimization problem, but considering the results of the previous
section, $n(\theta)$ is non-differentiable. In fact, the values of $\theta$
for which $n(\theta)$ is not differentiable correspond to the situations where
3 points are in contact with the limiting planes. We extend $n(\theta)$ by
continuity for $\theta=0$ by defining

$n(0)=\frac{\frac{1}{z_{2}}-\frac{1}{z_{1}}}{\frac{1}{z_{1}}+\frac{1}{z_{2}}-\frac{2}{f}}\left(\frac{f}{c}\right),$

where

$z_{1}=\max_{i=1,2,3}X^{i}_{3},~{}~{}z_{2}=\min_{i=1,2,3}X^{i}_{3}.$

This formula can be directly obtained by using conjugation formulas or by
taking $\theta=0$ in equation (24).

###### Example 1

Figure 9: Enumeration of the 3 candidates configurations for Example 1.
---
(a)
(b)
Figure 10: Example 1 graph of $n(\theta)$ for $\theta\in[-0.6,0.6]$ (a) and
$\theta\in[0,0.06]$ (b). Labels give the different types of contact.

We consider a lens with focal length $f=5.10^{-2}\mathrm{m}$ and a confusion
circle $c=3.10^{-5}m$ (commonly used value for 24x36 cameras). The vertices
have coordinates

$\mathbf{X}^{1}=\left(\begin{array}[]{r}0\\\ -1\\\
1\end{array}\right),~{}\mathbf{X}^{2}=\left(\begin{array}[]{r}0\\\ 3\\\
1\end{array}\right),~{}\mathbf{X}^{3}=\left(\begin{array}[]{r}0\\\ 0\\\
1.5\end{array}\right).$

Figure 9 shows the desired depth of field region (dashed zone) and the three
candidates hinge lines corresponding to contacts $E_{ij}V_{k}$:

  * •

$E_{12}V_{3}$, obtained when the edge $[{X}^{1},{X}^{2}]$ is in contact with
$\mathrm{SFP_{1}}$ and vertex ${X}^{3}$ with $\mathrm{SFP_{2}}$.

  * •

$E_{13}V_{2}$, obtained when the edge $[{X}^{1},{X}^{3}]$ is in contact with
$\mathrm{SFP_{2}}$ and vertex ${X}^{2}$ with $\mathrm{SFP_{1}}$.

  * •

$E_{23}V_{1}$, obtained when the edge $[{X}^{2},{X}^{3}]$ is in contact with
$\mathrm{SFP_{1}}$ and vertex ${X}^{1}$ with $\mathrm{SFP_{2}}$.

The associated values of $\theta$ and $n(\theta)$ are given in Table 1,

Contact | $\theta$ | $n(\theta)$
---|---|---
$E_{12}V_{3}$ | 0.0166674 | 9.52
$E_{13}V_{2}$ | 0.025002 | 7.16
$E_{23}V_{1}$ | 0.0500209 | 28.28
Table 1: Value of $\theta$ for each possible optimal contact and corresponding
f-number $n(\theta)$ for Example 1.

which shows that contact $E_{13}V_{2}$ seems to give the minimum f-number.
Condition (15) is verified, since

$\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{2}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{2}\|}=1,~{}\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{3}\|}=1.34,$
$\frac{\|\mathbf{X}^{2}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{2}-\mathbf{X}^{3}\|}=1.48.$

Hence, the derivative of $n(\theta)$ cannot vanish and the minimum f-number is
necessarily reached for $\theta^{*}=0.025002$.

We can confirm this by considering the graph of $n(\theta)$ depicted on Figure
10. In the zone of interest, the function $n(\theta)$ is almost piecewise
affine. For each point in the interior of curved segments of the graph, the
value of $\theta$ is such that the contact of $\mathcal{X}$ with the limiting
planes is of type $V_{i}V_{j}$. Clearly, the derivative of $n(\theta)$ does
not vanish. The possible optimal values of $\theta$, corresponding to contacts
of type $E_{i}jV_{k}$, are the abscissa of angular points of the graph, marked
with red dots. The graph confirms that the minimal value of $n(\theta)$ is
reached for contact $E_{12}V_{3}$.

The minimal f-number is equal to $n(\theta^{*})=7.16$. By comparison, the
f-number without tilt optimization is $n(0)=28.74$. This example highlights
the important gain in terms of f-number reduction with the optimized tilt
angle.

###### Example 2

Contact | $\theta$ | $n(\theta)$
---|---|---
$E_{12}V_{3}$ | 0.185269 | 29.49
$E_{23}V_{1}$ | 0.100419 | 83.67
$E_{13}V_{2}$ | 0.235825 | 47.04
Table 2: Value of $\theta$ for each possible optimal contact and corresponding
f-number $n(\theta)$ for Example 2.
---
(a)
(b)
Figure 11: Example 2 graph of $n(\theta)$ for $\theta\in[-0.6,0.6]$ (a) and
$\theta\in[0,0.3]$ (b). Labels give the different types of contact.

We consider the same lens and confusion circle as in Example 1
($f=5.10^{-2}\mathrm{m}$, $c=3.10^{-5}m$) but the vertices have coordinates

$\mathbf{X}^{1}=\left(\begin{array}[]{r}0\\\ -0.1\\\
0.12\end{array}\right),~{}\mathbf{X}^{2}=\left(\begin{array}[]{r}0\\\ 0\\\
0.19\end{array}\right),~{}\mathbf{X}^{3}=\left(\begin{array}[]{r}0\\\
-0.0525\\\ 0.17\end{array}\right).$

The object is almost ten times smaller than the object of the previous example
(it has the size of a small pen), but it is also ten times closer: this is a
typical close up configuration. The values of $\theta$ and $n(\theta)$
associated to contacts of type $E_{ij}V_{k}$ are given in Table 2 which shows
that contact $E_{13}V_{2}$ seems to give the minimum f-number. We have

$\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{2}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{2}\|}=0.16,~{}\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{3}\|}=0.16,$
$\frac{\|\mathbf{X}^{2}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{2}-\mathbf{X}^{3}\|}=0.1775527,$

showing that condition (15) is still verified, even if the values are smaller
than the values of Example 1. Hence, the derivative of $n(\theta)$ cannot
vanish and the minimum f-number is reached for $\theta^{*}=0.185269$.

We can confirm this by considering the graph of $n(\theta)$ depicted on Figure
11. As in Example 1, the derivative of $n(\theta)$ does not vanish and the
graph confirms that the minimal value of $n(\theta)$ is reached for contact
$E_{12}V_{3}$.

The minimal f-number is equal to $n(\theta^{*})=29.49$. By comparison, the
f-number without tilt optimization is $n(0)=193.79$. Such a large value gives
an aperture of diameter $0.26$mm, almost equivalent to a pin hole ! Since the
maximum f-number of view camera lenses is never larger than 64, the object
cannot be in focus without using tilt.

This example also shows that Proposition 2 is still valid even if the object
is close to the lens and the obtained optimal tilt angle $0.185269$ (10.61
degrees) is large.

###### Example 3

---
(a)
(b)
Figure 12: Example 3 graphs of $n(\theta)$ for $\theta\in[-1,1]$.

In this example we consider an extremely flat triangle which is almost aligned
with the optical center. From the photographer’s point of view, this is an
unrealistic case. We consider the same optical parameters as before and
consider the following vertices

$\mathbf{X}^{1}=\left(\begin{array}[]{r}0\\\ 0\\\
1\end{array}\right),~{}\mathbf{X}^{2}=\left(\begin{array}[]{r}0\\\ h\\\
1.5\end{array}\right),~{}\mathbf{X}^{3}=\left(\begin{array}[]{r}0\\\ -h\\\
2\end{array}\right),$

for $h=0.01$ and we have

$\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{2}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{2}\|}=0.02,~{}\frac{\|\mathbf{X}^{1}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{1}-\mathbf{X}^{3}\|}=0.01,$
$\frac{\|\mathbf{X}^{2}\times\mathbf{X}^{3}\|}{\|\mathbf{X}^{2}-\mathbf{X}^{3}\|}=0.07,$

hence condition (15) is violated in configurations $V_{1}V_{2}$ and
$V_{1}V_{3}$. Since this condition is sufficient, the minimum value of
$n(\theta)$ could still occur for a contact of type $E_{ij}V_{k}$. However, we
can see in Figure 12a that $n(\theta)$ has a minimum at a differentiable point
in configuration $V_{1}V_{3}$.

###### Remark 7

Condition (15) is not necessary: in the proof of Proposition 2 (given in
Appendix A.3), it can be seen that (15) is a condition ensuring that the
polynomial $p(\theta)$ defined by equation (25) has no root in
$[-\pi/2,\pi/2]$. However, the relevant interval is smaller because
${X}^{i_{1}}$ and ${X}^{i_{2}}$ do not stay in contact with the limiting
planes for all values of $\theta$ in $[-\pi/2,\pi/2]$, and because all
vertices must be in front of the focal plane. Anyway, it is always possible to
construct absolutely unrealistic configurations. For example, when we consider
the above vertices with $h=0.005$, then no edge is in contact with the
limiting planes and vertices ${X}^{1}$ and ${X}^{3}$ stay in contact with the
limiting planes for all admissible values of $\theta$. The corresponding graph
of $n(\theta)$ is given in Figure 12b.

###  3.4 Depth of field optimization with respect to tilt and swing angles

As in the previous section, we consider the case of a thin lens and where the
optical and sensor centers coincide respectively with the front and rear
standard rotation centers. Without loss of generality, we consider that the
sensor plane has the normal $\mathbf{n^{S}}=(0,0,1)^{\top}$. The lens plane is
given by

$\mathrm{LP}=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{X}}^{\top}{\mathbf{n^{L}}}=0\right\\},$

where

$\mathbf{n^{L}}=(-\sin\phi\cos\theta,-\sin\theta,\cos\phi\cos\theta)^{\top}.$

A parametric equation of $\mathrm{HL}$ is given by

$\displaystyle\mathrm{HL}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;\exists\,t\in\mathbb{R},\;\mathbf{X}=\mathbf{W}(\theta,\phi)+t\mathbf{V}(\theta,\phi)\right\\},$

where the direction vector is given by

$\mathbf{V}(\theta,\phi)=\mathbf{n^{L}}\times\mathbf{n^{S}}=(-\sin\theta,\sin\phi\cos\theta,0)^{\top},$

and $\mathbf{W}(\theta,\phi)$ is the coordinate vector of a particular point
${W}(\theta,\phi)$ on $\mathrm{HL}$, obtained as the minimum norm solution of

$\displaystyle{\mathbf{W}(\theta)}^{\top}{\mathbf{n^{S}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{W}(\theta)}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=f.$

Consider, as depicted in Figure 7, the two planes of sharp focus
$\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ intersecting at HL, with normals
$\mathbf{n^{1}}$ and $\mathbf{n^{1}}$ respectively. The point $W(\theta,\phi)$
belongs to $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ and any direction vector
of $\mathrm{SFP_{1}}\cap\mathrm{SFP_{2}}$ is collinear to
$\mathbf{V}(\theta,\phi)$. Hence, we have

$\displaystyle\mathrm{SFP_{1}}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{W}(\theta,\phi))}^{\top}{\mathbf{n^{1}}}=0\right\\},$
$\displaystyle\mathrm{SFP_{2}}$
$\displaystyle=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{(\mathbf{X}-\mathbf{W}(\theta,\phi))}^{\top}{\mathbf{n^{2}}}=0\right\\},$

and

$(\mathbf{n^{1}}\times\mathbf{n^{2}})\times\mathbf{V}(\theta,\phi)=0.$ (18)

Using equation (7) the f-number is equal to

$N(\theta,\phi,\mathbf{n^{1}},\mathbf{n^{2}})=\frac{f}{c}\left|\frac{{(\mathbf{U}^{1}-\mathbf{U}^{2})}^{\top}{\mathbf{n^{S}}}}{{(\mathbf{U}^{1}+\mathbf{U}^{2})}^{\top}{\mathbf{n^{S}}}}\right|,$
(19)

where each coordinate vector $\mathbf{U}^{i}$ of point $U^{i}$, for $i=1,2$,
is obtained as a particular solution of the following system:

$\displaystyle{\mathbf{U}^{i}}^{\top}{\mathbf{n^{L}}}$ $\displaystyle=0,$
$\displaystyle{\mathbf{U}^{i}}^{\top}{\mathbf{n}^{i}}$
$\displaystyle={\mathbf{W}(\theta,\phi)}^{\top}{\mathbf{n}^{i}}.$

###### Remark 8

Figure 13: Position of the hinge line when both tilt and swing are used and
$\mathbf{n^{S}}=(0,0,1)^{\top}$. The LP and FFP planes have not been
represented for reasons of readability.

When $\mathbf{n^{S}}=(0,0,1)^{\top}$ the intersections of $\mathrm{HL}$ with
the $(L,X_{1})$ and the $(L,X_{2})$ axes can be determined, as depicted in
Figure 13. In this case, it is easy to show that the coordinates of the
minimum norm $\mathbf{W}(\theta,\phi)$ are given by

$\mathbf{W}(\theta,\phi)=-\frac{f}{\sin^{2}\phi\cos^{2}\theta+\sin^{2}\theta}(\sin\phi\cos\theta,\sin\theta,0)^{\top}.$

We note that up to a rotation of axis $(0,0,1)^{\top}$ and angle $\alpha$, we
recover a configuration where only a tilt angle $\psi$ is used, where these
two angles are respectively defined by

$\displaystyle\sin\psi$
$\displaystyle=\operatorname{sign}\theta\sqrt{\sin^{2}\phi\cos^{2}\theta+\sin^{2}\theta},$
$\displaystyle\sin\alpha$ $\displaystyle=\frac{\sin\phi\cos\theta}{\sin\psi}.$

####  3.4.1 Optimization problem

Consider a set 4 non coplanar points $\mathcal{X}=\\{{X}^{i}\\}_{i=1\dots 4}$,
which have to be within the depth of field region with minimal f-number and
denote by $\\{\mathbf{X}^{i}\\}_{i=1\dots 4}$ their respective coordinate
vectors. The optimization problem can be stated as follows: find

$(\theta^{*},\phi^{*},\mathbf{n^{1}}^{*},\mathbf{n^{2}}^{*})=\arg\min_{\theta,\phi,\mathbf{n^{1}},\mathbf{n^{2}}}N(\theta,\phi,\mathbf{n^{1}},\mathbf{n^{2}}),$
(20)

where the minimum is taken for $\theta,\phi,\mathbf{n^{1}}$ and
$\mathbf{n^{2}}$ such that equation (18) is verified and such that the points
$\\{{X}^{i}\\}_{i=1\dots 4}$ lie between $\mathrm{SFP_{1}}$ and
$\mathrm{SFP_{2}}$. This last set of constraints can be expressed in a way
similar to equations (12)-(13).

####  3.4.2 Analysis of configurations

Consider the function $n(\theta,\phi)$ defined by

$n(\theta,\phi)=\min_{\mathbf{n^{1}},\mathbf{n^{2}}}N(\theta,\phi,\mathbf{n^{1}},\mathbf{n^{2}}),$
(21)

where $\mathbf{n^{1}}$ and $\mathbf{n^{2}}$ are constrained as in the previous
section. Consider the two limiting planes $\mathrm{SFP_{1}}$ and
$\mathrm{SFP_{2}}$ with normals $\mathbf{n^{1}}$ and $\mathbf{n^{2}}$
satisfying the minimum in equation (21). Using the rotation argument of Remark
8 we can easily show that $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ are
necessary in contact with at least two vertices. However, for each value of
the pair $(\theta,\phi)$, we have three types of possible contact between the
tetrahedron formed by points $\\{{X}^{i}\\}_{i=1\dots 4}$ and the limiting
planes: vertex-vertex, edge-vertex, edge-edge or face-vertex. These
configurations can be analyzed as follows: let us consider a pair
$(\theta_{0},\phi_{0})$ and the corresponding type of contact:

  * •

Vertex-vertex: each limiting plane is in contact with only one vertex,
respectively $V_{i}$ and $V_{j}$. In this case, $n(\theta,\phi)$ is
differentiable at $(\theta_{0},\phi_{0})$ and there exists a curve
$\gamma_{vv}$ defined by

$\gamma_{vv}(t)=(\theta(t),\phi(t)),~{}\gamma_{vv}(0)=(\theta_{0},\phi_{0}),$

such that $\frac{d}{dt}n(\theta(t),\phi(t))$ exists and does not vanish for
$t=0$. This can be proved using again the rotation argument of Remark 8.
Hence, $n(\theta_{0},\phi_{0})$ cannot be minimal.

  * •

Edge-vertex: one of the two planes is in contact with edge $E_{ij}$ and the
other one is in contact with vertex $V_{k}$. In this case there is still a
degree of freedom since the plane in contact with $E_{ij}$ can rotate around
this edge in either directions while keeping the other plane in contact with
$V_{k}$ only. If the plane in contact with $E_{ij}$ is $\mathrm{SFP_{1}}$, its
normal $\mathbf{n^{1}}$ can be parameterized by using a single scalar
parameter $t$ and we obtain a family of planes defined by

$\mathrm{SFP_{1}}(t)=\left\\{\mathbf{X}\in\mathbb{R}^{3},\;{\mathbf{n^{1}}(t)}^{\top}{(\mathbf{X}-\mathbf{X}^{i})}=0\right\\}.$

For each value of $t$, the intersection of $\mathrm{SFP_{1}}(t)$ with
$\mathrm{PSLP}$ defines a Hinge Line and thus a pair $(\theta(t),\phi(t))$ of
tilt and swing angles. Hence, there exists a parametric curve

$\gamma_{ev}(t)=(\theta(t),\phi(t)),~{}\gamma_{ev}(0)=(\theta_{0},\phi_{0}),$
(22)

along which $n(\theta,\phi)$ is differentiable. As we will see in the
numerical results, the curve $\gamma_{ev}(t)$ is almost a straight line when
$\theta$ and $\phi$ are small, and $\frac{d}{dt}n(\theta,\phi(t))$ does not
vanish for $t=0$.

  * •

Edge-edge: the limiting planes are respectively in contact with edges
$E_{ij}$, $E_{kl}$ connecting, respectively, vertices $V_{i},V_{j}$ and
vertices $V_{k},V_{l}$. There is no degree of freedom left since these edges
cannot be parallel (otherwise all points would be coplanar). Hence,
$n(\theta,\phi)$ is not differentiable at ($\theta_{0},\phi_{0}$).

  * •

Face-vertex: the limiting planes are respectively in contact with vertex
$V_{l}$ and with the face $F_{ijk}$ connecting vertices $V_{i},V_{j},V_{k}$.
As in the previous case, there is no degree of freedom left and
$n(\theta,\phi)$ is not differentiable at ($\theta_{0},\phi_{0}$).

We can already speculate that the first two configurations are necessary
suboptimal. Consequently we just have to compute the f-number associated with
each one of the 7 possible configurations of type edge-edge of face-vertex.

####  3.4.3 Numerical results

We have considered the flat object of Example 1, translated in plane
$X_{1}=-0.5$, and a complimentary point in order to form a tetrahedron. The
vertices have coordinates

$\mathbf{X}^{1}=\left(\begin{array}[]{r}-0.5\\\ -1\\\
1\end{array}\right),~{}\mathbf{X}^{2}=\left(\begin{array}[]{r}-0.5\\\ 3\\\
1\end{array}\right),$ $\mathbf{X}^{3}=\left(\begin{array}[]{r}-0.5\\\ 0\\\
1.5\end{array}\right),\mathbf{X}^{4}=\left(\begin{array}[]{r}1\\\ 1\\\
1.5\end{array}\right).$

Contact | $\theta$ | $\phi$ | $n(\theta,\phi)$
---|---|---|---
$E_{12}E_{34}$ | 0.021430 | -0.028582 | 12.35
$E_{23}E_{14}$ | 0.150568 | -0.203694 | 90.75
$E_{13}E_{24}$ | 0.030005 | -0.020010 | 8.64
$F_{123}V_{4}$ | 0 | 0.100167 | 88.18
$F_{243}V_{1}$ | 0.075070 | -0.050162 | 42.91
$F_{134}V_{2}$ | 0.033340 | -0.033358 | 9.64
$F_{124}V_{3}$ | 0.018751 | -0.012503 | 10.76
Table 3: Value of $\theta$ and $\phi$ for each possible optimal contact and
corresponding f-number $n(\theta,\phi)$.

All configurations of type edge-edge and face-vertex have been considered and
the corresponding values of $\theta,\phi$ and $n(\theta,\phi)$ are given in
Table 3. The $E_{13}E_{24}$ contact seems to give the minimum f-number.

Figure 14: Graph of $n(\theta,\phi)$ for
$(\theta,\phi)\in[-0.175,0.225]\times[-0.3225,0.2775]$. Figure 15: Level
curves of $n(\theta,\phi)$ and types of contact for
$(\theta,\phi)\in[-0.175,0.225]\times[-0.3225,0.2775]$. Figure 16: Level
curves and types of of $n(\theta,\phi)$ contact for
$(\theta,\phi)\in[-0.015,0.035]\times[-0.0375,-0.0075]$.

Figure 14 gives the graph of $n(\theta,\phi)$ and its level curves in the
vicinity of the minimum are depicted in Figures 15 and 16. In the interior of
each different shaded region, the $(\theta,\phi)$ pair is such that the
contact of $\mathcal{X}$ with the limiting planes is of type $V_{i}V_{j}$. The
possible optimal $(\theta,\phi)$ pairs, corresponding to contacts of type
$E_{ij}E_{kl}$ of $F_{ijk}V_{l}$, are marked with red dots. Notice that the
graph of $n(\theta,\phi)$ is almost polyhedral, i.e. in the interior of
regions of type $V_{i}V_{j}$, the gradient is almost constant and does not
vanish, as seen on the level curves. If confirms that the minimum cannot occur
in these regions, as announced in Section 3.4.2.

The frontiers between regions of type $V_{i}V_{j}$ are curves corresponding to
contacts of type $E_{ij}V_{k}$ and defined by Equation (22). The extremities
of these curves are $(\theta,\phi)$ pairs corresponding to contacts of type
$E_{ij}E_{kl}$ or $F_{ijk}V_{l}$. For example, in Figure 16, the
$(\theta,\phi)$ pairs on the curve separating $V_{2}V_{3}$ and $V_{3}V_{4}$
regions correspond to the $E_{24}V_{3}$ contact. The extremities of this curve
are the two $(\theta,\phi)$ pairs corresponding to contacts $F_{124}V_{3}$ and
$E_{13}E_{24}$. Along this curve, $n(\theta,\phi)$ is strictly monotone as
shown by its level curves.

Finally, the convergence of its level curves in Figure 16 confirms that the
minimum of $n(\theta,\phi)$ is reached for the $E_{13}E_{24}$ contact. Hence,
the optimal angles are $(\theta^{*},\phi^{*})=(0.030005,-0.020010)$ and the
minimal f-number is equal to $n(\theta^{*},\phi^{*})=8.64$. By comparison, the
f-number without tilt and swing optimization is $n(0,0)=28.74$. This example
highlights again the important gain in terms of f-number reduction with the
optimized tilt and swing angles. In our experience, the optimal configuration
for general polyhedrons can be of type edge-edge or face-vertex.

##  4 Trends and conclusion

In this paper, we have given the optimal solution of the most challenging
issue in view camera photography: bring an object of arbitrary shape into
focus and at the same time minimize the f-number. This problem takes the form
of a continuous optimization problem where the objective function (the
f-number) and the constraints are non-linear with respect to the design
variables. When the object is a convex polyhedron, we have shown that this
optimization problem does not need to be solved by classical methods. Under
realistic hypotheses, the optimal solution always occurs when the maximum
number of constraints are saturated. Such a situation corresponds to a small
number of configurations (seven when the object is a tetrahedron). Hence, the
exact solution is found by comparing the value of the f-number for each
configuration.

The linear algebra framework allowed us to efficiently implement the
algorithms in a numerical computer algebra software. The camera software is
able to interact with a robotised view camera prototype, which is actually
used by our partner photographer. With the robotised camera, the time elapsed
in the focusing process is often orders of magnitude smaller than the
systematic trial and error technique.

The client/server architecture of the software allows us to rapidly develop
new problem solvers by validating them first on a virtual camera before
implementing them on the prototype. We are currently working on the fine
calibration of some extrinsic parameters of the camera, in order to improve
the precision of the acquisition of 3D points of the object.

###### Acknowledgements.

This work has been partly funded by the Innovation and Technological Transfer
Center of Région Ile de France.

##  Appendix A Appendix

###  A.1 Computation of the depth of field region

Figure 17: Construction of four approximate intersections of cones with
$\mathrm{SP}$

(a)

(b)

Figure 18: (a) close-up of a particular intersection exhibiting the vertices
of cones for a given directrix $\mathcal{D}$. (b) geometrical construction
allowing to derive the depth of field formula.

In order to explain the kind of approximation used, we have represented in
Figure 17 the geometric construction of the image space limits corresponding
to the depth of field region. Let us consider the cones whose base is the
pupil and having an intersection with $\mathrm{SP}$ of diameter $c$. The image
space limits are the locus of the vertex of such cones. The key point,
suggested in Bigler  and Evens , is the way the diameter of the intersection
is measured.

For a given line $\mathcal{D}$ passing through $L$ and a circle $\mathcal{C}$
of center $L$ in $\mathrm{LP}$ let us call
$\mathcal{K}(\mathcal{C},\mathcal{D})$ the set of cones with directrix
$\mathcal{D}$ and base $\mathcal{C}$. For a given directrix $\mathcal{D}$ let
us call $A$ its intersection with $SP$, as depicted in Figure 18a. Instead of
considering the intersection of cones of directrix $\mathcal{D}$ with
$\mathrm{SP}$, we consider their intersections with the plane passing through
$A$ and parallel to $\mathrm{LP}$. By construction, all intersections are
circles, and there exists only two cones $K_{1}$ and $K_{2}$ in
$\mathcal{K}(\mathcal{C},\mathcal{D})$ such that this intersection has a
diameter equal to $c$, with their respective vertices $A_{1},A_{2}$ on each
side of $\mathrm{SP}$, respectively marked in Figure 18a by a red and a green
spot. Moreover, for all cones in $\mathcal{K}(\mathcal{C},\mathcal{D})$ only
those with a vertex lying on the segment $[A_{1},A_{2}]$ have an ”approximate”
intersection of diameter less that $c$.

The classical laws of homothety show that for any directrix $\mathcal{D}$, the
locus of the vertices of cones $K_{1}$ and $K_{2}$ will be on two parallel
planes located in front of and behind $\mathrm{SP}$, as illustrated by a red
and a green frame in Figure 17a. Hence, the depth of field region in the
object space is the reciprocal image of the region between parallel planes
$\mathrm{SP_{1}}$ and $\mathrm{SP_{2}}$ as depicted in Figure 7.

Formulas (5) and (6) are obtained by considering the directrix that is
orthogonal to $\mathrm{LP}$, as depicted in Figure 18b. If we note $p=AL$,
$p_{1}=A_{1}L$, $p_{2}=A_{2}L$, by considering similar triangles, we have

$\displaystyle\frac{p_{1}-p}{p_{1}}=\frac{p-p_{2}}{p_{2}}=\frac{Nc}{f},$ (23)

which gives immediately

$p=\frac{2p_{1}p_{2}}{p_{1}+p_{2}},$

and by substituting $p$ in (23), we obtain

$\frac{p_{1}-p_{2}}{p_{1}+p_{2}}=\frac{Nc}{f},$

which allows to obtain (5).

###  A.2 Proof of Proposition 1

Without loss of generality, we consider that the optimal $\theta$ is positive.
Suppose now that only one constraint is active in (12). Then there exists
$i_{1}$ such that ${(\mathbf{X}^{i_{1}}-\mathbf{W})}^{\top}{\mathbf{n^{1}}}=0$
and the first order optimality condition is verified: if we define

$g(\mathbf{a},\theta)=-{(\mathbf{X}^{i}-\mathbf{W}(\theta))}^{\top}{\mathbf{n^{1}}}=X^{i_{1}}_{2}-a_{1}X^{i_{1}}_{3}+\frac{f}{\sin\theta},$

there exists $\lambda_{1}\geq 0$ such that the Kuhn and Tucker condition

$\nabla N(\mathbf{a},\theta)+\lambda_{1}\nabla g(\mathbf{a},\theta)=0,$

is verified. Hence, we have

$\frac{2}{(2\cot\theta-(a_{1}+a_{2}))^{2}}\left(\begin{array}[]{c}\cot\theta-
a_{2}\\\ -\cot\theta+a_{1}\\\
\frac{a_{1}-a_{2}}{\sin^{2}\theta}\end{array}\right)+\lambda_{1}\left(\begin{array}[]{c}-X^{i_{1}}_{3}\\\
0\\\ -\frac{\cos\theta}{\sin^{2}\theta}\end{array}\right),$

and necessarily, $a_{1}=\cot\theta$ so that $N(a,\theta)$ reaches its upper
bound and thus is not minimal. We obtain the same contradiction when only a
constraint in (13) is active, or only two constraints in (12), or only two
constraints in (13).∎

###  A.3 Proof of Proposition 2

Without loss of generality we suppose that $\theta\geq 0$. Suppose that the
minimum of $N(\mathbf{a},\theta)$ is reached with only vertices $i_{1}$ and
$i_{2}$ respectively in contact with limiting planes $\mathrm{SFP_{1}}$ and
$\mathrm{SFP_{2}}$. The values of $a_{1}$ and $a_{2}$ can be determined as the
following functions of $\theta$

$a_{1}(\theta)=\frac{X_{2}^{i_{1}}+\frac{f}{\sin\theta}}{X_{3}^{i_{1}}},~{}a_{1}(\theta)=\frac{X_{2}^{i_{2}}+\frac{f}{\sin\theta}}{X_{3}^{i_{2}}},$

and straightforward computations give

$N(\mathbf{a}(\theta),\theta)=\frac{\left(\frac{X^{i_{1}}_{2}}{X^{i_{1}}_{3}}-\frac{X^{i_{2}}_{2}}{X^{i_{2}}_{3}}\right)\sin\theta+\left(\frac{f}{X^{i_{1}}_{3}}-\frac{f}{X^{i_{2}}_{3}}\right)}{2\cos\theta-\left(\frac{X^{i_{1}}_{2}}{X^{i_{1}}_{3}}+\frac{X^{i_{2}}_{2}}{X^{i_{2}}_{3}}\right)\sin\theta-\left(\frac{f}{X^{i_{1}}_{3}}+\frac{f}{X^{i_{2}}_{3}}\right)}\left(\frac{f}{c}\right).$
(24)

In order to prove the result, we just have to check that the derivative of
$N(\mathbf{a}(\theta),\theta)$ with respect to $\theta$ cannot vanish for
$\theta\in[0,\frac{\pi}{2}]$. The total derivative of
$N(\mathbf{a}(\theta),\theta))$ with respect to $\theta$ is given by

$\frac{d}{d\theta}N(\mathbf{a}(\theta),\theta)=\\\
2\frac{\left(\frac{X^{i_{1}}_{2}}{X^{i_{1}}_{3}}-\frac{X^{i_{2}}_{2}}{X^{i_{2}}_{3}}\right)-f\left(\frac{X^{i_{1}}_{2}-X^{i_{2}}_{2}}{X^{i_{1}}_{3}X^{i_{2}}_{3}}\right)\cos\theta+\left(\frac{f}{X^{i_{1}}_{3}}-\frac{f}{X^{i_{2}}_{3}}\right)\sin\theta}{\left(2\cos\theta-\left(\frac{X^{i_{1}}_{2}}{X^{i_{1}}_{3}}+\frac{X^{i_{2}}_{2}}{X^{i_{2}}_{3}}\right)\sin\theta-\left(\frac{f}{X^{i_{1}}_{3}}+\frac{f}{X^{i_{2}}_{3}}\right)\right)^{2}}\left(\frac{f}{c}\right)$

and its numerator is proportional to the trigonometrical polynomial

$p(\theta)=b_{0}+b_{1}\cos\theta+b_{2}\sin\theta,$ (25)

where $b_{0}=X^{i_{1}}_{2}X^{i_{2}}_{3}-X^{i_{2}}_{2}X^{i_{1}}_{3}$,
$b_{1}=-f\left(X^{i_{2}}_{2}-X^{i_{1}}_{2}\right)$,
$b_{2}=f(X^{i_{2}}_{3}-X^{i_{1}}_{3})$. It can be easily shown by using the
Schwartz inequality that $p(\theta)$ does not vanish provided that

$b_{0}^{2}>b_{1}^{2}+b_{2}^{2}.$ (26)

Since $X_{1}^{i_{2}}=X_{1}^{i_{1}}=0$, whe have
$b_{0}^{2}=\|X_{1}^{i_{1}}\times X_{1}^{i_{1}}\|^{2}$ and
$b_{1}^{2}+b_{2}^{2}=f^{2}\|X_{1}^{i_{1}}-X_{1}^{i_{1}}\|^{2}$. Hence (26) is
equivalent to condition (15), this ends the proof.∎

###  A.4 Depth of field region approximation used by A. Merklinger

Figure 19: Depth of field region approximation using the hyperfocal.

In his book (Merklinger , Chapter 7) A. Merklinger has proposed the following
approximation based on the assumption of distant objects and small tilt
angles: if $h$ is the distance from $\mathrm{SFP_{1}}$ to $\mathrm{SFP_{2}}$,
measured in a direction parallel to the sensor plane at distance $z$ from the
lens plane, as depicted in Figure 19, we have

$\frac{h}{2z}\approx\frac{f}{H\sin\theta},$ (27)

where $H$ is the hyperfocal distance (for a definition see Ray  p. 221),
related to the f-number $N$ by the formula

$H=\frac{f^{2}}{Nc},$

and $c$ is the diameter of the circle of confusion. Using (27), the f-number
$N$ can be approximated by

$\tilde{N}=\sin\theta\frac{h}{2z}\frac{f}{c}.$

Since the slopes of $\mathrm{SFP_{1}}$ and $\mathrm{SFP_{2}}$ are respectively
given by $a_{1}$ and $a_{2}$, we have

$\frac{h}{z}={a_{1}-a_{2}},$

and we obtain immediately

$\tilde{N}=(a_{1}-a_{2})\sin\theta\left(\frac{f}{2c}\right),$

which is the same as (16).

## References

  * (1) E. Bigler.  Depth of field and Scheimpflug rule : a minimalist geometrical approach.  http://www.galerie-photo.com/profondeur-de-champ-scheimpflug-english.html, 2002.
  * (2) L. Evens.  View camera geometry.  http://www.math.northwestern.edu/~len/photos/pages/vc.pdf, 2008\.
  * (3) L. Larmore.  Introduction to Photographic Principles.  Dover Publication Inc., New York, 1965.
  * (4) A. Merklinger.  Focusing the view camera.  Bedford, Nova Scotia, 1996.  http://www.trenholm.org/hmmerk/FVC161.pdf.
  * (5) S. F. Ray.  Applied Photographic Optics.  Focal Press, 2002.  third edition.
  * (6) T. Scheimpflug.  Improved Method and Apparatus for the Systematic Alteration or Distortion of Plane Pictures and Images by Means of Lenses and Mirrors for Photography and for other purposes.  GB Patent No. 1196, 1904.
  * (7) U. Tillmans.  Creative Large Format: Basics and Applications.  Sinar AG. Feuerthalen, Switzerland, 1997.
  * (8) R. Wheeler.  Notes on view camera geometry, 2003.  http://www.bobwheeler.com/photo/ViewCam.pdf.


# Towards Supervised and Unsupervised Neural Machine Translation Baselines for
Nigerian Pidgin

Orevaoghene Ahia
InstaDeep
<EMAIL_ADDRESS>
&Kelechi Ogueji11footnotemark: 1
InstaDeep
<EMAIL_ADDRESS>
Equal Contribution

###### Abstract

Nigerian Pidgin is arguably the most widely spoken language in Nigeria.
Variants of this language are also spoken across West and Central Africa,
making it a very important language. This work aims to establish supervised
and unsupervised neural machine translation (NMT) baselines between English
and Nigerian Pidgin. We implement and compare NMT models with different
tokenization methods, creating a solid foundation for future works.

##  1 Introduction

Over 500 languages are spoken in Nigeria, but Nigerian Pidgin is the uniting
language in the country. Between three and five million people are estimated
to use this language as a first language in performing their daily activities.
Nigerian Pidgin is also considered a second language to up to 75 million
people in Nigeria, accounting for about half of the country’s population
according to BBC (2017).

The language is considered an informal lingua franca and offers several
benefits to the country. In 2020, 65% of Nigeria’s population is estimated to
have access to the internet according to Statista (2019). However, over 58.4%
of the internet’s content is in English language, while Nigerian languages,
such as Igbo, Yoruba and Hausa, account for less than 0.1% of internet content
according to W3Techs (2020). For Nigerians to truly harness the advantages the
internet offers, it is imperative that English content is able to be
translated to Nigerian languages, and vice versa.

This work is a first attempt towards using contemporary neural machine
translation (NMT) techniques to perform machine translation for Nigerian
Pidgin, establishing solid baselines that will ease and spur future work. We
evaluate the performance of supervised and unsupervised neural machine
translation models using word-level and the subword-level tokenization of
Sennrich et al. (2015).

##  2 Related Work

Some work has been done on developing neural machine translation baselines for
African languages. Abbott & Martinus (2018) implemented a transformer model
which significantly outperformed existing statistical machine translation
architectures from English to South-African Setswana. Also, Martinus & Abbott
(2019) went further, to train neural machine translation models from English
to five South African languages using two different architectures -
convolutional sequence-to-sequence and transformer. Their results showed that
neural machine translation models are very promising for African languages.

The only known natural language processing work done on any variant of Pidgin
English is by Ogueji & Ahia (2019). The authors provided the largest known
Nigerian Pidgin English corpus and trained the first ever translation models
between both languages via unsupervised neural machine translation due to the
absence of parallel training data at the time.

##  3 Methodology

All baseline models were trained using the Transformer architecture of Vaswani
et al. (2017). We experiment with both word-level and Byte Pair Encoding (BPE)
subword-level tokenization methods for the supervised models. We learned 4000
byte pair encoding tokens, following the findings of Martinus & Abbott (2019).
For the unuspervised model, we experiment with only word-level tokenization.

###  3.1 Dataset

The dataset used for the supervised was obtained from the JW300 large-scale,
parallel corpus for Machine Translation (MT) by Agić & Vulić (2019). The train
set contained 20214 sentence pairs, while the validation contained 1000
sentence pairs. Both the supervised and unsupervised models were evaluated on
a test set of 2101 sentences preprocessed by the
Masakhane111https://github.com/masakhane-
io/masakhane/tree/master/jw300_utils/test/ group. The model with the highest
test BLEU score is selected as the best.

###  3.2 Models

Unsupervised model training followed Ogueji & Ahia (2019) which used a
Transformer of 4 encoder and 4 decoder layers with 10 attention heads.
Embedding dimension was set to 300.

Supervised model training was performed with the open-source machine
translation toolkit JoeyNMT by Kreutzer et al. (2019). For the byte pair
encoding, embedding dimension was set to 256, while the embedding dimension
was set to 300 for the word-level tokenization. The Transformer used for the
byte pair encoding model had 6 encoder and 6 decoder layers, with 4 attention
heads. For word-level, the encoder and decoder each had 4 layers with 10
attention heads for fair comparison to the unsupervised model. The models were
each trained for 200 epochs on an Amazon EC2 p3.2xlarge instance.

##  4 Results

###  4.1 Quantitative

English to Pidgin:

Table 1: BLEU Scores (English to Pidgin) Test BLEU score | Model
---|---
5.18 | Unsupervised (Word-Level)
17.73 | Supervised (Word-Level)
24.29 | Supervised (Byte Pair Encoding)

Pidgin to English:

Table 2: BLEU Scores (Pidgin to English) Test BLEU score | Model
---|---
7.93 | Unsupervised (Word-Level)
24.67 | Supervised (Word-Level)
13.00 | Supervised (Byte Pair Encoding)

For the word-level tokenization English to Pidgin models, the supervised model
outperforms the unsupervised model, achieving a BLEU score of 17.73 in
comparison to the BLEU score of 5.18 achieved by the unsupervised model. The
supervised model trained with byte pair encoding tokenization outperforms both
word-level tokenization models, achieving a BLEU score of 24.29.

Taking a look at the results from the word-level tokenization Pidgin to
English models, the supervised model outperforms the unsupervised model,
achieving a BLEU score of 24.67 in comparison to the BLEU score of 7.93
achieved by the unsupervised model. The supervised model trained with byte
pair encoding tokenization achieved a BLEU score of 13.00. One thing that is
worthy of note is that word-level tokenization methods seem to perform better
on Pidgin to English translation models, in comparison to English to Pidgin
translation models.

###  4.2 Qualitative

When analyzed by L1 speakers, the translation qualities were rated very well.
In particular, the unsupervised model makes many translations that did not
exactly match the reference translation, but conveyed the same meaning. More
analysis and translation examples are in the Appendix.

##  5 Conclusion

There is an increasing need to use neural machine translation techniques for
African languages. Due to the low-resourced nature of these languages, these
techniques can help build useful translation models that could hopefully help
with the preservation and discoverability of these languages.

Future works include establishing qualitative metrics and the use of pre-
trained models to bolster these translation models.

Code, data, trained models and result translations are available here -
https://github.com/orevaoghene/pidgin-baseline

#### Acknowledgments

Special thanks to the Masakhane group for catalysing this work.

## References

  * Abbott & Martinus (2018) Jade Z Abbott and Laura Martinus.  Towards neural machine translation for african languages.  _NeurIPS Workshop on Machine Learning for the Developing World_ , 2018\.
  * Agić & Vulić (2019) Željko Agić and Ivan Vulić.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy, 2019.  doi: 10.18653/v1/P19-1310.  URL https://www.aclweb.org/anthology/P19-1310.
  * BBC (2017) BBC.  Bbc starts pidgin digital service for west africa audiences.  https://www.bbc.com/news/world-africa-40975399, 2017.
  * Kreutzer et al. (2019) Julia Kreutzer, Joost Bastings, and Stefan Riezler.  Joey NMT: A minimalist NMT toolkit for novices.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_ , Hong Kong, China, 2019.
  * Martinus & Abbott (2019) Laura Martinus and Jade Z Abbott.  A focus on neural machine translation for african languages.  _ArviX Preprint_ , 2019.
  * Ogueji & Ahia (2019) Kelechi Ogueji and Orevaoghene Ahia.  Pidginunmt: Unsupervised neural machine translation from west african pidgin to english, 2019.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ , 2015.
  * Statista (2019) Statista.  Internet user penetration in nigeria from 2017 to 2023.  https://www.statista.com/statistics/484918/internet-user-reach-nigeria/, 2019\.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pp. 5998–6008, 2017.
  * W3Techs (2020) W3Techs.  Usage of content languages for websites.  https://w3techs.com/technologies/overview/content_language/all, 2020\.

##  Appendix A Appendix

###  A.1 English to Pidgin translations

Unsupervised (Word-Level):

Source | How has holy spirit helped the Governing Body ?
---|---
Reference | How holy spirit don take help Governing Body ?
Model Translation | ibrahim c how dey do word wey dey guide the governing body .
Source | What can we learn from Jesus ’ counsel ?
Reference | Wetin we fit learn from this advice ?
Model Translation | wetin we don learn from jesus ’ counsel
Source | One student began coming to the kingdom hall .
Reference | One of my student come start to come kingdom hall .
Model Translation | one student wey begin dey come di kingdom hall .
Table 3: Unsupervised (Word-Level) Results from English to Nigerian Pidgin

Supervised (Word-Level):

Source | How has holy spirit helped the Governing Body ?
---|---
Reference | How holy spirit don take help Governing Body ?
Model Translation | How holy spirit take help Governing Body ?
Source | What can we learn from Jesus ’ counsel ?
Reference | Wetin we fit learn from this advice ?
Model Translation | Wetin we fit learn from Jesus example ?
Source | One student began coming to the kingdom hall .
Reference | One of my student come start to come kingdom hall .
Model Translation | One day , e start to Kingdom Hall .
Table 4: Supervised (Word-Level) Results from English to Nigerian Pidgin

Supervised (Byte Pair Encoding):

Source | How has holy spirit helped the Governing Body ?
---|---
Reference | How holy spirit don take help Governing Body ?
Model Translation | How holy spirit take help Governing Body ?
Source | What can we learn from Jesus ’ counsel ?
Reference | Wetin we fit learn from this advice ?
Model Translation | Wetin we fit learn from Jesus example ?
Source | One student began coming to the kingdom hall .
Reference | One of my student come start to come kingdom hall .
Model Translation | One woman come start to dey go meeting .
Table 5: Supervised (Byte Pair Encoding) Results from English to Nigerian
Pidgin

##### Discussions:

The following insights can be drawn from the example translations shown in the
tables above:

  1. 1.

The unsupervised model performed poorly at some simple translation examples,
such as the first translation example.

  2. 2.

For all translation models, the model makes hypothesis that are grammatically
and qualitatively correct, but do not exactly match the reference translation,
such as the second translation example.

  3. 3.

Surprisingly, the unsupervised model performs better at some relatively simple
translation examples than both supervised models. The third example is a
typical such case.

  4. 4.

The supervised translation models seem to perform better at longer example
translations than the unsupervised example.

###  A.2 Pidgin to English translations

Unsupervised (Word-Level):

Source | How holy spirit don take help Governing Body ?
---|---
Reference | How has holy spirit helped the Governing Body ?
Model Translation | how holy spirit is to help governing body .
Source | Wetin we fit learn from this advice ?
Reference | What can we learn from Jesus ’ counsel ?
Model Translation | what should we learn from this advice ?
Table 6: Unsupervised (Word-Level) Results from English to Nigerian Pidgin

Supervised (Word-Level):

Source | How holy spirit don take help Governing Body ?
---|---
Reference | How has holy spirit helped the Governing Body ?
Model Translation | how has holy spirit the governing body ?
Source | Wetin we fit learn from this advice ?
Reference | What can we learn from Jesus ’ counsel ?
Model Translation | What can we learn from this ?
Table 7: Supervised (Word-Level) Results from English to Nigerian Pidgin

Supervised (Byte Pair Encoding):

Source | How holy spirit don take help Governing Body ?
---|---
Reference | How has holy spirit helped the Governing Body ?
Model Translation | 5 , 6 . ( a ) how did holy spirit help the governing body ?
Source | Wetin we fit learn from this advice ?
Reference | What can we learn from Jesus ’ counsel ?
Model Translation | Wtin can we learn from this advice ?
Table 8: Supervised (Byte Pair Encoding) Results from English to Nigerian
Pidgin