CRF for Multi View Object Class Recognition and Segmentation

Document Sample
CRF for Multi View Object Class Recognition and Segmentation Powered By Docstoc
					    3D LayoutCRF for Multi-View Object Class Recognition and Segmentation

                  Derek Hoiem                                              Carsten Rother, John Winn
 Robotics Institute, Carnegie Mellon University                  Microsoft Research Cambridge, Cambridge, UK

   We introduce an approach to accurately detect and
segment partially occluded objects in various viewpoints
and scales. Our main contribution is a novel framework
for combining object-level descriptions (such as position,
shape, and color) with pixel-level appearance, boundary,
and occlusion reasoning. In training, we exploit a rough
                                                                            (a) Image                     (b) Parts/Object
3D object model to learn physically localized part appear-
ances. To ﬁnd and segment objects in an image, we gener-         Figure 1. We introduce the 3D LayoutCRF algorithm, which com-
                                                                 bines pixel-level and object-level reasoning to detect, segment, and
ate proposals based on the appearance and layout of local
                                                                 describe the object.
parts. The proposals are then reﬁned after incorporating
object-level information, and overlapping objects compete        cost in a CRF, while allowing efﬁcient inference with graph
for pixels to produce a ﬁnal description and segmentation        cuts. Altogether, we are able, not only to detect objects
of objects in the scene. A further contribution is a novel       across viewpoints and scales, but to label the pixels of the
instance penalty, which is handled very efﬁciently during        object into parts and describe the position, bounding box,
inference. We experimentally validate our approach on the        viewpoint, and color of the object! In Figure 1, we show an
challenging PASCAL’06 car database.                              example of our results on a test image.
                                                                    The main challenge in detecting and segmenting objects
1. Introduction                                                  across viewpoint and scale is that there is a huge space of
                                                                 possible solutions. How do we get from simple pixels to
   In this paper, we address the problem of detecting and        a complete description of the object? Our approach is to
segmenting objects of a known class when seen from arbi-         build up to our ﬁnal model in several steps, with each step
trary viewpoints, even when the object is partially occluded.    adding new information and providing a more precise hy-
This task is extremely challenging, since inferring the posi-    pothesis about the objects. First, for several wide viewpoint
tion, orientation and visibility of an unknown number of         and scale ranges, we generate a set of proposals by labeling
objects involves reasoning within a high dimensional latent      the pixels of the image into parts while maintaining a lo-
space.                                                           cal consistency of neighboring parts. This gives us a rough
   Recently, Winn and Shotton [16] have introduced the           idea of the viewpoint (e.g. within 45 degrees), scale (within
Layout Conditional Random Field (LayoutCRF) algorithm            a factor of 2), and position (within several pixels) of the
to detect and segment objects while maintaining a consis-        object. To more precisely deﬁne each proposed object, we
tent layout of parts (e.g., a nose above a mouth in a face)      then enforce a global consistency of parts with respect to the
and reasoning about occlusions. They demonstrate suc-            object bounding box and search for the most likely part la-
cess in detecting and segmenting side views of cars, but         beling and object description. After this reﬁnement step, we
their method cannot handle multiple viewpoints or multiple       compute a color model of each object (based on its current
scales.                                                          segmentation estimate) and of the background surrounding
   Our main contribution is to relax this restriction, using a   the object. This gives us several proposals at different view-
rough 3D model of the object class to register parts across      points and scales, each with a precise object description
instances during training, allowing detection of cars in a       and a pixel labeling into parts. Some of these proposals,
continuous range of viewpoints and scales. We also extend        however, will be incorrect, and others will be contradictory,
the object model to include a description of the color of        claiming the very same pixels as parts. To decide which
the object. Further, we show how to include a per-instance       proposals are valid, we assign a per-instance cost to each
object and ﬁnd the best overall solution, considering both
how well each object explains the image and how likely we
are to see the object in a given position. In past CRF formu-
lations, label smoothing terms have been unfairly burdened
with the task of removing false positives. The incorporation
of an instance cost provides a much more reasonable way of
determining whether an object proposal has sufﬁcient evi-
    A key idea in our approach is to use a coarse 3D model to
roughly correspond physical parts across instances at differ-
ent viewpoints. The beneﬁts of 3D object models have been
demonstrated by Ratan et al. [10], who ﬁnd the object pose
that provides the best appearance match with a template,
Thomas et al. [14], who use an implicit 3D object model to
improve detection, and Kushal and Ponce [6], who ﬁnd ob-
jects in cluttered photographs based on 3D models obtained
from stereo pairs. Our method allows us to take advantage
of currently available large datasets (e.g. LabelMe [12])           Figure 2. The 3D LayoutCRF model. The part labels h (orange)
of hand-segmented images, avoiding the need for multiple            are conditioned on the image x (dark blue) and connected 4-wise
views [14] or stereo pairs [6] of the same object instance.         with their neighbors. These pairwise potentials encourage neigh-
Additionally, our 3D part correspondence enables feature            boring parts to belong to the same object instance provided that
sharing [15, 9] across viewpoints, rather than requiring sep-       they are consistent with the part layout of that instance. Each ob-
arate appearance models for dozens of discrete viewpoints,          ject instance has a set of variables Tm (green) relating to its posi-
as in [13].                                                         tion, viewpoint, color distribution and visibility. These instance
    Although the above mentioned approaches tackle multi-           variables affect the expected location/visiblity of the instance’s
                                                                    parts via a rough 3D model. Each set of instance variables Tm
ple viewpoint detection and others have attempted to detect
                                                                    is connected to all of the part labels.
and segment objects from a single viewpoint (e.g. [1]), ours
is the ﬁrst, to our knowledge, to simultaneously detect and
segment objects in a large range of viewpoints and scales.
The key to our success is the ability to reason about local
part appearance and occlusion relationships while maintain-
ing a globally consistent description of the object.                      Average Back/Side Segments                    3D Model

2. The Model
   We aim assign all pixels of an image x to an object in-
stance or to the background. For each object instance, we
also aim to capture the position, scale and viewpoint of that
object. Hence, our model contains both pixel-level vari-
ables h = {hi } and object-level variables T = {Tm }. An
overview of the entire model is shown in Figure 2.
   At the pixel level, the part label hi indicates the object in-
stance that the pixel belongs to and the part of that instance
the pixel lies on. Instances are numbered {0, 1, . . . , M }
where the background is indicated by 0, and M foreground
instances by {1, . . . , M }. Each foreground instance is sub-
divided into H parts. Rather than deﬁning parts according
to a 2D rectangle, as in the original LayoutCRF, we deﬁne                   Image                Initial Labels     Deformed Labels
them over the surface of a 3D solid (Figure 3).                     Figure 3. 3D LayoutCRF Part Assignment. During initializa-
   At the object level, the variables for the mth instance          tion, we use a rough 3D model (top) to consistently assign the
are denoted Tm and consist of the position and scale zm ,           same physical part across different instances in different view-
viewpoint Vm and color distribution Cm . The scale is               points. We then learn appearance models over those parts and use
anisotropic, so that instances of different aspect ratios can       them to relabel the parts (bottom), allowing the part grid to deform
be detected.                                                        slightly to better ﬁt each training instance.
    The probability distribution for all latent variables con-
ditioned on the image is given by                                                               (7,2)   (8,2)   (9,2)

                                 exp [−E(h, T | x; θ)]
          P (h, T | x; θ) =                                       (1)
                                       Z(x, θ)                                                  (7,3)   (8,3)   (9,3)

where θ are the model parameters and E is the energy:
                                                                                                (7,4)   (8,4)   (9,4)
                                 part appearance

     E(h, T | x; θ) =           φi (hi | x, {Tm })
                                      part layout

                     +          ψij (hi , hj | x, {Vm })
             inst. appearance      inst. layout      inst. cost

+            µi (hi , xi ; Cm )+ λi (hi , Tm )+ βinst (Vm ) . (2)
    m    i
                                                                        Figure 4. Layout consistency. From a given viewpoint V , object
    The part appearance potentials φi use local image infor-            parts are expected to appear in a particular two-dimensional or-
mation to detect which part is at pixel i. The part layout po-          dering (top). Neighboring part labels are layout consistent if they
tentials ψij encourage neighboring pixels to be layout con-             are consistent with this ordering (deﬁned over the 3D surface of
sistent, i.e. to have part labels belonging to the same object          the object). To allow for object deformation and small rotations,
instance and in the correct relative layout. The instance ap-           the diagonally-related part labels are also considered layout con-
pearance and instance layout potentials {µi , λi } favor part           sistent.
labelings that are consistent with the appearance, position,
scale and viewpoint of each object instance. Finally, the in-           right), mirroring the image appropriately when evaluating
stance cost βinst deﬁnes a prior on the existence of an object          features. Descriptions of the features used in the decision
at a particular viewpoint in the image. We will now look at             forests, along with details of the learning method are given
each of these potentials in more detail.                                in Sec. 3.1.

2.1. Part appearance
                                                                        2.2. Part layout
    The part appearance potential captures the mapping from
local appearance features of the image to object parts. In the              The part layout potential favors part labels which are lay-
original LayoutCRF, since parts were always seen from the               out consistent, as deﬁned in [16]. In essence, a part labeling
same viewpoint, this term had to represent only the intra-              is layout consistent if all pairs of neighboring parts are in
class variability in the appearance of a part. In the 3D Lay-           the correct relative position, for example, in a face a nose
outCRF, we have to consider how the appearance of a part                part is above a mouth part (see Figure 4 for a more detailed
varies with viewpoint. One possibility is to train the ap-              explanation). For layout consistency to be applicable, it is
pearance model to recognize parts independent of the view-              necessary for the object parts to appear in the same relative
point. However, this leads to much greater variability in               position for any visible region of an object. When we ﬁx
appearance and so reduces detection accuracy, while pro-                the viewpoint, this is a good assumption for most rigid ob-
viding less bottom-up information about the viewpoint. In-              ject classes. However, if we allow the viewpoint to change
stead, we choose to provide multiple appearance models for              arbitrarily, the relative position of the parts can also change
each part, one for each 45◦ viewing range V .                           arbitrarily. For example, if a face can appear upside-down,
    As in [16], the appearance models we use are deci-                  then a nose part can appear below a mouth part. To avoid
sion forests. While separate appearance models are learned              this, we ﬁx the viewpoint to be within a 45◦ range for any
for each viewpoint range, we share features between the                 proposed object instance. With this restriction on the view-
models by ensuring that the decision forests for each                   point, layout consistency of any object region is a reason-
φi (hi | x, T ; θ) have identical structures. This sharing of           able assumption, given that the LayoutCRF allows for small
features helps to reduce over-ﬁtting in the individual mod-             rotations/deformations. The assumption of a ﬁxed view-
els, as demonstrated by [15]. For symmetrical objects, we               point range also allows the appropriate appearance model
also enforce parameter sharing between pairs of viewpoints              to be selected for an object instance.
that mirror each other (e.g. car facing left and car facing                The part layout potential for a given viewpoint V , takes
the form                                                           ensures that parts are globally consistent with the position,
                                                                  scale and viewpoint of the instance. Unlike the single-
                            0          Layout Consistent
                                                                  viewpoint LayoutCRF, the global position of parts can no
                             βoe .eij   Object Edge
ψij (hi , hj | x, V ; θ) =                                         longer be speciﬁed using a 2D rectangular coordinate frame.
                            βoo .eij   Object Occlusion
                                                                  We now specify the likelihood of each part given its position
                             βinc       Inconsistent
                                                                   and the object’s position, scale, and viewpoint. Thus, our
where eij is an edge cost that encourages object boundaries        object representation is expanded into viewpoint and scale
to align with image contrast edges (see [16]) and the four         space. The quantization for determining instance layout is
cases are:                                                         ﬁner than for part appearance sharing. In our experiments,
Layout Consistent: Both hi and hj are layout consistent            we subdivide each viewpoint range of 45 degrees and height
foreground labels as seen from viewpoint V , or both are           range of 2 into three subviewpoints, three height ranges,
background labels.                                                 and two aspect ratio (bounding box width:height) ranges.
Object Edge: One label is the background, the other is an             The instance layout potential is a look-up table for each
edge part (i.e., a part that lies on the edge of the object when   quantized viewpoint Vm ,
seen from viewpoint V ).
                                                                                                       P (hi |loc(zm , i), Vm )
Object Occlusion: One label is an interior (non-edge) part          λi (hi , Tm ) = −δ(hi ∈ m) log                              (4)
label, the other is the background label or an edge part la-                                                    P (hi )
bel. This represents the case where an object is occluded by
                                                                   where δ(hi ∈ m) is as speciﬁed above, loc(zm , i) returns
another object or a ‘background’ object.
                                                                   the position of pixel i in the object-coordinate system given
Inconsistent: Both labels are interior part labels which are
                                                                   that the object is at position/scale zm , and P (hi ) is the part
not layout consistent.
                                                                   prior. During the proposal stage, when object-level infor-
   For the experiments in this paper, we set the cost param-
                                                                   mation is unavailable, we instead apply a constant penalty
eters to {βoe = 3, βoo = 6, βinc = ∞} when generating
                                                                   βbg for assigning a pixel to background, in order to offset
proposals and {βoe = 1, βoo = 2, βinc = ∞} for the later
                                                                   the low prior probability of any individual part.
stages (when instance layout can also be considered).
                                                                   Instance cost: We introduce a per-instance cost βinst (Vm )
2.3. Object Instance Model                                         to the MRF formulation, which acts as a prior favoring im-
                                                                   age interpretations with fewer object instances. Effectively,
    The instance model ensures that the part labeling is con-      it determines whether the total evidence for an object out-
sistent with the color, position, scale and viewpoint of the       weighs our prior bias against its existence. The instance
instance. It also provides a prior on the existence of an          cost is commonly used in object classiﬁcation (e.g. [13]),
object at a particular viewpoint, through the use of a per-        and can be justiﬁed as the odds ratio in a log-likelihood ratio
instance cost.                                                     test or as the object description length in the MDL principle.
Instance appearance: Though colors may vary widely                     The use of an instance cost is a more natural way to re-
within an object class, any particular instance typically con-     move false detections than relying on smoothing terms, re-
sists of a small set of colors. For instance, red and blue car     sulting in better segmentations. It also provides the ability
doors are common, but a single car rarely has both. Thus,          to determine that disconnected regions are part of the same
we require the color of the parts to be consistent with the        object (for example, when a lamp post divides a car in two).
overall color distribution of an instance. We represent the        We deﬁne a cost that depends only on viewpoint; depen-
color distribution for instance m as a mixture of Gaussians        dence on scale and position could allow methods such as
model with parameters Cm , as used in [11]. We also learn          [3] to be employed. In Sec. 4.1, we demonstrate improve-
a localized color distribution C0 for the background. The          ment due to our use of the instance penalty.
instance appearance potential is deﬁned to be
                                           PMoG (xi |Cm )          3. Training and Inference
  µi (hi , xi ; Cm ) = −βc δ(hi ∈ m) log                  (3)
                                           PMoG (xi |C0 )          3.1. Training
where PMoG is the mixture of Gaussians model and δ(hi ∈               The full training process is summarized in Figure 5.
m) is 1 if hi is a part of instance m and 0 otherwise. Since       Learning the 3D Model: We create the 3D model (shown
this potential depends on the test image, it is learned during     in Figure 3) by space carving [7] from thresholded segmen-
inference (see Section 3.2). In our experiments, its weight        tations of the rear and side views of cars, assuming an or-
βc is set to 0.25.                                                 thographic projection. To do this, we ﬁrst center and rescale
Instance layout: While layout consistency ensures that             (according to height) the segmentations of each viewpoint.
parts have a reasonable local layout, instance consistency         We then take the mean of the segmentations and threshold
                         TRAINING                                                          INFERENCE
    1. Gather training examples with segmented and                    1. Generate Proposals (scale/viewpoint separately)
       viewpoint-labeled objects                                           (a) Compute P(h | x)
    2. Construct rough 3D object model                                    (b) Label pixels into parts using only part-level
    3. Assign part labels to training examples                                 appearance and consistency (TRW-S)
    4. Learn part appearance (randomized decision trees)                   (c) Connected components (by layout consis-
    5. Learn instance layout (simple counting over part lo-                    tency) become proposals
       cations)                                                       2. Reﬁne Proposals (instance/scale/viewpoint separately)
    6. Reﬁne labeling (run inference steps 1 and 2); go to               For each proposal, iterate until convergence:
       step 4 (one iteration)                                              (a) Find all likely object conﬁgurations {T}        ˆ
      Figure 5. Training procedure for 3D Layout CRF.                                                ˆ ˆ
                                                                                           ˆ (s.t. P(T | h) > 0.01)
                                                                               given parts h
                                                                                                      ˆ          ˆ
                                                                          (b) Find most likely parts h given {T} (TRW-S)
(at 0.5) to get an “average” segmentation from each view.                Compute object and background color distributions,
Assuming an orthographic projection, we then carve voxels                object appearance terms
out of a solid cube. Finally, we assign parts to the surface          3. Create Final Labeling (instance/scale/viewpoint jointly)
of the 3D object model, projecting a grid onto each side.                Input: part labeling for each proposal with unary
   To assign parts to a new training instance, we require a              potentials
segmentation and orientation (obtained by clicking on the                Assign object labels m to pixels using complete
corner of the car). We rotate and scale (separately in each              model (alpha-expansion graph cuts)
axis) our 3D prototype to match the orientation and seg-
mentation of the training instance as closely as possible and           Figure 6. Inference algorithm for 3D Layout CRF.
back-project the part labels onto the object segment, again
assuming an orthographic projection.                              dure is outlined in Figure 6 and illustrated in Figure 7.
Choice of image features: Our local parts appearances are         1) Generate proposals: We use sequential tree-reweighted
based on RGB intensity, ﬁlter responses of random patches         message passing (TRW-S) [4] to obtain the MAP solution
selected from the training images (similarly to [15]) and a       and create layout-consistent connected components of parts
distance transform of a Canny edge image. The latter can          to get our initial proposals.
effectively model unique edge appearances, such as the cir-       2) Reﬁne proposals: To reﬁne each proposal, we iteratively
cle of a wheel, and long-range properties, such as that a         (1) estimate the distribution of likely instance descriptions
pixel in large uniform regions will have a high distance to       (T) and marginalize to get the instance consistency terms
the nearest edge, and, thus, is unlikely to be part of a car or   for that proposal; and (2) ﬁnd the most likely parts given
person.                                                           the current instance estimate (also using TRW-S). By main-
Training Decision Trees: We apply the randomized trees            taining a distribution of instance descriptions during this it-
method [8] to estimate the likelihood of a pixel label given      erative process, we robustly converge to a good solution.
its image features. Randomized trees allows efﬁcient large-       3) Combine proposals: To determine which proposals are
label learning (we have 120 parts in our experiments) and         valid and the ﬁnal segmentations of the objects, we apply
is robust to over-ﬁtting. We learn a single set of 25 ran-        alpha-expansion [5], with each expansion being a potential
domized trees, each with 250 leaf nodes, on a subset of           switch to a different object label. It can be shown that each
our data. We then re-estimate the parameters of the trees,        expansion is submodular, making graph cuts well suited for
without changing the structure, using the millions of pix-        this phase of the inference. The proposal and reﬁnement
els in our training set. By sharing the tree structure across     stages take most of the computational time (typically 1-5
viewpoint models, we reduce over-ﬁtting and allow more            minutes per scale in a 120x160 image). The ﬁnal alpha-
efﬁcient training and testing.                                    expansion stage requires only a few seconds.
                                                                  Applying the instance cost: We are the ﬁrst, to our knowl-
3.2. Inference                                                    edge, to show that an instance cost can be handled appro-
   To optimize the full objective function in one step is in-     priately with the alpha expansion procedure (i.e., that the
tractable and prone to poor locally optimal solutions. There-     objective function is expansion submodular).
fore, the inference is split into three different optimization       The instance cost is deﬁned as a cost per instance that is
steps. In the ﬁrst two steps, each potential instance in each     present in the image. If at least one pixel is labeled as part of
viewpoint/scale range is optimized individually. The ﬁnal         an object, then that object must be visible, incurring a ﬁxed
step operates on the full objective function that considers       cost; otherwise, the object may be invisible (non-existent)
total evidence across the entire image. The inference proce-      at no cost. Formally we may write this as hard constraint
      (a) Input Image             (b) Initial Parts       (c) Connected Comps               (d) Reﬁned Proposals         (e) Final Result
Figure 7. Inference illustration. From left to right, we show the input image, the initial part labelings (Figure 6, step 1b) that enforce
local consistency (4 of 8 total viewpoints per scale), the corresponding layout consistent connected components (step 1c), four (of 24 total)
reﬁned proposals (step 2), and the ﬁnal labeling after performing inference over the full model.

which is added to our objective function in Equation 2                                1

                        ∞ ((1 − sm ) IPm (hi )) ,              (5)
               m    i

where the instance part function IPm has a binary output
indicating whether hi is a part of instance m or not. The in-               Recall   0.8
stance variable sm is 1 if instance m exist in the image, oth-
erwise 0. Note that this function prohibits the conﬁguration
that sm = 0 and a part of instance m is present in the image.                        0.7
                                                                                                                   Shotton et al.
This term is indeed submodular [5], i.e. E(0, 0)+E(1, 1) ≤
                                                                                                                   Fergus et al.
E(0, 1) + E(1, 0). It is E(0, 0) = E(1, 1) = E(0, 1) = 0                                                           Agarwal & Roth
and E(1, 0) = ∞, where E(hi , sm ) is the pairwise term                                                            Leibe et al.
between a pixel node and an instance node.                                                                         LayoutCRF
                                                                                                                   3D LayoutCRF
Learning the instance appearance: Since the color ap-                                0.5
pearance of an object is instance speciﬁc the respective pa-                            0       0.1      0.2      0.3      0.4      0.5
rameters in Equation 3 are learned during inference, in the                            Figure 8. Precision-recall on UIUC car test set.
end of the reﬁnement stage (step 2 in Figure 6). We gen-
erate two proposed segmentations: a conservative estimate                ments is to demonstrate the value of each of these contribu-
(increase βbg by 0.05) in which all pixels are highly likely             tions.
to be object and a loose estimate (decrease βbg by 1) such
that all pixels outside of it are highly unlikely to be object.          4.1. Comparison to the Original LayoutCRF
We deﬁne a local background region of pixels within an en-
larged bounding box (by ten pixels on each side), excluding                  The key contribution of our algorithm is the ability to de-
pixels in the loosely estimated segment. The object color                tect and segment objects in the presence of viewpoint and
distribution and background color distribution are each esti-            scale variation. To do this, we have introduced many modi-
mated over their respective regions using a mixture of three             ﬁcations to the original LayoutCRF inference algorithm and
diagonal-covariance Gaussians in L*a*b* space. The class-                improved the basic model by including an instance cost.
conditional log likelihood ratio of the color likelihoods is             To compare, we perform the experiment described by Winn
factored into our appearance term.                                       and Shotton [16] on the UIUC car dataset. Note that, since
                                                                         this dataset is single-scale with only side views of cars, we
                                                                         do not incorporate our 3D model for this experiment.
4. Experiments
                                                                             In Figure 8, we show that our algorithm achieves higher
    In this paper, we have shown how to reason about object-             recall (by about 8%) than Winn and Shotton at the high-
level properties, such as viewpoint, size, and color, while              precision regions of the precision-recall curve, with sim-
also reasoning about pixel-level appearance and part con-                ilar recall elsewhere. The beneﬁt of incorporating an in-
sistency. We have also introduced a 3D model to allow part               stance cost into our model can be seen in the segmenta-
assignments on different instances to correspond roughly to              tion and qualitative results (see Figure 9). The instance
physical parts on the object. Finally, we have shown how to              cost allows the smoothness and contrast costs to be reduced,
incorporate a per-instance cost into the CRF, allowing ob-               since they are no longer responsible for removing false posi-
ject proposals to be rejected or accepted based on the entire            tives. Thus, our algorithm has a segmentation accuracy (av-
evidence, instead of relying on local pairwise smoothing                 erage intersection-union ratio of ground truth and inferred
costs to remove false positives. Our goal in these experi-               object regions) of 0.77, compared to 0.66 for the original
      (a) Image          (b) Parts/Object      (c) Segmentation            (d) Image          (e) Parts/Object      (f) Segmentation
Figure 9. Test results on the UIUC dataset. Note the accurate segmentations and the ability to determine that disconnected car regions
can be explained by a single instance. In (b), parts are labeled with separate colors, bounding boxes indicate the estimated extent of an
object, and arrows indicate the estimated orientation. The instance cost allows disconnected object regions to be explained by a single
instance (left, and right bottom row). The orientation is incorrectly estimated in (right bottom row), leading to a poorer segmentation.
LayoutCRF algorithm. More importantly, the instance cost               tween 26 and 38 pixels tall. This constraint is due to the
allows our algorithm to correctly assign disconnected car              high computational cost of our current algorithm (it still
parts to a single instance, when they are separated by an              takes 1-10 minutes per image, depending on the number of
occlusion (see Figure 9 for examples). The instance cost               initial proposals) and not due to any fundamental limitations
allows the algorithm to follow Occam’s razor – that if the             of our approach. When searching within a 26-38 pixel scale
same pixels can be explained as well by the presence of one            range, we produce separate proposals for each 45-degree
car as by two, then the single-car hypothesis is preferred.            range of viewpoints and repeat with the mirrored the image
                                                                       to cover the full 360 degree range (taking advantage of car
4.2. Multi-view Car Detection                                          symmetry).

    To demonstrate our ability to recognize and segment cars
in a variety of viewpoints, we experiment on images from               Results: Considering the large interclass variability, heav-
the PASCAL 2006 challenge [2], a very difﬁcult dataset. In             ily occluded objects, and viewpoint and scale variation in
Figure 10, we show some example detections and segmen-                 the dataset, our quantitative and qualitative results are quite
tations at different viewpoints and scales on test images.             good. We achieve equal precision-recall at 61%. For refer-
Training: We trained using roughly 700 pre-segmented                   ence, the highest reported results [2] in the 2006 PASCAL
cars from the LabelMe database [12] and 300 cars from the              challenge had an equal precision-recall rate of about 45%
PASCAL training set, which we manually segmented. We                   (but note that this rate is for the full-scale test set, which is
train appearance models for four viewpoint ranges (45 de-              a much more difﬁcult test). In Figure 10, we demonstrate
grees each) with a scale range of 26 to 38 pixels tall. When           the ability to accurately detect, segment, and determine the
estimating our instance consistency terms, we rescale the              viewpoint of cars in a wide variety of cases. We also show
cars to 30-34 pixels tall, divide them into groups with view-          several examples of failure. Often the mistakes, such as
point ranges of 15 degrees, and subdivide those into two               getting viewpoint wrong by 180 degrees or thinking that a
groups according to aspect ratio (width to height of bound-            double-decker bus is actually two cars are reasonable in the
ing box). Thus, during the reﬁnement stage, we are able to             absence of high-level contextual information.
accurately recover the viewpoint and bounding box of the
cars. We set βbg = 4.75 and an instance cost for each view-
                                                                          We also measure the value of our 3D model and of
point at α ∗ {1000, 750, 1100, 1100}, where α determines
                                                                       modeling the color distribution of the object. For the for-
the precision-recall trade-off. We set the weight of the ob-
                                                                       mer, we assign a 2D grid of parts for each viewpoint range
ject color term to 0.25. We have not found the algorithm to
                                                                       and relabel based on appearance, as is described in [16].
be highly sensitive to these parameters, except βbg , which
                                                                       After learning appearance models under this part-labeling
must be set sufﬁciently high to allow high recall, but low
                                                                       method, we then run our inference keeping other aspects of
enough so that the entire image is not initially assigned to
                                                                       the algorithm equal (e.g., we include instance costs and the
object parts (i.e., no pixels assigned to background).
                                                                       object color term). Our 3D model outperforms the 2D grid
Testing: To create our test set, we downsample the PAS-                method of initial part assignment in accuracy (by about 5%
CAL car test set to 160x120 pixels and test on the ﬁrst 150            recall at the equal precision-recall point of 60%), produces
images that contain cars at least 26 pixels tall. In a multi-
                                                     √                 better segmentations, and a more precise viewpoint estima-
scale search, downsampling the image in steps of 2, we                 tion. Similarly, including the color model improves recall
process only those scales for which at least one car is be-            by about 5% at 60% precision and improves segmentation.
            Figure 10. Results of multi-view car detection and segmentation on test images of the challenging Pascal Dataset.

5. Discussion and Future Work                                           [3] D. Hoiem, A. Efros, and M. Hebert. Putting objects in perspective.
                                                                            In CVPR, 2006.
   We introduce a method which we believe is the ﬁrst to                [4] V. Kolmogorov. Convergent tree-reweighted message passing for en-
combine multi-viewpoint class recognition with segmenta-                    ergy minimization. IEEE Trans. PAMI, 28(10):1568–1583, 2006.
tion. Our 3D LayoutCRF model makes it possible to reason                [5] V. Kolmogorov and R. Zabih. What energy functions can be mini-
                                                                            mized via graph cuts? IEEE Trans. PAMI, 26(2):147–159, 2004.
about object-level properties, such as viewpoint, size and              [6] A. Kushal and J. Ponce. Modeling 3d objects from stereo views and
color, while also reasoning about pixel-level appearance,                   recognizing them in photographs. In ECCV, 2006.
part consistency and occlusion. Another important contri-               [7] K. N. Kutulakos and S. Seitz. A theory of shape by space carving. In
bution is an instance cost, which improves segmentation ac-                 TR692, Computer Science Dept., U. Rochester. May 1998.
                                                                        [8] V. Lepetit, P. Lagger, and P. Fua. Randomized trees for real-time
curacy and allows non-contiguous regions to be assigned to                  keypoint recognition. In CVPR, June 2005.
the same object.                                                        [9] A. Opelt, A. Pinz, and A. Zisserman. Incremental learning of object
   Some conceptually simple (but perhaps technically dif-                   detectors using a visual shape alphabet. In CVPR, 2006.
ﬁcult) extensions include reducing the currently-prohibitive           [10] A. L. Ratan, W. E. L. Grimson, and I. William M. Wells. Object de-
computational time, modeling a larger number of objects,                    tection and localization by dynamic template warping. Int. J. Com-
                                                                            puter Vision, 36(2):131–147, 2000.
and modeling scale dependencies (e.g., as in [3]). Major               [11] C. Rother, V. Kolmogorov, and A. Blake. GrabCut -interactive fore-
challenges include extension to non-rigid or articulated ob-                ground extraction using iterated graph cuts. In ACM SIGGRAPH
jects and integration with methods that are more appropriate                2004.
for objects without a well-deﬁned shape, such as buildings             [12] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman. La-
                                                                            belMe: a database and web-based tool for image annotation. MIT AI
or grass.                                                                   Lab Memo AIM-2005-025, 2005.
Acknowledgements: We would like to thank Vladimir                      [13] H. Schneiderman and T. Kanade. A statistical method for 3D object
Kolmogorov for insights on the submodularity of the in-                     detection applied to faces and cars. In CVPR, 2000.
stance cost and for improving the TRW-S inference speed.               [14] A. Thomas, V. Ferrar, B. Leibe, T. Tuytelaars, B. Schiel, and
                                                                            L. Van Gool. Towards multi-view object class detection. In CVPR,
References                                                                  2006.
                                                                       [15] A. Torralba, K. Murphy, and W. Freeman. Sharing features: efﬁcient
 [1] E. Borenstein, E. Sharon, and S. Ullman. Combining top-down and        boosting procedures for multiclass object detection. CVPR, 2004.
     bottom-up segmentation. In CVPR, 2004.                            [16] J. Winn and J. Shotton. The layout consistent random ﬁeld for rec-
 [2] M. Everingham, A. Zisserman, C. Williams, and L. V. Gool. The          ognizing and segmenting partially occluded objects. In CVPR, 2006.
     Pascal VOC2006 results. Technical report, 2006.

Shared By: