basicstyle=, commentstyle=, frame=single,
stringstyle=, numbers=left,numbersep=5pt,
numberstyle=, breaklines=true, tabsize=3

cisco[1][mathescape=true]size=fbox, enhanced, breakable, listing only, listing

cisco2col[1][mathescape=true,escapeinside=(**)]size=fbox, colback = blue!5, enhanced, breakable, listing only, listing
options=style=tcblatex,basicstyle=,tabsize=2, multicols = 2,numbers = left,  xleftmargin = 1em,
showstringspaces = false,#1

, Watch me Explode:
A Light in the Dark for Revealing MPLS

Yves Vanaubel^∗, Jean-Romain Luttringer^, Pascal Mérindol^,  Jean-Jacques
Pansiot^, Benoit Donnet^∗

∗ Montefiore Institute, Université de Liège – Belgium

Icube, Université de Strasbourg – France

Internet topology discovery has been a recurrent research topic for nearly 20
years now. Usually, it works by sending hop-limited probes (i.e., )
towards a set of destinations to collect topological data in order to infer the
Internet topology at a given scale (e.g., at the router or the AS level).
However, comes with multiple limitations, in particular with layer-2
clouds such as MPLS that might hide their content to exploration.
Thus, the resulting Internet topology data and models are incomplete and

In this paper, we introduce (Trace the
Naughty Tunnels), an extension to
Paris traceroute for revealing most (if not all) MPLS tunnels along a path. works in two basic stages. First, along with probes, it looks for
evidences of the potential presence of hidden tunnels. Those evidences are
surprising patterns in the output, e.g., abrupt and significant TTL
shifts. Second, if alarms are triggered due to the presence of such evidences,
launches additional and dedicated probing for possibly revealing the
content of the hidden tunnel. We validate through emulation with GNS3 and
tune its parameters through a dedicated measurement campaign. We also largely
deploy on the Archipelago platform and provide a quantification of tunnels,
updating so the state of the art vision of MPLS tunnels. Finally, and its
validation platform are fully and publicly available, as well as the collected
data and scripts used for processing data.

§ INTRODUCTION

For now twenty years, the Internet topology discovery has attracted a lot of
attention from the research community [1, 2].  First, numerous
tools have been proposed to better capture the Internet at the IP interface
level (mainly based on ) and at the router level (by aggregating IP
interfaces of a router through alias resolution).  Second, the data
collected has been used to model the Internet [3], but also to
have a better knowledge of the network ecosystem and how it is organized by

However, despite the work done so far, a lot of issues still need to be fixed,
especially in data collection processes based on .  For instance,
collecting data about Layer-2 devices connecting routers is still an open
question, although it has been addressed previously with a, nowadays, deprecated
tool (i.e., IGMP-based probing) [4].  Another example is the
relationship between traditional network hardware and the so-called
middleboxes [5, 6].  Finally, MPLS
tunnels [7] also have an impact on topology discovery as they
allow to hide internal hops [8, 9].

This paper focuses on the interaction between and MPLS.  In a
nutshell, MPLS has been designed to reduce the time required to make forwarding
decisions thanks to the insertion of labels (called Label Stack
Entries, or LSE) before the IP header.[Although MPLS can also  be used
with IPv6, we consider only IPv4 in this paper.] In an MPLS network, packets are
forwarded using an exact match lookup of a 20-bit value found in the LSE. At
each MPLS hop, the label of the incoming packet is replaced by a corresponding
outgoing label found in an MPLS switching table. The MPLS forwarding engine is
lighter than the IP forwarding engine because finding an exact match for a label
is simpler than finding the longest matching prefix for an IP address. Some MPLS
tunnels may be revealed to because MPLS routers are able to generate
ICMP messages when the MPLS TTL expires and the ICMP message embeds
the LSE, revealing so the presence of the tunnel [10, 8].
However, MPLS supports optional features that make tunnels more or less
invisible to . Such features modify the way routers process the IP
and MPLS TTL of a packet. By carefully analyzing some MPLS related patterns like
TTLs (e.g., the quoted forward TTL, the returned TTL of both error and standard
replies), one can identify and possibly discover the L3-hops hidden within an
MPLS cloud. A first attempt has been already proposed for revealing so-called
Invisible tunnels [9]. Here we are going several steps further
by providing new revelation techniques (in particular for dealing with the
ultimate hop popping feature), and its validation with multiple MPLS and BGP
configurations (by emulating MPLS network through GNS3).

This paper aims at plugging the gaps in identifying and revealing the content of
MPLS tunnels. This is done by introducing (Trace the
Naughty Tunnels), an open-source
extension for Paris traceroute [11] including techniques for
inferring and revealing MPLS tunnels content.  Compared to our previous
work [8, 9], this paper provides multiple contributions:

* we strongly revise the MPLS tunnel classification as proposed
by Donnet et al. [8]. In particular, when possible, we subdivide
the “Invisible Tunnel” class in two more accurate categories, “Invisible
PHP” and “Invisible UHP”. We show that actually those tunnels can be
systematically revealed when they are built with basic P2P LDP [12] or
RSVP-TE [13] circuits (and can be at least detected if constructed
with more complex technologies such as P2MP VPRN [14]).  We also explain
why most “Opaque” tunnels content cannot be revealed in practice.  Finally,
we refine and even correct  previous quantification of each tunnel class with
large-scale measurements performed in the wild;

* we complement the state of the art with -based
measurement techniques able to reveal most (or at least detect all)
MPLS tunnels, even those that were built for hiding their content.  While our
previous work [9] required to target suspect and
pre-analyzed zones in the Internet (i.e., considering high degree nodes and
their neighbors visible in the ITDK dataset [15]), we provide here
measurement techniques fully integrated in . We associate with each
category of the classification indicators or triggers that are
used to determine, on the fly, the potential presence of a tunnel and possibly
its nature. In particular, in this paper, we are able to identify the presence
of the newly introduced “UHP Invisible” tunnel class thanks to the
duplication of an IP address in the output.  When a trigger is
pulled during a exploration, an MPLS
revelation [9] is launched with the objective of
revealing the tunnel content.  We validate the indicators, triggers, and
revelations using GNS-3, an emulator running the actual IOS of real routers in
a virtualized environment[See
<https://gns3.com/> Note that it is also possible to emulate other router
brands, e.g., Juniper, with GNS-3.], on a large set of realistic
configurations. We also show, through measurements, that our techniques are
efficient in terms of cost (i.e., the additional amount of probes injected is
reasonable, specially compared to the quality of new data discovered) and
errors (false positives and false negatives);

* we implement those techniques within
Scamper [16], the state of the art network measurements
toolbox as a Paris traceroute extension, called , and deploy it on the
Archipelago infrastructure [17].  aims at replacing the old version
of Scamper and is, thus, subject to run every day towards millions of
destinations.  As such, we believe will be useful to study MPLS
deployment and usage over time, increasing so our knowledge and culture on
this technology;

* we analyze the data collected, the efficiency of in doing
so (for tuning it to its best set of calibration parameters) and report a new
quantification on MPLS deployment in the wild, correcting and updating so
previous results that erroneously underestimated or overestimated
the prevalence of some tunnel classes [8];

* we work in a reproducibility perspective.  As such, all our
code (, GNS-3, data processing and analysis) as well as our collected
dataset are made available.[See
<http://www.montefiore.ulg.ac.be/ bdonnet/mpls>]

The remainder of this paper is organized as follows: Sec. <ref>
provides the required technical background for this paper;
Sec. <ref> revises the MPLS taxonomy initially introduced by
Donnet et al. [8] in the light of newly understood MPLS behaviors;
Sec. <ref> formally introduces , our extension to for
revealing the content of all MPLS tunnels; Sec. <ref> discusses parameters and its calibration, while Sec. <ref> presents results of
the deployment over the Archipelago architecture; Sec. <ref>
positions our work with respect to the state of the art; finally, Sec. <ref>
concludes this paper by summarizing its main achievements.

§ BACKGROUND

This section discusses the technical background required for the paper.
Sec. <ref> explains how hardware brand can be inferred
from collected TTLs. Sec. <ref> provides the basics
of MPLS labels and introduces the MPLS control plane while
Sec. <ref> focuses on the MPLS data plane and MPLS TTL processing.

§.§ Network Fingerprinting

Router Signature     Router Brand and OS

$< 255, 255 >$     Cisco (IOS, IOS XR)

$< 255, 64 >$      Juniper (Junos)

$< 128, 128 >$     Juniper (JunosE)

$< 64, 64 >$     Brocade, Alcatel, Linux

Summary of main router signature, the first initial TTL of the pair
corresponds to ICMP , while the second is for ICMP .

Vanaubel et al. [18] have presented a router fingerprinting
technique that classifies networking devices based on their hardware and
operating system (OS). This method infers initial TTL values used by a router
when forging different kinds of packets. It then builds the router
signature, i.e., the $n$-tuple of $n$ initial TTLs. A basic pair-signature
(with $n$ = 2) simply uses the initial TTL of two different messages: an ICMP
message elicited by a probe, and an ICMP message obtained from an probe.
Table <ref> summarizes the main router
signatures, with associated router brands and router OSes. This feature is
really interesting since the two most deployed router brands, Cisco and Juniper,
have distinct MPLS behaviors and signatures.

20Label 3TC
1S 8LSE-TTL

The MPLS label stack entry (LSE) format.

§.§ MPLS Basics and Control Plane

MPLS routers, i.e., Label Switching Routers (LSRs), exchange labeled
packets over Label Switched Paths (LSPs). In practice, those packets are
tagged with one or more label stack entries (LSE) inserted between the
frame header (data-link layer) and the IP packet (network layer). Each LSE is
made of four fields as illustrated by Fig. <ref>:
an MPLS label used for forwarding the packet to the next router, a Traffic Class
field for quality of service, priority, and Explicit Congestion
Notification [19], a bottom of stack flag bit (to indicate whether the
current LSE is the last in the stack [20])[To simplify the
presentation we will consider  only one LSE in the remainder of this paper], and
a time-to-live (LSE-TTL) field having the same purpose as the IP-TTL
field [21] (i.e., avoiding routing loops).

Labels may be allocated through the Label Distribution Protocol
(LDP) [12]. Each LSR announces to its neighbors the association
between a prefix in its routing table and a label it has chosen for a given
Forwarding Equivalent Class (a FEC is a destination prefix by default),
populating so a Label Forwarding Information Table (LFIB) in each LSR.
With LDP, a router advertises the same label to all its neighbors for a given
LDP is mainly used for scalability reasons (e.g., to limit BGP-IGP interactions
to edge routers) and to avoid anomalies for the transit traffic such as iBGP
deflection issues. Indeed, LDP deployed tunnels use the routes computed by
the IGP (without any interest at the first, and naive, glance) as the LFIB is
built on top of the IGP FIB. Labels can also be distributed through
RSVP-TE [13], when MPLS is used for Traffic Engineering (TE) purposes.
In practice, most operators deploying RSVP-TE tunnels use
LDP [9] as a default labeling protocol.

With LDP, MPLS has two ways of binding labels to destination prefixes:

* through ordered LSP control (default configuration of Juniper
routers [22]);

* through independent LSP control (default configuration of Cisco
routers <cit.>).

In the former mode, a LSR only binds a label to a prefix if this prefix is local
(typically, the exit point of the LSR), or if it has received a label binding
proposal from the IGP next hop towards this prefix. This mode is thus iterative
as each intermediate upstream LSR waits for a proposal of its downstream LSR (to
build the LSP from the exit to the entry point).  Juniper routers use this mode
as default and only propose labels for loopback IP addresses. In the second
mode, the Cisco default one, a LSR creates a label binding for each prefix it
has in its RIB (connected or – redistributed in – IGP routes only) and
distributes it to all its neighbors. This mode does not require any proposal
from downstream LSRs.  Consequently, a label proposal is sent to all neighbors
without ensuring that the LSP is enabled up to the exit point of the tunnel. LSP
setup takes less time but may lead to uncommon situations in which an LSP can end
abruptly before reaching the exit point (see Sec. <ref> for details.)

The last LSR towards a FEC is the Egress Label Edge Router (the Egress LER
– PE$_2$ in Fig. <ref>). Depending on its configuration, two
labeling modes may be performed.  The default mode [9] is
Penultimate Hop Popping (PHP), where the Egress advertises an Implicit
NULL label (label value of 3 [20] – see
Table <ref> for details on reserved MPLS label
values). The previous LSR (Penultimate Hop LSR,PH – P$_3$ in
Fig. <ref>) is in charge of removing the LSE to reduce the
load on the Egress. In the Ultimate Hop Popping (UHP), the Egress LER
advertises an Explicit NULL label (label value of 0 [20] – see
Table <ref> for details on reserved MPLS label
values). The PH will use this Explicit NULL label and the Egress LER will be
responsible for its removal. Labels assigned by LSRs other than the Egress LER
are distinct from Implicit or Explicit NULL labels. The Ending Hop LSR
(EH) is the LSR in charge of removing the label, it can be the PH in case of
PHP, the Egress LER in case of UHP or possibly another LSR in the case of
independent LSP control.

Table <ref> provides a summary of main acronyms used
by MPLS and their corresponding concept in the classic IP world.

Acronym     Meaning     IP

LSR     Label Switching Router     3*Router

PH LSR     Penultimate Hop LSR

EH      Ending Hop LSR

LER     Label Edge Router        Border Router

LSP     Label Switching Path     Tunnel

LSE     Label Sack Entry     Header

LSE-TTL     LSE Time-to-Live     IP-TTL

LDP     Label Distribution Protocol     2*Signaling

RSVP-TE     ReSerVetation Protocol – Traffic Engineering     (control plane)

LIB      Label Information Base                        RIB

LFIB     Label Forwarding Information Base     FIB

PHP     Penultimate Hop Popping     2*Decapsulation

UHP     Ultimate Hop Popping

FEC     Forwarding Equivalent Class     QoS Class

MPLS terminology with its matching in the classic IP world.

§.§ MPLS Data Plane and TTL processing

Illustration of MPLS vocabulary and relationship between MPLS and
. The figure is made of three parts.  The upper part represents the
network topology we use, throughout the paper to illustrate concepts.  In
particular, with respect to MPLS, P$_1$ is the LSP First Hop (FH), while P$_3$ is
the Penultimate Hop (PH). In case of PHP, P$_3$ is the Ending Hop (EH)
and  is responsible for removing the LSE. In case of UHP, the LSE is removed
by the Egress LER (PE$_2$). The middle part of the figure presents the MPLS
Tunnel classification, as observed with (this classification
is a revision of the original one proposed by Donnet et al.) Finally, the
bottom part of the figure provides triggers and indicators of an MPLS tunnel
presence when probing with . The relationship between the
trigger/indicator and the observation made with probing is provided in red.
Additional information (such as path length) are provided. This
is used in Sec. <ref> for illustrating .

Depending on its location along the LSP, a LSR applies one of the three
following operations:

* (Sec. <ref>).  The first MPLS router, i.e., the
tunnel entry point pushes one or several LSEs in the IP packet that turns into
an MPLS one. The Ingress Label Edge Router (Ingress LER) associates the
packet FEC to its LSP.

* (Sec. <ref>).  Within the LSP, each LSR makes a
label lookup in the LFIB, swaps the incoming label with its corresponding
outgoing label, and sends the MPLS packet further along the LSP.

* (Sec. <ref>).  The EH, the last LSR of the LSP,
deletes the LSE, and converts the MPLS packet back into an IP one. The EH can
be the Egress Label Edge Router (the Egress LER) when UHP is enabled or
the PH otherwise.

Fig. <ref> (above part) illustrates the main vocabulary
associated to MPLS tunnels.

§.§.§ LSP Entry Behavior

When an IP packet enters an MPLS cloud, the Ingress LER binds a label to the
packet thanks to a lookup into its LFIB, depending on the packet FEC, e.g., its
IP destination prefix.  Prior to pushing the LSE into the packet, the Ingress
LER has to initialize the LSE-TTL (see Fig. <ref>).
Two behaviors can then be configured: either the Ingress LER resets the LSE-TTL
to an arbitrary value (255,  ) or it copies the current IP-TTL value
into the LSE-TTL (, the default behavior).  Operators can configure
this operation using the option provided by the router
manufacturer [21]. In the former case, the LSP is called a pipe
LSP, while, in the latter case, a uniform one.

Once the LSE-TTL has been initialized, the LSE is pushed on the packet and then
sent to an outgoing interface of the Ingress LER. In most cases, except for a
given Juniper OS (i.e., Olive), the IP-TTL is decremented before being
encapsulated into the MPLS header.

§.§.§ LSP Internal Behavior

Upon an MPLS packet arrival, an LSR decrements its LSE-TTL.  If it does not
expire, the LSR looks up the label in its LFIB.  It then swaps the top LSE with
the one provided by the LFIB. The operation is actually a swap only if the
outgoing label returned by the LFIB is neither Implicit NULL nor
empty[In practice the actual label used for the forwarding is then
greater than or equal to 0 (this specific value being reserved for Explicit NULL
tunnel ending, i.e. for UHP) but excluding by design the reserved value $3$ that
is dedicated for Implicit NULL.]. Otherwise, it is a operation as described
in the next subsection. Finally, the packet is sent to the outgoing interface of
the LSR with a new label, both according to the LFIB.

If the LSE-TTL expires, the LSR, in the fashion of any IP router, forges an ICMP
that is sent back to the packet originator.  It is worth to notice
that a LSR may implement RFC 4950 [24] (as it should be the case in
all recent OSes).  If so, it means that the LSR will quote the full MPLS LSE
stack of the expired packet in the ICMP message.

ICMP processing in MPLS tunnels varies according to the ICMP type of message.
ICMP Information messages (e.g., ) are directly sent to the
destination (e.g., originator of the ) if the IP FIB allows for it
(otherwise no replies are generated). On the contrary, ICMP Error messages
(e.g., ) are generally forwarded to the Egress LER that will be in
charge of forwarding the packet through its IP plane [8].
Differences between Juniper and Cisco OS and configurations are discussed in
details in Sec. <ref>.

Value     Meaning     Mechanism

3*0     3*IPv4 Explicit NULL     downstream LSR should pop label

immediately (UHP – popped packet

is an IPv4 datagram)

1     Router Alert           deliver to control plane, do not forward

3*2     3*IPv6 Explicit NULL     downstream LSR should pop label

immediately (UHP – popped packet

is an IPv6 datagram)

3*3     3*Implicit NULL     pop immediately and treat as an IPv4

packet (PHP)

Reserved label values in MPLS and their meanings.

§.§.§ LSP Exit Behavior

At the MPLS packet arrival, the EH decrements the LSE-TTL. If this TTL
does not expire, the EH then pops the LSE stack after having determined the new

Applying PHP comes with the advantage of reducing the load on the Egress LER,
especially if it is the root of a large LSP-tree. This means that, when using
PHP, the last MPLS operation (i.e., ) is performed one hop before the Egress
LER, on the PH. On the contrary, UHP is generally used only when the ISP
implements more sophisticated traffic engineering operations or wants to make
the tunnel content and semantics more transparent to the customers.[The
UHP feature has been recently made available  on Juniper routers when LSPs are
set with LDP.  However, PHP remains the rule on
Juniper <cit.>.]

When leaving a tunnel, the router has to decide which is the correct TTL value
(IP-TTL or LSE-TTL) to copy in the IP header.  If the Ingress LER has activated
the option, the EH should pick the IP-TTL of the incoming packet
while the LSE-TTL should be selected otherwise. This way, the resulting outgoing
TTL cannot be greater than the incoming one: in the former case, internal hops
are not counted because the tunnel is hidden while they are for the latter case.
In both cases, the TTL behavior remains monotonic. In order to synchronize both
ends of the tunnel without any message exchange, two mechanisms might be used
for selecting the IP-TTL at the EH:

* applying a IP-TTLLSE-TTL operation (solution implemented for
Cisco PHP configurations [23]);

* assuming  the Ingress configuration (or not) is the same as
the local configuration (solution implemented by some JunOS and also in some
Cisco UHP configuration).

Applying the IP-TTLLSE-TTL is the best option because it correctly
supports heterogeneous configurations in any case while, at the same
time, mitigating forwarding loop without exchanging signalization messages.

This IP-TTLLSE-TTL behavior might be used for detecting the presence
of hidden MPLS tunnels [9].  Indeed, it is likely that the EH
generating the ICMP message will use the same MPLS cloud back to
reply to the vantage point.  In that case, when the reply will leave the MPLS
cloud, the returning EH ($P_1$ in Fig. <ref>) will choose to
copy the LSE-TTL in the IP-TTL, as the IP-TTL has been initialized at its
maximum value on the Egress of the forward tunnel ($255$ for a Cisco router –
see Sec. <ref>).  As a consequence, while the forward
path hides the MPLS cloud because the IP-TTLLSE-TTL operated on the
forward PH ($P_3$) selects the IP-TTL which is lower, the return path indicates
its presence because the returning PH ($P_1$), on the contrary, selects the
LSE-TTL.  In general, a sufficient condition for this pattern to occur is if the
returning Ingress, which is the forward EH, re-uses the MPLS cloud back.

In practice, it is interesting to mention that this MPLS behavior is strongly
dependent on the implementation and the configuration.  For instance, on some
Juniper OS routers (at least with JunOS Olive) or when the UHP option is
activated on some Cisco IOS (at least with the 15.2 version), the
IP-TTLLSE-TTL operation is not – systematically – applied. The EH
assumes that the propagation configuration is homogeneous among LERs. When it is
not the case (at one end of the tunnel and at the
other end), the PH (for PHP routers without IP-TTLLSE-TTL) or the
Egress LER (for the Cisco UHP configuration) will use the IP-TTL instead of the
LSE-TTL, leading so to a so-called jump effect with (i.e., as
many hops as the LSP length are skipped after the tunnel).  Except when
explicitly stated, we will consider homogeneous configurations (e.g.,
on the whole tunnel) in the remainder of the paper. Finally, it is
worth noticing that mixing UHP and PHP (hybrid configurations) can also result
in uncommon behaviors.[Those behaviors are described and
discussed in details in the Appendix, at the end of this paper.]

§ REVISITING MPLS TUNNELS TAXONOMY

According to whether LSRs implement RFC4950 or not
(Sec. <ref>) and whether they activate the option
or not (Sec. <ref>), MPLS tunnels can be revealed to
following Donnet et al. [8] taxonomy.

Explicit tunnels are those with RFC4950 and the option
activated (this is the default configuration). As such, they are fully visible
with , including labels along the LSP.  Implicit tunnels
activate the option but do not implement the RFC4950. No IP
information is missed but LSRs are viewed as ordinary IP routers, leading to a
lack of “semantic” in the output. Opaque tunnels are
obscured from as the option is disabled while the
RFC4950 is implemented and, more decisive, the EH that pops the last label has
not received an Explicit or Implicit NULL proposal for the given FEC (making the
LSP end in a non controlled fashion). Consequently, the EH can be seen while the
remainder of the tunnel is hidden.  Finally, Invisible tunnels are hidden
as the option is activated (RFC4950 may be implemented or not).

As illustrated in Fig. <ref> (middle part), Explicit tunnels
are the ideal case as all the MPLS information comes natively with .
For Implicit tunnels, Donnet et al. [8] have proposed techniques for
identifying the tunnel based on the way LSRs process ICMP messages (see
Sec. <ref> – the so-called ) and the IP-TTL quoted in
the message (the so-called qTTL) that is increased by one at each
subsequent LSR of the LSP due to the option (ICMP are
generated based on the LSE-TTL while the IP-TTL of the probe is left unchanged
within the LSP and, thus, quoted as such in the ICMP ).

Opaque tunnels are only encountered with Cisco LSPs and are a consequence of the
way labels are distributed with LDP (see Sec. <ref>). Indeed, a
label proposal may be sent to all neighbors without ensuring that the LSP is
enabled up to the Egress LER, leading so to Opaque tunnels because an LSP can
end abruptly without reaching the Egress LER (where the prefix is injected in
the IGP) that should bind an Explicit (UHP) or Implicit NULL label (PHP).  As
illustrated in Fig. <ref>, Opaque tunnels and their length can
be identified thanks to the LSE-TTL. These LSPs end without a standard terminating
label (Implicit or Explicit NULL) and so they break with the last MPLS
header of the neighbor that may not be an MPLS speaker. Thanks to our large
scale campaign and cross-validation with our emulation platform, we realized
that the vast majority of Opaque tunnels seems to be caused by
Carrier-of-Carriers VPN [26] or similar technologies. Indeed, they
provoke an abrupt tunnel ending (the bottom label is necessarily carried up to
the end of the tunnel to determine the correct outgoing VPN), and unfortunately
lead to non revealable tunnels as we will show later.

The behavior, for Invisible tunnel, is different according to the
way the LSE is popped from the packet (i.e., UHP or PHP), as illustrated in
Fig. <ref>.   Invisible tunnels are problematic, as they lead
to a false vision of the Internet topology, creating false links, and spoiling
graph metrics, such as the node degree distribution [9]. In
this paper, we revisit the original taxonomy by doing a clear distinction
between Invisible tunnels produced with PHP and UHP. In Donnet et
al. [8],  the class “Invisible” only covered PHP.
Vanaubel et al. [9] have since proposed techniques for
revealing the content of Invisible MPLS tunnels only in the case of PHP.

With Invisible UHP tunnels, the behavior is clearly different, at least for
Cisco routers using the 15.2 IOS[While it is now possible to enable UHP
with Juniper for LDP, is not able to make a distinction between the two
because the visible and revealed patterns are the same.]. Upon reception of a
packet with IP-TTL of 1, the Egress LER does not decrement this TTL, but,
rather, forwards the packet to the next hop ($CE_2$ in the example), so that the
Egress does not show up in the trace. In contrast, the next hop will appear
twice: once for the probe that should have expired at the Egress and once at the
next probe. UHP indeed provokes a surprising pattern, a duplicated IP at two
successive hops, illustrated as “Invisible UHP” in
Fig. <ref>. This duplicated IP addresses might be
misunderstood as a forwarding loop.

On the contrary, PHP moves the function at the PH, one hop before the
tunnel end. This PH does not decrement the IP-TTL whatever its value is.
Except for some JunOS, the packet is still MPLS switched because the LSE-TTL has
not expired on it. It is somehow surprising because for Explicit and Implicit
tunnels, the PH replies on its own. It is because, in this cases, the LSE-TTL has also expired.
In Fig. <ref>, we can see that there is no more asymmetry in
path length for router P$_3$ proving so its reply does not follow a via
the Egress. On the contrary, any other LSR on the LSP builds a message when the LSE-TTL expires and then continues to MPLS switch their reply
error packet to the Egress LER unless the  command has been activated for Cisco routers. It seems to be just
an option for Juniper routers with the  command.

Note that Opaque and Invisible UHP tunnels are Cisco tunnels (signature
$<255,255>$) due to specific implementations. Invisible PHP are Juniper
(signature $<255,64>$), Linux boxes (signature $<64,64>$), or Cisco tunnels but
they do not behave exactly the same as we will explain later.

Sec. <ref> extends techniques for revealing MPLS tunnels by proposing
and implementing integrated measurement techniques for all tunnels (i.e.,
Explicit, Implicit, Opaque, and both UHP and PHP Invisible ones) in a single
tool called .

§ DESIGN AND REPRODUCIBILITY

This section introduces our tool, (Trace the
Naughty Tunnels), able to reveal MPLS
tunnels along a path.  is an extension to Paris
Traceroute [11] so that we avoid most of the problems related
to load balancing.  has been implemented within Scamper [16], the
state-of-the-art network measurements toolbox, and is freely
available.<ref> Sec. <ref> provides
an overview of , while Sec. <ref> and Sec. <ref>
focus on techniques for revealing hidden tunnels and how those techniques are
triggered. Finally, Sec. <ref> explains how we validated on
a GNS-3 platform<ref>, an emulator running the
actual OS of real routers in a virtualized environment.

§.§ Overview

[escapeinside=(**), label=tnt.overview.trace_tunnel,
caption=Pseudo-code for ]
Codes := 0, None ; 1, LSE ; 2, qTTL ; 3, (**); 4, LSE-TTL;
5, (**); 6, (**); 7, (**).
prev_hop, cur_hop, next_hop = None

for (ttl=STARTING_TTL, !halt(ttl, target), ttl++)
state, tun_code = None
next_hop = trace_hop(ttl)

#first check uniform tunnel evidence with indicators
tun_code = check_indicators(cur_hop) (**)
#possibly fires (**) with triggers or opaques tunnels
if (tun_code == None)
tun_code = check_triggers(prev_hop, cur_hop, next_hop)
#check if cur_hop does not belong to a uniform LSP
if (tun_code != None)
#potential hidden tunnel to reveal
state = reveal_tunnel(prev_hop, cur_hop, tun_code)
elif (tun_code == LSE-TTL)
#potential opaque tunnel to reveal
state = reveal_tunnel(prev_hop, cur_hop, tun_code)

#hop by hop and tunnel display
dump(cur_hop, tun_code, state)

#sliding pair of IP addresses
prev_hop = cur_hop #candidate ingress LER
cur_hop = next_hop #candidate egress LER

is conceptually illustrated in Listing <ref>. At
the macroscopic scale, the  function is a
simple loop that fires probes towards each processed target. consists in
collecting, in a hop-by-hop fashion,
intermediate IP addresses ( function) between the vantage
point and the target. Tracing a particular destination ends when the
function returns true: the target has been reached or a gap has
been encountered (e.g., five consecutive non-responding hops). uses a
moving window of two hops such that, at each iteration, it considers a potential
Ingress LER (i.e., ) and a potential Egress LER (i.e.,
) for possibly revealing an Invisible tunnel between them.
Indicators allow to check if the current hop does not belong to a uniform
tunnel, i.e., a visible one (see line <ref>).

For each pair of collected IP addresses with , checks
for the presence of tunnels through so called indicators and
triggers. The former provides reliable indications about the presence of
an MPLS tunnel without necessarily requiring additional probing. Generally,
indicators correspond to uniform tunnels (or to the last hop of an Opaque
tunnel), and are, mostly, basic evidence of visible MPLS presence such, as LSEs
quoted in the ICMP packet (see Sec. <ref> for details).
Triggers are mainly unsigned values suggesting the potential presence of
Invisible tunnels through a large shifting in path length asymmetry (see
Sec. <ref> for details). When exceeding a given threshold
$\mathcal{T}$, such triggers fire path revelation methods (function
) between the potential Ingress and Egress LERs as
developed in Sec. <ref>. If intermediate hops are found, they are
stored in a global stack structure named .

is  a parameter used to avoid tracing repeatedly the
nodes close to the vantage point [27], usually  $\in

Finally, at each loop iteration, the collected data is dumped into a warts file,
the Scamper file format for storing IPv4/IPv6 records. This job is
performed by the  function. It writes potential revealed hops
(available in the global stack structure ), and any
useful information, such as tags, identifying the tunnel's type, and
revelation method, if any.

§.§ Indicators and Triggers

[escapeinside=(**), label=tnt.trig.check_indicators,
caption=Pseudo-code for checking indicators]
code check_indicators(hop):
#hop must exist
if (hop == None)
return None

if (is_mpls(hop))
if ((**) < hop.lse_ttl < 255) (**)
#opaque tunnel are both indicators and triggers
return LSE-TTL (**)
#explicit tunnel
return LSE(**)

if (hop.qttl > 1)(**)
#implicit tunnel
return qTTL(**)

#retrieve path length from raw TTLs
(**) = path_len(hop.ttl_te)
(**) = path_len(hop.ttl_er)

#(**) will be turned into (**) for junOS signatures
if (|(**) - (**)| (*$>$ *)        !signature_is_junOS(hop))(**)
#implicit tunnel
return (**)

return None

Tunnels indicators are pieces of evidence of MPLS tunnel presence and concern
cases where tunnels (or parts of them) can be directly retrieved from the
original . They are used for Explicit tunnels and uniform/visible
tunnels in general.  Explicit tunnels are indicated through LSEs directly quoted
in the ICMP message – See line <ref> in
Listing <ref> and output on
Fig. <ref>.  It is worth noting that
Fig. <ref> highlights the main patterns looks for firing
or not additional path revelation in a simple scenario where forward and return
paths are symmetrical.

The indicator for Opaque tunnels consists in a single hop LSP with the quoted
LSE-TTL not being equal to $1$, due to the way labels are distributed within
Cisco routers (see Sec. <ref>) or the way Cisco routers deal with
VPRN tunnel ending.[Juniper routers never lead to Opaque indicators
because they behave differently as discussed in Sec. <ref>]  This
is illustrated in Fig. <ref> where we get a value of $252$
because the LSP is actually $3$ hops long. This surprising quoted LSE-TTL is an
evidence in itself. It is illustrated in lines <ref>
to <ref> in Listing <ref>, where a hop
is tagged as Opaque if the quoted LSE-TTL is between a minimum threshold,
(see Sec. <ref> for fixing a value for the threshold) and 254
(LSE-TTL is initialized to 255 [21]). Note that this pattern resulting
from an Opaque tunnel is both an indicator and a trigger: passively
understands the tunnel is incomplete and try to reveal its content with new
active measurements.

Implicit tunnels are detected through qTTL and/or indicators [8]. First, if the IP-TTL quoted in an ICMP message (qTTL) is greater than one, it likely reveals the option at
the Ingress LER of an LSP. For each subsequent probe within the LSP,
the qTTL will be one greater, resulting in an increasing sequence of qTTL
values. This indicator is considered in line <ref> in
Listing <ref>. Second, the indicator relies on
the fact that, by default, LSRs send ICMP messages to the Egress
LER which, in turns, forwards the packets to the probing source.
However, they reply directly to other kinds of probes (e.g., ) using
their own IP forwarding table, if available. As a result, return
paths are generally shorter for messages than for replies.
Thereby, is the signature related to the difference in these lengths.
This is illustrated in Fig. <ref> (Implicit and Explicit
tunnels follow the same behavior except for  RFC4950 implementation).
On P$_1$, we have (P$_1$) = - = 9 - 3 = 6. With a symmetric
example, one can formalize the pattern for an LSR P$_i$ in an LSP of
length $LL$ as follows:

\begin{equation}
\uturn(P_i) = 2 \times (LL - i + 1) .
\label{eqn.uturn}
\end{equation}

Due to the iBGP path heterogeneity (the IGP tie-break rule in particular), the
BGP return path taken by the ICMP message can be different from the
BGP return path taken by the reply. This is illustrated in
Fig. <ref> where the two return paths in blue and red can
differ even outside the AS (L”$^{TE}_R$ can be distinct from L”$^{ER}_R$). As
a result, and because it may differ at each intermediate hop, the indicator does not necessarily follow exactly Eqn. <ref>. A small
variation may then appear in practice. In particular, a value of $0$ can hide a
true Implicit hop.

For JunOS routers, the situation is quite different. It turns out that, by
default (i.e., without enabling the  feature – see
Appendix <ref> for details), these routers send replies directly to the source, without forwarding them to the egress LER. The
indicator becomes then useless. Moreover, for routers having the JunOS
signature, the indicator and the trigger are computed in the same
way. Thus, to avoid any confusion, introduces an exception for such OS
signatures (line <ref> in Listing <ref>),
and first considers the difference as a trigger, and then falls back to an
indicator if the revelation fails (not shown in
Listing <ref> for clarity).
In addition, when  is enabled, replies start
with a TTL of $254$, implying a bigger difference with replies, as
it can be seen in Fig. <ref>:
$\uturn(P_1)=$-$=10-3=7$ instead of $6$ if $P_1$ runs a Cisco OS.

Implicit tunnels.

Invisible tunnels.

Indicators and triggers illustration for Implicit and Invisible
tunnels. Notations L'$^{\textrm{x}}_y$ and L”$^{\textrm{x}}_y$ refer to a
given sub-length of an ICMP packet  on the  path (y being
the forward or return path and x being a or ICMP
packet, see Fig. <ref>). For example, L'$^{TE}_R$
gives the return path of the within the MPLS cloud, while
L”$^{TE}_R$ is the return path of the between the MPLS
cloud and the vantage point. Consequently, we have =
L'$^{TE}_R$ +  L”$^{TE}_R$.

[escapeinside=(**), label=tnt.trig.check_trigger,
caption=Pseudo-code for checking triggers]
code check_triggers(prev_hop, cur_hop, next_hop):
#prev_hop and cur_hop must exist
#duplicate IP checked on cur_hop and next_hop
if (prev_hop == None or cur_hop == None or prev_hop == cur_hop)
return None

if (cur_hop == next_hop)(**)
#invisible UHP tunnel
return (**)
#retrieve path length from raw TTLs
(**) = path_len(cur_hop.ttl_te)(**)
(**) = path_len(cur_hop.ttl_er)
(**) = cur_hop.probe_ttl(**)

if (sign_is_junOS(cur_hop))(**)
#for the JunOS signature
if ((**) - (**) (*$\geq$ *))(**)
#invisible PHP tunnel with JunOS
return (**)
#for other signatures (raw TTLs are initialized the same)
if ((**) - (**) (*$\geq$ *))(**)
#invisible PHP tunnel with other known OS
return (**)

return None

Indicators are MPLS passive pieces of evidence that can also prevent
from firing new probes (with the exception of LSE-TTL that is also a
trigger for Opaque tunnels).
On the contrary, triggers are active patterns suggesting the presence of
Invisible tunnels (both PHP and UHP) that could be revealed using additional
probing (see Sec. <ref>).
Listing <ref> provides the pseudo-code for checking

First, we look for potential Invisible UHP tunnel (line <ref>). As explained
in Sec. <ref>, Invisible UHP tunnels occur with Cisco routers
using IOS 15.2.  When receiving a packet with an IP-TTL of 1, the Egress LER
does not decrement the TTL but, rather, forwards it directly to the next hop.
Consequently, the Egress LER does not appear in the trace while, on the
contrary, the next hop (CE$_2$ in Fig. <ref>) appears twice
(duplicate IP address in the trace output).

The two remaining triggers, (Return Tunnel Length
Analysis [9] – see Table <ref> for a
summary of acronyms used by ) and (Forward/Return Path Length
Analysis [9]), work by using three path lengths, which are
(the path length), (the path length), and
(the forward path length). More precisely, is the
difference between the and the return path lengths,
while is the difference between the forward and the return path lengths
(obtained based on probe and reply messages). tries to capture
significative differences between these lengths to infer the presence of MPLS
tunnels, relying on two common practices of LSRs, in particular the EH,
developed in the previous subsection. Both triggers are based on the idea that
replies sent back to the vantage point are also likely to cross back the MPLS
cloud, which will apply the IP-TTLLSE-TTL operation at the EH of the
return tunnel. These triggers respectively infer the exact () or
approximate () return path length. Indeed, is subject to BGP path
asymmetry (and so, to false positives or negatives) in opposition to when
it applies (it may produce some false alarms but only due to ECMP). In the
absence of Invisible tunnel, we expect those triggers to have a value equal or
close to $0$. Indeed, in such a case, we should have L'$^{ER}_R =$ L'$^{TE}_F=$
L'$^{TE}_R=1$ if BGP does not interfere (see Fig. <ref>).
Therefore, any significant deviation from this value is interpreted as the
potential presence of an Invisible MPLS cloud, and thus, brings to trigger
additional path revelation techniques (see Sec. <ref>). In
practice (look at Fig. <ref>), we expect to have
L'$^{ER}_R =$ L'$^{TE}_F = 1$ (due to the Min for the return
tunnel and the pipe mode for the forward tunnel) while L'$^{TE}_R$ directly
provides the actual return tunnel length (with a value $\geq 1$). It is due to
the Min operation applied by the EH of the return tunnel, which selects
the LSE-TTL of the reply, and keeps the IP-TTL for the packet. Indeed, in the case of the message, the return Ingress LER
(i.e., the forward Egress LER) initializes the LSE-TTL with the same value as
the IP-TTL, meaning 255. For packets, the IP-TTL is set to 64. is not subject to any BGP asymmetry because we have L”$^{ER}_R =$ L”$^{TE}_R$,
i.e. BGP return paths have the same length. Indeed, the two messages use the
same physical path, the only difference being the Min operation applied
at the EH of the return tunnel, if any.

To check for those triggers, we first extract the three key distances thanks to
the reply IP-TTLs received by the vantage point (lines <ref>
to <ref> in Listing <ref>). As explained by
Vanaubel et al. [9], only works with JunOS routers,
while is more generic. Therefore, prior to estimate the triggers, uses
network fingerprinting (see Sec. <ref>) to determine
the router brand of the potential Egress LER (line <ref> in
Listing <ref>).

In the presence of a JunOS hardware, is compared to , as in case of
an Invisible tunnel, is supposed to be greater than .
Indeed, with this routing platform, and packets have
different initial TTL values (see Table <ref>),
and the trigger can exploit the TTL gap between those two kinds of
messages caused by the IP-TTLLSE-TTL behavior at the Egress LER (the
appears longer than as the Min operation results in a
different pick). This difference represents the number of LSRs in the return
LSP, and is compared to a pre-defined threshold (line <ref> in
Listing <ref>). This threshold (see Sec. <ref>
for the parameter calibration) filters all the LSPs shorter than the limit it
defines. In the case depicted in Fig. <ref>:

\begin{eqnarray*}
\rtla(PE_2) &=& L^{ER}_{R} - L^{TE}_R\\
&=& L'^{ER}_R - L'^{TE}_R\\
&=& 6 - 3\\
&=& 3 .
\end{eqnarray*}

Indeed, for the message, we have

\begin{eqnarray*}
\textrm{IP-TTL} & = & 64\\
& = & \textsc{Min}(\textrm{IP-TTL}=64,\textrm{LSE-TTL}=252) .
\end{eqnarray*}

instead of

\begin{eqnarray*}
\textrm{IP-TTL} & = & 252\\
& = & \textsc{Min}(\textrm{IP-TTL}=255,\textrm{LSE-TTL}=252) .
\end{eqnarray*}

for the reply. Note that an invisible shadow effect also
applies for after the Invisible tunnel, as the trigger will still be
positive for a few nodes after the egress LER.

is more generic and applies thus to any configuration. allows to
compare, at the AS granularity, the length distribution of forward (i.e., )
and return paths (i.e., ). Return paths are expected to be longer than
forward ones, as the tunnel hops are not counted in the forward paths while they
are taken into account in the return paths (due to the IP-TTLLSE-TTL
behavior at the return Egress LER).Then, we can statistically analyze their
length difference and check if a shift appears (see Line <ref> in
Listing <ref>). This is illustrated in
Fig. <ref> (“Invisible PHP”) in which is 3 while is equal to 6, leading so to an estimation of the return tunnel length of 3.  In
general, when no IP hops are hidden, we expect that the resulting distribution
will look like a normal distribution centered in 0 (i.e., forward and return
paths have, on average, a similar length). If we rather observe a significant
and generalized shift towards positive values, it means the AS makes probably
use of the option. In order to deal with path asymmetry, uses a
threshold, (see Sec. <ref> for calibrating this parameter),
greater than $0$ to avoid generating too much false positives (revelation
attempt with no tunnel). The Min effect also results in an invisible
shadow after the hidden LSP: $\frpla(CE_2)=2$ and $\frpla(CE_3)=1$, etc until
the situation returns to normal. Note that the and shadows are the
reasons why does not look for consecutive Invisible tunnels in a trace.
Finally, for Invisible UHP, one can observe that no Min shift applies
on the return path, as only the duplicate effect is visible.

Threshold calibration will be discussed in details in Sec. <ref>. The
optimal calibration can provide a 80/20 % success/error rates (errors being due
to the BGP and ECMP noises). Moreover, the order in which considers
indicators and triggers, their codes, reflects their reliability, and so, their
respective success rates (and their resulting states): the lower the code (i.e.
the higher its priority), the more reliable (and higher the revelation success
rate). Thus, if a hop matches simultaneously multiple triggers (and for example), it is tagged with the one having the highest priority (i.e., in our example).

Acronym     Meaning     Usage

Forward/Return Path Length Analysis     2*Trigger

Return Tunnel Length Analysis

Direct Path Revelation     2*Path Revelation

Backward Recursive Path Revelation

Summary of acronyms used by .

§.§ Hidden Tunnels Revelation

[escapeinside=(**), label=tnt.revelation.reveal_tunnel,
caption=Pseudo-code for revealing Invisible tunnels]
state reveal_tunnel(ingress, egress, tun_code):
#ingress and egress hops must exist
if (ingress == None or egress == None)
return None
buddy_bit = False
#standard (**) towards the candidate egress
target = egress
route = trace(REV_STARTING_TTL, target)(**)

if (last_hop(route) != egress)
#the target does not respond (revelation is not possible)
return (**)(**)
else if (ingress (*$\notin$*) route)
#the forwarding path differs (revelation is not possible)
return (**)(**)
else if (distance(ingress, egress, route) > 1)
#path segment revelation with push_segment_to_revelation_stack(ingress, egress, route)
return (**)(**)
ttl = ingress.probe_ttl + 1
revealed_ip = extract_hop(ttl, route)

for iTR=0;;
if (revealed_ip == target)
if (tun_code != (**) || buddy_bit)
#no more progression in the revelation
#try with the buddy for the (**) trigger
target = buddy(revealed_ip)(**)
buddy_bit = True
#a new hop has been revealed
target = revealed_ip
buddy_bit = False

revealed_ip = traceHop(ttl, target)

if (iTR == 0)
#no revelation (fail)
return NOTHING_TO_REVEAL(**)
if (iTR == 1)
#single hop revealed LSP ((*$\approx$*) )
return 1HOP_LSP(**)
#hop by hop revelation with return (**)(**)

Listing <ref> offers a simplified view of the tunnel revelation. The first step consists in launching a standard towards the candidate Egress[We use the term candidate as, at
this point, we are not completely sure an MPLS tunnel is hidden there.]
(line <ref> in Listing <ref>).
is the starting TTL used for the revelation, which
corresponds to 2 hops before the candidate Ingress hop, by default.  During this
first attempt, may fail to reach the candidate Egress
(line <ref>), and/or the candidate Ingress
(line <ref>) when collecting the active data. Otherwise, may
reveal a tunnel and four additional output states can arise:

* an LSP composed of at least 2 LSRs is revealed in the first trace
towards the Egress (line <ref> – , Direct Path
Revelation [9] – Table <ref>
for a summary of acronyms used by );

* an LSP having more than one LSR is revealed using several iterations
(line <ref> – , Backward Recursive Path
Revelation [9]).

* nothing is revealed, the candidate Ingress and Egress are still
consecutive IP addresses in the trace towards the candidate Egress
(line <ref>);

* a single-hop LSP is revealed (line <ref>) although
several iterations have been tried:  and cannot be distinguished for
one hop LSPs.

With the default UHP configuration on Cisco IOS 15.2, an additional test, called
buddy (line <ref>), is required to retrieve the outgoing IP
interface of the Egress LER (the right interface, in green, on PE$_2$ in
Fig. <ref>), and thanks to its retrieval, force replies from
its incoming IP interface (the left one, in red, on PE$_2$ in
Fig. <ref>). The  function assumes a
point-to-point connection between the Egress LER and the next hop (IP addresses
on this point-to-point link are called buddies). In most cases, the
corresponding IP addresses belong to a /31 or a /30
prefix [4, 28].  Note that according to the IP address
submitted to , this function may require additional probing to
infer the correct prefix in use.
Besides, specific UDP probing is necessary in order to provoke
messages. Such error messages, as ones, enable to get
the incoming interface of the targeted router (instead of ones that
are indexed with the target IP).

(Direct Path Revelation) works when there is no MPLS tunneling for internal
IGP prefixes other than loopback addresses, i.e., the traffic to internal IP
prefixes is not MPLS encapsulated (default Juniper configuration but can also be
easily configured on Cisco devices – see Sec. <ref>) .
With PHP, (Backward Recursive Path Revelation) works because the target
(PE$_2$.left on Fig. <ref>) belongs to a prefix being also
advertised by the PH. Thus, the probe is popped one hop before the PH (P$_3$ on
Fig. <ref>), and it appears in the trace towards the Egress
incoming IP interface, e.g., PE$_2$.left on Fig. <ref>. is then applied recursively on the newly discovered interface until no new IP
address is revealed. works also natively with UHP on IOS 12.4 (i.e.,
without the  function), for the same reason as for PHP: the
prefix is local and shifts the end of the tunnel one hop before and, in this
implementation, the EH replies directly. On the contrary, needs to use the
function at each step for IOS 15.2 enabling UHP, because the EH
silently forwards the packet one hop ahead. Vanaubel et al. [9]
provides more details on and .

§.§ Reproducibility and Practical BGP Configurations

We use the  GNS3  emulation environment for several purposes.
First, we aim at verifying that the inference assumptions we considered in the
wild are correct and reproducible in a controlled environment. Second, some of
the phenomena we exploit to reveal tunnels in the wild have been directly
discovered in our testbed. Indeed, using our testbed we reverse-engineered the
TTL processing (considering many MPLS configurations, we study the operation in particular) of some common OSes used by many real routers. Finally,
it is also useful for debugging to test its features in this controllable
environment. Generally speaking, we aim at reproducing with GNS3 all common
behaviors observed in the wild, and, on the opposite, we also expect
to encounter in the wild all basic behaviors (based on standard MPLS and BGP
configurations) we build and setup within GNS3.

In practice, we have considered four distinct router OSes: two  Cisco
standard IOS (12.4 and 15.2), and  two virtualized versions of
JunOS (Olive and VMX, the only Juniper OSes we succeeded to emulate within
GNS3). We envision in a near future to also test the IOS XR and some other
Juniper OSes, if possible, but we believe that our tests are already
representative enough of most behaviors existing in the wild.

In our emulations, topologies (see Fig. <ref>) are configured
as follows.  We assumed that LERs are AS Provider-Edge (PE) routers, i.e., AS
border routers of the ISP running (e)BGP sessions.   Two main configurations are
then possible to enable transit tunneling at the edges. Either the BGP next-hop
can be the loopback IP address of the PE itself (with
command), or it belongs to the eBGP neighbor – and in that case the connected
subnet or the IP address should be redistributed in the ISP. In both cases,
there exists a LDP mapping, at each Ingress LER and for any transit forwarding
equivalent class (FEC) between the BGP next-hop, the IGP next-hop, and the local
MPLS label to be pushed. According to the configuration at the Egress LER, when
the Ingress LER is in pipe mode (see Sec. <ref>), distinct
kinds of tunnels emerge: Opaque, UHP Invisible, or PHP Invisible.

We consider the simplest possible configurations, i.e., homogeneous in terms of OS
and MPLS+BGP configurations. They are consistent and symmetric MPLS
configurations both in terms of signaling (LDP with the independent  model using
all IGP connected prefix – Cisco default mode – xor the ordered model using
only loopback addresses – Juniper default mode)[See
Sec. <ref>] and the propagation operation in use (pipe xor
uniform)[See Sec. <ref>] at the domain scale.
Using heterogeneous configurations, we discovered many intriguing corner cases
that are discussed in Appendix <ref>. Some of them may result in
incorrect TTL processing and other in hiding even more the tunnel to . In some rare
cases, only the Brute Force option of is able to fire the path revelation
that exposes tunnels.

The BGP configuration is also standard: the Egress LER enables the next-hop-self
feature and so the transit traffic is tunneled via this IP address. All LSRs
also have a global IGP routing table thanks to a route reflector (they can
answer natively to ping requests) or a redistribution in the IGP routing control
plane. The AS scale BGP prefix is advertised using a global aggregation and the
BGP inter-domain link is addressed by the neighbor but can be redistributed in
the IGP as a connected one.

Opaque tunnels show up when enabling the  command towards the BGP neighbor whose IP address is redistributed
statically in the IGP. works also with Cisco IOS when enabling the
command. Eventually, the
command  allows for revealing
UHP tunnels without the use of the buddy.

Table <ref> provides a summary of capacities considering
several MPLS usages.  In particular, it provides many information about the way
is able to collect information about tunnels in default cases (i.e.,
standard configurations). For example, it shows that is able to
discriminate between Cisco Invisible UHP and PHP tunnels while it is not the
case for Juniper routers (the  is not colored). Indeed, for both
UHP/PHP Juniper configurations, the trigger and the revelation methods are the
same (RTLA and ). Moreover, we also show when our basic set of techniques
need to be extended for enabling revelation and distinction among different
classes: we use the symbol $++$ to enforce these new requirements. In
particular, for revealing UHP Cisco tunnels, extends with additional
function and UDP probing.

Configurations     Pop     Cisco iOS15.2     Juniper VMX

P2P circuits         PHP     FRPLA,     RTLA,

(e.g. LDP or         UHP     DUP_IP, ++     RTLA,

RSVP-TE tunnels)

P2MP overlays     PHP     LSE-TTL, -     RTLA++, -

(e.g. VPRN: CsC or     UHP     LSE-TTL++, -     N/A

VPN BGP-MPLS)

revelation () and detection () capacities
according to the OS and the MPLS tunnel flavor (i.e. the MPLS L3 tunneling
underlying technologies for a given usage). In particular, this table provides
the default indicator/trigger and the default path revelation method (when it

Another kind of MPLS technology that conduct to surprising patterns is
VPRN tunnels. They also lead to Opaque tunnels for Cisco routers
and twisted traces with Juniper routers.

As shown by Table <ref>, is able to reveal the content of
all basic point-to-point (P2P) MPLS circuits (i.e., LDP or RSP-TE tunnels).
However, things are not as easy when point-to-multipoint (P2MP) tunnels join the
party (second part of Table <ref>), leading typically to
Opaque tunnels.  In such a situation, is only able to detect their presence
but without revealing their content. Indeed, such tunnels may result from
various network configurations, e.g., heterogeneous routing devices (a
combination of Juniper and Cisco devices in particular), specific BGP edge
configurations as already introduced, or VPRN (Virtual Private Routed
Network) [14]. As for the two former configurations, the Opaque
pattern arises with VPRN because of an abrupt tunnel ending, i.e., the LSP ends
without a standard ending label (Implicit or Explicit NULL). Indeed, the last
hop towards the Egress contains at least one label (two with UHP, the top label
then being 0). The inner label is used to identify the VPN and the associated
VRF containing routes. By definition, the VPN label value is neither Explicit
NULL nor Implicit NULL. Upon receiving this non-terminating label, the Egress
behaves as if the tunnel did not end in a controlled fashion.

Using our GNS-3 platform, it appears that VPRN content cannot be revealed with
, while other Opaque tunnels configurations (i.e., routing devices
heterogeneity, BGP edge configuration) can.  The mechanism behind the absence of
content revelation can be explained by the IP address collected by from the
source IP field in the ICMP reply.  Usually, the collected address is the one
assigned to the physical incoming interface of the Egress PE. In the VPRN case, the
collected IP is the one assigned to the interface on which the VRF is attached.
In practice, this corresponds to the outgoing interface towards the VPN at
the customer's side. Said otherwise, collects the outgoing address instead
of the incoming one. Because the incoming address is the only one that
enables a successful revelation, this type of Opaque tunnels cannot be revealed
yet.  Table <ref> in Sec. <ref> will show
that, in the wild, VPRNs are the most prevalent case of Opaque tunnels.

Whatever the kind of probes sent to or through the VPRN, the IP address visible
to (or  in general) is the outgoing address. Despite
its expired TTL, it is likely that the probe arriving on the Egress PE will be
pushed to the VRF of the VPN and its associated interface before generating the error
message (the VPN being identified with the MPLS label contained in the packet).
Then, the interface where the packet actually expires is the one associated to
the VRF. However, as shown in Table <ref>, we are able to
distinguish UHP and PHP configurations (thanks to LSE-TTL++), because the bottom
label is equal to $255$ for UHP and lower with PHP.

With Juniper VPN, there is no Opaque indicator resulting from VPRN or any other
configurations. A first explanation is that Juniper routers, on the contrary to
the independent mode enabled by default with Cisco routers, do not inject the
whole IGP in LDP, but only their loopback address using the ordered mode (see
Sec. <ref>). This mode limits the probability to face a
non-controlled tunnel ending. However, with VPRN configurations, a Juniper
Egress LER deals with the same packet level situation as with Cisco routers. Up
to the end of the tunnel, the packet is still MPLS encapsulated with the
end-to-end VPN non terminating label at the bottom of the stack. Juniper routers
do not, however, produce an Opaque indicator in that situation. Indeed, packets
destined to the VPN are handled in a specific way with Juniper devices: they are
IP packets forwarded directly to the next-hop without looking at or manipulating
the IP-TTL whatever its value.

The outcome of such a sliding packet is twofold. Firstly, the Egress hop is
hidden in the transit trace, as with Cisco UHP but without the duplicated IP.
Secondly, when performing a direct trace (even with UDP) targeting the first
address of the path within the VPN, i.e the IP interface of the Egress LER
belonging to the VPN, one can see that this address and its buddy appear in the
wrong order. Indeed, in the trace, the two addresses are switched, meaning that
the CE IP address appears before the Egress one. Being forwarded without
inspecting the IP-TTL, probes targeting IP addresses belonging to the VPN are
automatically forwarded to the CE router, where they expire. The next probe,
having a greater TTL, follows the same path as the one before, but can be
forwarded back to the Egress LER by the CE router before expiring. This loop
results in the two addresses being switched regarding their actual location in
the path. Finally, one can infer the loop because two additional artifacts
compared to RTLA (RTLA++) are visible: the TTL that deviates from its monotony and
subsequent IP addresses also raise alarms due to potential conflicting

Additional details on the validation through GNS3 emulation can be found in
the Appendix, at the end of this paper. All topologies and scripts developed are
available for reproducibility.<ref>

§ CALIBRATION AND PROBING COST

Sec. <ref> shows that relies mainly on four parameters when looking for
tunnels indicators or triggers: for Opaque tunnels, for
Implicit tunnels, and and for PHP/UHP Invisible tunnels.  This
section aims at calibrating those parameters (Sec. <ref>), as well
as evaluating the probing cost associated to (Sec. <ref>).

§.§ Measurement Setup

We deployed on three vantage points (VPs) over the Archipelago
infrastructure [17].  VPs were located in Europe (Belgium), North America
(San Diego), and Asia (Tokyo).

was run on April 6, 2018 towards a set of 10,000 destinations
(randomly chosen among the whole set of Archipelago destinations list).  Each VP
had its own list of destinations, without any overlapping.

Distribution of abnormal LSE-TTL values received at vantage points

From indicators and triggers described in Sec. <ref> (see
Listing <ref> and <ref>), it is
obvious that is equivalent to for Juniper routers. However, the
will not have the same value than . $=0$ by design as any
difference between and replies for the Cisco router
signature indicates LSE-/IP-TTL shifting. In practice, we reinforce the
condition by looking for at least two consecutive hops having a cumulated $\geq 3$. Besides, we have observed that abnormal[Abnormal here means
“different from 1” which is the LSE-TTL value that should be obtained in ICMP
messages.] LSE-TTL values oscillate between 236 and 254, the
main proportion being located between 250 and 254, as shown in Fig. <ref>.  It suggests thus that, in
the majority of the cases, Opaque tunnels are rather short. Consequently, a
value of 236 for would be enough for detecting the presence of an
Opaque tunnel and launching additional measurements for possibly revealing its content.

For our tests, we varied and between 0 and 4. A full measurement
campaign was launched for each pair of parameter value (thus, a total of 25
measurement runs). Moreover for each pair, if no trigger is pulled, a so called
brute force revelation is undertaken: /BRPR  are launched (with the use of
the buddy if required). This brute force data is used as a basis to evaluate
the quality and cost of each threshold value.

§.§ Calibration

Receiver operating characteristic (ROC) curve providing the
efficiency of according to values for Invisible tunnels parameters.
$\mathcal{T}_{R_x}$ refers to with the value $x$, while
$\mathcal{T}_{F_y}$ to with the value $y$.

With the help of well calibrated thresholds, the results associated to and triggers allows for a binary classification. These triggers provide a
prediction, while the results of additional probing gives the true facts when
some conditions apply (see resulting states of
Listing <ref>), i.e., being or not a tunnel. With
that in mind, one can assess the performance of and triggers
through the analysis of True Positive Rate (TPR) and False Positive Rate (FPR):
we plot the results on a Receiver Operating Characteristic (ROC) curve in
Fig. <ref>. We define TPR as the ratio of success
to the number of links being actually MPLS tunnels (having a length greater than
1): triggers additional probing and actually reveals Invisible tunnels (we
have $TPR+FNR=1$, i.e., when adding to False Negative Rate, we obtain all links
being long enough tunnels). FPR is defined as the ratio of failure to the
amount of standard IP links: it triggers for additional probing but without
revealing anything (we have $FPR+TNR=1$, i.e., when adding to True Negative
Rate, we obtain all IP links without tunnels). Here, our brute force data gives
the ground data that we consider reliable (i.e., revelation is fired at each hop
and if nothing is revealed, we consider that there is no tunnel – we do not
consider inconclusive cases where we obtain states or – see Listing <ref>).  The ROC curve is obtained
by varying the and parameters between 0 and 4. The red dotted
diagonal provides the separation between positive results for (above part
of the graph) and negative results (below part of the graph). Finally, the black
dotted line is the interpolation of measurement results (at the exception of
$\mathcal{T}_{R_0}$ values which appear as being outliers, as expected).

We observe that the results are essentially positive for .  Some results,
between ($\mathcal{T}_{R_1}$, $\mathcal{T}_{F_3}$) and ($\mathcal{T}_{R_2}$,
$\mathcal{T}_{F_3}$), are even reasonably close to the perfect classification
(upper left corner) and, thus, are considered as the best choice for defining
our thresholds and . We obtain a compromise close to 80%-20%:
while we expect to reveal at least 80% of existing tunnels (MPLS links), has a controlled overhead of 20%, i.e., it fires useless additional probing for
an average limited to two actual IP links on ten.

§.§ Probing Cost

Probing cost associated to according to and thresholds.

Fig. <ref> illustrates the probing cost associated to .
In particular, it focuses on additional measurements triggered by or
for revealing Invisible tunnels. The light grey zone (labeled as
“Original” on Fig. <ref>) corresponds to probes associated
to standard . The green, orange, and dark grey zones correspond to
probes sent when additional measurements are triggered by or . In
particular, the green zone corresponds to additional measurements that were able
to reveal the content of an Invisible tunnel. On the contrary, the orange zone
refers to additional measurements that failed, i.e., no Invisible tunnel content
was revealed.  Finally, the dark grey zone refers to inconclusive revelation:
the trigger has led to additional measurements but was unable to reach the
potential Egress LER (i.e., the IP address that engaged the trigger –
in Listing <ref> – generally due to
unresponsive IP interface) or was unable to reach again the candidate
Ingress LER (i.e.,  in
Listing <ref>) because the destination has changed
(ECMP or BGP routing noises).

If the amount of probes sent for actually revealing the content of an Invisible
tunnel remains almost stable whatever the values for and are,
one can observe a very slow decrease meaning that there are less revealed
tunnels for high values.  Further, the additional traffic generated by erroneous
trigger (orange) or by inconclusive revelation (dark grey) clearly decreases while
increases.  This result is aligned with Sec. <ref>
in which the best values for are between 2 and 3. Note that is
more generic but less reliable than other triggers.
On the contrary, the threshold has a minor effect on the amount of probes
sent because it is more specific and more reliable.

Hatched zones (orange, dark grey, and green) correspond to the amount of probes
sent using brute force.
First, on the contrary to normal behavior (i.e., revelation launched according
to triggers), the amount of probes sent increases with (the impact of
is quite negligible), as well as the amount of inconclusive revelation.
Second, the amount of probes having revealed an Invisible tunnel is low compared
to standard behavior.

Generally speaking, one can observe that the overhead of is quite limited
compared to a basic active campaign and considering the information gathered. In
particular, if using correct parameters to limit both useless probes and missed
tunnels (e.g., $\mathcal{T}_{R_1}$, $\mathcal{T}_{F_3}$), our tool generates
less than 10% of additional probing compared to the underlying campaign for
reaching a satisfying compromise where 80% of tunnels are revealed.

§ TUNNELS QUANTIFICATION WITH

This section aims at discussing how and its features behave in the wild
Internet. In particular, it analyzes the success rate of each indicator and
trigger with respect to possible revelation techniques.
Sec. <ref> describes the measurement setup, while
Sec. <ref> discusses the results obtained.

§.§ Measurement Setup

We deployed on the Archipelago infrastructure [17] on April 23,
2018 with parameters fixed to $3$ and to $1$, according to
results discussed in Sec. <ref>.

has been deployed over 28 vantage points, scattered all around the world:
Europe (9), North America (11), South America (1), Asia (4), and
Australia (3).  The overall set of destinations, nearly 2,800,000 IP addresses, is
inherited from the Archipelago dataset and spread over the 28 vantage
points to speed up the probing process.

A total of 522,049 distinct IP addresses (excluding targets) has
been collected, with 28,350 being non publicly routable addresses (and thus
excluded from our dataset). Each collected routable IP address has been
pinged, only once per vantage point, allowing us to collect additional data for
fingerprinting (see Sec. <ref>).  Our dataset and our
post-processing scripts are freely available.<ref>

§.§ Results

2c|2*Status     3c# probes

2c|original     63,559,385     7,109,075     $-$

4*90attempt     revealed          2,190,275
206,842     19,181

no revelation     1,640,224          $-$         556

4,174,404     $-$         9,888

1,790,900     $-$         7,326

Raw number of probes sent by over the set of 28 vantage

Table <ref> provides the amount of probes sent by
-like probing in , , and buddy bit exploration. The
row “original” refers to standard based revelation (i.e.,
nothing to reveal, Explicit, or Implicit tunnels).

The main results from Table <ref> is the amount of
probes involved in inconclusive revelation, split between (was
unable to reach the potential Egress LER) and (did not cross
the potential Ingress LER). In particular, involved twice more
probes than revealed tunnels. Those particular inconclusive revelations might be
explained by ICMP rate limiting between the probe and additional
probing (both  and BRPR/DPR). Another explanation is that those
potential Egress LERs respond to initial with an IP address that is
not globally announced. As such, additional probing following will
fail as no route is available to reach them.

2*Tunnel Type     2*Indicator/Trigger     4c|Revelation Technique     2*# Tunnels

DPR     BRPR         Mix

Explicit     LSE headers     -     -     -     -     150,036

2*Implicit     qTTL     -     -     -     -     2,689

-     -     -     -     7,216

Opaque     LSE-TTL     22     17     43     -     3,346

2*Invisible PHP           11,268     1,191     2,595     279       15,333

5,903	     2,555     3,260     1,012     12,730

Invisible UHP          1,609     1,531     686     296     4,122

2c|Total     18,802     5,294     6,584     1,587     195,525

Raw number of tunnels discovered by per tunnel type (see
Sec. <ref>). Color code for indicators/triggers is identical
to Fig. <ref>. No additional revelation technique is
necessary for Explicit and Implicit tunnels.

Table <ref> provides the number of MPLS tunnels
discovered by , per tunnel type as indicated in the first column. The
indicators/triggers are provided, as well as the additional revelation technique
used. Without any surprise, Explicit tunnels are the most present category
(76% of tunnels discovered).

Implicit tunnels represent 5% of the whole dataset, with the indicator
providing more results than qTTL. However, those results must be taken with care
as is subject to false positive (implicit tunnels are likely to be
overestimated because of possible confusion with RTLA for Juniper routers),
while qTTL is much more reliable [29]. Compared to previous works, it
is clear that is class is not as prevalent as expected at the time (because both
we correct and improve our methodology and the RFC4950 is likely to be more and
more deployed).

Opaque tunnels are less prevalent (1.7% of tunnels discovered).  This is
somewhat expected as Opaque tunnels are the results of particular label
distribution within Cisco MPLS clouds. This confirms previous empirical
results <cit.>.  It is also worth noticing that additional
revelation techniques (DPR or BRPR) does not perform well with such tunnels
(content of 98% of Opaque tunnels cannot be revealed). Indeed, as already
discussed earlier, this result can be explained because Carrier-of-Carriers
VPN [26] or similar VPRN are not possible to reveal but can only be
detected. We deduce from our campaign results that the vast majority of Opaque
tunnels seems to arise from Cisco VPRN.[In this paper, does not
look for Juniper VPRN as its indicator, RTLA++, is less reliable (See
Sec. <ref>).]

The proportion of Invisible tunnels is not negligible (16% of tunnels in our
dataset). Those measurements clearly contradicts our previous work suggesting
that Invisible tunnels were probably 40 to 50 times less numerous than Explicit
ones <cit.>. More precisely, Invisible PHP is the most
prominent configuration (87% of Invisible tunnels belongs to the Invisible PHP
category), confirming so our past survey [9]. appears as
being the most efficient trigger. This is partially due to the order of triggers
in the code because it favors high ranked trigger compared to low ranked
(in case both apply, we prefer to use the most reliable, i.e., the less subject
to any interference such a BGP asymmetry). As indicated in
Listing <ref> (Sec. <ref>), we first check for
as it is more reliable than . DPR works better than BRPR, which is
obvious as it is triggered by (Juniper routers). For Invisible UHP, it is
worth noticing that the buddy bit, prior to BRPR or DPR revelation, was required
in nearly 25% of the cases. In other cases, a simple BRPR or DPR revelation was
enough to get the tunnel content. UHP seems to be often filtered for a
particular FEC, e.g., only /32 host loopback addresses are advertised in LDP
with UHP while other FEC are advertised with PHP (BRPR) or are not injected at
all (DPR).

The column labeled “mix” corresponds to tunnels partially revealed thanks to
BRPR and partially with DPR. Typically, it comes from heterogeneous MPLS
clouds. For instance, operators may deploy both Juniper and Cisco hardware
without any homogeneous prefixes distribution (i.e., local prefix for Juniper,
all prefixes for Cisco – See Sec. <ref> for details). Note that
it is also possible that the UHP and PHP label popping techniques co-exist when
using our backward recursive path revelation (BRPR). Although not explained in
Sec. <ref> for clarity reasons, can deal with those more complex
situations, making the tool quite robust to pitfalls encountered in the wild
Internet (5% of the Invisible tunnels encountered).

Finally, the column labeled “” corresponds to one hop tunnels where
DPR and BRPR cannot be distinguished. This large proportion (20%) of very short
Invisible tunnels is aligned with previous works that already noticed the
proportion of short Explicit tunnels [8, 10, 30].

§ RELATED WORK

For years now, has been used as the main tool for discovering the
Internet topology [1].  Multiple extensions have been provided to
circumvent limits.

Doubletree [27, 31] has been proposed for improving the cooperation
between scattered vantage points, reducing so the probing
redundancy. Paris traceroute [11] has been developed for
fixing issues related to IP load balancing, avoiding so false links between IP
interfaces.  [5] extends for revealing
the presence of middleboxes along a path.  YARRP [32] provides
techniques for speeding up the probing process.  Reverse
[33] is able to provide the reverse path (i.e.,
from the target back to the vantage point).  Passenger [34] and
Discarte [35] extend with the IP record route option.
Marchetta et al. [36] have proposed to use the ICMP Parameter
Problem in addition to Record Route option in .  Finally,
[37] mimics for discovering

is also in the scope of the hidden router issue, i.e., any device
that does not decrement the TTL causing the device to be transparent to
probing.  Discarte and Passenger, through the use of IP Record Route
Option, allows, to some extent, to reveal hidden routers along a path.
Drago [38] considers the ICMP Timestamp for also detecting
hidden routers.  goes beyond those solutions as it does not rely on ICMP
messages and IP option that are, generally, filtered by operators either locally
(i.e., the option/message is turned off on the router) or for transit packets
(i.e., edge routers do not forward those particular packets).[It has
been, however, demonstrated recently that IP Record Route option might still
find a suitable usage in Internet measurements if used with
prudence [39].] only relies on standard messages
(/and ) that are implemented and used by the
vast majority of routers and, as such, has the potential to reveal much more

MPLS tunnels discovery has been the subject of several researches those last
years.  In particular, Sommers et al. [10] examined the
characteristics of MPLS deployments that are explicitly identified using RFC4950
extensions, as observed in CAIDA's topology data.  Donnet et al. [8]
proposed the first classification of MPLS tunnels according to the relationship
between MPLS and .  This paper is a revision of Donnet et al.'s work
in light of a deeper understanding of MPLS mechanisms, in particular for hidden
tunnels (Opaque, Invisible PHP and UHP). More recently, Vanaubel et
al. [9] have proposed techniques for inferring and possibly
revealing hidden tunnels: , , , and . and were
initially not used as triggers for measurements (as we are doing in this paper
with by extending those techniques in many aspects) but rather as a way to
infer an hidden tunnel length. Vanaubel et al. directed their measurements
towards pre-identified high degree routers with the ITDK dataset used as a
source for triggering specific measurements (as they were suspected to be the
exit point of a large number of hidden MPLS tunnels). As such, Vanaubel et al.
did not provide any integrated measurement tool, on the contrary to , in
which MPLS tunnels are discovered on the fly (does not rely on any kind of
external dataset).

§ CONCLUSION

In this paper, we revise the MPLS classification proposed by Donnet et
al. [8] and introduce (Trace the
Naughty Tunnels) that is an extension
to Paris traceroute for revealing MPLS tunnels along a path. As such, has
the potential to reveal more complete information on the exact Internet
topology. We provide accurate IP level tracing functions leading so to better
Internet models. For instance, it has been shown that Invisible tunnels have an
impact on Internet basic graph properties [9]). Our fully
integrated tool reveals, or at least detect, all kind of tunnels in two simple
stages: first, it uses indicators and triggers to respectively classify and
possibly tag tunnels as hidden, second it reveals the tagged tunnel content if
any. has the capacity to unveil the MPLS ecosystem deployed by operators.
Recent works on MPLS discovery have revealed that MPLS is largely deployed by
most ISP [8, 30, 10]. By running on a daily (or
nearly daily) basis from the Archipelago platform, we expect to see numerous
researches using our tool and data to mitigate the impact of MPLS on the
Internet topology. has been developed with a reproducibility perspective.
As such, it is freely available, as well as our dataset and scripts used for
processing data.<ref>

§ ACKNOWLEDGMENTS

Authors would like to thank kc claffy and her team at Caida for letting
them deploying on the Archipelago infrastructure.  In addition, part of Mr.
Vanaubel's work was supported by an internship at Caida, under the
direction of Young Hyun.

[1]
B. Donnet and T. Friedman, “Internet topology discovery: a survey,”
IEEE Communications Surveys and Tutorials, vol. 9, no. 4, pp. 2–15,
December 2007.

[2]
H. Haddadi, G. Iannaccone, A. Moore, R. Mortier, and M. Rio, “Network
topologies: Inference, modeling and generation,” IEEE Communications
Surveys and Tutorials, vol. 10, no. 2, pp. 48–69, April 2008.

[3]
R. Pastor-Satorras and A. Vespignani, Evolution and Structure of the
Internet: A Statistical Physics Approach.1em plus 0.5em minus
0.4emCambridge University Press, 2004.

[4]
P. Mérindol, B. Donnet, O. Bonaventure, and J.-J. Pansiot, “On the impact of
layer-2 on node degree distribution,” in Proc. ACM Internet
Measurement Conference (IMC), November 2010.

[5]
G. Detal, b. Hesmans, O. Bonaventure, Y. Vanaubel, and B. Donnet, “Revealing
middlebox interference with tracebox,” in Proc. ACM Internet
Measurement Conference (IMC), October 2013.

[6]
K. Edeline and B. Donnet, “A first look at the prevalence and persistence of
middleboxes in the wild,” in Proc. International Teletraffic Congress
(ITC), September 2017.

[7]
E. Rosen, A. Viswanathan, and R. Callon, “Multiprotocol label switching
architecture,” Internet Engineering Task Force, RFC 3031, January

[8]
B. Donnet, M. Luckie, P. Mérindol, and J.-J. Pansiot, “Revealing MPLS
tunnels obscured from traceroute,” ACM SIGCOMM Computer
Communication Review, vol. 42, no. 2, pp. 87–93, April 2012.

[9]
Y. Vanaubel, P. Mérindol, J.-J. Pansiot, and B. Donnet, “Through the
wormhole: Tracking invisible MPLS tunnels,” in In Proc. ACM Internet
Measurement Conference (IMC), November 2017.

[10]
J. Sommers, B. Eriksson, and P. Barford, “On the prevalence and
characteristics of MPLS deployments in the open Internet,” in
Proc. ACM Internet Measurement Conference (IMC), November 2011.

[11]
B. Augustin, X. Cuvellier, B. Orgogozo, F. Viger, T. Friedman, M. Latapy,
C. Magnien, and R. Teixeira, “Avoiding traceroute anomalies with Paris
traceroute,” in Proc. ACM Internet Measurement Conference (IMC),
October 2006.

[12]
L. Andersson, I. Minei, and T. Thomas, “LDP specification,” Internet
Engineering Task Force, RFC 5036, October 2007.

[13]
D. Awduche, L. Berger, D. Gan, T. Li, V. Srinivasan, and G. Swallow,
“RSVP-TE: Extensions to RSVP for LSP tunnels,” Internet
Engineering Task Force, RFC 3209, December 2001.

[14]
B. Gleeson, A. Lin, J. Heinanen, G. Armitage, and A. Malis, “A framework for
IP based virtual private networks,” Internet Engineering Task
Force, RFC 2764, February 2000.

[15]
Center for Applied Data Analysis, “The CAIDA UCSD internet topology data
kit,” March 2016, see

[16]
M. Luckie, “Scamper: a scalable and extensible packet prober for active
measurement of the Internet,” in Proc. ACM Internet Measurement
Conference (IMC), November 2010.

[17]
k. claffy, Y. Hyun, K. Keys, M. Fomenkov, and D. Krioukov, “Internet
mapping: from art to science,” in Proc. IEEE Cybersecurity
Application and Technologies Conference for Homeland Security (CATCH),
March 2009.

[18]
Y. Vanaubel, J.-J. Pansiot, P. Mérindol, and B. Donnet, “Network
fingerprinting: TTL-based router signature,” in Proc. ACM Internet
Measurement Conference (IMC), October 2013.

[19]
L. Andersson and R. Asati, “Multiprotocol label switching (MPLS) label stack
entry: EXP field renamed to traffic class field,” Internet Engineering
Task Force, RFC 5462, February 2009.

[20]
E. Rosen, D. Tappan, G. Fedorkow, Y. Rekhter, D. Farinacci, T. Li, and
A. Conta, “MPLS label stack encoding,” Internet Engineering Task
Force, RFC 3032, January 2001.

[21]
P. Agarwal and B. Akyol, “Time-to-live (TTL) processing in multiprotocol
label switching (MPLS) networks,” Internet Engineering Task Force,
RFC 3443, January 2003.

[22]
D. Aydin, “CISCO vs. Juniper MPLS,” June 2014, see

[23]
L. De Ghein, MPLS Fundamental: A Comprehensive Introduction to MPLS
(Theory and Practice).1em plus 0.5em minus 0.4emCISCO Press,
November 2006.

[24]
R. Bonica, D. Gan, D. Tappan, and C. Pignataro, “ICMP extensions for
multiprotocol label switching,” Internet Engineering Task Force,
RFC 4950, August 2007.

[25]
T. Fiola and J. Panagos, This Week: Deploying MPLS, ser. Junos
Networking Technologies Series.1em plus 0.5em minus 0.4emJuniper Networks Books, April 2011.

[26]
E. Rosen and Y. Rekhter, “BGP/MPLS IP virtual private networks (VPNs),”
Internet Engineering Task Force, RFC 4364, February 2006.

[27]
B. Donnet, P. Raoult, T. Friedman, and M. Crovella, “Efficient algorithms for
large-scale topology discovery,” in Proc. ACM SIGMETRICS, June

[28]
J.-F. Grailet, F. Tarissan, and B. Donnet, “TreeNET: Discovering and
connecting subnets,” in Proc. Traffic Monitoring and Analysis Workshop
(TMA), April 2016.

[29]
G. Davila Revelo, M. A. Ricci, B. Donnet, and J. I. Alvarez-Hamelin,
“Unveiling the MPLS structure on Internet topology,” in Proc.
Traffic Monitoring and Analysis Workshop (TMA), April 2016.

[30]
Y. Vanaubel, P. Mérindol, J.-J. Pansiot, and B. Donnet, “MPLS under the
microscope: Revealing actual transit path diversity,” in Proc. ACM
Internet Measurement Conference (IMC), October 2015.

[31]
R. Beverly, A. Berger, and G. Xie, “Primitives for active Internet topology
mapping: Toward high-frequency characterization,” in Proc. ACM
Internet Measurement Conference (IMC), November 2010.

[32]
R. Beverly, “Yarrp'ing the Internet: Randomized high-speed active topology
discovery,” in Proc. ACM Internet Measurement Conference (IMC),
November 2016.

[33]
E. Katz-Bassett, H. Madhyastha, V. Adhikari, C. Scott, J. Sherry, P. van Wesep,
A. Krishnamurthy, and T. Anderson, “Reverse traceroute,” in Proc.
USENIX Symposium on Networked Systems Design and Implementations (NSDI),
June 2010, see <https://www.revtr.ccs.neu.edu>.

[34]
R. Sherwood and N. Spring, “Touring the internet in a TCP sidecar,” in
Proc. ACM Internet Measurement Conference (IMC), October 2006.

[35]
R. Sherwood, A. Bender, and N. Spring, “Discarte: a disjunctive Internet
cartographer,” in Proc. ACM SIGCOMM, August 2008.

[36]
P. Marchetta, W. de Donato, V. Persico, and A. Pescapé, “Experimenting with
alternative path tracing solutions,” in Proc. IEEE Symposium on
Computers and Communications (ISCC, July 2015.

[37]
M. E. Tozal and K. Sarac, “TraceNET: an Internet topology data
collector,” in Proc. ACM Internet Measurement Conference (IMC),
November 2010.

[38]
P. Marchetta and A. Pescapé, “DRAGO: Detecting, quantifying and locating
hidden routers in traceroute IP paths,” in Proc. Global Internet
Symposium (GI), April 2013.

[39]
B. J. Goodchild, Y.-C. Chiu, R. Hansen, H. Lua, M. Calder, M. Luckie, W. Lloyd,
D. Choffnes, and E. Katz-Bassett, “The record route option is an option!”
in In Proc. ACM Internet Measurement Conference (IMC), November

This appendix illustrates the validation of through GNS-3 emulations.
Multiple configurations have been tested (and even more are proposed on the
website<ref> and can be setup using the scripts
and the data online). Note that we use the version 2.1.5 of GNS3 to export the
so-called portable configurations. is able to deal with all those
configurations (both in the wild and with the ones emulated in GNS3), making it
a pretty robust tool. However, in this report we use another version of our tool
to simplify the output. The output of slightly differ but the conclusions
are the same.

Cisco topology.  PE1 is the Ingress LER, PE2 the
Egress LER, the LSP is set up between PE1 and the EH (P3 or PE2).  The target (i.e.,
the argument of  function – See
Listing <ref>) is the loopback address of CE3.

Juniper topology.  PE1 is the Ingress LER, PE2 the
Egress LER, the LSP is set up between PE1 and the EH (P3 or PE2).  The target (i.e.,
the argument of  function – See
Listing <ref>) is the loopback address of CE3.

Topology used for GNS-3 tests

§ P2P CIRCUITS

§.§ Explicit Tunnels Validation

We first review Explicit tunnels, i.e., tunnels with RFC4950 and enabled (see Sec. <ref>).

In the following, we distinguish Cisco (Appendix <ref>)
and Juniper IP topologies (Appendix <ref>) and
configurations. In particular, with Cisco configurations, PHP (LSE popped by P3)
is distinguished from UHP (LSE popped by Egress LER).

For each case, we provide the configuration of routers as well as the simplified output. Indicators and triggers (see Sec. <ref>) are provided, as well
as raw ICMP and ICMP TTLs.

§.§.§ Cisco Explicit Configurations

All configurations presented here were run on the IP topology provided by
Fig. <ref>.

The first example provides an Explicit tunnel deployed with PHP, under Cisco IOS 15.2.
The behavior is the one expected.

[title=IOS 15.2 – Explicit PHP]
version 15.2
mpls label protocol ldp
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Explicit PHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 27.083 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 19.895 ms
3  left.P1 (10.1.0.2)  <247,253> [frpla = 6][qttl = 1][uturn = 6][MPLS LSE | Label : 19 | LSE-TTL : 1]  80.598 ms
4  left.P2 (10.2.0.2)  <248,252> [frpla = 4][qttl = 2][uturn = 4][MPLS LSE | Label : 20 | LSE-TTL : 1]  69.875 ms
5  left.P3 (10.3.0.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 20 | LSE-TTL : 1]  68.98 ms
6  left.PE2 (10.4.0.2)  <250,250> [frpla = 0][qttl = 1][uturn = 0] 78.17 ms
7  left.CE2 (192.168.2.2)  <249,249> [frpla = 0][qttl = 1][uturn = 0] 78.957 ms
8  192.168.4.2 (192.168.4.2)  <248,248> [frpla = 0][qttl = 1][uturn = 0] 110.598 ms

The next two configurations illustrate UHP with both IOS 12.4 and IOS 15.2.
works as expected and shows two examples of MPLS TTL processing specifically with UHP.
With the 12.4 IOS, we see the null label while it is hidden with the 15.2 IOS.
In addition, we can see that UHP tunnels show a signature different from PHP tunnels.
This difference results from the way time-exceeded messages are handled by the LSRs. In both cases,
the time-exceeded message is forwarded to the EH which replies using its own IP forwarding table.
The EH changes depending on the configuration: P3 for PHP (here the EH is the PH), and PE2 for UHP
(here the EH is the Egress LER). Indeed, we can see that the difference disappears at the respective EH.

[title=IOS 12.4 – Explicit UHP]
version 12.4
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 12.4
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 12.4
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 12.4
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 12.4
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 12.4 – Explicit UHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 22.651 ms
2  192.168.8.2 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 230.326 ms
3  left.P1 (10.1.0.2)  <247,253> [frpla = 6][qttl = 1][uturn = 6][MPLS LSE | Label : 22 | LSE-TTL : 1]  263.686 ms
4  left.P2 (10.2.0.2)  <248,252> [frpla = 4][qttl = 2][uturn = 4][MPLS LSE | Label : 22 | LSE-TTL : 1]  358.238 ms
5  left.P3 (10.3.0.2)  <249,251> [frpla = 2][qttl = 3][uturn = 2][MPLS LSE | Label : 16 | LSE-TTL : 1]  374.214 ms
6  left.PE2 (10.4.0.2)  <250,250> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 0 | LSE-TTL : 1]  418.696 ms
7  left.CE2 (192.168.2.2)  <249,249> [frpla = 0][qttl = 1][uturn = 0] 655.848 ms
8  192.168.4.2 (192.168.4.2)  <248,248> [frpla = 0][qttl = 1][uturn = 0] 513.054 ms

[title=IOS 15.2 – Explicit UHP]
version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Explicit UHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 7.64 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 39.87 ms
3  left.P1 (10.1.0.2)  <247,253> [frpla = 6][qttl = 1][uturn = 6][MPLS LSE | Label : 19 | LSE-TTL : 1]  100.632 ms
4  left.P2 (10.2.0.2)  <248,252> [frpla = 4][qttl = 2][uturn = 4][MPLS LSE | Label : 20 | LSE-TTL : 1]  80.453 ms
5  left.P3 (10.3.0.2)  <249,251> [frpla = 2][qttl = 3][uturn = 2][MPLS LSE | Label : 20 | LSE-TTL : 1]  100.815 ms
6  left.PE2 (10.4.0.2)  <250,250> [frpla = 0][qttl = 1][uturn = 0] 109.089 ms
7  left.CE2 (192.168.2.2)  <249,249> [frpla = 0][qttl = 1][uturn = 0] 98.817 ms
8  192.168.4.2 (192.168.4.2)  <248,248> [frpla = 0][qttl = 1][uturn = 0] 119.842 ms

§.§.§ Juniper Explicit Configurations

All configurations presented here were run on the topology provided by
Fig. <ref>.

For Explicit tunnels, Juniper Olive and VMX behave the same.  We first provide
the configuration and output for Explicit tunnels without effect.

[title=VMX – Explicit PHP (default configuration)]
propagate ttl

propagate ttl

propagate ttl

propagate ttl

propagate ttl

[title=running over VMX - Explicit PHP (default configuration)]
Launching TNT: 192.168.2.102 (192.168.2.102)

1  CE1 ( 172.16.0.5)  <255,64> [frpla = 0][qttl = 1][uturn = 0] 2.682 ms
2  PE1 ( 172.16.0.2)  <254,63> [frpla = 0][qttl = 1][uturn = 0] 4.603 ms
3  left.P1 (192.168.1.2)  <253,62> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299824 | LSE-TTL : 1]  6.362 ms
4  left.P2 (192.168.1.6)  <252,61> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299792 | LSE-TTL : 1]  8.451 ms
5  left.P3 (192.168.1.10)  <251,60> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299792 | LSE-TTL : 1]  8.557 ms
6  left.PE2 (192.168.1.14)  <250,59> [frpla = 0][qttl = 1][uturn = 0] 8.285 ms
7  CE2 (192.168.2.2)  <249,58> [frpla = 0][qttl = 1][uturn = 0] 8.09 ms
8  CE3 (192.168.2.102)  <248,57> [frpla = 0][qttl = 1][uturn = 0] 8.142 ms

On the contrary to Cisco configuration, Juniper does not exhibit the effect.
When the LSE-TTL of a packet expires, the LSR does not send the ICMP
to the EH which then forwards the packets on its own to the
probing source, it replies the same with respect to other probes (e.g.,
) using its own IP forwarding table if available – resulting in
general in a shorter return path (see Sec. <ref>). The configuration must be
explicitly stated with the  as provided below.

[title=VMX – Explicit PHP ( configuration)]
propagate ttl

propagate ttl

propagate ttl

propagate ttl

propagate ttl

[title=running over VMX – Explicit PHP ( configuration)]
Launching TNT: 192.168.2.102 (192.168.2.102)

1  CE1 ( 172.16.0.5)  <255,64> [frpla = 0][qttl = 1][uturn = 0] 2.034 ms
2  PE1 ( 172.16.0.2)  <254,63> [frpla = 0][qttl = 1][uturn = 0] 4.646 ms
3  left.P1 (192.168.1.2)  <246,62> [frpla = 7][qttl = 1][uturn = 7][MPLS LSE | Label : 299824 | LSE-TTL : 1]  11.424 ms
4  left.P2 (192.168.1.6)  <247,61> [frpla = 5][qttl = 1][uturn = 5][MPLS LSE | Label : 299824 | LSE-TTL : 1]  7.994 ms
5  left.P3 (192.168.1.10)  <251,60> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299824 | LSE-TTL : 1]  6.252 ms
6  left.PE2 (192.168.1.14)  <250,59> [frpla = 0][qttl = 1][uturn = 0] 8.585 ms
7  CE2 (192.168.2.2)  <249,58> [frpla = 0][qttl = 1][uturn = 0] 9.369 ms
8  CE3 (192.168.2.102)  <248,57> [frpla = 0][qttl = 1][uturn = 0] 9.232 ms

§.§ Opaque Tunnels Validation (Cisco only)

Opaque tunnels only occur with Cisco routers, in some particular configuration
(see Sec. <ref> for details). The topology used for GNS-3
emulation is the one provided by Fig. <ref>.  We only show
tests for IOS 15.2 as the situation is the same with IOS 12.4. In our example,
we were able to reveal the content of the Opaque tunnel through BRPR, on the
contrary to in the wild deployment where Opaque tunnels revelation did not
work that much (see Sec. <ref>).  We see thus here a difference
between theory and practice.

[title=IOS 15.2 – Opaque PHP]
version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 192.168.8.1 remote-as 1024

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 192.168.6.1 remote-as 2048
neighbor 192.168.6.1 ebgp-multihop 2

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Opaque PHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 25.164 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 40.06 ms

OPAQUE | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] left.P1 (10.1.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 40.008 ms  - step 2
2.2 [REVEALED] left.P2 (10.2.0.2)  <252,252> [frpla = 0][qttl = 1][uturn = 0] 40.058 ms  - step 1
2.3 [REVEALED] left.P3 (10.3.0.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0] 90.301 ms  - step 0

3  left.PE2 (10.4.0.2)  <250,250> [frpla = 3][qttl = 1][uturn = 0][MPLS LSE | Label : 16 | LSE-TTL : 252]  110.408 ms
4  left.CE2 (192.168.2.2)  <250,250> [frpla = 2][qttl = 1][uturn = 0] 80.195 ms
5  192.168.4.2 (192.168.4.2)  <250,250> [frpla = 1][qttl = 1][uturn = 0] 132.331 ms

§.§ Invisible Tunnels Validation

This section discusses Invisible tunnels, i.e., tunnels with
the option enabled (see Sec. <ref>).

We do a distinction between Cisco
(Appendix <ref>) and Juniper configurations
(Appendix <ref>).  PHP (LSE popped by P3) is also
distinguished from UHP (LSE popped by Egress LER).

For each case, we provide the configuration of routers as well as the output.  Indicators and triggers (see Sec. <ref>) are provided, as well
as ICMP and ICMP TTLs.

§.§.§ Invisible Cisco Configurations

All configurations presented here were run on the topology provided by
Fig. <ref>.

[title=IOS 15.2 – Invisible PHP]
version 15.2
mpls label protocol ldp
no propagate-ttl
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Invisible PHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 7.52 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 29.927 ms

FRPLA | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] left.P1 (10.1.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 50.051 ms  - step 2
2.2 [REVEALED] left.P2 (10.2.0.2)  <252,252> [frpla = 0][qttl = 1][uturn = 0] 60.102 ms  - step 1
2.3 [REVEALED] left.P3 (10.3.0.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0] 59.876 ms  - step 0

3  left.PE2 (10.4.0.2)  <250,250> [frpla = 3][qttl = 1][uturn = 0] 80.38 ms
4  left.CE2 (192.168.2.2)  <250,250> [frpla = 2][qttl = 1][uturn = 0] 69.89 ms
5  192.168.4.2 (192.168.4.2)  <250,250> [frpla = 1][qttl = 1][uturn = 0] 99.833 ms

The configuration for running standard Cisco Invisible UHP tunnels is provided
below. Such a configuration might be revealed through BRPR thanks to the trigger.

[title=IOS 15.2 – Invisible UHP]
version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over  IOS 15.2 – Invisible UHP]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 3.157 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 29.92 ms

Duplicate IP (Egress : 192.168.2.2) | Length estimation : 1 | Revealed : 4 (difference : 3)
2.1 [REVEALED] left.P1 (10.1.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 50.043 ms  - step 4  (Buddy used)
2.2 [REVEALED] left.P2 (10.2.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 49.778 ms  - step 3  (Buddy used)
2.3 [REVEALED] left.P3 (10.3.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 69.834 ms  - step 2  (Buddy used)
2.4 [REVEALED] left.PE2 (10.4.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 80.594 ms  - step 1  (Buddy used)

3  left.CE2 (192.168.2.2)  <252,252> [frpla = 1][qttl = 1][uturn = 0] 80.08 ms
4  left.CE2 (192.168.2.2)  <252,252> [frpla = 0][qttl = 1][uturn = 0] 89.891 ms
5  192.168.4.2 (192.168.4.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0] 107.579 ms

With Cisco routers, it is possible to mimic an Invisible UHP tunnel with a
Juniper per loopback configuration (i.e., by filtering addresses to /32 border
prefixes), meaning that the tunnel content might be revealed through DPR, thanks
to the trigger.  Such a configuration is achieved with the

[title=IOS 15.2 – Invisible UHP ( configuration)]
version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
mpls ldp label
allocate global host-routes
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
mpls ldp label
allocate global host-routes
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
mpls ldp label
allocate global host-routes
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
mpls ldp label
allocate global host-routes
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
mpls ldp label
allocate global host-routes
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Invisible UHP ( configuration)]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 8.091 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 39.867 ms

Duplicate IP (Egress : 10.1.0.2) | Length estimation : 1 | Revealed : 4 (difference : 3)
2.1 [REVEALED] left.P1 (10.1.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 39.788 ms  - step 2
2.2 [REVEALED] left.P2 (10.2.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 49.573 ms  - step 2
2.3 [REVEALED] left.P3 (10.3.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 70.094 ms  - step 2
2.4 [REVEALED] left.PE2 (10.4.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 89.171 ms  - step 1  ( Buddy used )

3  left.CE2 (192.168.2.2)  <252,252> [frpla = 1][qttl = 1][uturn = 0] 120.546 ms
4  left.CE2 (192.168.2.2)  <252,252> [frpla = 0][qttl = 1][uturn = 0] 89.892 ms
5  192.168.4.2 (192.168.4.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0] 117.301 ms

It is also possible to build Invisible UHP tunnel in which the buddy mechanism
is not necessary (as we discover in the wild). Simply running BRPR will make the
tunnel content visible.  This configuration might be achieved with the
command to enable Ultimate Hop Popping for external destinations only:

[title=IOS 15.2 – Invisible UHP ( configuration)]
version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null for BRPR-wo-buddy
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self
ip access-list standard BRPR-wo-buddy
permit 10.9.0.1
deny any

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Invisible UHP ( configuration)]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  192.168.3.2 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 7.299 ms
2  192.168.8.2 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 14.921 ms

Duplicate IP (Egress : 10.4.0.2) | Length estimation : 3 | Revealed : 4 (difference : 1)
2.1 [REVEALED] 10.1.0.2 (10.1.0.2)  <253,253> [frpla = 0][qttl = 1][uturn = 0] 36.443 ms  - step 3
2.2 [REVEALED] 10.2.0.2 (10.2.0.2)  <252,252> [frpla = 0][qttl = 1][uturn = 0] 35.879 ms  - step 2
2.3 [REVEALED] 10.3.0.2 (10.3.0.2)  <251,251> [frpla = 0][qttl = 1][uturn = 0] 66.288 ms  - step 1
2.4 [REVEALED] 10.4.0.2 (10.4.0.2)  <250,250> [frpla = 0][qttl = 1][uturn = 0] 64.19 ms  - step 0

3  CE2 (192.168.2.2)  <250,250> [frpla = 3][qttl = 1][uturn = 0] 116.643 ms
4  CE2 (192.168.2.2)  <250,250> [frpla = 2][qttl = 1][uturn = 0] 99.93 ms
5  192.168.4.2 (192.168.4.2)  <250,250> [frpla = 1][qttl = 1][uturn = 0] 94.185 ms

§.§.§ Juniper Invisible Configurations

All configurations presented here were run on the topology provided by
Fig. <ref>.

Juniper, with Olive OS, does not apply the IP-TTLLSE-TTL at the exit
of the MPLS cloud. As such, the trigger does not provide the return
tunnel length but is equal to 1 because the ingress LER process the incoming IP
TTL in a distinct way with respect to the origin of the packet (locally generated or
not).  Invisible PHP tunnel can, then, be revealed through DPR.  Juniper LSR
can be configured as followed:

[title=JunOS Olive – Invisible PHP]

[title=running over JunOS Olive – Invisible PHP]
Launching TNT: 192.168.2.102 (192.168.2.102)

1  CE1 ( 172.16.0.5)  <255,64> [frpla = 0][qttl = 1][uturn = 0] 0.638 ms
2  PE1 ( 172.16.0.2)  <254,63> [frpla = 0][qttl = 1][uturn = 0] 1.898 ms

FRPLA | Length estimation : 1 | Revealed : 3 (difference : 2)
2.1 [REVEALED] left.P1 (192.168.1.2)  <253,62> [frpla = 0][qttl = 1][uturn = 0] 3.039 ms  - step 0
2.2 [REVEALED] left.P2 (192.168.1.6)  <252,61> [frpla = 0][qttl = 1][uturn = 0] 3.951 ms  - step 0
2.3 [REVEALED] left.P3 (192.168.1.10)  <252,61> [frpla = 0][qttl = 1][uturn = 0] 4.906 ms  - step 0

3  left.PE2 (192.168.1.14)  <252,61> [frpla = 1][qttl = 1][uturn = 0] 7.043 ms
4  CE2 (192.168.2.2)  <252,61> [frpla = 0][qttl = 1][uturn = 0] 6.891 ms
5  CE3 (192.168.2.102)  <251,60> [frpla = 0][qttl = 1][uturn = 0] 8.978 ms

On the contrary to Olive, VMX applies the IP-TTLLSE-TTL function.  As
such, the behavior observed is the theoretical one.  It is worth noting that
configuring Juniper VMX for Invisible MPLS tunnels is identical than with Olive.
Invisible tunnels are, now, revealed through DPR, with the trigger.

[title=JunOS VMX – Invisible PHP]

[title=running over JunOS VMX – Invisible PHP]
Launching TNT: 192.168.2.102 (192.168.2.102)

1  CE1 ( 172.16.0.5)  <255,64> [frpla = 0][qttl = 1][uturn = 0] 0.96 ms
2  PE1 ( 172.16.0.2)  <254,63> [frpla = 0][qttl = 1][uturn = 0] 1.66 ms

RTLA | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] left.P1 (192.168.1.2)  <253,62> [frpla = 0][qttl = 1][uturn = 0] 8.8 ms  - step 0
2.2 [REVEALED] left.P2 (192.168.1.6)  <252,62> [frpla = 0][qttl = 1][uturn = 0] 2.134 ms  - step 0
2.3 [REVEALED] left.P3 (192.168.1.10)  <251,62> [frpla = 0][qttl = 1][uturn = 0] 3.352 ms  - step 0

3  left.PE2 (192.168.1.14)  <250,62> [frpla = 3][rtl = 3(3)][qttl = 1][uturn = 3] 4.569 ms
4  CE2 (192.168.2.2)  <250,61> [frpla = 2][rtl = 2(-1)][qttl = 1][uturn = 2] 4.625 ms
5  CE3 (192.168.2.102)  <250,60> [frpla = 1][rtl = 1(-1)][qttl = 1][uturn = 1] 4.355 ms

§.§ Corner Cases: Heterogeneous Propagation Configuration

This section discusses corner cases, i.e., unlikely configurations that may
arise when MPLS is not homogeneously configured throughout the tunnel.  ,
like , cannot deal with those situations, but these abnormal shiftings
have not been clearly encountered in practice.

§.§.§ Cisco Jumpy Configurations

The following Cisco configuration (for IOS 15.2) is supposed to build an UHP
Invisible tunnel.  However, on the contrary to the configuration provided in
Appendix <ref>, the management of LSE-TTL is
heterogeneous over the tunnel.  Indeed, in this case, the Ingress LER is not
configured with the (on the contrary to the Egress LER and other
routers in the tunnel).  As such, the IP-TTLLSE-TTL operation is not
– systematically – applied on the Egress while it is expected to be from the
Ingress. The EH assumes that the propagation configuration is homogeneous among
LERs, which is not the case here.  Therefore, the Egress LER will use the IP-TTL
instead of the LSE-TTL when popping the LSE.  As consequence, and as shown by
the output, we observe that

* the MPLS tunnel is actually Explicit;

* a number of hops equal to the tunnel length after the MPLS
tunnel are missing (here, only CE2 is missing as the platform is too short – see
Fig. <ref> for the Cisco topology we use), leading to a
so-called jump effect.

We call such a configuration Explicit Jump and it can be observed in the
qTTL of the last hop (2 instead of one plus the skipped hop).

[title=IOS 15.2 – Explicit Jump (heterogeneous configuration)]
version 15.2
mpls label protocol ldp
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.8.1 remote-as 1024
neighbor 192.168.8.1 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
redistribute connected
redistribute ospf 10
neighbor 10.12.0.1 remote-as 3333
neighbor 10.12.0.1 next-hop-self
neighbor 192.168.2.2 remote-as 2048
neighbor 192.168.2.2 next-hop-self

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

version 15.2
mpls label protocol ldp
no  propagate-ttl
mpls ldp explicit-null
router bgp 3333
neighbor 10.12.0.1 remote-as 3333

[title=running over IOS 15.2 – Explicit Jump (heterogeneous configuration)]
Launching TNT: 192.168.7.1 (192.168.7.1)

1  left.CE1 (192.168.3.2)  <255,255> [frpla = 0][qttl = 1][uturn = 0] 8.407 ms
2  left.PE1 (192.168.8.2)  <254,254> [frpla = 0][qttl = 1][uturn = 0] 29.477 ms
3  left.P1 (10.1.0.2)  <250,253> [frpla = 3][qttl = 1][uturn = 3][MPLS LSE | Label : 19 | LSE-TTL : 1]  79.929 ms
4  left.P2 (10.2.0.2)  <250,252> [frpla = 2][qttl = 2][uturn = 2][MPLS LSE | Label : 20 | LSE-TTL : 1]  80.573 ms
5  left.P3 (10.3.0.2)  <250,251> [frpla = 1][qttl = 3][uturn = 1][MPLS LSE | Label : 20 | LSE-TTL : 1]  109.577 ms
6  left.PE2 (10.4.0.2)  <250,250> [frpla = 0][qttl = 1][uturn = 0] 79.766 ms
7  192.168.4.2 (192.168.4.2)  <250,250> [frpla = -1][qttl = 2][uturn = 0] 109.357 ms

§.§.§ Juniper Jumpy Configurations

In the fashion of Cisco, Juniper with the Olive OS (this is not possible with
VMX) allows to configure an Explicit Jump tunnel with PHP.  The configuration provided
below shows such an MPLS tunnel. The EH is configured with the
option, while other routers are configured with .  As
such, P3 will not apply the IP-TTLLSE-TTL when popping the label,
leading so to a jump effect that is nearly as long as the tunnel itself (the
Egress LER and CE2 are missing plus the qTTl at 2 on the last hop).

[title=Olive – Explicit Jump (heterogeneous configuration)]
propagate ttl

propagate ttl

propagate ttl

[title=running over Olive – Explicit (heterogeneous configuration)]
Launching TNT: 192.168.2.102 (192.168.2.102)

1  CE1 (172.16.0.5) <255,64> [frpla = 0][qttl = 1][uturn = 0] 0.622 ms
2  PE1 (172.16.0.2) <254,63> [frpla = 0][qttl = 1][uturn = 0] 1.749 ms
3  left.P1 (192.168.1.2) <253,62> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299824 | LSE-TTL : 1] 2.799 ms
4  left.P2 (192.168.1.6) <252,252> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299792 | LSE-TTL : 1] 3.725 ms
5  left.P3 (192.168.1.10) <251,251> [frpla = 0][qttl = 1][uturn = 0][MPLS LSE | Label : 299776 | LSE-TTL : 1] 7.784 ms
6  CE3 (192.168.2.102)  <248,57> [frpla = 2][qttl = 2][uturn = 0] 8.884 ms

The last configuration is Juniper Olive with an Invisible Jump
configuration.  This is somewhat equivalent to the Explicit Jump but for
Invisible tunnels.  In that case, when P3 (PHP is configured) will pop the LSE,
it will not apply the IP-TTLLSE-TTL.  As a result, will see the
Ingress LER (PE1) and several hops after P3 will be missed (Egress LER and CE2).
The tunnel is invisible and triggers do not work. One can notice a qTTL of 250
on the last hop of our platform: it means that can miss an entire
path of 255 minus the length of the tunnel!

[title=Olive – Invisible Jump configuration (heterogeneous configuration)]
no-propagate ttl

propagate ttl

no-propagate ttl

[title=running over Olive – Invisible Jump (heterogeneous configuration)]
Launching TNT: 192.168.2.102 (192.168.2.102)
1  CE1 ( 172.16.0.5)  <255,64> [frpla = 0][qttl = 1][uturn = 0] 0.515 ms
2  PE1 ( 172.16.0.2)  <254,63> [frpla = 0][qttl = 1][uturn = 0] 1.712 ms
3  CE3 (192.168.2.102)  <251,60> [frpla = 2][qttl = 250][uturn = 0] 8.553 ms

§.§ RSVP-TE (All Invisible)

In this section, we introduce PHP and UHP invisible configurations with both
Cisco and Juniper OSes but considering RSVP-TE instead of LDP.
There is no much to say as all the resulting behaviors are the same as with LDP
configurations described previously. The conclusion is then simple: is able
to reveal TE tunnels as LDP ones whatever the OS and the popping configuration
is (i.e. PHP or UHP).

§.§.§ Cisco Config – PHP

[title=cisco IOS 15.2 – RSVP-TE PHP]
version 15.2
no propagate-ttl
mpls traffic-eng tunnels

interface Tunnel0
ip unnumbered Loopback0
tunnel mode mpls traffic-eng
tunnel destination 10.9.0.1
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng priority 1 1
tunnel mpls traffic-eng bandwidth 500
tunnel mpls traffic-eng path-option 1 dynamic

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

[title=running over 15.2 – RSVP-TE PHP]
Launching TraceTunnel: 192.168.7.1 (192.168.7.1)

1  192.168.3.2 (192.168.3.2)  <255,255> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]
2  192.168.8.2 (192.168.8.2)  <254,254> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]

FRPLA | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] 10.1.0.2 (   10.1.0.2)  <253,253> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]  - step 2
2.2 [REVEALED] 10.2.0.2 (   10.2.0.2)  <252,252> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]  - step 1
2.3 [REVEALED] 10.3.0.2 (   10.3.0.2)  <251,251> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]  - step 0

3  10.4.0.2 (   10.4.0.2)  <250,250> [ frpla = 3 ][ qttl = 1 ][ uturn = 0 ]
4  CE2 (192.168.2.2)  <250,250> [ frpla = 2 ][ qttl = 1 ][ uturn = 0 ]
5  192.168.4.2 (192.168.4.2)  <250,250> [ frpla = 1 ][ qttl = 1 ][ uturn = 0 ]

The tunnel is easily revealed as for LDP tunnels.

§.§.§ Juniper Config – PHP

[title=Juniper – RSVP-TE PHP]
interface all;

label-switched-path PE1-to-PE2
to 192.168.1.105;

interface all;

shortcuts lsp-metric-into-summary;

interface all;

label-switched-path PE2-toPE1
to 192.168.1.101;

interface all;
interface ge-0/0/2.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

[title=running over Juniper – RSVP-TE PHP]
Launching TraceTunnel: 192.168.2.102 (192.168.2.102)
1  CE1 ( 172.16.0.5)  <255,64> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]
2  PE1 ( 172.16.0.2)  <254,63> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]

RTL | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] left.P1 (192.168.1.2)  <253,62> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0
2.2 [REVEALED] left.P2 (192.168.1.6)  <252,61> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0
2.3 [REVEALED] left.P3 (192.168.1.10)  <251,60> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0

3  left.PE2 (192.168.1.14)  <250,62> [ frpla = 3 ][ rtla = 3(3) ][ qttl = 1 ][ uturn = 3 ]
4  CE2 (192.168.2.2)  <250,61> [ frpla = 2 ][ rtla = 0(-1) ][ qttl = 1 ][ uturn = 0 ]
5  CE3 (192.168.2.102)  <250,60> [ frpla = 1 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]

§.§.§ Cisco Config – UHP

[title=cisco IOS 15.2 – RSVP-TE UHP]
version 15.2
no propagate-ttl
mpls traffic-eng tunnels

interface Tunnel0
ip unnumbered Loopback0
tunnel mode mpls traffic-eng
tunnel destination 10.9.0.1
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng priority 1 1
tunnel mpls traffic-eng bandwidth 500
tunnel mpls traffic-eng path-option 1 dynamic

version 15.2
no propagate-ttl
mpls traffic-eng tunnels
mpls ldp explicit-null

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

version 15.2
no propagate-ttl
mpls traffic-eng tunnels

version 15.2
no propagate-ttl
mpls traffic-eng tunnels
mpls traffic-eng signalling interpret explicit-null verbatim

[title=running over 15.2 – RSVP-TE UHP]
Launching TraceTunnel: 192.168.7.1 (192.168.7.1)
1  192.168.3.2 (192.168.3.2)  <255,255> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]
2  192.168.8.2 (192.168.8.2)  <254,254> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]

Duplicate IP (Egress : 10.4.0.2) | Length estimation : 3 | Revealed : 4 (difference : 1)
2.1 [REVEALED] 10.1.0.2 (   10.1.0.2)  <253,253> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ] - step 4
2.2 [REVEALED] 10.2.0.2 (   10.2.0.2)  <252,252> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ] - step 3
2.3 [REVEALED] 10.3.0.2 (   10.3.0.2)  <251,251> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ] - step 2
2.4 [REVEALED] 10.4.0.2 (   10.4.0.2)  <250,250> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ] - step 1  ( Buddy used )

3  CE2 (192.168.2.2)  <250,250> [ frpla = 3 ][ qttl = 1 ][ uturn = 0 ][ meta = 3, 0, 0 ]
4  CE2 (192.168.2.2)  <250,250> [ frpla = 2 ][ qttl = 1 ][ uturn = 0 ][ meta = -1, 0, 0 ]
5  192.168.4.2 (192.168.4.2)  <250,250> [ frpla = 1 ][ qttl = 1 ][ uturn = 0 ][ meta = 0, 0, 0 ]

§.§.§ Juniper Config – UHP

[title=Juniper – RSVP-TE UHP]
interface all;

label-switched-path PE1-to-PE2
to 192.168.1.105;

interface all;

shortcuts lsp-metric-into-summary;

interface all;

label-switched-path PE2-toPE1
to 192.168.1.101;

interface all;
interface ge-0/0/2.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

interface all;

interface ge-0/0/1.0;
interface ge-0/0/3.0;

[title=running over Juniper – RSVP-TE UHP]
Launching TraceTunnel: 192.168.2.102 (192.168.2.102)
1  CE1 ( 172.16.0.5)  <255,64> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]
2  PE1 ( 172.16.0.2)  <254,63> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]

RTL | Length estimation : 3 | Revealed : 3 (difference : 0)
2.1 [REVEALED] left.P1 (192.168.1.2)  <253,62> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0
2.2 [REVEALED] left.P2 (192.168.1.6)  <252,61> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0
2.3 [REVEALED] left.P3 (192.168.1.10)  <251,60> [ frpla = 0 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ] - step 0

3  left.PE2 (192.168.1.14)  <250,62> [ frpla = 3 ][ rtla = 3(3) ][ qttl = 1 ][ uturn = 3 ]
4  CE2 (192.168.2.2)  <250,61> [ frpla = 2 ][ rtla = 0(-1) ][ qttl = 1 ][ uturn = 0 ]]
5  CE3 (192.168.2.102)  <250,60> [ frpla = 1 ][ rtla = 0(0) ][ qttl = 1 ][ uturn = 0 ]

Again, as one can observe, there is no difference in the trace revelation
regarding the LDP case (in any configurations).

§ P2MP CIRCUITS (ALL INVISIBLE)

In this section, we study the VPRN case. In addition to MPLS, it requires the
MP-BGP feature to build P2MP tunnels. We divide the analysis in two invisible
tunnels sub-cases: the Implicit Null model (relying on the PHP option) and the
Explicit Null one (enabling the UHP option). Note that we simplify a bit the
given configurations for obvious readability purposes (as in the previous TE
section). Moreover the IP configuration is not exactly the same as for previous
cases as its is a more complex MPLS usage.

§.§ VPN BGP MPLS – Implicit Null Model

§.§.§ Cisco Config – PHP

[title=cisco IOS 15.2 – MPLS BGP VPN PHP]
version 15.2
no propagate-ttl
mpls label protocol ldp

ip vrf VPN_Y
rd 1:2
route-target export 1:2
route-target import 1:2

interface GigabitEthernet4/0
description PEtoCEVPN
ip vrf forwarding VPN_Y

router rip
address-family ipv4 vrf VPN_Y
redistribute bgp 1 metric 1
network 10.0.0.0

router bgp 1
address-family vpnv4
neighbor 10.0.0.131 activate
neighbor 10.0.0.131 send-community extended
neighbor 10.0.0.132 activate
neighbor 10.0.0.132 send-community extended
neighbor 10.0.0.130 activate
neighbor 10.0.0.130 send-community extended

address-family ipv4 vrf VPN_Y
redistribute rip
no synchronization

version 15.2
no propagate-ttl
ip vrf VPN_Y
rd 1:2
route-target export 1:2
route-target import 1:2

interface GigabitEthernet4/0
description PEtoCEVPN
ip vrf forwarding VPN_Y

router rip
address-family ipv4 vrf VPN_Y
redistribute bgp 1 metric 1
network 10.0.0.0

router bgp 1
address-family vpnv4
neighbor 10.0.0.130 activate
neighbor 10.0.0.130 send-community extended
neighbor 10.0.0.132 activate
neighbor 10.0.0.132 send-community extended
neighbor 10.0.0.133 activate
neighbor 10.0.0.133 send-community extended

address-family ipv4 vrf VPN_Y
redistribute rip
no synchronization

[title=running over 15.2 – VPN BGP MPLS PHP]
Launching TraceTunnel: 10.0.1.103 (10.0.1.103)
1  192.168.3.2 (192.168.3.2)  <255,255> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]
2  10.0.0.54 (  10.0.0.54)  <254,254> [ frpla = 0 ][ qttl = 1 ][ uturn = 0 ]

OPAQUE | Length estimation : 3 | Revealed : 0 (difference : 3)

3  10.0.0.58 (  10.0.0.58)  <250,250> [ frpla = 3 ][ qttl = 1 ][ uturn = 0 ][MPLS LSE | Label : 28 | mTTL : 252 ]
4  10.0.0.57 (  10.0.0.57)  <249,249> [ frpla = 3 ][ qttl = 1 ][ uturn = 0 ]

The bottom label is revealed and equal to 252. This is the only visible MPLS
indication. No revelation is working on such Opaque tunnels.
Whatever the kind of probes sent to or through the VPRN, the IP address visible
to (or  in general) is the outgoing address. Despite its
expired TTL, it is likely that the probe arriving on the Egress PE will be
pushed to the VRF of the VPN and its associated interface before generating the
error message (the VPN being identified with the MPLS label contained in the