Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hessler, Peter
Biidirectional Forwarding Detection (BFD) is a protocol that allows detecting faults in links or routes. This is similar to GRE keep-alives, but is actually supported on real routers. Contrary to traditional link-state detection, BFD works on the next-hop IP address; so one can detect failures of some peers that do not affect the link state. Internet links fail. This is a truism as old as Internet links. When a link fails, traffic gets dropped until the failure is detected and traffic can be re-routed. Detection of failures can be quite tricky however, since they are not always directly visible. Most systems use link state or a form of keep-alives for detection of failures. Link state detection does not help when there are active devices between a router and the other system, such as a switch or long distance links which use MPLS. The in-protocol BGP timers can also be quite long (a common default is 90 seconds) which is a lot of traffic when one are sending 10Gbps or even faster rates. BFD is a new protocol that exists outside of existing routing protocols, but can communicate the status to all protocols. This allows for a single keep-alive to detect the health of a single link, without having to depend on a keep-alive in each and every protocol being used. As this is part of the "parent" interface, this does not introduce another layer in the network configuration. And since the link-state is only per next-hop IP, one can mix and match BFD and non-BFD neighbours on the same interface. This is extremely useful for routers connected to an Internet Exchange Point, which can have hundreds of peers spread over 10 or more physical locations. A clever description of this is described in a draft RFC, which introduces automagic configuration of BFD between parties allowing for stronger resilience when there are many potential neighbouring networks without the overhead of manual configuration. I will be discussing of the implementation of the BFD protocol for OpenBSD, problems discovered in both the protocol and network stack, use cases and production experience.
 My name is Peter Hussler. I'm working on implementing the bidirectional forwarding detection protocol for OpenVSD. I apologize for my voice. I have a bit of a head cold that I've been working on. So to tell you about BFD, we have to tell you about the life before BFD and what problem is it trying to solve. BFD is a protocol that is designed for big iron routers for connecting one router to another and making sure that there is a full data path between the two. Traditionally what you do is you have a single ethernet cable that you plug in to each router and they're right next to each other. So you just can depend on the link state of the connection. In a fiber connection, you see the light from both sides or in a copper ethernet, you see the electrical connectivity. And this is pretty common. But you cannot always depend on this. Maybe there will be a failure in the connection that keeps the link up, but the machine is actually down. Or there may be active devices in the path between you and your neighbor. A common case would be like if you're at an internet exchange and you're connected to the switch at the IXP and then your neighbor that you want to talk to is connected to either the same switch or even a different switch at that IXP. Or you have a long reach dark fiber cable between you and them and there are active paths in the middle that will always show you link state. Yeah? Good. The echo is pretty rough. That means you're coming to my recording. Yeah, but my record is in the back. Oh, sorry. All right. Okay. So what is BFD? It's a collection of RFCs. There's two that are mandatory to implement. The first one simply defines bi-directional forwarding detection. And what it does is it does a health check on the actual forwarding plane of the two devices. For those of you who've been doing a lot of networks and have been using tunneling, it works similar to GRE Keepalives and where it just sends a hello packet every so often and then it receives the response of, yes, I got you. It is protocol independent so it can work on anything. In the wild, I've seen it most commonly used in association with BGP. An interesting thing about the original BFD RFC is that it doesn't tell you what the packet is in. It is just a set of data that you continue. And so the RFC that directly follows it talks about actually implementing it inside of an IP packet. In that case, it's just a simple UDP packet with a well-known port and it just passes it back and forth that way. The reason why they split it up like that is so you can embed the BFD actual features into any sort of packet you want. It can either be something in OSPF or it can be in IP for just standard usage. It can be in Sonet if you want. It can be pretty much anywhere. So why do you care? In the BGP, when you're checking the health of your neighbor, the timers are really commonly around 90 seconds to detect failure if your neighbor is no longer available. That can turn into a huge amount of data when you're at high data rates. How much is that at 10 gigabits or even at 100 gigabits? This can be just massive, massive problems for you. And one problem is that in BGP, the fastest you can do is three seconds because the lowest you can do is one second and you have to lose three packets in order for it to detect the other side is down. And as a side note, the vast majority of the big iron hardware that has 100 gigabit connectors can't do BGP packets every second or cannot do BGP hello packets every second because they're standard CPUs have less computing power than this cup of tea. So as I said, this is very commonly found on big iron routers like Cisco, your high end Juniper, et cetera. The specs talk about microseconds and that's not milliseconds, the one that you're used to normally. It's the u or the mu or whatever, the Greek s not the MS. Yes. Because of how OpenBSD is architected in the background, as an implementation detail, we are extremely unlikely to support anything faster than every 50 milliseconds. And even that is very aggressive. Many for our use cases, that's not a big worry. So there are two modes in BFT. The first one is called asynchronous or active and it's just your standard keep alive. You send the hello, it sends back, I received it. And you just keep sending that back and forth. It behaves just like any other keep alive protocol you've ever heard of. The other one is on demand mode and it is implementation defined, but what most implementations are doing is they're actually monitoring the actual packet counters on that interface. And when it's, and so as long as it's receiving packets from its neighbor, then it just keeps resetting the timer. If it hasn't received the packet from the neighbor, then it manually sends out the hello packet, which when you are running at extremely high data rates or if you have a huge amount of neighbors configured on your system, this can be very beneficial because it lowers the amount of traffic that you have to actually generate and send out. There's an interesting and incredibly stupid part to the spec. This is a direct quotation from the IPv4 and specifically within the UDP part. It mandates which ephemeral port range the sending port must be. And it is not the standard ephemeral port range. This is the high upper ephemeral port range, which is a side note I discovered was defined because of FTP and ports in this range for FTP should not be firewalled. Thank you 1985 or whenever that was. It recommends that the source port is globally unique for all of the BFDPers and that you can reuse it, but you should minimize your reuse. But none of that matters because you must use the mechanisms actually built into VFD to identify which session is which. So we defined all these crazy rules and unfortunately you have to follow them because these are musts in the spec, but they ultimately just don't matter. I discovered that this actually matters because if you don't set your port in that range, you cannot bring up against a juniper. It's a must, okay. All right, I guess. Another delightful part of the spec is there is an optional encryption portion where you can ensure that the packet you're sending your neighbor is the same as the packet they actually receive. And so they have two modes for most of them. Key for the SHA1 authentication is keyed SHA1 and meticulous keyed SHA1. And in this, the authentication header includes the type, the length, the sequence number and the key itself, and then there's some magic done on the packet to encrypt it or to hash it. For meticulous keyed, the sequence number must be incremented for every packet, which means for non-meticulous, it's optional. So even though the entire point of this is to avoid replay attacks, it's kind of an optional thing, whether you are avoiding a replay attack or not. As a implementation detail, we will always increment the sequence number even if you're in the non-meticulous mode because that is legal. You're allowed to increase it any time you want to. And just for receiving the packets, we'll just have to have, if it's non-meticulous, then it's allowed to equal the last one. Interestingly, both of the RFCs are full of these sort of very interesting requirements. It seems like I get the real feeling it was hardware engineers who worked on these specs and not necessarily network engineers. The large parts of it were if you read it, you get the feeling it was really easy to implement this into an ASIC, but doesn't actually matter on the wire. It doesn't match what a software engineer would, or like a software-based network engineer would expect. Currently, I've implemented all the musts of the 5880 and 5881, which is the base BFD implementation and the single hop IPv4 and IPv6. I can do a turn up against the Juniper device, and I can do all the regular health checks with that. When I pull a link, I see it fail correctly within the appropriate amount of time. I can do, it's all the basic configuration options are there. A few bits are hard-coded just because I haven't felt like actually dealing with that yet. Some basic logging, quite a bit of this will need to be removed because it's still printfdebugging, printfdevelopment style. I am using route messages to indicate certain state changes within BFD, which I'll go into a bit later. Nicely, because of the mechanisms I'm using, even though this is entirely within the kernel, you can use pf to block or otherwise manipulate the packets. They show up as regular packets. They enter all the regular interfaces, and I'm not bypassing the stack to either send or receive, which definitely simplifies my life a lot and keeps it to a generic solution for making that fast as opposed to, oh, God, I have to process every single packet received on the interface. My development for this was a lot of exploration that needed to happen to fully understand all the details of this. This is one of the first time I've implemented a network protocol actually in the kernel. There was a lot of just poking at exploration. One of the original designs from us was to do interface to interface for your standard link between two routers. Because we have a number of pieces that support interface link manipulation, then it seemed like that would be a very good place to start to at least get it up, get it running, and to get that working. Unfortunately, as I was implementing it, I discovered that manipulating the interface state was at the exact same place as interacting with the actual route state itself. Since one of my end goals was to get this to have on an IX, where I can connect to multiple peers at the same time, I decided to just skip working on the interfaces itself and move it all into the routing subsystem. Because of that, I've ended up migrating it to the routing subsystem. This is definitely a much better location for it because it's almost exclusively to do with the next top IP address, with your neighbor IP address, that you want to send all of your routes across. You want to send all of your routes to that address. In the OpenVC kernel, when we notice that the next top neighbor IP address is no longer accessible for whatever reason, then we have all these built-in mechanisms that will mark it down. It'll say this is an invalid next hop, and then either the built-in kernel routing table will just fail over to whatever has an overlapping route, or it'll signal to a routing daemon, such as BGPD, and then BGP will then make its, will recalculate all of the routes because its next top is no longer valid, and it'll be able to redistribute the routes as appropriate. So as I mentioned that the interface link state was as equally as difficult to work as the route next top link state. I tried working with that, but this is hard-coded in quite a few places, so I decided just to punt on this problem, and I'll come back to it at a later point in time. What I did do was I was able to, I created two new route flags that I'll be using to signal the BFD state. For now, because it kind of matches, and I have both upper and lower case available, I decided to use F as my BFD state, capital F is when it's up, lower case F when it's down, and I also have some special route messages that allow me to tell that I can announce that the specific details of that BFD state, so that way an application that cares, such as BGPD, can then monitor that and then not have to decipher all of the route flag messages itself. You can listen only for those messages globally as opposed to listening for just a subset of nets. I've run into some very interesting problems. I had an interesting problem originally where I was leaking an M buff for every packet I received, which created a really interesting thing as at almost exactly 90 minutes after I brought up the session, we would panic because we'd ran out of memory. It was just strange that it was almost exactly like two of the second 90 minutes after I brought up the session. After using your standard free functions, I was able to work around that. Now it's a very interesting thing where somewhere deep in SO receive, I receive a panic after about eight hours or so, I haven't touched SO receive at all. This will require some very interesting debugging to figure out what is going on there. My guess is I'm stopping in memory somewhere. It's relatively easy to do, but since it's nothing that I've touched, I'm confused. It's always in SO receive somewhere deep inside it. I reconfigure BFD either because I'm typing too fast in my search, in my shell, and I enter before I actually read what I'm command I'm setting, or if I just rerun it for whatever reason, it gets into a weird half-configured, half-unconfigured mode that needs to be fixed. That's just a lot of looking at the details of how I'm spinning up and actually connecting BFD to the system. Since all of that will have to be completely rewritten when I migrate fully to the route side, I'm not really super worried about that. Okay, so the question is, what do I mean by manipulating or going to the route side? So right now, I'm configuring everything with ifconfig, and it is a pointer that hangs off of the ifp pointer structure. Yeah, I can see some grimaces. As I said before, this is completely the wrong place. What my plan is is to have this be attributes on the next hop, and then the actual user interface side will be within the, you'll configure it through route. Okay, yeah, so it will be attached to a route, because we'll have to have a route to whatever sort of thing that we're monitoring. Okay, can I. that we need that word, we got the valid, but the correct one is the other word is the other one, the main valid, but that's actually what we need to serve on the dot. Correct. That is something that will need to be worked on. And right now in the implementation that's completely ignored for now. But when it's actually moved into the root subsystem properly is that knowledge will need to be maintained. And then we'll need to do, we'll need to have some, probably have some hooks for when we learn identical routes and have not necessarily empath knowledge because empath is primarily for the, where the priority is the same. Because in OpenBSD the way we implement that is if you learn through BGP you get one priority. In terms of OSPF you get a different priority. If it's statically configured it's at a yet third different priority and the lowest number priority would win. And so yeah we would need to look at that and make sure that, make sure that we handle that case correctly. Possibly, possibly. This is something that will. Yeah, those are details that we'll have to pay attention to definitely. Right, so that's what I'll be doing with the route message. Yeah. Right, precisely that's, that is the situation that I'm in now. Is that with the monitoring is because as I said I'm punting on that issue. I have the route message and then anyone who will have knowledge of BFD unfortunately will have to make small modifications to the daemons that care about this to understand it more. We can also take a look at maybe integrating the BFD up down flag state as well. But yeah this is like we'll definitely have some special knowledge for those protocols but then the question is do we care about the static route. So if you, if you manually configure and you're not using a routing daemon what happens in that situation like it and or like if it's plugged in directly on the same subnet. Right. But if that link is down so if you unplug the cable we see that and then we fail over the normal route elections in the kernel side. Yeah. Yeah. Yeah. Yeah. Yeah. and I have a simple minimal, all I need is the BFD damage. I don't need any other stuff. I'm just going to stop here. Yeah, so this is how I'm doing it in some of my testing with, against our Juniper, is exactly that. That only refers to routes that have a next top of my BFD destination, my BFD peer. I'm also thinking about manipulating it so the BFD peer itself is offline. And that one, I agree, maybe we would be the only ones doing that. I want to look and investigate that a lot more before we would go into that sort of situation. Because if no one else is doing it, maybe it's not worth it. Okay, yeah, so what we do in OpenBSD is that we have the different priorities. If we mark it as not up, remove the up flag, the routes stay, but then they're no longer valid. And so we just remove it that way. Yeah, so that way when the peer goes back up, they're then available. And you can keep using them. I have state D solve the problem when I want to design this on my own. It doesn't necessarily solve it when I want to connect to other big iron implementations. Because trying to get someone to run a state D on like say their Cisco or their Juniper is not going to work out so well. Yeah, and then also BFD is designed for the ultra ultra high performance. As I mentioned, it's in milliseconds. And the other one is if loss of view, if loss of view would be a PGP session, the signal to PGP, PGP does its thing. They go, oh, my peer is down. It's starting to establish a mutation. It's still not, no, better to track all the routes. But the FD does the same thing. There's no data for static routes. Right. I have state D and build back all the way. In limited ways. The relady also has a not so well known node. Yeah. Yeah, it can monitor routes. It can check relady and monitor 45 model. Right. Yeah. That's a different thing than BFD. I think that BFD, as you said, can do it against the system. Exactly. Exactly. This is something that you don't necessarily need to, the configuration is much simpler, because you just say IP and go. I was just going to point out that the principle we survive, BFD is a device we're testing. And it's testing something else. Right. Yeah. Yeah. It's telling the other parts of the local system if something is accessible or non-accessible. The BFD code is a principle that is forever available to the people. Now you've got to put the same stuff. Right. Yeah. It's going to teach people. I've noticed. No, but this isn't finished yet. There are a lot of details to be aware of. Yeah. About 50% of the code I've written so far needs to be thrown away. It needs to be thrown away very, very quickly. Yeah. Yeah. Yeah. Yeah. Yeah. The code is rather horrific right now. There are some interesting naming schemes that I'm using, which are basically I'm copying the RFC so I don't need to translate from RFC to my code and understand which is happening. Yeah. It's colorful. Colorful is a good word. At least there are no intentional spellings in it like the HTML protocol or HTML spec. And it's classified as a fairly wide-ranging system in general. So why does it need to be done in general? For two primary reasons. One, the performance requirements. In Useland, it'll get dropped at high load. And then secondly, I'm actually manipulating the actual route, the status of the route flags. And you can't really do that cleanly with the outside of the kernel. Yeah. And so the other thing is that this only supports one period of time. I have most of the framework to handle multiple peers, but I just haven't finished that yet. And once I realize it was the wrong place, I will migrate it somewhere nice, get that all working, and then start working on the multiple peers and multiple configurations because that will completely, completely change. So just to give you a little bit of the simple setup, this is kind of a cleaned up version of it's config output. You see here you have two BFD lines. It tells me the source, which is my IP, destination, which is my neighbor. I'm in the active mode, also known as asynchronous, but that's a stupid name. And we're doing it basically at one second for both the transmit and the receive. And the multiplier is three. The multiplier is an interesting bit where when the session goes down, you have to be up for the multiplier times the transmit, time frame. And then if you keep flapping, the multiplier increases it and how often before it goes back up again. So if you go up for three seconds and then you go down again, you have to be up for nine seconds before it marks it up, et cetera, et cetera, et cetera. I was going to say, well, there are limits within it of how many times to multiply. I think it's just you can only do it, you only multiply by itself a total number of times. No, no, no, no, no. We've seen this before. Oh yeah, yeah, definitely, definitely. So that's just a very simple, this is completely wrong, completely wrong place, as I said. It doesn't belong in a config. But I'll show you what I have. Oh, nice, it just barely fits. So this is the routing table that I have. And you can see here the capital F means that I have brought it up against my neighbor. That is Juniper MX80 in our test lab. And then this is also the Juniper side that shows the configuration. You can see it's been up for almost an hour and a half. It shows a variety of information. Our, so you set in the BFT packet, you set a discriminator, which is your unique session ID. Juniper, like the vast majority of people, just do i++ for every time they're configured. We do arc for random. So that way we try to make it nonpredictable. And the Juniper thankfully understands it and doesn't fight too much about that. Future plans, fix the bugs, fix all the bugs, of course. Yeah, as I said, we need to migrate from the interface subsystem to over to routing. And as part of our discussion earlier, is that it's a much, much better place to put it because we're actively care about routes and not necessarily which interface it's on. That being said, if it does change interfaces, then you mark it down and you redo everything. It's appropriate. The user interface user experience is right now really bad. That clearly needs to become better. Thankfully, I think I have enough information to be able to explain what all the issues would be at an upcoming hackathon so we can have a nice catfight and argue about that. Definitely need to support multiple peers over the same interface. Right now it's limited to one source and destination on one interface. We can do multiple interfaces, but that only solves a small amount of the problem. We need to add support for the encryption, which is that meticulous and non-meticulous features. Right now they've only specified Simple, which is just a pre-shared key that you just splat in. They also have an MD5, which I believe was a historical implementation. And then they support SHA-1. I have not looked very hard, but I didn't see any other encryption methods. Unfortunately, right now SHA-1 is a must support. So we still have to deal with that algorithm, even though it's not the best. And as I said, we do want to work, or rather I want to work on the actual manipulation of the up-down states in the routing table. Whether or not we do that is both a decision that we should make and possibly a technical limitation that we would run into. We do want to work on adding this knowledge into the various routing daemons so they can make their own intelligent decisions on the content and get them working so they can do their own route decisions based on the BFD state. Possibly integrates into SwitchD, VXLands. That way those layers understand more about the actual connectivity to the various endpoints that they're trying to talk to. And that way we can signal this in a way where it brings the, so we would know when one of them is up or down. There's also a multi-hop feature available in BFD. And that allows for very interesting things where you can say, I want to monitor this IP. Then I have three possible routes to it. And you actually health monitor each of the three different routes. Because it's standard routing, you can only make decisions based on the first next hop. If they all converge after that and go through the same path, you can't really do much about that. Or like, Bell Canada here in Ontario, where you're going to be here, just off your next stop. Yeah, that would definitely be very, very handy to have. Is, yeah, the comment was, one of the local providers, Bell Canada? Yeah. Bell Canada, your BGP session is a multi-hop session. Canada's not that third world, but it's damn near. Yeah, telecom wise, telecom wise, of course. It's bad. Why? Well, yourshall φo funnily changed lap sinceApplausePK And now, because I represent our graduates, we're gonnappe pops up at 0 white, but with a global dummy. You get one access to a section, where we can actually boot all of the time we have left. Here, have a switch. That's frightening. Yeah, thankfully in Frankfurt we have no such issues. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, yeah. Yeah, you can, you can do it. And so, well, when you, when you configure it, you say what the minimum and the maximum values are. And so there is a negotiation of the timers. Yes, they do heading. Yeah. Right there. It tests quite a bit of the forwarding path but not 100% of it, which is unfortunate because that was, that was the design behind it. And I think it's, on the bright side it does detect when the actual remote site is up. So if your remote site has crashed, then, you know, thankfully it goes down at that point. But there are, there are some limitations about how, about where exactly is processed. And yeah, as I said, like, okay, this cup is too empty and like it still has more computing power than most of the routing CPUs and those devices. Yeah. I glanced at the RFC and said, oh, different RFC. I'm putting you aside for now. Yeah, it's, yeah, doing it over, over, yeah, aggregated links. Trunk, legs, whatever. Essentially you need to monitor each member. And I'm not sure if I think in that case then it is encapsulated inside special Ethernet frames and not necessarily within UDP itself. So that's one of the reasons why they've extracted it out and made them different RFCs. That is definitely something we'll have to look at. And, but I have, I have designed the code in a way where the actual BFD processing monitoring, it basically, it takes, it takes the configuration or it takes the, what should I be encapsulated in as, as an external thing. So it just, it still works independently and just uses whatever method that you gave it to do. Yeah. Ooh, fun times. Fun times. If you, if it's easy to do it hardware, it should be hard to do in software. Yep. Yep. So thankfully that's right at the end. Are there any more questions? Why wouldn't you aggregate the technology running it over lag? You want to know that when it's down to the greatest extent. I think don't have to be any of you, why is that? It is, you know, really, if there's virtually no value add to it. Well, the value add there is that you can tell that if you're, if you're aggregating like 10, 10 gig links and you go from, from 100 gigs total to 90 gigs total, you want to know that. And you want to be able to make decisions based on that. And you have a kitchen sink, so you want to solve everything with the kitchen sink. And like, like, like, like, lack P binds them together, but isn't necessarily, it doesn't give be, be GP this information. It doesn't tell you when it's degraded, so to speak. I'm not sure of a one member popping out really is still in scope for this. I get a burden. Yeah. It's, it's, it's, that's completely out of scope for now. Yeah, definitely. I have. Where. You can tell because you get no traffic. Yeah. So you're putting your traffic for way, way, way. And you're being. You're back. Right. Yeah. There are certainly interesting. Yeah. Well, not in the, in this. I have a useful underlying thing there. Technology wise. Well, I would argue that. Yes, welcome to the internet. Yeah. Right. Well, part of the, part of the concern there is that how does the, how does lack P do the hashing and are you getting lucky and then just going over just the one link that's working, but you're still losing the other end links. Right. So thankfully it's not listed as a must within RFC 5880. So maybe we'll support it. Maybe we won't. Question in the back. Yeah. Yeah. Yeah. Yeah. And since it has knowledge of the, since the other side has to actually respond back over the same link. Yeah. Right. Right. Yeah. Yeah. Yeah. And either that or you may have interesting results over trunk. Yeah. Other questions. All right. Thank you.
 My name is Peter Hussler.  I'm working on implementing the bidirectional forwarding detection protocol for OpenVSD.  I apologize for my voice.  I have a bit of a head cold that I've been working on.  So to tell you about BFD, we have to tell you about the life before BFD and what problem  is it trying to solve.  BFD is a protocol that is designed for big iron routers for connecting one router to  another and making sure that there is a full data path between the two.  Traditionally what you do is you have a single ethernet cable that you plug in to each router  and they're right next to each other.  So you just can depend on the link state of the connection.  In a fiber connection, you see the light from both sides or in a copper ethernet, you see  the electrical connectivity.  And this is pretty common.  But you cannot always depend on this.  Maybe there will be a failure in the connection that keeps the link up, but the machine is  actually down.  Or there may be active devices in the path between you and your neighbor.  A common case would be like if you're at an internet exchange and you're connected to  the switch at the IXP and then your neighbor that you want to talk to is connected to either  the same switch or even a different switch at that IXP.  Or you have a long reach dark fiber cable between you and them and there are active  paths in the middle that will always show you link state.  Yeah?  Good.  The echo is pretty rough.  That means you're coming to my recording.  Yeah, but my record is in the back.  Oh, sorry.  All right.  Okay.  So what is BFD?  It's a collection of RFCs.  There's two that are mandatory to implement.  The first one simply defines bi-directional forwarding detection.  And what it does is it does a health check on the actual forwarding plane of the two  devices.  For those of you who've been doing a lot of networks and have been using tunneling,  it works similar to GRE Keepalives and where it just sends a hello packet every so often  and then it receives the response of, yes, I got you.  It is protocol independent so it can work on anything.  In the wild, I've seen it most commonly used in association with BGP.  An interesting thing about the original BFD RFC is that it doesn't tell you what the  packet is in.  It is just a set of data that you continue.  And so the RFC that directly follows it talks about actually implementing it inside of an  IP packet.  In that case, it's just a simple UDP packet with a well-known port and it just passes  it back and forth that way.  The reason why they split it up like that is so you can embed the BFD actual features  into any sort of packet you want.  It can either be something in OSPF or it can be in IP for just standard usage.  It can be in Sonet if you want.  It can be pretty much anywhere.  So why do you care?  In the BGP, when you're checking the health of your neighbor, the timers are really commonly  around 90 seconds to detect failure if your neighbor is no longer available.  That can turn into a huge amount of data when you're at high data rates.  How much is that at 10 gigabits or even at 100 gigabits?  This can be just massive, massive problems for you.  And one problem is that in BGP, the fastest you can do is three seconds because the lowest  you can do is one second and you have to lose three packets in order for it to detect the  other side is down.  And as a side note, the vast majority of the big iron hardware that has 100 gigabit connectors  can't do BGP packets every second or cannot do BGP hello packets every second because  they're standard CPUs have less computing power than this cup of tea.  So as I said, this is very commonly found on big iron routers like Cisco, your high  end Juniper, et cetera.  The specs talk about microseconds and that's not milliseconds, the one that you're used  to normally.  It's the u or the mu or whatever, the Greek s not the MS.  Yes.  Because of how OpenBSD is architected in the background, as an implementation detail, we  are extremely unlikely to support anything faster than every 50 milliseconds.  And even that is very aggressive.  Many for our use cases, that's not a big worry.  So there are two modes in BFT.  The first one is called asynchronous or active and it's just your standard keep alive.  You send the hello, it sends back, I received it.  And you just keep sending that back and forth.  It behaves just like any other keep alive protocol you've ever heard of.  The other one is on demand mode and it is implementation defined, but what most implementations  are doing is they're actually monitoring the actual packet counters on that interface.  And when it's, and so as long as it's receiving packets from its neighbor, then it just keeps  resetting the timer.  If it hasn't received the packet from the neighbor, then it manually sends out the hello  packet, which when you are running at extremely high data rates or if you have a huge amount  of neighbors configured on your system, this can be very beneficial because it lowers the  amount of traffic that you have to actually generate and send out.  There's an interesting and incredibly stupid part to the spec.  This is a direct quotation from the IPv4 and specifically within the UDP part.  It mandates which ephemeral port range the sending port must be.  And it is not the standard ephemeral port range.  This is the high upper ephemeral port range, which is a side note I discovered was defined  because of FTP and ports in this range for FTP should not be firewalled.  Thank you 1985 or whenever that was.  It recommends that the source port is globally unique for all of the BFDPers and that you  can reuse it, but you should minimize your reuse.  But none of that matters because you must use the mechanisms actually built into VFD  to identify which session is which.  So we defined all these crazy rules and unfortunately you have to follow them because these are  musts in the spec, but they ultimately just don't matter.  I discovered that this actually matters because if you don't set your port in that range,  you cannot bring up against a juniper.  It's a must, okay.  All right, I guess.  Another delightful part of the spec is there is an optional encryption portion where you  can ensure that the packet you're sending your neighbor is the same as the packet they  actually receive.  And so they have two modes for most of them.  Key for the SHA1 authentication is keyed SHA1 and meticulous keyed SHA1.  And in this, the authentication header includes the type, the length, the sequence number  and the key itself, and then there's some magic done on the packet to encrypt it or  to hash it.  For meticulous keyed, the sequence number must be incremented for every packet, which  means for non-meticulous, it's optional.  So even though the entire point of this is to avoid replay attacks, it's kind of an optional  thing, whether you are avoiding a replay attack or not.  As a implementation detail, we will always increment the sequence number even if you're  in the non-meticulous mode because that is legal.  You're allowed to increase it any time you want to.  And just for receiving the packets, we'll just have to have, if it's non-meticulous,  then it's allowed to equal the last one.  Interestingly, both of the RFCs are full of these sort of very interesting requirements.  It seems like I get the real feeling it was hardware engineers who worked on these specs  and not necessarily network engineers.  The large parts of it were if you read it, you get the feeling it was really easy to  implement this into an ASIC, but doesn't actually matter on the wire.  It doesn't match what a software engineer would, or like a software-based network engineer  would expect.  Currently, I've implemented all the musts of the 5880 and 5881, which is the base BFD  implementation and the single hop IPv4 and IPv6.  I can do a turn up against the Juniper device, and I can do all the regular health checks  with that.  When I pull a link, I see it fail correctly within the appropriate amount of time.  I can do, it's all the basic configuration options are there.  A few bits are hard-coded just because I haven't felt like actually dealing with that yet.  Some basic logging, quite a bit of this will need to be removed because it's still printfdebugging,  printfdevelopment style.  I am using route messages to indicate certain state changes within BFD, which I'll go  into a bit later.  Nicely, because of the mechanisms I'm using, even though this is entirely within the kernel,  you can use pf to block or otherwise manipulate the packets.  They show up as regular packets.  They enter all the regular interfaces, and I'm not bypassing the stack to either send  or receive, which definitely simplifies my life a lot and keeps it to a generic solution  for making that fast as opposed to, oh, God, I have to process every single packet received  on the interface.  My development for this was a lot of exploration that needed to happen to fully understand all  the details of this.  This is one of the first time I've implemented a network protocol actually in the kernel.  There was a lot of just poking at exploration.  One of the original designs from us was to do interface to interface for your standard  link between two routers.  Because we have a number of pieces that support interface link manipulation, then it seemed  like that would be a very good place to start to at least get it up, get it running, and  to get that working.  Unfortunately, as I was implementing it, I discovered that manipulating the interface  state was at the exact same place as interacting with the actual route state itself.  Since one of my end goals was to get this to have on an IX, where I can connect to multiple  peers at the same time, I decided to just skip working on the interfaces itself and  move it all into the routing subsystem.  Because of that, I've ended up migrating it to the routing subsystem.  This is definitely a much better location for it because it's almost exclusively to do  with the next top IP address, with your neighbor IP address, that you want to send all of your  routes across.  You want to send all of your routes to that address.  In the OpenVC kernel, when we notice that the next top neighbor IP address is no longer  accessible for whatever reason, then we have all these built-in mechanisms that will mark  it down.  It'll say this is an invalid next hop, and then either the built-in kernel routing table  will just fail over to whatever has an overlapping route, or it'll signal to a routing daemon,  such as BGPD, and then BGP will then make its, will recalculate all of the routes because  its next top is no longer valid, and it'll be able to redistribute the routes as appropriate.  So as I mentioned that the interface link state was as equally as difficult to work as  the route next top link state.  I tried working with that, but this is hard-coded in quite a few places, so I decided just  to punt on this problem, and I'll come back to it at a later point in time.  What I did do was I was able to, I created two new route flags that I'll be using to  signal the BFD state.  For now, because it kind of matches, and I have both upper and lower case available,  I decided to use F as my BFD state, capital F is when it's up, lower case F when it's  down, and I also have some special route messages that allow me to tell that I can announce  that the specific details of that BFD state, so that way an application that cares, such  as BGPD, can then monitor that and then not have to decipher all of the route flag messages  itself.  You can listen only for those messages globally as opposed to listening for just a subset  of nets.  I've run into some very interesting problems.  I had an interesting problem originally where I was leaking an M buff for every packet I  received, which created a really interesting thing as at almost exactly 90 minutes after  I brought up the session, we would panic because we'd ran out of memory.  It was just strange that it was almost exactly like two of the second 90 minutes after I  brought up the session.  After using your standard free functions, I was able to work around that.  Now it's a very interesting thing where somewhere deep in SO receive, I receive a panic after  about eight hours or so, I haven't touched SO receive at all.  This will require some very interesting debugging to figure out what is going on there.  My guess is I'm stopping in memory somewhere.  It's relatively easy to do, but since it's nothing that I've touched, I'm confused.  It's always in SO receive somewhere deep inside it.  I reconfigure BFD either because I'm typing too fast in my search, in my shell, and I  enter before I actually read what I'm command I'm setting, or if I just rerun it for whatever  reason, it gets into a weird half-configured, half-unconfigured mode that needs to be fixed.  That's just a lot of looking at the details of how I'm spinning up and actually connecting  BFD to the system.  Since all of that will have to be completely rewritten when I migrate fully to the route  side, I'm not really super worried about that.  Okay, so the question is, what do I mean by manipulating or going to the route side?  So right now, I'm configuring everything with ifconfig, and it is a pointer that hangs  off of the ifp pointer structure.  Yeah, I can see some grimaces.  As I said before, this is completely the wrong place.  What my plan is is to have this be attributes on the next hop, and then the actual user  interface side will be within the, you'll configure it through route.  Okay, yeah, so it will be attached to a route, because we'll have to have a route to whatever  sort of thing that we're monitoring.  Okay, can I.  that we need that word, we got the valid, but the correct one is the other word is  the other one, the main valid, but that's actually what we need to serve on the dot.  Correct. That is something that will need to be worked on. And right now in the implementation  that's completely ignored for now. But when it's actually moved into the root subsystem  properly is that knowledge will need to be maintained. And then we'll need to do, we'll  need to have some, probably have some hooks for when we learn identical routes and have  not necessarily empath knowledge because empath is primarily for the, where the priority is  the same. Because in OpenBSD the way we implement that is if you learn through BGP you get  one priority. In terms of OSPF you get a different priority. If it's statically configured  it's at a yet third different priority and the lowest number priority would win. And  so yeah we would need to look at that and make sure that, make sure that we handle that  case correctly. Possibly, possibly. This is something that will.  Yeah, those are details that we'll have to pay attention to definitely.  Right, so that's what I'll be doing with the route message.  Yeah. Right, precisely that's, that is the situation  that I'm in now. Is that with the monitoring is because as I said I'm punting on that issue.  I have the route message and then anyone who will have knowledge of BFD unfortunately will  have to make small modifications to the daemons that care about this to understand it more.  We can also take a look at maybe integrating the BFD up down flag state as well. But yeah  this is like we'll definitely have some special knowledge for those protocols but then the  question is do we care about the static route. So if you, if you manually configure and you're  not using a routing daemon what happens in that situation like it and or like if it's  plugged in directly on the same subnet. Right.  But if that link is down so if you unplug the cable we see that and then we fail over  the normal route elections in the kernel side. Yeah.  Yeah.  Yeah. Yeah. Yeah.  and I have a simple minimal, all I need is the BFD damage.  I don't need any other stuff.  I'm just going to stop here.  Yeah, so this is how I'm doing it in some of my testing  with, against our Juniper, is exactly that.  That only refers to routes that have a next top  of my BFD destination, my BFD peer.  I'm also thinking about manipulating it  so the BFD peer itself is offline.  And that one, I agree, maybe we would be the only ones  doing that.  I want to look and investigate that a lot more  before we would go into that sort of situation.  Because if no one else is doing it, maybe it's not worth it.  Okay, yeah, so what we do in OpenBSD is that we have  the different priorities.  If we mark it as not up, remove the up flag,  the routes stay, but then they're no longer valid.  And so we just remove it that way.  Yeah, so that way when the peer goes back up,  they're then available.  And you can keep using them.  I have state D solve the problem when I want to design  this on my own.  It doesn't necessarily solve it when I want to connect  to other big iron implementations.  Because trying to get someone to run a state D  on like say their Cisco or their Juniper  is not going to work out so well.  Yeah, and then also BFD is designed  for the ultra ultra high performance.  As I mentioned, it's in milliseconds.  And the other one is if loss of view,  if loss of view would be a PGP session,  the signal to PGP, PGP does its thing.  They go, oh, my peer is down.  It's starting to establish a mutation.  It's still not, no, better to track all the routes.  But the FD does the same thing.  There's no data for static routes.  Right.  I have state D and build back all the way.  In limited ways.  The relady also has a not so well known node.  Yeah.  Yeah, it can monitor routes.  It can check relady and monitor 45 model.  Right.  Yeah.  That's a different thing than BFD.  I think that BFD, as you said, can do it against the system.  Exactly.  Exactly.  This is something that you don't necessarily need to,  the configuration is much simpler, because you just say IP  and go.  I was just going to point out that the principle  we survive, BFD is a device we're testing.  And it's testing something else.  Right.  Yeah.  Yeah.  It's telling the other parts of the local system  if something is accessible or non-accessible.  The BFD code is a principle that is forever  available to the people.  Now you've got to put the same stuff.  Right.  Yeah.  It's going to teach people.  I've noticed.  No, but this isn't finished yet.  There are a lot of details to be aware of.  Yeah.  About 50% of the code I've written so far  needs to be thrown away.  It needs to be thrown away very, very quickly.  Yeah.  Yeah.  Yeah.  Yeah.  Yeah.  The code is rather horrific right now.  There are some interesting naming schemes  that I'm using, which are basically  I'm copying the RFC so I don't need to translate from RFC to my  code and understand which is happening.  Yeah.  It's colorful.  Colorful is a good word.  At least there are no intentional spellings in it  like the HTML protocol or HTML spec.  And it's classified as a fairly wide-ranging  system in general.  So why does it need to be done in general?  For two primary reasons.  One, the performance requirements.  In Useland, it'll get dropped at high load.  And then secondly, I'm actually manipulating the actual route,  the status of the route flags.  And you can't really do that cleanly  with the outside of the kernel.  Yeah.  And so the other thing is that this only supports  one period of time.  I have most of the framework to handle multiple peers,  but I just haven't finished that yet.  And once I realize it was the wrong place,  I will migrate it somewhere nice, get that all working,  and then start working on the multiple peers  and multiple configurations because that will completely,  completely change.  So just to give you a little bit of the simple setup,  this is kind of a cleaned up version of it's config output.  You see here you have two BFD lines.  It tells me the source, which is my IP, destination,  which is my neighbor.  I'm in the active mode, also known as asynchronous,  but that's a stupid name.  And we're doing it basically at one second  for both the transmit and the receive.  And the multiplier is three.  The multiplier is an interesting bit  where when the session goes down, you  have to be up for the multiplier times the transmit,  time frame.  And then if you keep flapping, the multiplier increases it  and how often before it goes back up again.  So if you go up for three seconds and then you go down again,  you have to be up for nine seconds before it  marks it up, et cetera, et cetera, et cetera.  I was going to say, well, there are limits within it  of how many times to multiply.  I think it's just you can only do it, you only multiply by itself  a total number of times.  No, no, no, no, no.  We've seen this before.  Oh yeah, yeah, definitely, definitely.  So that's just a very simple, this is completely wrong,  completely wrong place, as I said.  It doesn't belong in a config.  But I'll show you what I have.  Oh, nice, it just barely fits.  So this is the routing table that I have.  And you can see here the capital F  means that I have brought it up against my neighbor.  That is Juniper MX80 in our test lab.  And then this is also the Juniper side that shows  the configuration.  You can see it's been up for almost an hour and a half.  It shows a variety of information.  Our, so you set in the BFT packet,  you set a discriminator, which is your unique session ID.  Juniper, like the vast majority of people,  just do i++ for every time they're configured.  We do arc for random.  So that way we try to make it nonpredictable.  And the Juniper thankfully understands it  and doesn't fight too much about that.  Future plans, fix the bugs, fix all the bugs, of course.  Yeah, as I said, we need to migrate from the interface  subsystem to over to routing.  And as part of our discussion earlier,  is that it's a much, much better place  to put it because we're actively care about routes  and not necessarily which interface it's on.  That being said, if it does change interfaces,  then you mark it down and you redo everything.  It's appropriate.  The user interface user experience  is right now really bad.  That clearly needs to become better.  Thankfully, I think I have enough information  to be able to explain what all the issues would be  at an upcoming hackathon so we can have a nice catfight  and argue about that.  Definitely need to support multiple peers  over the same interface.  Right now it's limited to one source and destination  on one interface.  We can do multiple interfaces, but that only solves  a small amount of the problem.  We need to add support for the encryption, which  is that meticulous and non-meticulous features.  Right now they've only specified Simple,  which is just a pre-shared key that you just splat in.  They also have an MD5, which I believe  was a historical implementation.  And then they support SHA-1.  I have not looked very hard, but I  didn't see any other encryption methods.  Unfortunately, right now SHA-1 is a must support.  So we still have to deal with that algorithm,  even though it's not the best.  And as I said, we do want to work,  or rather I want to work on the actual manipulation  of the up-down states in the routing table.  Whether or not we do that is both a decision  that we should make and possibly a technical limitation  that we would run into.  We do want to work on adding this knowledge  into the various routing daemons so they can make  their own intelligent decisions on the content  and get them working so they can do their own route  decisions based on the BFD state.  Possibly integrates into SwitchD, VXLands.  That way those layers understand more  about the actual connectivity to the various endpoints  that they're trying to talk to.  And that way we can signal this in a way where it brings the,  so we would know when one of them is up or down.  There's also a multi-hop feature available in BFD.  And that allows for very interesting things  where you can say, I want to monitor this IP.  Then I have three possible routes to it.  And you actually health monitor each of the three  different routes.  Because it's standard routing, you  can only make decisions based on the first next hop.  If they all converge after that and go through the same path,  you can't really do much about that.  Or like, Bell Canada here in Ontario, where you're  going to be here, just off your next stop.  Yeah, that would definitely be very, very handy to have.  Is, yeah, the comment was, one of the local providers,  Bell Canada?  Yeah.  Bell Canada, your BGP session is a multi-hop session.  Canada's not that third world, but it's damn near.  Yeah, telecom wise, telecom wise, of course.  It's bad.  Why?  Well, yourshall φo funnily changed lap since ApplausePK And now, because I represent our  graduates, we're gonnappe pops up at 0 white,  but with a global dummy.  You get one access to a section, where we can  actually boot all of the time we have left.  Here, have a switch.  That's frightening.  Yeah, thankfully in Frankfurt we have no such issues.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, yeah.  Yeah, you can, you can do it.  And so, well, when you, when you configure it, you say what the minimum and the maximum values are.  And so there is a negotiation of the timers.  Yes, they do heading.  Yeah.  Right there.  It tests quite a bit of the forwarding path but not 100% of it, which is unfortunate because that was, that was the design behind it.  And I think it's, on the bright side it does detect when the actual remote site is up.  So if your remote site has crashed, then, you know, thankfully it goes down at that point.  But there are, there are some limitations about how, about where exactly is processed.  And yeah, as I said, like, okay, this cup is too empty and like it still has more computing power than most of the routing CPUs and those devices.  Yeah.  I glanced at the RFC and said, oh, different RFC.  I'm putting you aside for now.  Yeah, it's, yeah, doing it over, over, yeah, aggregated links.  Trunk, legs, whatever.  Essentially you need to monitor each member.  And I'm not sure if I think in that case then it is encapsulated inside special Ethernet frames and not necessarily within UDP itself.  So that's one of the reasons why they've extracted it out and made them different RFCs.  That is definitely something we'll have to look at.  And, but I have, I have designed the code in a way where the actual BFD processing monitoring, it basically, it takes, it takes the configuration or it takes the, what should I be encapsulated in as, as an external thing.  So it just, it still works independently and just uses whatever method that you gave it to do.  Yeah.  Ooh, fun times.  Fun times.  If you, if it's easy to do it hardware, it should be hard to do in software.  Yep.  Yep.  So thankfully that's right at the end.  Are there any more questions?  Why wouldn't you aggregate the technology running it over lag?  You want to know that when it's down to the greatest extent.  I think don't have to be any of you, why is that?  It is, you know, really, if there's virtually no value add to it.  Well, the value add there is that you can tell that if you're, if you're aggregating like 10, 10 gig links and you go from, from 100 gigs total to 90 gigs total, you want to know that.  And you want to be able to make decisions based on that.  And you have a kitchen sink, so you want to solve everything with the kitchen sink.  And like, like, like, like, lack P binds them together, but isn't necessarily, it doesn't give be, be GP this information.  It doesn't tell you when it's degraded, so to speak.  I'm not sure of a one member popping out really is still in scope for this.  I get a burden.  Yeah.  It's, it's, it's, that's completely out of scope for now.  Yeah, definitely.  I have.  Where.  You can tell because you get no traffic.  Yeah.  So you're putting your traffic for way, way, way.  And you're being.  You're back.  Right.  Yeah.  There are certainly interesting.  Yeah.  Well, not in the, in this.  I have a useful underlying thing there.  Technology wise.  Well, I would argue that.  Yes, welcome to the internet.  Yeah.  Right.  Well, part of the, part of the concern there is that how does the, how does lack P do the  hashing and are you getting lucky and then just going over just the one link that's working,  but you're still losing the other end links.  Right.  So thankfully it's not listed as a must within RFC 5880.  So maybe we'll support it.  Maybe we won't.  Question in the back.  Yeah.  Yeah.  Yeah.  Yeah.  And since it has knowledge of the, since the other side has to actually respond back  over the same link.  Yeah.  Right.  Right.  Yeah.  Yeah.  Yeah.  And either that or you may have interesting results over trunk.  Yeah.  Other questions.  All right. Thank you.
Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_0_2711.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_2712_5231.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_5232_6899.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_6900_8579.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_8580_15935.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_15936_16679.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_16680_16751.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_16752_16811.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_16812_17123.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_17124_18647.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_18648_19979.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_19980_24071.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_24072_25727.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_25728_26075.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_26076_26795.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_26796_26915.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_26916_28463.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_28464_28595.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_28596_29735.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_29736_29915.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_29916_30803.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_30804_30899.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_30900_32231.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_32232_40991.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_40992_41771.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_41772_43007.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_43008_44135.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_44136_48623.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_48624_51935.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_51936_52259.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_52260_52379.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_52380_52475.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_52476_52847.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_52848_53447.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_53448_54983.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_54984_55847.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_55848_56495.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_56496_56867.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_56868_57179.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_57180_59375.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_59376_59879.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_59880_60659.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_60660_60851.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_60852_61055.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_61056_61139.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_61140_63059.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_63060_63935.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_63936_64559.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_64560_65003.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_65004_65543.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_65544_66611.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_66612_67367.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_67368_67679.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_67680_68831.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_68832_69107.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_69108_69503.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_69504_69563.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_69564_71063.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_71064_71855.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_71856_73559.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_73560_73931.png Bidirectional_Forwarding_Detection_BFD_implementat_nLq6T8zUryo_73932_74153.png