While the industry is preaching cloud in every market, and users, whether eagerly, reluctantly or unwittingly start falling in line everywhere, few seem to talk about the actual fabric of the Internet and the ever remaining complexity that undeniably sits between a customer and their mission critical cloud services.

As a result there is precious little acceptance left for connectivity issues of any kind; never mind that even a fairly large service provider in the northern outskirts of Europe, can influence the global flow of traffic between backbone providers or alleviate transatlantic congestion, to about the same extent as the nearby conference hotel, where said service provider host their customer training sessions, can keep United Airlines (Only due to a "pressing, unforeseen and otherwise undisclosed matter of passenger safety", of course. I'm sure you understand.) from cancelling the customer team's outbound flight, despite the hotel's obvious and undisputed obligation to make good the non-refundable room reservations.

In slightly simpler terms, customers increasingly expect that service providers take it upon them to ensure continuous service availability and performance, at least beyond the customer's own Internet uplink. With the competition never being further away than one click of the mouse, I feel multihoming is rapidly becoming an absolute requirement for even the smallest provider of SaaS or other Internet dependant service - Save being built on top of third party infrastructure, in which case the same requirement falls to said third party.

Unfortunately, BGP is as complex, cumbersome and scary as ever, and just does not seem like a good fit for a relatively small operation without a team of dedicated wan people. As such, I've been looking hard for a fairly robust, simple and inexpensive solution that will allow dynamic routing via two Internet Service Providers without running full BGP from your own AS.

Now, I think I finally have an idea for a usable design that will at least keep dynamic routing protocols at an arms length, and for the most part, within the respective domain of the two ISPs.

Outline of suggested network design

The setup makes use of a couple of notable mechanics of the BGP routing protocol and the F5 Networks BigIP appliance. First out, BGP will give heed to the most specific applicable route object. This allows both ISPs to announce the larger /22 route in addition to a more specific /23 route each, effectively cutting the block into two halfs during normal operation. As the two /23 routes combined account for the entire /22 block; as long as both ISPs are reachable, the /22 routes should be ignored, thus letting traffic flow according to the /23 routes to each of the sites along the green markers.

If, say, the link to ISP 2 goes down, AS 200 should withdraw both routes (or withdraw the small route and downgrade the other, as they can still announce on behalf of it's peer AS 1000) and traffic will start to flow to the /22 route already announced by AS 1000, with any luck, following the red marker.

Since both BigIP appliances are reachable from both ISP gateways in the same layer 2 segment, the rerouted traffic will be bridged across the Lan 2 Lan links, and end up at the same BigIP appliance as before, although they will arrived on one of the two ports facing the Lan 2 Lan rather than the one facing the ISP 2 gateway.

Here the second of he two mentioned mechanics come into play. BigIP will by default transmit replies on the network port where a given request originated, and similarly (or so I hope) direct traffic to the originating gateway. Admittedly, I have not yet tested the part about replies being forwarded to the originating router; I suppose it could be necessary with some sort of rule to update the static gateway route entry, maybe use a simple local routing protocol or perhaps even let a VIP interface on BigIP (or if possible, on the remaining healthy ISP gateway) take over the IP address of the unhealthy ISP gateway.

In this specific layout, the fact that the two ISPs already have direct peering seem to add to the robustness of the solution. Most likely these are already announcing our routes to their respective peers on behalf of each other. This can both help reduce the immediate impact of an outage and speed up propagation of changes necessary to regain full reach-ability. If the AS 200 of ISP 2 becomes mostly unavailable, though somehow maintains connectivity to Site B as well as the BGP session to ISP 1, any traffic destined for ISP 2 that arrives at AS 1000 will simply be forwarded to AS 200 and join the path of the green marker from there.

Of course, if any of these scenarios are reversed and faults occur on the ISP 1 side instead, effects mirroring those previously outlined should present themselves, and traffic should similarly start to flow along the path of the blue marker.

As for the rest of the design, it should be possible, beneficial even, to combine this with DNS round-robin to distribute initial requests across the two sites. The setup is also meant to tolerate failure of any internal component up to and including a full site blackout.

So. Think this should work ok? Any reason it should not be done this way? Did I miss any big gotchas? See any possible improvements? Or perhaps even have an altogether better alternative? I am, as they say, all ears :)


Your thinking around the external BGP advertisements is sound, although as other have suggested, I'd highly recommend you run BGP with your providers for automated route-withdrawal on failure. Consider also the need to take a site offline for maintenance, or the need to add, move or migrate providers in the future.

Having said that, how about an alternative design:

  1. Rather than de-aggregating your prefixes and further polluting the already bloated DFZ, advertise the same /22 out both providers. (Search: BGP Anycast)

  2. Rather than stretching a layer 2-domain across two physically separate sites, make each of those links routed, using something like OSPF and BFD, so that your fail-overs are faster (sub-second), having your sites go split-brain doesn't affect anything and when the day comes that you move one of your sites to a location that is too far away for Layer 2 adjacency, your design will still stand-up.

  3. Configure both your BIG-IP LTMs with the same VIPs, so that either will function, regardless of what is happening with your upstream service-provider(s). Consider having secondary server pools from the neighbouring site so that in the event of maintenance on the back-end, the LTMs can happily continue sending clients to the other site. The VIP subnet would be advertised into OSPF/BGP (preferably from the BIG-IP) making the local LTM the preferred target for traffic from either ISP Gateway. During a maintenance window, this route would be withdrawn but the BIG-IP, and the neighbour site would take over (Search: F5 Route Health Injection)

  4. I don't see anything between the LTMs and the ISP Gateways that is stateful (unless the ISP Gateways themselves are), so return traffic can leave via the nearest ISP gateway regardless of where it originated. For bonus points, if you take a limited BGP feed from each provider (only their customers and local peers) and share that between your Internet Gateways you will provide the best performance to those clients.

BGP doesn't have to be a complex beast, and like any design: Good documentation should address any concerns about supportability into the future. If your management aren't supportive of building it right, then I'd advise that SaaS is not a space they should be getting into. Outsource to a player like AWS and make it Someone Else's Problem™

| improve this answer | |
  • Clear points, I appreciate that. There are reasons for not using some large IaaS, which would be a convenient solution, I agree. Tho, truth be told, my employer has been doing SaaS a lot longer than Amazon. And as for "doing it right", BPG arguably does add complexity, and complexity is the nemesis of availability. The current design with multiple paths to a single ISP isn't horrible, but if you think about it .. deploying BGP on our end will simply put us in the role ISP1 has today. Only fewer peers, no experience and less resources. I'm not sure why we would think that we could do better – Roy Jul 15 '15 at 11:33
  • Curious, is there something about BGP that makes is better at detecting loss of reachability than any other protocol or mechanism that might be deployed to detect failures? – Roy Jul 15 '15 at 11:39
  • 4) That holds true in my draft design as well, doesnt it? Both ISPs will accept traffic sourced anywhere within the /22 – Roy Jul 15 '15 at 11:41
  • 3) Of course. And thats a carrier grade N+1 ring connecting the four sites (Site A + standby and Site B + standby) – Roy Jul 15 '15 at 11:42
  • 2) The layer 2 seems to be working great for national deployments such as this. Reduces complexity, gives great flexibility and very fast failover should one leg in the ring go offline – Roy Jul 15 '15 at 11:43

Regarding availability, depending on your SLA requirements, the manual failover from one ISP to another by calling the ISP and requesting the route be removed sounds like a procedure that would exceed an SLA. BGP routes do not change rapidly like internal routing protocols which makes me worried as to how long the change is going to take. Further if a ISP is having issues the reality they'll have staff available to satisfy your request in a timely manner also sounds risky to me.

Depending on SLA's I would want to have both ISP1 and ISP2 at both SiteA and SiteB. I would then want to run BGP with both ISP's. This would allow for automatic failover. However, it sounds like the option we all want to see is out of scope/budget.

I would just suggest monitoring the BGP paths with something like thousandeyes.com. You can set this up to send you email alerts should a path change, which may allow you to detect an issue before the ISP reports this to you.

The diagram you have up, looks like you have Cisco ASR which makes me think ASR1001 or something similar. If so, that's a great device to run BGP w/ your service provider. The only other device I see you could run BGP on (not sure how well it works) is the F5. This is with the mindset of keeping costs low.

The unique challenge here is a different ISP at each location and attempting a level of redundancy for inbound traffic to your SaaS.

Another idea would be to advertise only one prefix from one ISP and another prefix from another ISP. Setup the F5 with a VIP in each prefix at their respective locations prefix A at Site A with ISP1 and then prefix B at Site B with ISP2. Then use DNS to fail clients or load balance clients to your SaaS. If you set the TTL value on your DNS entries to 30 seconds, this could be near automatic. DynDNS has a load balancing option and a certain health check (is the server up, if not change DNS to the standby IP). Further you could write your own health checks to DynDNS's API and make the change that way should your script detect an event that would execute a command to change DNS. I would see making a DNS change as being much quicker versus calling your ISP to have BGP routes removed from being advertised because ISP's can be challenging to get things done and BGP does not converge fast at all.

| improve this answer | |
  • I don't imagine the last resort manual failover scenario to be a very common one. If one of the ISPs lose all paths to us, including the direct peerings between ISP1 and ISP2 they they are likely utterly down, in which case BGP will do its magic. If they lose direct communication to us they will simply keep the announce active, perhaps update the metric and forward traffic to the other ISP. – Roy Jul 15 '15 at 10:34
  • The SLA target is 99.999%. Curious. What would we gain from having additional com links from ISP1 to site B and ISP2 to site A? We have a carrier grade ring network on our end (N+1 redundancy), and the two ISPs have one ring each. The three rings already intersect on at least 2 locations (Site A, Site B, Local Internet Exchange, National Internet Exchange) – Roy Jul 15 '15 at 10:41
  • DNS based traffic management doesn't look too robust when done on a small scale. You simply don't know what caching is being done around the world. Some ISPs tend to ignore TTL values and cache for hours. I am considering a design which uses DNS round robin to distribute connections across the BigIP cluster. Some web browsers and corporate proxy servers will selectively use the other listed address if one fails to respond. I believe the BigIP can update the DNS records if one of the prefixes becomes unroutable. Hmm. Im just not sure how much this will add availability in plausible scenarios. – Roy Jul 15 '15 at 10:50
  • Of course, the round robin DNS would be on top of the multihoming. In a typical failover scenario, the load balancers on one site would run all the IP addresses from both prefixes. – Roy Jul 15 '15 at 10:53
  • I might add that we're currently using DnsMadeEasy for some small amount of Global Traffic Management through DNS. I believe their product is called Global Traffic Director. We have the ability to update DNS records per geographic region, and of course initiated through their APIs. – Roy Jul 15 '15 at 10:57

As long as you are fully meshed and the you can backhaul your own traffic arriving at one site but destined to the other, sure, it will work. So if your peering session drops with either ISP, your more specific will be withdrawn (you HOPE!) and the traffic will flow to the aggregate route on the other ISP and you can then backhaul the traffic to the other site.

The most frustrating problem I see is network providers that advertise unreachable routes. Either through some screwup internally or whatever, they announce a route to their peers that they can not actually reach inside their network. There isn't much of a workaround for those cases.

As for the F5 handing reply traffic back via the interface on which it was received, most load balancers can do that. I don't specifically know if the F5 will but I know Citrix Netscaler will and A10 will so it seems to be a rather standard feature.

| improve this answer | |
  • Actually, I'm hoping not to run BGP sessions on my routers at all, provided that the ISPs can update their routes based on the reachability of my respective endpoints, and announce/withdraw accordingly. I'm also hoping that the direct peering between the two ISPs will help prevent the unreachable route scenario. If all direct paths to me are down, they might still be able to reach me through their direct peering with my other ISP. That make any sense? – Roy Jul 7 '15 at 22:04
  • It would be suboptimal really. You would want to run BGP (even if from a private ASN with each provider) and simply take a default route from the provider and announce both the respective /23 and the aggregate /22 from each. That is an extremely minimal BGP configuration and does not require owning an ASN. We're talking only a few lines of BGP config and taking only one route from each provider (default). – GeorgeB Jul 7 '15 at 22:08
  • The problem doing it without BGP is that the provider would have to statically route to you. If your link drops, they might not notice depending on how the connection to you is provisioned and they might continue announcing the route to your more specific. With BGP, the route goes away if the link drops. – GeorgeB Jul 7 '15 at 22:17
  • 1
    BGP doesn't have to be "cumbersome, complex and scary." What you're describing isn't particularly complicated. And you don't need a team of WAN experts, either. If you can get F5s to work, BGP shouldn't be any harder. – Ron Trunk Jul 7 '15 at 22:24
  • You're preaching to the choir, but it's not my decision. An AS# is the one thing I do have, what I lack is equipment/software, resources and management approval. As for detecting if links are up and endpoints reachable to determine available/best paths, both we and more importantly, our primary ISP has been doing that for years without any notable issues. We can also influence their BGP metrics or force our routes to be withdrawn through automated requests or by phone, should we need to. The bottom line is, I have the option of multi homing without BGP or staying multipathed to a single ISP. – Roy Jul 7 '15 at 23:24

Your Answer

