






So with that …

Why soft switching is awesome:





First, what are the latency costs of soft switching?




What about throughput?




Soft Packet Classification:

















SR-IOV is not Passthrough (Response to “The Rise of Soft Switching: Part 1”)

[Note: I’ve been working with networking at the virtual edge for over 3 years now, and every time SR-IOV has come up, it has been in the context of passthrough (bypassing the virtual switch).  However, Sean Varley of Intel wrote to point out that the two should not be conflated.  He further points out the potential benefits that SR-IOV can provide.  Rather than try and rehash an already great note, I figured I would just include it here in its entirety.   So first, sorry for screwing that up.  Mea culpa.  Second, thanks to Sean for providing the clarification.  And finally, passthrough still sucks ;)]

I’d like to clarify a technological fact that you may have overlooked in this recent blog post that I think has done a disservice to the SR-IOV technology and partly isn’t even relevant to the point you are trying make with this post.
The fact is essentially this – SR-IOV is NOT a kernel or soft switch bypass technology.  I agree that there is a great misperception in the industry that it is only relevant under these circumstances, however, as a specification it merely allows for multiple child functions to be instantiated under a parent function in a standard manner under the PCIe interface.  This has much more to do with identifying, associating, and managing NIC partitions than anything else.  By erroneously applying this basic functionality to a single specific use case of soft switch bypass, you are pigeon holing it to a small space and perpetuating a significant myth in the industry.
I have attached a single diagram that shows the wide variety of uses for SR-IOV in emulated, local kernel, and even non virtualized environments where SR-IOV is a significant enabling technology.  The essence of NIC partitioning is very simple:  I want a single big pipe to look like a bunch of smaller pipes.  SR-IOV doesn’t do this but it allows a standard interface to be applied to a partition that enables every OS in the industry to recognize and enumerate that partition as an interface to bind whatever service it likes to according to a standard.  All other partitioning technologies that exist today (Flex10, vNIC, etc.) are proprietary technologies that actually clutter the market place with vertical, inherently incompatible (amongst vendors) solutions to NIC partitioning.

Openflow is the answer! (now .. what was the question again?)

Wow, over the last couple of weeks there has been an escalation in the confusion around Openflow. The problem is on both sides of the metaphorical isle.   Those behind the hype engine are spinning out of control with fanciful tales about bending the laws of physics (“Openflow will keep the icecream in your refrigerator cold during a power outage!”). And those opposed to OpenFlow (for whatever reason) are using the hype to build non-existant strawmen to attack (“Openflow cannot bend the laws of physics, therefore it isn’t useful”).

So for this post, I figured I’d go through some of the dubious claims and bogus strawmen and try and beat some sense into the discussion.  Each of the claims I talk to below were pulled from some article/blog/interview I ran into over the last few weeks.

Claim: “Openflow provides a more powerful interface for managing the network than exists today”

Patently false. APIs expose functionality, not create it. Openflow is a subset of what switching chipsets offer today. If you want more power, sign an NDA with Broadcom and write to their SDK directly.

What Openflow does attempt to do is provide a standard interface above the box. If you believe that SDN is a powerful concept (and I do), then you do need some interface that is sufficiently expressive and hopefully widely supported.

This is almost not worth saying, but clearly Openflow is more expressive than a CLI. CLIs are fine for manual configuration, but they totally blow for building automated systems. The shortcomings are blindingly obvious, for example traditional CLIs have no clear data schema or state semantics and they change all the time because the designers are trying to solve an HCI not an API versioning problem. However, I don’t think there is any real disagreement on this point.

Claim: “Openflow will make hardware simpler”

I highly doubt this. Even with Openflow, a practical network deployment needs a bunch of stuff. It needs to do lookups over common headers (minimally L2/L3/L4), it may need hash-based multi-pathing, it may need port groups, it may need tunneling, it may need fine-grained link timers. If you grab a chip from Broadcom, I’m not exactly sure what you’d throw away if using Openflow.

What Openflow may discourage is stupid shit like creating new tags as an excuse for hardware stickyness (“you want new feature X? It’s implemented using the Suxen tag(tm) and only our new chip supports it.”). This is because an Openflow-like interface can  effectively flatten layers. For example, I don’t have to use the MPLs tag for MPLS per se. I can use it for network virtualization (for instance) or for identifying aggregates that correspond to the same security group. However, that doesn’t mean hardware is simpler. Just that the design isn’t redundant to fulfill a business need. (Post facto note:  There are some great points regarding this issue in the comments.  We’ll work to spin this out into a separate post.)

Or more to the point. There are an awful lot of fields and bits in a header. And an awful lot of lookup capacity in a switch. If you shuffle around what fields mean what, you can almost certainly do what you want without having to change how the hardware parses the packet.

Claim: “Openflow will commoditize the hardware layer”

This is total Kaiser Soze-esque nonsense (yup, that’s the picture at the top of the post). To begin with, networking hardware is already on its way to horizontal integration. The reason that Arista is successful has nothing to do with Openflow (they strictly don’t support it and never have), it has nothing to do with SDN, it’s because Ken Duda, and Andy Bechtolsheim are total bad-asses and have built a great product. Oh, and because merchant silicon has come of age, meaning you don’t need need an ASIC team to be successful in the market.

The difference between OpenFlow and what Arista supports is that with Openflow you choose to build to an industry standard interface, and not a proprietary one. Whether or not you care is, of course, totally up to you.

So Openflow does provide another layer of horizontal integration and once the ecosystem develops that it is, in my opinion, a very good thing. But the ecosystem is still embryonic, so it will take some time before the benefits can be realized.

I think the power of merchant silicon and the rise of the “commodity” fabric are a far greater threat to the crusty scale-up network model. Oddly, Openflow has become a relatively significant distraction. That said, as this area matures, creating a horizontal layer “above the box” will grow in significance.

Claim: “Openflow does not map efficiently to existing hardware”

True! Older versions of Openflow did not map well to existing silicon chips. This is a *major* focus of the existing design effort; to make Openflow more flexible. As with any design effort, the trade-off is between having a future proof roadmap, and having a practical design that can have tangible benefits now. So we’re trying to thread the needle practically but with some foresight. It will take a little patience to see how this evolves.

Claim: “Openflow reduces complexity”

This is a meaningless statement. Openflow is like USB, it doesn’t “do” anything new. A case can be made for SDN to reduce complexity, but it does this by constraining the distribution model and providing a development platform that doesn’t suck. No magic there.

Claim: “Openflow will obviate the need for traditional distributed routing protocols”

I hear this a lot, but I just don’t buy it. There are certainly those within the Openflow community who disagree with me, so please take this for what it’s worth (very little …)

In my opinion, traditional distributed protocols are very good at what they do, scaling, routing packets in complex graphs, converging, etc. If I were to build a fabric, I’d sure as hell do it with an distributed routing protocol (again, not everyone agrees on this point). What traditional protocols suck at is distributing all the other state in the network. What state is that? If you look at the state in a switching chip, a lot of it isn’t traditionally populated through distributed protocols. For example, the ACL tables, the tunnels, many types of tags (e.g. VLAN), etc.

Going forward, it may be the case the distributed routing protocols get pulled into the controller. Why? Because controllers have much higher cpu density and therefore can run the computation for multiple switches. A multicore server will kick the crap out of the standard switch management CPU. Like I described in a previous post, this is no different than the evolution from distance vector to link state, it’s just taking it a step further. However, the protocol certainly is still distributed.

Claim: “Openflow cannot scale because of X”

I’ve addressed this at length in a previous post.  Still, I’m going to have to call Doug Gourlay out.  And I quote …

“I honestly don’t think it’s going to work,” Arista’s Gourlay said. “The flow setup rates on Stanford’s largest production OpenFlow network are 500 flows per second. We’ve had to deal with networks with a million flows being set up in seconds. I’m not sure it’s going to scale up to that.”

Other than being a basic logical fallacy (Stanford’s largest production Openflow network has absolutely nothing to do with the scaling properties of the Openflow or SDN architecture),  there appears to be an implicit assumption that flow-setups are in some way a limiting resource.  This clearly isn’t the case if they don’t leave the datapath which is a valid (and popular) deployment model for Openflow.  Doug’s a great guy, at a great company, but this is a careless statement, and it doesn’t help further the dialog.

Claim: “Openflow helps virtualize the network”

Again, this is almost a meaningless statement. A compiler helps virtualize the network if it is being used to write code to that effect. The fact is, the pioneers of network virtualization, such as VMWare, Amazon, and Cisco, don’t use Openflow.

So yes, you can use Openflow for virtualization (that’s a help, I guess … ). And open standards are a good thing. But no, you certainly don’t need to.

OK, that’s enough for now.

To be very clear, I’m a huge fan of Openflow. And I’m a huge fan of SDN.  Yet, neither is a panacea, and neither is a system or even a solution. One is an effort to provide a standardized interface for building awesome systems (Openflow). The other is a philosophical model for how to build those systems (SDN). It will be up to the systems themselves to validate the approach.



How many times have you been involved in the following conversation?









Alright, on to the focus of this post, background.




















Is that all?


Until then …

The Scaling Implications of SDN

In my experience, it’s rare to have a discussion about SDN without someone getting their panties in a bunch over scaling. Nevermind that SDN networks have already been implemented that scale to tens of thousands of ports. And nevermind that distributed systems are built today that manage hundreds of thousands of entities and petabytes of data. It remains a perennial point of contention.

So here is a hand-wavy – probably not totally convincing – but hopefully offers some intuitive understanding – attempt at an explanation of the scaling limits of SDN.

But first! Some history: Unfortunately, I think I am partially to blame for the sorry state of affairs. In some of the earliest writeups we would describe an SDN controller as “logically centralized”. The intended meaning of this was something along the lines of “it has a centralized programmatic model, but was really distributed”.

Now, what does “logically centralized” actually mean? Nothing. It’s a nonsense term and the result of sloppy thinking. Either you’re centralized, or you’re distributed (thanks to Teemu Koponen for pointing that out). So by logically centralized, I guess we really meant “distributed”, and that is what we should have said. Whoops!

Clearly, a centralized controller cannot scale. Perhaps it can handle a really large network. Hell, I’ve heard claims of single node SDN networks that scale to thousands of switches and tens of thousands of ports. Whether or not that’s true, at some point either CPU or memory will run out.

However, controllers can (and should!) be distributed. And in that case, I would argue that there are no inherent bottlenecks. Trivially, you can distribute controllers such that the total amount of available CPU and memory for the control path is equivalent to the sum total of management cpu’s on the switches themselves. Controllers can distribute state amongst themselves like network nodes do today.

Still not convinced? Let’s deconstruct this …

Latency: How might SDN affect latency? That depends on how it is used. In some implementations, datapath traffic never leaves the hardware. In which case, datapath latency should be identical to traditional networking. Other implementations will forward the first packet of a flow up to the controller and then cache the forward decisions on the datapath. In this case, the flow setup will have to pay an RTT to the controller. The following paragraph will discuss how much that might be.

Whether or not data traffic is sent to the controller, control traffic often is (for example BGP from a neighboring network, or the signalling of a port status change). In this case the additional overhead is the RTT to the controller + the cost of the computation to determine what to do next. How much that is will vary wildlyby deployment. It’s possible to keep this sub millisecond (200-300us is not unusual). I’ve seen as little as 70us claimed, but I doubt that could be maintained under any real load.

So you may not want to pay this latency per flow (though in many cases it probably doesn’t matter). However, there isn’t any appreciable overhead for responding to network events. And as I’ll describe later, it probably will reduce the total time needed to disseminate control traffic.

Datapath Scaling: The datapath is where the original OpenFlow design was broken. This is because OpenFlow used the abstraction of a switch datapath as a single TCAM. Clearly, this can be problematic for some forwarding rulesets. Assume, for example, that you are building an SDN application which would integrate with BGP, and map prefixes to tags with QoS policies. Implemented this within a single flow table would require (RIB * QoS rules) entries. Ouch.

However, later versions of OpenFlow support multiple tables which take care of this Cartesian explosion nightmare. Rather than having to multiply all the rules together, they can be placed in separate tables limiting the maximum size requirements of a single table (by a lot!).

Still, many modern implementations of SDN dispense with the abstraction of a flow altogether and program the switch hardware tables directly. I think we’ll see a lot more of this going forward.

Convergence on failure:  Alright, what happens on link (or node) failure? Traditionally, when a link fails, the information of the failed link is propagated through flooding. Flooding, of course, scales linearly with the distance of the longest loop free path. If you compare this with SDN, instead of being flooded the information goes to one of the controllers, which sends the updated routing tables to the affected switches.

A few things to note:

  • With SDN, the controller should only be updating the switches whose tables have actually changed.
  • With SDN, the total cost is link propagation to the controller + cost of computation + time to update to affected switches.
  • In a distributed SDN case, an implementation may distributed the link update among the controller clusters which then do the computation for their piece of the network.

Unfortunately for SDN, when in-band control is in play, things are a lot uglier. Inband control basically means that the datapath is used both for SDN control traffic and data. So a failure on the network may affect connectivity to the controller which must be patched up somehow before the controller can fix the problem for the rest of the network.

Patching up the control channel, in this case, is generally done with an IGP, so we’re strictly worse off than if we just used an IGP to begin with. Suck.

So bottom line, if convergence time is important, out of band control should be used. It’s simple to build a cheap, reliable out of band control network using traditional gear (the amount of traffic is minimal).

Computation and Memory: It’s easy to see that the latest multi-core server available from your friendly  server vendor can beat the pants off of whatever crap embedded cpu you’ll find in most networking gear (800mhz-1Ghz is common).

I think Nick McKeown said it best.  If we look historically, early routing protocols used distance vector routing algorithms (like RIP) which have very low computational requirements, but suck in a bunch of other ways (convergence times, split infinity etc.). As cpu’s became more powerful, it was possible for each node to compute single source shortest path to all destinations on each failure, which is the standard approach used today.

Calculating more routes (e.g. all pairs shortest path, or multiple source to all destinations) on stronger CPUs really  follows as a natural evolution. The clear “next step”. And a beefy server with lots of memory can compute *a lot* of routes, and quickly. Further, we already know how to distribute the algorithms.

Some parting notes: The goal of SDN is not to scale a simple network fabric. This is something that distributed routing algorithms do just fine. The goal of SDN is to provide a design paradigm which allows the creation of more sophisticated control paradigms: TE, security, virtualization, service interposition, etc. Perhaps that means just manipulating state at the edge of the network while traditional protocols are used in the core. Or perhaps that means using SDN throughout. In eithercase, SDN architecturally should be able to handle the scale. Remember, it’s the same amount of state that’s being passed around.

I think the right question to ask is not “does it scale” but “is it worth the hassle of building networks this way”? Building an SDN network requires some thought. In addition to the physical network nodes, a control network (probably), and controller servers need to be thrown into the mix.

So, is it worth it?

That, is up to you to decide.   Ultimately the answer will rely on what gets built using SDN and who wants to run it.  I think it’s still too early in the development cycle to derive a meaningful prediction.

Presentation on SDN and High-Level Network Abstractions

If you haven’t heard the talk, or seen the slides, these are a must.  Professor Shenker, one of the creators of SDN, put together this talk about why programming to high-level abstractions is better than protocol design, and why SDN helps achieve it.

An Extremely Brief Conceptual Introduction to Open vSwitch


Open vSwitch is one of my favorite open source projects.   For those of you who aren’t familiar with it, it’s a switch stack which can by run both as a soft switch (vswitch) within a virtualized environment)and as the control stack for hardware switches. Good stuff.

However, the real kung-fu is that Open vSwitch is built for programmatic state distribution. It does this through two interfaces. One being OpenFlow (with a ton of extensions) for managing the forwarding behaviour of the fast path, and the second being a JSON-RPC based config protocol used for less time critical configuration (tunnels, QoS, NetFlow, etc.). You can view the schema for the config protocol here.


So, what might you do with something like Open vSwitch? Well, the idea is to enable the creation of automated network infrastructure for virtualized environments.

For example, you could use a centralized control system to manage network policies policies that migrate with VMs. This has already been done within the XenServer environment with the Citrix Distributed vSwitch Controller.

Open vSwitch also supports programmatic control of address remapping (L2 and L3 a la OpenFlow), and programmatic control of tunnels as well as multiple tunnel types (e.g. GRE, IPsec, and CAPWAP). These come in handy for various network virtualization functions such as supporting mobility or an L2 service model across subnets.

Open vSwitch is already used in a bunch of production environments. It is most commonly used as a vswitch in large cloud deployments (many thousands of servers) for automated VLAN, policy, and tunnel management. However, I know of a number of deployments which use it as a simple OpenFlow switch, or a more sophisticated programmatic switch to control hardware environments.


Open vSwitch is fast. Damn fast. Some performance tests have shown it to be faster than the native Linux bridge. Open vSwitch uses flow-caching (when running in software), so even under complex configurations the common case should be blazingly fast. Open vSwitch also has highly optimized tunneling implementations.


Open vSwitch is primarily developed and deployed in Linux (however there are ports to other OSes, particularly those used in embedded environments). It is commonly used with both Xen and KVM (there are production environments using both). Further, it has been integrated into a number of cloud management systems including Xen Cloud Platform, OpenQRM, and OpenNebula (a long with a bunch of proprietary CMSes).  It’s currently being integrated into Open Stack.  You can track the progress here.

Why Do I Care?

Mostly because with Open vSwitch, as with other distributed switch solutions, it’s possible to build really sophisticated networks with not-so-sophisticated hardware.  For example, an L3 network from your neighborhood OEM or other other low-cost hardware vendor (check out, for example, Pronto), plus Open vSwitch and a bit of programming can equate to a cheap, bad-ass network for virtual deployments.  But that, my friends, is a topic for another post.