RoCE as a performance accelerator
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Shwartsman, Slava
RDMA (Remote Direct Memory Access) and its RoCE (RDMA over Converged Ethernet) implementation, currently available in FreeBSD, is growing in popularity as a way to transfer large amounts of data with high bandwidth, low latency and minimal CPU involvement. This session will describe the RDMA stack update done in FreeBSD, focusing on the addition of RoCEv2 (Routable RoCE) and complementary protocols. We will also present the performance benefits of RDMA, demonstrating the enhancement of several storage protocols relaying on RoCE transport, starting with what is already part of FreeBSD – iSER, an iSCSI extension for RDMA, and moving on to more opportunities available on other platforms which can be introduced to FreeBSD.
 So, first of all, thank you for joining me today. We are going to talk a little bit about Rocky as a performance accelerator. So a few words about me. My name is Slava Schwartzman. I live in Israel in Kvarsaba, which actually translates to Grandpas Village, but it's a medium-sized city in Israel. I'm a software engineer at Malamux Technologies. My main interests are the networking stack and the InfiniBand. I'll be mostly focusing on Rocky and RDMA. I'm a new committer at Source, and my mentors are Konstantin Belosov, Kib, and Hans-Peter Sulaski. So, today we are going to talk a little bit about Rocky, or what is Rocky, and where it comes from, and a little bit about history and its current implementation in previous day. We'll talk a little bit about complementary protocols, and why do we need it for Rocky. And then we'll have a look at some graphs and some performance acceleration for storage protocols using RDMA. So before starting to talk about RDMA or Rocky, we need to understand a little bit about its history. Rocky actually comes from the InfiniBand world, and RDMA is coming from the InfiniBand world. InfiniBand is basically a networking stack, just like Ethernet, but it's on the pre-ations for the layer one, two, and three, three, and four. And its main characteristics are that it has low latency, high bandwidth, and very low CPU utilization. It can achieve those benefits or characteristics by using RDMA. RDMA, which stands for Remote Direct Memory Access. I'm pretty sure that most of you are familiar with DMMA, basically the ability for hardware to directly access memory regions without any involvement of the CPU. RDMA basically allows remote direct memory access, basically allows the HCA to access memory on remote hosts. The current implementation of RDMA allows hardware offload, basically packet processing in the hardware instead of doing them in the kernel itself. And of course, RDMA was designed with security concerns, like protection domains, for example. They have user level access and exchange of keys. The benefits of RDMA are minimal CPU involvement, of course, since the packet processing is actually being offloaded into the hardware, hence the CPU doesn't have to work too hard or there is minimal involvement from the CPU. Kernel bypass, for example. We allow user applications to fully bypass our kernel and use, again, those memory regions. Scutter gather entry support. Basically RDMA was designed with Scutter gather entry support as definition. Basically allows you to not allocate continuous memory and just use pieces and Scutter gather them. Zero copy basically stands for the fact that unlike traditional sockets where the buffer that being allocated by the application needs to be copied along the networking stack in RDMA, this buffer is used by the RDMA in itself and copied and transferred to the remote host. So Rocky? So with that concerning mind that to achieve the benefits of RDMA, you need to change the entire architecture of your network is not something trivial to ask. And hence Rocky or RDMA overconverged Ethernet was introduced. Basically RDMA overconverged Ethernet allows you to use RDMA benefits over an existing Ethernet fabrics. Rocky is a standard protocol, standardized by the IBTA, which stands for Infiniband Trade Association and both Infiniband and Rocky APIs share the same common MPI. It's interestingly enough to mention that both FreeBSD and Linux, for example, will share the same API because those are the same APIs. And that's it. I wanted to say something about it, but it's enough. So here is the header layout of the Rocky protocol. Basically you can see that the Infiniband things are encapsulated inside in an Ethernet header. The Ether type will indicate that the next packet is IBGRH, which stands for Infiniband Global Routing Header, which actually contains the layer 3 addresses for Infiniband. And the next header is IBBTH, Infiniband-based transport header, which is the layer 4 header. The main problem with Rocky, or as we will see in a second, with Rocky V1, is the fact that it's limited to its domain, meaning it's not routable. And hence, Rocky V2 was introduced. Rocky V2 is basically a straightforward extension of the Rocky V1 protocol, which allows it to be routed between different subnets. The change that was done is basically to the headers and telf. We can see this in this diagram. Instead of having IBGRH at the layer 3, we now have an IP header. So the Ether type in Rocky V2 will indicate that the next header is actually an IP header. And the IP header will indicate that the next header will be UDP header, which will have a specific destination port indicating that this packet is Rocky V2. And we still have the IBBTH and the Infiniband payload inside. So the current status in FreeBSD is that we have Rocky V2 support both for FreeBSD11 and FreeBSD12. We currently need special hardware support, of course, when hardware support is working. And in order to enjoy the RDMA capabilities, you will have to rebuild world and kernel with off-ed yes. So we're currently actually working on make the build kernel with off-ed yes as a default, where if you would like, you can change it, of course, to build without off-ed. So let's have a quick look at some APIs or as we call them, verbs. So you can find them on user-included Infiniband verbs, of course, after building the whole kernel with off-ed. And let's have a look at one of those examples, which will be IBVRGMR. IBVRGMR stands for Infiniband verbs, register, memory region. This functionality actually enables you to pin a memory for later use by the hardware. What this function accepts as input are the protection domain, which is basically an entity that connects this memory region to a specific QP or QS. It will have the address and the length of this memory region, and the access user and the access are the user-level flags, which will, again, for security reasons, of course. Each IBVRGMR will be followed by IBVRGMR, of course, to free this pin memory. Taking a quick look and in the trace of calling, for example, IBVRGMR, you will actually work with verbs, which is the API for the user application, which will have a callback called REGMR, which every RDMA user space provider will have to implement, which will be MLX5 REGMR, for example, which will call IBVRGMR, which is, again, a callback, back to the verbs, which will actually issue a right command for the hardware, for the kernel, of course. So going back and forth from the verbs to the RDMA provider, which is in our case MLX5, and back to verbs makes it basically a generic API from the user application perspective and back a generic API to the kernel code itself. After handling the CIS call of write, IBU verbs will have its character device with an implementation of the right callback, which will call REG user MR, which is, again, a callback that has to be implemented by the RDMA driver, or the driver of the specific vendor, which will call the MMAP single, which is the actual pinning of that memory. So for some better use case or simpler use case, we have pushed to FreeBSD, both 11 and 12, of course, to contrib off at lib RDMA and lib IBVRGMR examples. We have multiple user space applications, which can be used to investigate or to play with RDMA. Also we have CIS contrib RDMA key arping, which is a kernel application example of how can we use, basically, RDMA in the kernel column. And recently, actually thanks for Philip, we just pushed the Perftest, which is a micro benchmarking tool for RDMA. Questions so far? Awesome. So when taking such a protocol as RDMA from the infiniband world and putting it into Ethernet, we have a lot of benefits. Like, like just said, we have low latency, we have low CPU utilization, we have great bandwidth, but on the other hand, we encounter into some difficulties. One of those difficulties is the fact that Ethernet is a lossy fabric. In the next part, we will talk what does it mean that Ethernet is a lossy fabric, and how can we manage that by introducing a few complementary protocols? And why do we care about it for Rocky, for example? So the fact that Ethernet is a lossy fabric means that Ethernet allows packets to be dropped when buffers overflow. And actually Ethernet is using this to indicate that a congestion has occurred in the fabric. From a rocky perspective, when a packet is being dropped, since everything is being offloaded to the hardware, it means that there is a firmer flow, which is very expensive from a performance perspective. And another thing that I will demonstrate in the upcoming slide is what exactly is being happening when this transmission happens. So we can manage those packet loss by or those congestions in two manners. First of all, we can improve the hardware to work faster when the transmission flows are needed. Basically, we were working on this on Connectic 6, and I will explain what is the difference between what's going on now and what it's going to do. Or the other approach can be to do a congestion control on the link layer. So from a rocky perspective, when a packet is being dropped, PSN stands for packet sequence number. When you see, for example, if PSN number two was dropped, we will have to retransmit not only this second packet, we will have to retransmit everything from this packet and on. This is again because of the way that Trocki works. Important thing to notice is the fact that in the infinite world, this is the difference between infinite and Internet, there is no such thing as packet loss, basically. The idea of infinite is a little bit different. They work with something, they exchange data to transmit as much as can, some different algorithm that we can talk a little bit about later. But basically the problem that in Internet, we will have to retransmit everything from this packet and on. One of the mechanisms to avoid that or to manage that will be flow control or global pause. Basically what global pause means is that whenever the buffer will reach a certain threshold, a special packet called pause will descend from the receiver to the center side, asking him to completely stop any traffic. Of course, global pause is a link level flow control and I'll explain it in a second how it works. So, basically, let's imagine that we have single receiver and multiple transmitters to the same receiver guys. So those transmitters send a lot of traffic to this receiver up to the fact that his buffers are starting to get to some point in his threshold. When this happens, this guy will send a pause frame but instead of sending it for each of the senders, he will send it to the switch which is directly connected to him and only when his buffers will get to this certain threshold, he will send the pause frames back to the senders asking them to stop down. The problem, basically this is a high level view of that. The problem with that is that when we have multiple senders and single receiver, we have multiple types of flows and each of these flows has its own thing. So for example, if we don't want a specific flow to be paused at all, for example, if we have video traffic and, I don't know, HTTP traffic, for example. So we need a way to distinguish between those flows and deal with that. One of the approaches to do this will be priority flow control or PFC. So basically, we have something called PCP which is a priority code point. It's a mechanism to put a priority inside the shim, inside the dot one q shim which is also called a villain header and to distinguish between different types of flows. In this mechanism with PFC, instead of stopping all the traffic at once from the sender side to the receiver, we can stop each and every priority. Basically how it works, that we have multiple queues right now, multiple buffers, and we can stop the traffic on each buffer by itself instead of stopping altogether. Again, this protocol PFC is link level, just like the previous one, just like pause. So two problems with both PFC and global pause is the fact that first, it's link level. And the second one, that the granularity is not that good. Still, we have only eight priorities. And the third, but maybe a little bit different thing, is the fact that in PFC and global pause, what will happen is that we will stop sending any traffic at all, meaning instead of slowing down. What basically happens behind the scenes is that when a sender receives a pause packet, he will start a counter which will start sending traffic again only when this counter expires and he didn't receive another pause frame. So the next mechanism called ECN, or explicit congestion notification, is basically an extension of the IP protocol which allows an end-to-end congestion notification almost without dropping any packets. This mechanism works a little bit different. It uses DCQCN, which is very much like red, and red stands for random early detection. Basically the idea is that instead of sending a pause frame or a pause asking for the sender to slow down his traffic on a certain threshold, using red, what happens is that as much as if the longer the queue of the packets in the buffers, the higher probability of marking a packet as a congestion experience gets higher. Basically unlike PFC and global pause, what happens with ECN is that the intermediate devices like switches and routers can mark a specific flow that a congestion has occurred and when the receiver side will get this packet, a regular packet from the flow with congestion experience, he will send a special packet called CMP or congestion notification packet to the sender side asking him to slow down. So from ECN what we get is much better granularity and as you can see, instead of stopping the traffic completely, he will ask him to just slow down. So as a little summary for what we've been talking so far is first of all that ROKV2 is erudible, unlike ROKV1 which was introduced in the previous generations of previous D. ROKV is scalable. ROKV is actually being implemented right now in not implemented, sorry, is used in one of the biggest cloud providers like Azure. ROKV works on Ethernet available distances, meaning that all distances that are available for Ethernet will be available for ROKV2. ROKV works with all advanced Ethernet signaling rates, meaning that ROKV is working on 100 gigabit Ethernet. Common traffic management and monitoring tools work with ROKV, for example TCP-DAM has support for sniffing ROKV packets, even in free BSD. It is something that was pushed after updating lead pickup. ROKV requires lossless network. This is true, but we have great mechanisms for supporting that. ROKV is supported by multiple vendors. It is, it's not only Melanox. And ROKV interoperability between different vendors is regularly tested by IBTA, by the InfiniBend Association, actually this is happening twice a year and being published for general availability. Any questions? So ROKV has a lot of application. We can, we can accelerate MPI applications, for example, we can accelerate HPC applications. And one of the interesting points that we have is that ROKV can accelerate storage protocols. All those storage protocols that everyone are using like iSCSI, like NFS, like NVME, like Samba, which I'm not going to talk about. But basically all of them can be accelerated using ROKV. So let's start from the first example, which is actually already implemented for free BSD, which is ISER. ISER is basically an iSCSI extension for RDMA. We are using iSCSI and instead of using the TCP stack as the transport layer, we're actually using RDMA. The benefits, of course, just like any other RDMA or ROKV application, will be high bandwidth, high IOPS, low latency, we will reduce CPU usage, and of course all the iSCSI management tools are available. So here are some ISER performance examples on both on Connectix 4 and Connectix 4LX for 25 and 100 gig. So from this graph, you can see that using one small thing that I have to mention, those performance results are wore down on Linux and not on free BSD, unfortunately. So as you can see that using 512 bytes as a block size for IO, you can see that we can get three times better performance from IO perspective. We can get almost three times better latency using ISER and almost four times better bandwidth using latency. Again, I'm comparing the same devices, the same PCs, the same computers, the same everything, just using RDMA and using regular TCP. Using CPU performance, we can see that we can get almost four and a half times better from CPU utilization. NFS over RDMA, which is unfortunately not available yet for free BSD, basically the idea that when there was, and I'm pretty sure that most of you, if not all, are familiar with NFS. When NFS was introduced, it was introduced on 10 megabit Ethernet and the migration to 100 megabit actually showed much performance improvements. On the other hand, the next generation didn't show that much improvement on the other and from the CPU utilization perspectives, NFS was much worse when migrating to faster networks. So here I'll show some performance results done by Oracle actually. So you can see again that we gain almost four times better on the read IOPS and much better on the right. You can see that almost three times better. This on 16 was on one thread, sorry, and on six threads, as you can see here, using big block sizes and for 4K and 8K, we see the same picture basically from NFS over RDMA point of view. The next protocol that I would like to talk about is NVME over fabrics, which again is unfortunately not available for free BSD yet. Only NVME is nonvolatile memory express, allows you to have an SSD device basically attached to your PCI. NVME over fabrics allows you to write using RDMA to NVME devices from remote. So here are some performance measurements for Connectix 5, for example, and from this graph you can see that the performance for doing an NVME device, meaning NVME over fabric, is almost alike doing a local write. And again, from throughput and IOPS perspective, you can see that using NVME for NVME over fabrics, we can get to up to 4 million IOPS on small block sizes, and we get to almost saturating the whole link in 4K. Questions? Right, exactly, right, and even with the new generation like the SOC, we can offload the target completely. You can have the whole target inside the NIC itself. Bluefield. Can you go back to slide the... I couldn't read that. I couldn't read my eyes with that. I couldn't read the color. Which one? What each color means? So the blue one is RDMA write IOPS. The orange one is IP of write B write IOPS. It's actually not comparing real TCP to RDMA. It's basically comparing IP of write B to RDMA. And then RDMA read IOPS and IP of write B read IOPS. It's read write and RDMA versus IP of write B. Yes, of course. So for RDMA, I understand that this is basically the star will write something. Bear with me here, sorry. For NVME, if a write is coming in, the NIC is going to put the data into memory and the NIC is going to issue the piece management to the device, say, go write using this buffer. So I'm not an NVME expert. But as far as I understand how it works is basically that by using the NVME protocol, first of all, we have to understand that NVME is not scousing, right? Yeah. NVME is NVME. So as far as I understand, when we get the NVME message on the wire to the card itself, it will just put the data or the buffer directly on the NVME device over the PCI instead of going up to the CPU and back. Yeah, that's why I'm saying the NIC is putting the, it's doing the buffer for what comes in over the network and then the NIC is sending the right man to... Directly on the PCI to the NVME device. Other questions? Suppose I still do the double-fogging device and write something to RAM and NIC takes from RAM and send to the... So this is the point that I'm not sure about and I want to commit to. But as far as I understand, there will no be... No security interaction, but this double-DMA. So I'm not sure if there will be a double-DMA. This is what I'm not sure about, but I think that it will be a direct DMA to the NVME device itself. But this one I'll have to check because I'm not sure about it. Of course. Do you mean Rocky Multicast? You mean Rocky Multicast. Are DMA Multicast, basically? Yes. Yes. And actually we have the MCC application located in the same place. I think it's under RDMA applications, under LibRDMA. And MCC is basically a user space application which uses multicast for RDMA traffic. I do have one comment, I mean, it's not a message here, several complementary protocols to make the internet fabric to be lost as a step of policy. So this is something that we are trying to overcome harder so there won't be a need for those protocols. And basically we are trying to advance the way we are doing the transmission and we have to keep it to be less involved. So we are just setting the exact piece that was missing and cut it off just like the DC piece that can reduce the need for any complementary protocols. So, you know, future generations and connecting six LX and beyond and onwards, you want to need them for a piece of what's called I work for. No. It's a different protocol. It's also based on RDMA. And you are an expert here that's the pilot. So basically what will happen just like I demonstrated in the next generations, instead of resending everything from the lost packet and force, we can just resend the lost packet itself instead of resending everything. This is what many was trying to describe about the next generations. Right. But, but, but, right, of course. So basically about the complementary protocols, the best practice that we recommend will be to use PFC or global pause, which is, of course, PFC because it has better granularity and ECN because ECN still allows a little bit of traffic to be dropped. And when using global pause, there is no chance or PFC, of course, there is no chance for traffic loss. ECN is implemented in hardware, for example. Yes, of course. So I wasn't sure if I have enough time to talk about ECN because ECN is, but I do have. So talk about it a little bit. So what happens with ECN is, first of all, if you have two servers connected back to back, you will not have ECN at all. So what happens with ECN, the intermediate devices, the switches and routers must support ECN. What basically happens is when the threshold in the intermediate devices buffers gets to a certain point, this packet will be marked, like both bits will be turned on. And when the receiver side will get a regular packet with those two bits on, but in hardware, it will generate those special packets called C&P, which is a congestion notification packet, sending it directly to the sender, asking him to slow down the traffic. Okay. Awesome. Any further questions? So I think that it. Thank you, everybody, for joining me today.
 So, first of all, thank you for joining me today.  We are going to talk a little bit about Rocky as a performance accelerator.  So a few words about me.  My name is Slava Schwartzman.  I live in Israel in Kvarsaba, which actually translates to Grandpas Village, but it's a  medium-sized city in Israel.  I'm a software engineer at Malamux Technologies.  My main interests are the networking stack and the InfiniBand.  I'll be mostly focusing on Rocky and RDMA.  I'm a new committer at Source, and my mentors are Konstantin Belosov, Kib, and Hans-Peter  Sulaski.  So, today we are going to talk a little bit about Rocky, or what is Rocky, and where it  comes from, and a little bit about history and its current implementation in previous  day.  We'll talk a little bit about complementary protocols, and why do we need it for Rocky.  And then we'll have a look at some graphs and some performance acceleration for storage  protocols using RDMA.  So before starting to talk about RDMA or Rocky, we need to understand a little bit about its  history.  Rocky actually comes from the InfiniBand world, and RDMA is coming from the InfiniBand world.  InfiniBand is basically a networking stack, just like Ethernet, but it's on the pre-ations  for the layer one, two, and three, three, and four.  And its main characteristics are that it has low latency, high bandwidth, and very low  CPU utilization.  It can achieve those benefits or characteristics by using RDMA.  RDMA, which stands for Remote Direct Memory Access.  I'm pretty sure that most of you are familiar with DMMA, basically the ability for hardware  to directly access memory regions without any involvement of the CPU.  RDMA basically allows remote direct memory access, basically allows the HCA to access  memory on remote hosts.  The current implementation of RDMA allows hardware offload, basically packet processing  in the hardware instead of doing them in the kernel itself.  And of course, RDMA was designed with security concerns, like protection domains, for example.  They have user level access and exchange of keys.  The benefits of RDMA are minimal CPU involvement, of course, since the packet processing is  actually being offloaded into the hardware, hence the CPU doesn't have to work too hard  or there is minimal involvement from the CPU.  Kernel bypass, for example.  We allow user applications to fully bypass our kernel and use, again, those memory regions.  Scutter gather entry support.  Basically RDMA was designed with Scutter gather entry support as definition.  Basically allows you to not allocate continuous memory and just use pieces and Scutter gather  them.  Zero copy basically stands for the fact that unlike traditional sockets where the buffer  that being allocated by the application needs to be copied along the networking stack in  RDMA, this buffer is used by the RDMA in itself and copied and transferred to the remote host.  So Rocky?  So with that concerning mind that to achieve the benefits of RDMA, you need to change the  entire architecture of your network is not something trivial to ask.  And hence Rocky or RDMA overconverged Ethernet was introduced.  Basically RDMA overconverged Ethernet allows you to use RDMA benefits over an existing  Ethernet fabrics.  Rocky is a standard protocol, standardized by the IBTA, which stands for Infiniband Trade  Association and both Infiniband and Rocky APIs share the same common MPI.  It's interestingly enough to mention that both FreeBSD and Linux, for example, will share  the same API because those are the same APIs.  And that's it.  I wanted to say something about it, but it's enough.  So here is the header layout of the Rocky protocol.  Basically you can see that the Infiniband things are encapsulated inside in an Ethernet header.  The Ether type will indicate that the next packet is IBGRH, which stands for Infiniband  Global Routing Header, which actually contains the layer 3 addresses for Infiniband.  And the next header is IBBTH, Infiniband-based transport header, which is the layer 4 header.  The main problem with Rocky, or as we will see in a second, with Rocky V1, is the fact  that it's limited to its domain, meaning it's not routable.  And hence, Rocky V2 was introduced.  Rocky V2 is basically a straightforward extension of the Rocky V1 protocol, which allows it  to be routed between different subnets.  The change that was done is basically to the headers and telf.  We can see this in this diagram.  Instead of having IBGRH at the layer 3, we now have an IP header.  So the Ether type in Rocky V2 will indicate that the next header is actually an IP header.  And the IP header will indicate that the next header will be UDP header, which will have  a specific destination port indicating that this packet is Rocky V2.  And we still have the IBBTH and the Infiniband payload inside.  So the current status in FreeBSD is that we have Rocky V2 support both for FreeBSD11 and  FreeBSD12.  We currently need special hardware support, of course, when hardware support is working.  And in order to enjoy the RDMA capabilities, you will have to rebuild world and kernel  with off-ed yes.  So we're currently actually working on make the build kernel with off-ed yes as a default,  where if you would like, you can change it, of course, to build without off-ed.  So let's have a quick look at some APIs or as we call them, verbs.  So you can find them on user-included Infiniband verbs, of course, after building the whole  kernel with off-ed.  And let's have a look at one of those examples, which will be IBVRGMR.  IBVRGMR stands for Infiniband verbs, register, memory region.  This functionality actually enables you to pin a memory for later use by the hardware.  What this function accepts as input are the protection domain, which is basically an entity  that connects this memory region to a specific QP or QS.  It will have the address and the length of this memory region, and the access user and  the access are the user-level flags, which will, again, for security reasons, of course.  Each IBVRGMR will be followed by IBVRGMR, of course, to free this pin memory.  Taking a quick look and in the trace of calling, for example, IBVRGMR, you will actually work  with verbs, which is the API for the user application, which will have a callback called  REGMR, which every RDMA user space provider will have to implement, which will be MLX5  REGMR, for example, which will call IBVRGMR, which is, again, a callback, back to the verbs,  which will actually issue a right command for the hardware, for the kernel, of course.  So going back and forth from the verbs to the RDMA provider, which is in our case MLX5,  and back to verbs makes it basically a generic API from the user application perspective  and back a generic API to the kernel code itself.  After handling the CIS call of write, IBU verbs will have its character device with an implementation  of the right callback, which will call REG user MR, which is, again, a callback that has  to be implemented by the RDMA driver, or the driver of the specific vendor, which will call  the MMAP single, which is the actual pinning of that memory.  So for some better use case or simpler use case, we have pushed to FreeBSD, both 11 and  12, of course, to contrib off at lib RDMA and lib IBVRGMR examples.  We have multiple user space applications, which can be used to investigate or to play  with RDMA.  Also we have CIS contrib RDMA key arping, which is a kernel application example of how  can we use, basically, RDMA in the kernel column.  And recently, actually thanks for Philip, we just pushed the Perftest, which is a micro  benchmarking tool for RDMA.  Questions so far?  Awesome.  So when taking such a protocol as RDMA from the infiniband world and putting it into Ethernet,  we have a lot of benefits.  Like, like just said, we have low latency, we have low CPU utilization, we have great  bandwidth, but on the other hand, we encounter into some difficulties.  One of those difficulties is the fact that Ethernet is a lossy fabric.  In the next part, we will talk what does it mean that Ethernet is a lossy fabric, and  how can we manage that by introducing a few complementary protocols?  And why do we care about it for Rocky, for example?  So the fact that Ethernet is a lossy fabric means that Ethernet allows packets to be dropped  when buffers overflow.  And actually Ethernet is using this to indicate that a congestion has occurred in the fabric.  From a rocky perspective, when a packet is being dropped, since everything is being offloaded  to the hardware, it means that there is a firmer flow, which is very expensive from a  performance perspective.  And another thing that I will demonstrate in the upcoming slide is what exactly is being  happening when this transmission happens.  So we can manage those packet loss by or those congestions in two manners.  First of all, we can improve the hardware to work faster when the transmission flows  are needed.  Basically, we were working on this on Connectic 6, and I will explain what is the difference  between what's going on now and what it's going to do.  Or the other approach can be to do a congestion control on the link layer.  So from a rocky perspective, when a packet is being dropped, PSN stands for packet sequence  number.  When you see, for example, if PSN number two was dropped, we will have to retransmit not  only this second packet, we will have to retransmit everything from this packet and on.  This is again because of the way that Trocki works.  Important thing to notice is the fact that in the infinite world, this is the difference  between infinite and Internet, there is no such thing as packet loss, basically.  The idea of infinite is a little bit different.  They work with something, they exchange data to transmit as much as can, some different  algorithm that we can talk a little bit about later.  But basically the problem that in Internet, we will have to retransmit everything from  this packet and on.  One of the mechanisms to avoid that or to manage that will be flow control or global  pause.  Basically what global pause means is that whenever the buffer will reach a certain threshold,  a special packet called pause will descend from the receiver to the center side, asking  him to completely stop any traffic.  Of course, global pause is a link level flow control and I'll explain it in a second how  it works.  So, basically, let's imagine that we have single receiver and multiple transmitters  to the same receiver guys.  So those transmitters send a lot of traffic to this receiver up to the fact that his  buffers are starting to get to some point in his threshold.  When this happens, this guy will send a pause frame but instead of sending it for each of  the senders, he will send it to the switch which is directly connected to him and only  when his buffers will get to this certain threshold, he will send the pause frames back  to the senders asking them to stop down.  The problem, basically this is a high level view of that.  The problem with that is that when we have multiple senders and single receiver, we have  multiple types of flows and each of these flows has its own thing.  So for example, if we don't want a specific flow to be paused at all, for example, if  we have video traffic and, I don't know, HTTP traffic, for example.  So we need a way to distinguish between those flows and deal with that.  One of the approaches to do this will be priority flow control or PFC.  So basically, we have something called PCP which is a priority code point.  It's a mechanism to put a priority inside the shim, inside the dot one q shim which  is also called a villain header and to distinguish between different types of flows.  In this mechanism with PFC, instead of stopping all the traffic at once from the sender side  to the receiver, we can stop each and every priority.  Basically how it works, that we have multiple queues right now, multiple buffers, and we  can stop the traffic on each buffer by itself instead of stopping altogether.  Again, this protocol PFC is link level, just like the previous one, just like pause.  So two problems with both PFC and global pause is the fact that first, it's link level.  And the second one, that the granularity is not that good.  Still, we have only eight priorities.  And the third, but maybe a little bit different thing, is the fact that in PFC and global  pause, what will happen is that we will stop sending any traffic at all, meaning instead  of slowing down.  What basically happens behind the scenes is that when a sender receives a pause packet,  he will start a counter which will start sending traffic again only when this counter expires  and he didn't receive another pause frame.  So the next mechanism called ECN, or explicit congestion notification, is basically an extension  of the IP protocol which allows an end-to-end congestion notification almost without dropping  any packets.  This mechanism works a little bit different.  It uses DCQCN, which is very much like red, and red stands for random early detection.  Basically the idea is that instead of sending a pause frame or a pause asking for the sender  to slow down his traffic on a certain threshold, using red, what happens is that as much as  if the longer the queue of the packets in the buffers, the higher probability of marking  a packet as a congestion experience gets higher.  Basically unlike PFC and global pause, what happens with ECN is that the intermediate  devices like switches and routers can mark a specific flow that a congestion has occurred  and when the receiver side will get this packet, a regular packet from the flow with congestion  experience, he will send a special packet called CMP or congestion notification packet  to the sender side asking him to slow down.  So from ECN what we get is much better granularity and as you can see, instead of stopping the  traffic completely, he will ask him to just slow down.  So as a little summary for what we've been talking so far is first of all that ROKV2  is erudible, unlike ROKV1 which was introduced in the previous generations of previous D.  ROKV is scalable.  ROKV is actually being implemented right now in not implemented, sorry, is used in one  of the biggest cloud providers like Azure.  ROKV works on Ethernet available distances, meaning that all distances that are available  for Ethernet will be available for ROKV2.  ROKV works with all advanced Ethernet signaling rates, meaning that ROKV is working on 100  gigabit Ethernet.  Common traffic management and monitoring tools work with ROKV, for example TCP-DAM  has support for sniffing ROKV packets, even in free BSD.  It is something that was pushed after updating lead pickup.  ROKV requires lossless network.  This is true, but we have great mechanisms for supporting that.  ROKV is supported by multiple vendors.  It is, it's not only Melanox.  And ROKV interoperability between different vendors is regularly tested by IBTA, by the  InfiniBend Association, actually this is happening twice a year and being published for general  availability.  Any questions?  So ROKV has a lot of application.  We can, we can accelerate MPI applications, for example, we can accelerate HPC applications.  And one of the interesting points that we have is that ROKV can accelerate storage protocols.  All those storage protocols that everyone are using like iSCSI, like NFS, like NVME,  like Samba, which I'm not going to talk about.  But basically all of them can be accelerated using ROKV.  So let's start from the first example, which is actually already implemented for free BSD,  which is ISER.  ISER is basically an iSCSI extension for RDMA.  We are using iSCSI and instead of using the TCP stack as the transport layer, we're actually  using RDMA.  The benefits, of course, just like any other RDMA or ROKV application, will be high bandwidth,  high IOPS, low latency, we will reduce CPU usage, and of course all the iSCSI management  tools are available.  So here are some ISER performance examples on both on Connectix 4 and Connectix 4LX for  25 and 100 gig.  So from this graph, you can see that using one small thing that I have to mention, those  performance results are wore down on Linux and not on free BSD, unfortunately.  So as you can see that using 512 bytes as a block size for IO, you can see that we can  get three times better performance from IO perspective.  We can get almost three times better latency using ISER and almost four times better bandwidth  using latency.  Again, I'm comparing the same devices, the same PCs, the same computers, the same everything,  just using RDMA and using regular TCP.  Using CPU performance, we can see that we can get almost four and a half times better  from CPU utilization.  NFS over RDMA, which is unfortunately not available yet for free BSD, basically the  idea that when there was, and I'm pretty sure that most of you, if not all, are familiar  with NFS.  When NFS was introduced, it was introduced on 10 megabit Ethernet and the migration to  100 megabit actually showed much performance improvements.  On the other hand, the next generation didn't show that much improvement on the other and  from the CPU utilization perspectives, NFS was much worse when migrating to faster networks.  So here I'll show some performance results done by Oracle actually.  So you can see again that we gain almost four times better on the read IOPS and much better  on the right.  You can see that almost three times better.  This on 16 was on one thread, sorry, and on six threads, as you can see here, using big  block sizes and for 4K and 8K, we see the same picture basically from NFS over RDMA point  of view.  The next protocol that I would like to talk about is NVME over fabrics, which again is  unfortunately not available for free BSD yet.  Only NVME is nonvolatile memory express, allows you to have an SSD device basically attached  to your PCI.  NVME over fabrics allows you to write using RDMA to NVME devices from remote.  So here are some performance measurements for Connectix 5, for example, and from this  graph you can see that the performance for doing an NVME device, meaning NVME over fabric,  is almost alike doing a local write.  And again, from throughput and IOPS perspective, you can see that using NVME for NVME over  fabrics, we can get to up to 4 million IOPS on small block sizes, and we get to almost  saturating the whole link in 4K.  Questions?  Right, exactly, right, and even with the new generation like the SOC, we can offload the  target completely.  You can have the whole target inside the NIC itself.  Bluefield.  Can you go back to slide the...  I couldn't read that.  I couldn't read my eyes with that.  I couldn't read the color.  Which one?  What each color means?  So the blue one is RDMA write IOPS.  The orange one is IP of write B write IOPS.  It's actually not comparing real TCP to RDMA.  It's basically comparing IP of write B to RDMA.  And then RDMA read IOPS and IP of write B read IOPS.  It's read write and RDMA versus IP of write B.  Yes, of course.  So for RDMA, I understand that this is basically the star will write something.  Bear with me here, sorry.  For NVME, if a write is coming in, the NIC is going to put the data into memory and  the NIC is going to issue the piece management to the device, say, go write using this buffer.  So I'm not an NVME expert.  But as far as I understand how it works is basically that by using the NVME protocol,  first of all, we have to understand that NVME is not scousing, right?  Yeah.  NVME is NVME.  So as far as I understand, when we get the NVME message on the wire to the card itself,  it will just put the data or the buffer directly on the NVME device over the PCI instead of  going up to the CPU and back.  Yeah, that's why I'm saying the NIC is putting the, it's doing the buffer for what comes  in over the network and then the NIC is sending the right man to...  Directly on the PCI to the NVME device.  Other questions?  Suppose I still do the double-fogging device and write something to RAM and NIC takes from  RAM and send to the...  So this is the point that I'm not sure about and I want to commit to.  But as far as I understand, there will no be...  No security interaction, but this double-DMA.  So I'm not sure if there will be a double-DMA.  This is what I'm not sure about, but I think that it will be a direct DMA to the NVME device  itself.  But this one I'll have to check because I'm not sure about it.  Of course.  Do you mean Rocky Multicast?  You mean Rocky Multicast.  Are DMA Multicast, basically?  Yes.  Yes.  And actually we have the MCC application located in the same place.  I think it's under RDMA applications, under LibRDMA.  And MCC is basically a user space application which uses multicast for RDMA traffic.  I do have one comment, I mean, it's not a message here, several complementary protocols  to make the internet fabric to be lost as a step of policy.  So this is something that we are trying to overcome harder so there won't be a need for  those protocols.  And basically we are trying to advance the way we are doing the transmission and we  have to keep it to be less involved.  So we are just setting the exact piece that was missing and cut it off just like the  DC piece that can reduce the need for any complementary protocols.  So, you know, future generations and connecting six LX and beyond and onwards, you want to  need them for a piece of what's called I work for.  No.  It's a different protocol.  It's also based on RDMA.  And you are an expert here that's the pilot.  So basically what will happen just like I demonstrated in the next generations, instead  of resending everything from the lost packet and force, we can just resend the lost packet  itself instead of resending everything.  This is what many was trying to describe about the next generations.  Right.  But, but, but, right, of course.  So basically about the complementary protocols, the best practice that we recommend will  be to use PFC or global pause, which is, of course, PFC because it has better granularity  and ECN because ECN still allows a little bit of traffic to be dropped.  And when using global pause, there is no chance or PFC, of course, there is no chance for  traffic loss.  ECN is implemented in hardware, for example.  Yes, of course.  So I wasn't sure if I have enough time to talk about ECN because ECN is, but I do have.  So talk about it a little bit.  So what happens with ECN is, first of all, if you have two servers connected back to  back, you will not have ECN at all.  So what happens with ECN, the intermediate devices, the switches and routers must support  ECN.  What basically happens is when the threshold in the intermediate devices buffers gets to  a certain point, this packet will be marked, like both bits will be turned on.  And when the receiver side will get a regular packet with those two bits on, but in hardware,  it will generate those special packets called C&P, which is a congestion notification packet,  sending it directly to the sender, asking him to slow down the traffic.  Okay.  Awesome.  Any further questions?  So I think that it.  Thank you, everybody, for joining me today.
Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_0_107.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_108_251.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_252_479.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_480_1235.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_1236_2039.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_2040_2771.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_2772_4043.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_4044_5159.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_5160_5711.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_5712_6239.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_6240_8567.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_8568_9023.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_9024_9839.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_9840_10763.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_10764_11255.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_11256_12503.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_12504_13943.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_13944_14711.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_14712_16523.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_16524_16967.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_16968_18911.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_18912_20351.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_20352_21131.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_21132_22055.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_22056_22223.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_22224_23123.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_23124_24071.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_24072_25967.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_25968_28355.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_28356_28727.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_28728_31127.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_31128_31979.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_31980_32879.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_32880_33095.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_33096_33851.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_33852_34055.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_34056_34499.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_34500_34739.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_34740_35759.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_35760_35963.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_35964_36587.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_36588_36887.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_36888_37619.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_37620_37679.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_37680_38219.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_38220_39791.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_39792_39839.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_39840_39887.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_39888_44831.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_44832_44879.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_44880_44927.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_44928_44987.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_44988_46355.png Slava_Shwartsman__RoCE_as_a_performance_accelerator_--_BSDCan_2018_h1NyBaPHuyE_46356_49976.png