Faster memory reclamation with DPDK RCU
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Brown, Nathan
DPDK added a RCU library with a novel method to reclaim resources. We have been running tests to understand the performance differences between the DPDK RCU and the user space RCU library. In our tests, we find that DPDK RCU can perform reclamation faster and perform significantly better when pre-emptive readers are involved. Other than the performance, DPDK RCU has several advantages such as not requiring a background thread for reclaiming resources and the ability to integrate with existing libraries without having to modify the application. This talk will present various testing done on DPDK RCU and the user space RCU library and their results. It will go into the details of pre-emptive reader problem, which affects use cases beyond DPDK, and show that DPDK RCU library can reduce the reclamation time
 Hi, my name is Nathan Brown and I'm a software engineer at ARM. I'm really excited to talk to you today about faster memory reclamation with dbdkRCU. Specifically, I'll be comparing the dbdk and usespaceRCU libraries. First, I'm going to start off with a brief background of what RCU is and what problem it solves. Then, I'll talk about key advantages dbdkRCU brings to the table. Next, I'll walk us through some performance results I have gathered comparing dbdk and usespaceRCU and then we'll have some time set aside for Q&A. Before I get started, I'd like to thank my mentors at ARM, Darmic, and Hanapa, as well as the broader networking team I'm on. With all of their help and feedback, this presentation wouldn't have been possible. So thank you. Alright, let's get started. What is RCU? Well, with most things, we can start with a data structure. But since we're implementing a multi-threaded and high-performance application, we're going to want this to be a lock-free data structure. And while in general, RCU can work with any data structure you provide, fair simplicity in this presentation, let's just stick to a link list that has three elements. A, then B, followed by C. Now we can have multiple readers traversing through this link list, accessing elements, more or less doing what they please. And because it's lock-free, we can also have a writer thread that is about to do some updates. However, this raises a very interesting question. How can the writer thread safely remove and free element B? Well, the lock-free link list algorithm tells us how to remove element B from the list. We can atomically update A's next pointer to point to C instead of B. In this way, new readers such as reader1 will only be able to see elements A, then C in the list, and won't be able to discover B. However, current readers like reader2 and reader3, that are already accessing element B, will still be able to see a well-formed list and traverse onwards to element C when they're ready. But this idea all that brings about a bit of confusion. How can we safely free element B? We know we can't do it right now, because if we free element B, then reader2 and or reader3 attempts to iterate forward in the list, they'll end up accessing element B after it's already been freed. In short, we'll have a use after free error. So we're going to have to wait. But how long do we wait? We need to wait until all readers have stopped referencing element B. Only at that point in time will it become safe to free element B. But programmatically, how can we do that? How do we know how long to wait? That's what RCU provides. RCU is a suite of mechanisms that enable you to safely reclaim memory in lock free context, by letting you know how long you need to wait until that memory is safe to free. The way it works is that readers report quiescent states to signal when they're no longer accessing shared state. So when reader2 and reader3 in our previous example stopped accessing elements on the list, they both entered a quiescent state. We have grace periods for deleted elements, which is the time between the element being deleted and all readers reporting a quiescent state. It sends up being the lower bound on the delay between deleting the element and being able to reclaim its memory. In short, it's only safe to reclaim the memory after this grace period ends. We also like to measure the delete free delay, which is the actual delay between deletion and freeing for an element. While in theory this is lower bounded by the grace period, the actual realized delete free delay may become higher than that due to an RCU's implementation. Pictorially, it may look something like this. We have reader1 and reader2 entering quiescent states and accessing the data structure, quiescent states, data structure, quiescent states, so on and so forth. When the delete happens, which is the far left bar here, the time until the grace period is up is how long it takes for both readers to enter the quiescent state in green. However, the freeing may occur much later, and thus the delete free delay may become longer than the actual grace period. Use-based RCU is a most popular use-based implementation for RCU at this point in time. It's available at this URL. Its API is based on the Linux kernel RCU API. In it, synchronizeRCU is a function that starts detecting and blocks until a grace period is over. So, in our element b example, the writer might delete element b, then call synchronizeRCU to start and block until the grace period is over, then once synchronizeRCU returns, since the grace period is over, that means that the memory occupied by element b is now safe to reclaim. So, the writer is able to free element b. However, we're blocking in the writer thread, and that might not be the most desirable thing. So use-based RCU also provides the API of callRCU where you provide a callback, and that callback will be invoked by a background thread after the grace period is over. So, the writer might delete element b, and then call RCU and provide enough information such that the background thread can then execute the callback and free the memory associated with element b once it is safe to do so. However, with this background thread approach with use-based RCU, the grace period detection might not start immediately, and thus the delete free delay might end up being longer because the grace period detection doesn't start immediately. dbdkRCU, on the other hand, is based around the idea of tokens, where a token more or less correlates one to one to an element. So, you can get a token by RCU start, and this will start grace period detection for that token. Then, at a later point in time, you can call RCU check and provide a token given to you from RCU start. And RCU check will tell you if the grace period for that token is over. So, the way our writer might use this is it will delete element b from the list. Then it will get token b via RCU start. Now, it knows that it has to wait some time for the grace period to become over, so in the meantime, it's going to do some other useful work. Then it can come back and check, hey, is this grace period over? And if it is, it knows it's safe to free element b. Otherwise, it can continue doing useful work. Now, an immediate advantage you may have noticed is the increased flexibility dbdkRCU brings. We weren't blocking in our writer thread, but we also didn't have to rely on yet another background thread to manage our memory reclamation for us. Now, of course, there's nothing stopping us from having a background thread dedicated to memory reclamation, but we don't need to have one either. The way that we can use the background thread in dbdkRCU is we generate the tokens in the writer, then pass these tokens off to the background thread, then the background thread consistently checks the tokens and executes callbacks to reclaim memory as necessary. This is somewhat similar to how use-based RCUs background thread works. Something else I haven't mentioned yet is that dbdkRCU enables you to partition threads into different RCU groups. For example, let's take 10 threads. Let's say we know that threads 1 through 5 are accessing hashmap A and only those threads. Likewise, threads 5 through B are accessing hashmap B. Thread 5 is a bit unique here in that it can access both hashmaps. However, thread 7 is a bit slow and it's accessing hashmap B and it's taking some time. It's not going to report its quiescent state as quickly as say threads 1 through 5. So if thread 7 is never going to access hashmap A, but it's the one running slow, why should grace periods become elongated for hashmap A? They don't have to be. We know that thread 7 isn't ever going to access hashmap A and it has no information about the contents of hashmap A or its elements. And dbdk allows you to model this with these different RCU groups. However, with useFaceRCU, all of the threads are global and treated the same. So it doesn't have the context needed in order to be able to provide this additional flexibility. Additionally, in our benchmarks, dbdkRCU ends up on average producing faster memory reclamation than useFaceRCU. This is great. Faster memory reclamation generally leads to a lower memory overhead and less memory fragmentation. Additionally, dbdkRCU is able to sustain this faster memory reclamation even with preemptible readers. When a reader can be preempted, it can't report a quiescent state which can elongate grace periods and delete free delays. However, even in the face of this, dbdkRCU is still able to perform faster than useFaceRCU, which is great. Finally, dbdkRCU is integrated into some dbdk data structures already, for example, RTE hash. This means that with some minor code modifications to the initialization of your data structure, you can start using dbdkRCU today and already start reaping the benefits. So a little bit about how I benchmark delete free delay. I took dbdk's lock free hash table, pre-populated it with four million elements, allowed multiple readers and a single writer to start accessing it randomly. So the readers will be looking up random elements and the writer will be randomly inserting and deleting random elements. In order to simulate the writer doing something else rather than just modifying this hash table, I limited it to about 5k deletes per second. In order to reach this limit with useFaceRCU, I needed to use the callRCU method of reclamation rather than blocking within the writer thread. This means that for useFaceRCU, there's an additional thread going on. So in order to have a bit more of an interesting comparison with dbdk, I ran dbdk under two different configurations. One where dbdk had a background thread where the writer thread would generate the token and then pass the token and callback information to this background thread, which would then reclaim the memory as soon as it was able to. And another one where there was no background thread for dbdk. Without the background thread, when a writer was performing a delete, it would check and see if it could reclaim memory for any previous deletes. And then if it was able to, it would invoke some callbacks and reclaim the memory at that point in time. So here's some results. On the y-axis, we have deleteFreeDelay in milliseconds, and on the x-axis, we're varying the number of readers. In this configuration, we're giving each reader its own core, so we don't have to worry about preemption yet. And here we can see that I have dbdk, then use space rcu, and then dbdk with a background thread being measured. And we can see, wow, the performance results are great. dbdk is consistently outperforming use space rcu, as is dbdk with a background thread. In fact, we see that dbdk rcu is 72 to 77% faster to reclaim memory than use space rcu. And if we add in a background thread, it becomes 79 to 88% faster. That's great. However, what happens if we put all of these readers on one core? When we have all these readers on one core, only one reader is able to make progress at a time, and the rest of the readers will end up being preempted. Here we can see a bit of what I was alluding to with the poorer performance once preemption comes into the picture. Use space rcu, for example, goes from 6.6 milliseconds of delete free delay all the way up to 154.7 milliseconds of delete free delay. That's a huge increase. And while dbdk and dbdk with a background thread also have increases as the level of preemption increases, it's not quite as bad as use space rcu. In fact, dbdk rcu ends up being 41 to 61% faster to reclaim memory than use space rcu in this configuration. And once readers start being preempted, dbdk with a background thread and without a background thread end up performing very similarly. However, a one core workload isn't all realistic for every workload out there today. So I also took eight readers and spread them across various numbers of cores, ranging from one core to two, four, and eight cores. So here in the middle are the two most interesting graphs. So we have eight readers on two cores and eight readers on four cores. So we both have multiple cores and some amount of preemption on each core. And as we can see, dbdk with and without a background thread still consistently outperforms use space rcu in this configuration. dbk rcu ends up being 29 to 72% faster to reclaim memory than use space rcu. And with preemption, dbdk performs similarly both with and without a background thread. An interesting conclusion from this last point is that dbdk ends up waiting more for the readers to report their quiescent state than it is for waiting for a background thread to try and reclaim memory as fast as possible. So that implies that preempted readers are certainly hampering the performance of these rcu implementations. Thank you very much for your time. I'm looking forward to hearing your questions. And so as such, our implementation has been sort of tailor fit for dbdk. But the idea of tokens for rcu is broadly applicable. So if you want to make use of our implementation, it'll have to be within dbdk. But if you want to use the token-based rcu, it shouldn't be too difficult to end up porting the token implementation to some other projects as well. Okay. So you mean it's inside dbdk, but we can use it without using the rest of dbdk if needed. And the other part of the question is, is there something in the implementation which is specific to dbdk? How is it better than librcu for dbdk? I see. So the main advantages over librcu comes from the token design and the greater flexibility and better performance. So for those reasons, those are why we opted to write our own rcu implementation for dbdk rather than use librcu. And since we were writing the implementation for dbdk, we ended up using some of dbdk's other libraries like the ring data structure or some of the EAL functions. Okay. if we want to use it inside dbdk, we need to have this rcu integrated in the dbdk data structures. You said it's already interpreted for hash's data structure. What about others? I'm not sure about which others, but integrating into the data structures, only one of the easier ways to use rcu. You're still able to use the main API functions directly yourself, reporting quiescent states, getting tokens with rcu start and using rcu check to check if the grace period for the token has ended. And so we've integrated these functions into the rte hash data structure, for example, which you can, of course, draw inspiration from. Okay. Is there someone asking if you plan to write a paper about this work, about this new rcu? Yes, that is a project currently underway. Okay, cool. I suppose you are also discussing with the father of rcu, Mr. McKinsey, if I remember where. Yes. I believe he may have been involved with the reviewing of the presentation. Yes, right. Okay. Last question, you are talking about partitioning threads. Can you explain more about that? Sure. So libudurcu treats all threads the same, all of the reader threads the same, whereas with dbdkrcu you can form different groups of reader threads that are independent of each other. And so if one group ends up not performing as fast as another group, or perhaps there's more of them, so synchronization takes longer, you can end up separating them entirely.
 Hi, my name is Nathan Brown and I'm a software engineer at ARM.  I'm really excited to talk to you today about faster memory reclamation with dbdkRCU.  Specifically, I'll be comparing the dbdk and usespaceRCU libraries.  First, I'm going to start off with a brief background of what RCU is and what problem  it solves.  Then, I'll talk about key advantages dbdkRCU brings to the table.  Next, I'll walk us through some performance results I have gathered comparing dbdk and  usespaceRCU and then we'll have some time set aside for Q&A.  Before I get started, I'd like to thank my mentors at ARM, Darmic, and Hanapa, as well  as the broader networking team I'm on.  With all of their help and feedback, this presentation wouldn't have been possible.  So thank you.  Alright, let's get started.  What is RCU?  Well, with most things, we can start with a data structure.  But since we're implementing a multi-threaded and high-performance application, we're going  to want this to be a lock-free data structure.  And while in general, RCU can work with any data structure you provide, fair simplicity  in this presentation, let's just stick to a link list that has three elements.  A, then B, followed by C.  Now we can have multiple readers traversing through this link list, accessing elements,  more or less doing what they please.  And because it's lock-free, we can also have a writer thread that is about to do some updates.  However, this raises a very interesting question.  How can the writer thread safely remove and free element B?  Well, the lock-free link list algorithm tells us how to remove element B from the list.  We can atomically update A's next pointer to point to C instead of B. In this way, new  readers such as reader1 will only be able to see elements A, then C in the list, and  won't be able to discover B. However, current readers like reader2 and reader3, that are  already accessing element B, will still be able to see a well-formed list and traverse  onwards to element C when they're ready.  But this idea all that brings about a bit of confusion.  How can we safely free element B?  We know we can't do it right now, because if we free element B, then reader2 and or reader3  attempts to iterate forward in the list, they'll end up accessing element B after it's already  been freed.  In short, we'll have a use after free error.  So we're going to have to wait.  But how long do we wait?  We need to wait until all readers have stopped referencing element B. Only at that point in  time will it become safe to free element B.  But programmatically, how can we do that?  How do we know how long to wait?  That's what RCU provides.  RCU is a suite of mechanisms that enable you to safely reclaim memory in lock free context,  by letting you know how long you need to wait until that memory is safe to free.  The way it works is that readers report quiescent states to signal when they're no longer accessing  shared state.  So when reader2 and reader3 in our previous example stopped accessing elements on the  list, they both entered a quiescent state.  We have grace periods for deleted elements, which is the time between the element being  deleted and all readers reporting a quiescent state.  It sends up being the lower bound on the delay between deleting the element and being able  to reclaim its memory.  In short, it's only safe to reclaim the memory after this grace period ends.  We also like to measure the delete free delay, which is the actual delay between deletion  and freeing for an element.  While in theory this is lower bounded by the grace period, the actual realized delete free  delay may become higher than that due to an RCU's implementation.  Pictorially, it may look something like this.  We have reader1 and reader2 entering quiescent states and accessing the data structure, quiescent  states, data structure, quiescent states, so on and so forth.  When the delete happens, which is the far left bar here, the time until the grace period  is up is how long it takes for both readers to enter the quiescent state in green.  However, the freeing may occur much later, and thus the delete free delay may become  longer than the actual grace period.  Use-based RCU is a most popular use-based implementation for RCU at this point in time.  It's available at this URL.  Its API is based on the Linux kernel RCU API.  In it, synchronizeRCU is a function that starts detecting and blocks until a grace period  is over.  So, in our element b example, the writer might delete element b, then call synchronizeRCU  to start and block until the grace period is over, then once synchronizeRCU returns,  since the grace period is over, that means that the memory occupied by element b is now  safe to reclaim.  So, the writer is able to free element b.  However, we're blocking in the writer thread, and that might not be the most desirable thing.  So use-based RCU also provides the API of callRCU where you provide a callback, and  that callback will be invoked by a background thread after the grace period is over.  So, the writer might delete element b, and then call RCU and provide enough information  such that the background thread can then execute the callback and free the memory associated  with element b once it is safe to do so.  However, with this background thread approach with use-based RCU, the grace period detection  might not start immediately, and thus the delete free delay might end up being longer  because the grace period detection doesn't start immediately.  dbdkRCU, on the other hand, is based around the idea of tokens, where a token more or  less correlates one to one to an element.  So, you can get a token by RCU start, and this will start grace period detection for  that token.  Then, at a later point in time, you can call RCU check and provide a token given to you  from RCU start.  And RCU check will tell you if the grace period for that token is over.  So, the way our writer might use this is it will delete element b from the list.  Then it will get token b via RCU start.  Now, it knows that it has to wait some time for the grace period to become over, so in  the meantime, it's going to do some other useful work.  Then it can come back and check, hey, is this grace period over?  And if it is, it knows it's safe to free element b.  Otherwise, it can continue doing useful work.  Now, an immediate advantage you may have noticed is the increased flexibility dbdkRCU brings.  We weren't blocking in our writer thread, but we also didn't have to rely on yet another  background thread to manage our memory reclamation for us.  Now, of course, there's nothing stopping us from having a background thread dedicated  to memory reclamation, but we don't need to have one either.  The way that we can use the background thread in dbdkRCU is we generate the tokens in the  writer, then pass these tokens off to the background thread, then the background thread  consistently checks the tokens and executes callbacks to reclaim memory as necessary.  This is somewhat similar to how use-based RCUs background thread works.  Something else I haven't mentioned yet is that dbdkRCU enables you to partition threads  into different RCU groups.  For example, let's take 10 threads.  Let's say we know that threads 1 through 5 are accessing hashmap A and only those threads.  Likewise, threads 5 through B are accessing hashmap B.  Thread 5 is a bit unique here in that it can access both hashmaps.  However, thread 7 is a bit slow and it's accessing hashmap B and it's taking some time.  It's not going to report its quiescent state as quickly as say threads 1 through 5.  So if thread 7 is never going to access hashmap A, but it's the one running slow, why should  grace periods become elongated for hashmap A?  They don't have to be.  We know that thread 7 isn't ever going to access hashmap A and it has no information  about the contents of hashmap A or its elements.  And dbdk allows you to model this with these different RCU groups.  However, with useFaceRCU, all of the threads are global and treated the same.  So it doesn't have the context needed in order to be able to provide this additional flexibility.  Additionally, in our benchmarks, dbdkRCU ends up on average producing faster memory reclamation  than useFaceRCU.  This is great.  Faster memory reclamation generally leads to a lower memory overhead and less memory  fragmentation.  Additionally, dbdkRCU is able to sustain this faster memory reclamation even with preemptible  readers.  When a reader can be preempted, it can't report a quiescent state which can elongate  grace periods and delete free delays.  However, even in the face of this, dbdkRCU is still able to perform faster than useFaceRCU,  which is great.  Finally, dbdkRCU is integrated into some dbdk data structures already, for example,  RTE hash.  This means that with some minor code modifications to the initialization of your data structure,  you can start using dbdkRCU today and already start reaping the benefits.  So a little bit about how I benchmark delete free delay.  I took dbdk's lock free hash table, pre-populated it with four million elements, allowed multiple  readers and a single writer to start accessing it randomly.  So the readers will be looking up random elements and the writer will be randomly inserting  and deleting random elements.  In order to simulate the writer doing something else rather than just modifying this hash  table, I limited it to about 5k deletes per second.  In order to reach this limit with useFaceRCU, I needed to use the callRCU method of reclamation  rather than blocking within the writer thread.  This means that for useFaceRCU, there's an additional thread going on.  So in order to have a bit more of an interesting comparison with dbdk, I ran dbdk under two  different configurations.  One where dbdk had a background thread where the writer thread would generate the token  and then pass the token and callback information to this background thread, which would then  reclaim the memory as soon as it was able to.  And another one where there was no background thread for dbdk.  Without the background thread, when a writer was performing a delete, it would check and  see if it could reclaim memory for any previous deletes.  And then if it was able to, it would invoke some callbacks and reclaim the memory at that  point in time.  So here's some results.  On the y-axis, we have deleteFreeDelay in milliseconds, and on the x-axis, we're varying  the number of readers.  In this configuration, we're giving each reader its own core, so we don't have to worry about  preemption yet.  And here we can see that I have dbdk, then use space rcu, and then dbdk with a background  thread being measured.  And we can see, wow, the performance results are great.  dbdk is consistently outperforming use space rcu, as is dbdk with a background thread.  In fact, we see that dbdk rcu is 72 to 77% faster to reclaim memory than use space rcu.  And if we add in a background thread, it becomes 79 to 88% faster.  That's great.  However, what happens if we put all of these readers on one core?  When we have all these readers on one core, only one reader is able to make progress at  a time, and the rest of the readers will end up being preempted.  Here we can see a bit of what I was alluding to with the poorer performance once preemption  comes into the picture.  Use space rcu, for example, goes from 6.6 milliseconds of delete free delay all the way up to 154.7  milliseconds of delete free delay.  That's a huge increase.  And while dbdk and dbdk with a background thread also have increases as the level of  preemption increases, it's not quite as bad as use space rcu.  In fact, dbdk rcu ends up being 41 to 61% faster to reclaim memory than use space rcu  in this configuration.  And once readers start being preempted, dbdk with a background thread and without a background  thread end up performing very similarly.  However, a one core workload isn't all realistic for every workload out there today.  So I also took eight readers and spread them across various numbers of cores, ranging from  one core to two, four, and eight cores.  So here in the middle are the two most interesting graphs.  So we have eight readers on two cores and eight readers on four cores.  So we both have multiple cores and some amount of preemption on each core.  And as we can see, dbdk with and without a background thread still consistently outperforms  use space rcu in this configuration.  dbk rcu ends up being 29 to 72% faster to reclaim memory than use space rcu.  And with preemption, dbdk performs similarly both with and without a background thread.  An interesting conclusion from this last point is that dbdk ends up waiting more for the  readers to report their quiescent state than it is for waiting for a background thread  to try and reclaim memory as fast as possible.  So that implies that preempted readers are certainly hampering the performance of these  rcu implementations.  Thank you very much for your time.  I'm looking forward to hearing your questions.  And so as such, our implementation has been sort of tailor fit for dbdk.  But the idea of tokens for rcu is broadly applicable.  So if you want to make use of our implementation, it'll have to be within dbdk.  But if you want to use the token-based rcu, it shouldn't be too difficult to end up porting  the token implementation to some other projects as well.  Okay.  So you mean it's inside dbdk, but we can use it without using the rest of dbdk if needed.  And the other part of the question is, is there something in the implementation which  is specific to dbdk?  How is it better than librcu for dbdk?  I see.  So the main advantages over librcu comes from the token design and the greater flexibility  and better performance.  So for those reasons, those are why we opted to write our own rcu implementation for dbdk  rather than use librcu.  And since we were writing the implementation for dbdk, we ended up using some of dbdk's  other libraries like the ring data structure or some of the EAL functions.  Okay. if we want to use it inside dbdk, we need to have this rcu integrated in the dbdk  data structures.  You said it's already interpreted for hash's data structure.  What about others?  I'm not sure about which others, but integrating into the data structures, only one of the  easier ways to use rcu.  You're still able to use the main API functions directly yourself, reporting quiescent states,  getting tokens with rcu start and using rcu check to check if the grace period for the  token has ended.  And so we've integrated these functions into the rte hash data structure, for example,  which you can, of course, draw inspiration from.  Okay.  Is there someone asking if you plan to write a paper about this work, about this new rcu?  Yes, that is a project currently underway.  Okay, cool.  I suppose you are also discussing with the father of rcu, Mr. McKinsey, if I remember  where.  Yes.  I believe he may have been involved with the reviewing of the presentation.  Yes, right.  Okay.  Last question, you are talking about partitioning threads.  Can you explain more about that?  Sure.  So libudurcu treats all threads the same, all of the reader threads the same, whereas  with dbdkrcu you can form different groups of reader threads that are independent of  each other.  And so if one group ends up not performing as fast as another group, or perhaps there's  more of them, so synchronization takes longer, you can end up separating them entirely.
comparing_dpdk_rcu_and_user_space_rcu_library_0_155.png comparing_dpdk_rcu_and_user_space_rcu_library_156_551.png comparing_dpdk_rcu_and_user_space_rcu_library_552_2735.png comparing_dpdk_rcu_and_user_space_rcu_library_2736_3359.png comparing_dpdk_rcu_and_user_space_rcu_library_3360_4367.png comparing_dpdk_rcu_and_user_space_rcu_library_4368_4643.png comparing_dpdk_rcu_and_user_space_rcu_library_4644_5015.png comparing_dpdk_rcu_and_user_space_rcu_library_5016_6575.png comparing_dpdk_rcu_and_user_space_rcu_library_6576_6815.png comparing_dpdk_rcu_and_user_space_rcu_library_6816_11723.png comparing_dpdk_rcu_and_user_space_rcu_library_11724_14879.png comparing_dpdk_rcu_and_user_space_rcu_library_14880_16931.png comparing_dpdk_rcu_and_user_space_rcu_library_16932_19499.png comparing_dpdk_rcu_and_user_space_rcu_library_19500_21287.png comparing_dpdk_rcu_and_user_space_rcu_library_21288_21383.png comparing_dpdk_rcu_and_user_space_rcu_library_21384_27179.png comparing_dpdk_rcu_and_user_space_rcu_library_27180_27301.png