{"action":"created","issue":{"url":"https://api.github.com/repos/spdk/spdk/issues/297","repository_url":"https://api.github.com/repos/spdk/spdk","labels_url":"https://api.github.com/repos/spdk/spdk/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/spdk/spdk/issues/297/comments","events_url":"https://api.github.com/repos/spdk/spdk/issues/297/events","html_url":"https://github.com/spdk/spdk/issues/297","id":322666798,"number":297,"title":"Random write IOPS by Perf is over than NVMe device spec.","user":{"login":"tafukuyama","id":29906937,"avatar_url":"https://avatars2.githubusercontent.com/u/29906937?v=4","gravatar_id":"","url":"https://api.github.com/users/tafukuyama","html_url":"https://github.com/tafukuyama","followers_url":"https://api.github.com/users/tafukuyama/followers","following_url":"https://api.github.com/users/tafukuyama/following{/other_user}","gists_url":"https://api.github.com/users/tafukuyama/gists{/gist_id}","starred_url":"https://api.github.com/users/tafukuyama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tafukuyama/subscriptions","organizations_url":"https://api.github.com/users/tafukuyama/orgs","repos_url":"https://api.github.com/users/tafukuyama/repos","events_url":"https://api.github.com/users/tafukuyama/events{/privacy}","received_events_url":"https://api.github.com/users/tafukuyama/received_events","type":"User","site_admin":false},"labels":[{"id":235898967,"url":"https://api.github.com/repos/spdk/spdk/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":{"login":"tafukuyama","id":29906937,"avatar_url":"https://avatars2.githubusercontent.com/u/29906937?v=4","gravatar_id":"","url":"https://api.github.com/users/tafukuyama","html_url":"https://github.com/tafukuyama","followers_url":"https://api.github.com/users/tafukuyama/followers","following_url":"https://api.github.com/users/tafukuyama/following{/other_user}","gists_url":"https://api.github.com/users/tafukuyama/gists{/gist_id}","starred_url":"https://api.github.com/users/tafukuyama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tafukuyama/subscriptions","organizations_url":"https://api.github.com/users/tafukuyama/orgs","repos_url":"https://api.github.com/users/tafukuyama/repos","events_url":"https://api.github.com/users/tafukuyama/events{/privacy}","received_events_url":"https://api.github.com/users/tafukuyama/received_events","type":"User","site_admin":false},"assignees":[{"login":"tafukuyama","id":29906937,"avatar_url":"https://avatars2.githubusercontent.com/u/29906937?v=4","gravatar_id":"","url":"https://api.github.com/users/tafukuyama","html_url":"https://github.com/tafukuyama","followers_url":"https://api.github.com/users/tafukuyama/followers","following_url":"https://api.github.com/users/tafukuyama/following{/other_user}","gists_url":"https://api.github.com/users/tafukuyama/gists{/gist_id}","starred_url":"https://api.github.com/users/tafukuyama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tafukuyama/subscriptions","organizations_url":"https://api.github.com/users/tafukuyama/orgs","repos_url":"https://api.github.com/users/tafukuyama/repos","events_url":"https://api.github.com/users/tafukuyama/events{/privacy}","received_events_url":"https://api.github.com/users/tafukuyama/received_events","type":"User","site_admin":false}],"milestone":null,"comments":14,"created_at":"2018-05-14T04:11:41Z","updated_at":"2018-05-25T16:00:38Z","closed_at":null,"author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\n\r\nWe run spdk perf random write and found that reported IOPS is over than NVMe device spec.\r\nWe confirmed spdk perf random read looks reasonable IOPS.\r\nThis issue occurs only when random write.\r\n\r\nWe use  Intel® SSD DC P3700 Series 400GB for our test devices.\r\nAs far as looking below datasheet, random write IOPS for a single device is about 75KIOPS.\r\nhttps://www.intel.com/content/dam/support/us/en/documents/ssdc/hpssd/sb/Intel_SSD_DC_P3700_Series_PCIe_Product_Specification-005.pdf\r\n\r\nBut we get 175KIOPS when using spdk + perf with # of cores=1, iodepth=128, # of NVMe=1.\r\nWe tested spdk + fio_pulgin for our reference and it scored reasonable value.(85KIOPS)\r\nWe also observed same result in spdk + perf + aio (*1) (117KIOPS)\r\n*1: After executing \"/root/spdk-master/scripts/setup.sh reset\",  specify /dev/nvmeXn1 in pref command line.\r\n\r\n![image](https://user-images.githubusercontent.com/29906937/39975713-3a67a908-576b-11e8-8d4c-826685a40285.png)\r\n\r\n\\# /root/spdk-master/examples/nvme/perf/perf -q 128 -s 4096 -w randwrtie -t 600 -c 0x0001 -D -LL -r 'trtype:PCIe traddr:0000.01.00.0'\r\nStarting SPDK v18.04-pre / DPDK 18.02.1 initialization...\r\n[ DPDK EAL parameters: perf -c 0x0001 --file-prefix=spdk_pid27748 ]\r\nEAL: Multi-process socket /var/run/.spdk_pid27748_unix\r\nEAL: Probing VFIO support...\r\nInitializing NVMe Controllers\r\nEAL: PCI device 0000:01:00.0 on NUMA socket 0\r\nEAL:   probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:01:00.0\r\nAttached to NVMe Controller at 0000:01:00.0 [8086:0953]\r\nAssociating INTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) with lcore 0\r\nInitialization complete. Launching workers.\r\nStarting thread on core 0\r\n\\========================================================\r\n                                                                                            Latency(us)\r\nDevice Information                                     :       IOPS       MB/s    Average        min        max\r\nINTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) from core 0:  174892.30     683.17     731.81       4.69   21524.00\r\n\\========================================================\r\nTotal                                                  :  174892.30     683.17     731.81       4.69   21524.00\r\n\r\nSummary latency data for INTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) from core 0:\r\n\\=================================================================================\r\n  1.00000% :     6.918us\r\n 10.00000% :    13.592us\r\n 25.00000% :    33.646us\r\n 50.00000% :   444.709us\r\n 75.00000% :  1022.050us\r\n 90.00000% :  2122.118us\r\n 95.00000% :  2559.025us\r\n 98.00000% :  3183.177us\r\n 99.00000% :  3729.310us\r\n 99.50000% :  4431.482us\r\n 99.90000% :  6834.469us\r\n 99.99000% :  9924.023us\r\n 99.99900% : 12607.878us\r\n 99.99990% : 20222.537us\r\n 99.99999% : 21470.842us\r\n\r\n\r\n![image](https://user-images.githubusercontent.com/29906937/39975769-833616a6-576b-11e8-9139-7993b24a9ff0.png)\r\n\r\n\\# /root/spdk-master/examples/nvme/perf/perf -q 128 -s 4096 -w randwrtie -t 600 -c 0x000f -D -LL -r 'trtype:PCIe traddr:0000.01.00.0' -r 'trtype:PCIe traddr:0000.03.00.0' -r 'trtype:PCIe traddr:0000.82.00.0' -r 'trtype:PCIe traddr:0000.84.00.0'\r\nStarting SPDK v18.04-pre / DPDK 18.02.1 initialization...\r\n[ DPDK EAL parameters: perf -c 0x000f --file-prefix=spdk_pid18218 ]\r\nEAL: Multi-process socket /var/run/.spdk_pid18218_unix\r\nEAL: Probing VFIO support...\r\nInitializing NVMe Controllers\r\nEAL: PCI device 0000:01:00.0 on NUMA socket 0\r\nEAL:   probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:01:00.0\r\nAttached to NVMe Controller at 0000:01:00.0 [8086:0953]\r\nEAL: PCI device 0000:03:00.0 on NUMA socket 0\r\nEAL:   probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:03:00.0\r\nAttached to NVMe Controller at 0000:03:00.0 [8086:0953]\r\nEAL: PCI device 0000:82:00.0 on NUMA socket 1\r\nEAL:   probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:82:00.0\r\nAttached to NVMe Controller at 0000:82:00.0 [8086:0953]\r\nEAL: PCI device 0000:84:00.0 on NUMA socket 1\r\nEAL:   probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:84:00.0\r\nAttached to NVMe Controller at 0000:84:00.0 [8086:0953]\r\nAssociating INTEL SSDPE2MD400G4  (CVFT61220012400GGN  ) with lcore 3\r\nAssociating INTEL SSDPE2MD400G4  (CVFT4400002C400GGN  ) with lcore 2\r\nAssociating INTEL SSDPE2MD400G4  (CVFT7021006H400PGN  ) with lcore 1\r\nAssociating INTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) with lcore 0\r\nInitialization complete. Launching workers.\r\nStarting thread on core 3\r\nStarting thread on core 2\r\nStarting thread on core 1\r\nStarting thread on core 0\r\n\\========================================================\r\n                                                                                            Latency(us)\r\nDevice Information                                     :       IOPS       MB/s    Average        min        max\r\nINTEL SSDPE2MD400G4  (CVFT61220012400GGN  ) from core 3:  149532.80     584.11     855.93       5.04   21698.07\r\nINTEL SSDPE2MD400G4  (CVFT4400002C400GGN  ) from core 2:  145423.03     568.06     880.12       5.07   19903.95\r\nINTEL SSDPE2MD400G4  (CVFT7021006H400PGN  ) from core 1:  148992.53     582.00     859.01       4.70   19746.74\r\nINTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) from core 0:  146920.86     573.91     871.14       4.72   19918.79\r\n\\========================================================\r\nTotal                                                  :  590869.23    2308.08     866.44       4.70   21698.07\r\n\r\n\r\nSummary latency data for INTEL SSDPE2MD400G4  (CVFT61220012400GGN  ) from core 3:\r\n\\=================================================================================\r\n  1.00000% :     7.467us\r\n 10.00000% :    13.653us\r\n 25.00000% :    31.451us\r\n 50.00000% :   456.411us\r\n 75.00000% :  1365.333us\r\n 90.00000% :  2465.402us\r\n 95.00000% :  2949.120us\r\n 98.00000% :  3604.480us\r\n 99.00000% :  4213.029us\r\n 99.50000% :  4899.596us\r\n 99.90000% :  7614.659us\r\n 99.99000% : 10360.930us\r\n 99.99900% : 13918.598us\r\n 99.99990% : 19848.046us\r\n 99.99999% : 21720.503us\r\n\r\nSummary latency data for INTEL SSDPE2MD400G4  (CVFT4400002C400GGN  ) from core 2:\r\n\\=================================================================================\r\n  1.00000% :     7.345us\r\n 10.00000% :    12.739us\r\n 25.00000% :    27.550us\r\n 50.00000% :   470.065us\r\n 75.00000% :  1435.550us\r\n 90.00000% :  2527.817us\r\n 95.00000% :  2995.931us\r\n 98.00000% :  3635.688us\r\n 99.00000% :  4275.444us\r\n 99.50000% :  4930.804us\r\n 99.90000% :  7458.621us\r\n 99.99000% : 10298.514us\r\n 99.99900% : 13544.107us\r\n 99.99990% : 19348.724us\r\n 99.99999% : 19723.215us\r\n\r\nSummary latency data for INTEL SSDPE2MD400G4  (CVFT7021006H400PGN  ) from core 1:\r\n\\=================================================================================\r\n  1.00000% :     6.888us\r\n 10.00000% :    13.410us\r\n 25.00000% :    33.890us\r\n 50.00000% :   440.808us\r\n 75.00000% :  1326.324us\r\n 90.00000% :  2543.421us\r\n 95.00000% :  3042.743us\r\n 98.00000% :  3713.707us\r\n 99.00000% :  4337.859us\r\n 99.50000% :  4993.219us\r\n 99.90000% :  7333.790us\r\n 99.99000% : 10360.930us\r\n 99.99900% : 14105.844us\r\n 99.99990% : 19348.724us\r\n 99.99999% : 19848.046us\r\n\r\nSummary latency data for INTEL SSDPE2MD400G4  (CVFT4396003G400GGN  ) from core 0:\r\n\\=================================================================================\r\n  1.00000% :     6.857us\r\n 10.00000% :    13.166us\r\n 25.00000% :    32.427us\r\n 50.00000% :   452.510us\r\n 75.00000% :  1388.739us\r\n 90.00000% :  2527.817us\r\n 95.00000% :  3011.535us\r\n 98.00000% :  3698.103us\r\n 99.00000% :  4337.859us\r\n 99.50000% :  5024.427us\r\n 99.90000% :  7583.451us\r\n 99.99000% : 10485.760us\r\n 99.99900% : 12857.539us\r\n 99.99990% : 18474.910us\r\n 99.99999% : 19972.876us\r\n\r\n\r\n## Expected Behavior\r\n<!--- Tell us what should happen -->\r\nWe should get stable IOPS in specification.\r\n\r\n## Current Behavior\r\n<!--- Tell us what happens instead of the expected behavior -->\r\nPerf random write IOPS is over than NVMe device spec.\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\nN/A\r\n\r\n## Steps to Reproduce\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\n1. Build SPDK with DPDK\r\nWe build SPDK & DPDK in reference to below pages.\r\nhttps://github.com/spdk/spdk/blob/master/README.md\r\nhttp://dpdk.org/doc/quick-start\r\n2. Hugepages and Device Binding\r\nHUGEMEM=20480 /root/spdk-master/scripts/setup.sh\r\n3. Preconditioning of NVMe devices\r\n/root/spdk-master/examples/nvme/perf/perf -q 32 -s 4096 -w randwrite -t 5400 -c 0xf\r\n4. Run Perf\r\n/root/spdk-master/examples/nvme/perf/perf -q ${IODEPTH} -s 4096 -w randwrtie -t 600 -c 0x${COREMASK} -D -LL -r 'trtype:PCIe traddr:0000.01.00.0' -r 'trtype:PCIe traddr:0000.03.00.0' -r 'trtype:PCIe traddr:0000.82.00.0' -r 'trtype:PCIe traddr:0000.84.00.0'\r\n\r\n*We run perf repeatedly changing CPUMASK, IOPDEPTH parameters and the number of NVMe devices. EX: when the number of NVMe devices is 4, we adjust (CPUMASK : IODEPTH) as followings so that the value of \"(# of cores) * IODEPTH / (# of NVMe)\" set to be 128.\r\n    (ffff:32)->(00ff:64)->(000f:128)->(0003:256)->(0001:512)\r\nWe run perf in order of the most NVMes first and reduce the number of NVMes in subsequent test so that the NVMes we are using can stay active. We have 4 NVMe devices so we changed the number of target NVMes as 4, 2, 1.\r\n\r\n## Context (Environment including OS version, SPDK version, etc.)\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\nOS: Red Hat Enterprise Linux 7.3\r\nLinux Kernel: 4.15.1\r\nSPDK: 18.04 (Unzip and build spdk-master.zip downloaded on April 24th, 2018 from SPDK github.)\r\nDPDK: 18.02.1\r\nServe model: Supermicro® 6028R-TDWNR (M/B: X10DDW-iN)\r\nProcessor: Intel® Xeon® E5-2620v4 (8core/16thread/2.1GHz/20MB cache) x2\r\nMemory: 64GiB (DDR4-2400 8GiB x 8DIMMs)\r\nNVMe: Intel® SSD DC P3700 Series 400GB(2.5inch MLC PCIe NVMe 3.0) x4\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/spdk/spdk/issues/comments/392103995","html_url":"https://github.com/spdk/spdk/issues/297#issuecomment-392103995","issue_url":"https://api.github.com/repos/spdk/spdk/issues/297","id":392103995,"user":{"login":"jimharris","id":1102825,"avatar_url":"https://avatars0.githubusercontent.com/u/1102825?v=4","gravatar_id":"","url":"https://api.github.com/users/jimharris","html_url":"https://github.com/jimharris","followers_url":"https://api.github.com/users/jimharris/followers","following_url":"https://api.github.com/users/jimharris/following{/other_user}","gists_url":"https://api.github.com/users/jimharris/gists{/gist_id}","starred_url":"https://api.github.com/users/jimharris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jimharris/subscriptions","organizations_url":"https://api.github.com/users/jimharris/orgs","repos_url":"https://api.github.com/users/jimharris/repos","events_url":"https://api.github.com/users/jimharris/events{/privacy}","received_events_url":"https://api.github.com/users/jimharris/received_events","type":"User","site_admin":false},"created_at":"2018-05-25T16:00:37Z","updated_at":"2018-05-25T16:00:37Z","author_association":"CONTRIBUTOR","body":"Hi Taisuke-san,\r\n\r\nFor the 108K data point – it looks like this was run for 10 minutes (-t 600)?  Was this also with 90 minutes pre-conditioning?  I just want to make sure the higher performance is not related to lack of preconditioning or that the test was not run long enough.\r\n\r\nNote that even with using the stronger RNG – we still do not truly match fio’s randommap.  The randommap makes sure that every LBA is written to once, before any LBA is written to a second time.\r\n\r\nSo for this test below, it ran for 600 seconds at 107551 IO/s = 64.53M IO.  The disk size is 97.7M IO.  For fio, all 64.53M IO would be to different LBA – but for perf, at least some of those IO would be duplicates.  The duplicate I/O means the SSD has fewer LBAs to rearrange during garbage collection which could help performance.\r\n\r\nThe only way to confirm this is to write a randommap implementation for SPDK perf.  Is that something you or Shuhei-san would be interested in experimenting with?\r\n\r\n-Jim\r\n\r\n\r\nFrom: tafukuyama <notifications@github.com>\r\nReply-To: spdk/spdk <reply@reply.github.com>\r\nDate: Thursday, May 24, 2018 at 2:38 AM\r\nTo: spdk/spdk <spdk@noreply.github.com>\r\nCc: James Harris <james.r.harris@intel.com>, Comment <comment@noreply.github.com>\r\nSubject: Re: [spdk/spdk] Random write IOPS by Perf is over than NVMe device spec. (#297)\r\n\r\n\r\nHi Jim san,\r\n\r\nThank you for your advice!\r\nWe've confirmed offset_in_ios value becomes always the same by calling srand48_r on each IO.\r\nmax_offset_in_ios= 11310718, min_offset_in_ios= 11310718, entry->size_in_ios= 97677846\r\n\r\nSo we modified the patch as follows to call srand48_r(0, &_seed) once in work_fn() as Matsumoto san mentioned.\r\n\r\n--- a/examples/nvme/perf/perf.c\r\n\r\n+++ b/examples/nvme/perf/perf.c\r\n\r\n@@ -43,6 +43,7 @@\r\n\r\n#include \"spdk/histogram_data.h\"\r\n\r\n#include \"spdk/endian.h\"\r\n\r\n#include \"spdk/crc16.h\"\r\n\r\n+#include \"stdlib.h\"\r\n\r\n\r\n\r\n #if HAVE_LIBAIO\r\n\r\n#include <libaio.h>\r\n\r\n@@ -619,6 +620,7 @@\r\n\r\nstatic void io_complete(void *ctx, const struct spdk_nvme_cpl *completion);\r\n\r\n\r\n\r\n static __thread unsigned int seed = 0;\r\n\r\n+static __thread struct drand48_data _seed = {};\r\n\r\n\r\n\r\n static void\r\n\r\nsubmit_single_io(struct perf_task *task)\r\n\r\n@@ -627,9 +629,15 @@\r\n\r\n        int                     rc;\r\n\r\n        struct ns_worker_ctx    *ns_ctx = task->ns_ctx;\r\n\r\n        struct ns_entry         *entry = ns_ctx->entry;\r\n\r\n+       long int randnum;\r\n\r\n\r\n\r\n        if (g_is_random) {\r\n\r\n-               offset_in_ios = rand_r(&seed) % entry->size_in_ios;\r\n\r\n+               /* offset_in_ios = rand_r(&seed) % entry->size_in_ios; */\r\n\r\n+               lrand48_r(&_seed, &randnum);\r\n\r\n+               offset_in_ios = randnum << 31;\r\n\r\n+               lrand48_r(&_seed, &randnum);\r\n\r\n+               offset_in_ios |= randnum;\r\n\r\n+               offset_in_ios %= entry->size_in_ios;\r\n\r\n        } else {\r\n\r\n                offset_in_ios = ns_ctx->offset_in_ios++;\r\n\r\n                if (ns_ctx->offset_in_ios == entry->size_in_ios) {\r\n\r\n@@ -853,6 +861,7 @@\r\n\r\n        struct ns_worker_ctx *ns_ctx = NULL;\r\n\r\n\r\n\r\n        printf(\"Starting thread on core %u\\n\", worker->lcore);\r\n\r\n+       srand48_r(0, &_seed);\r\n\r\n\r\n\r\n        /* Allocate a queue pair for each namespace. */\r\n\r\n        ns_ctx = worker->ns_ctx;\r\n\r\n\r\n\r\nBefore I rerun perf w/ new patch, I confirmed offset_in_ios values are covered almost same range as rand_r.\r\nrand_r: max_offset_in_ios= 97677811, min_offset_in_ios= 5, entry->size_in_ios= 97677846\r\nlrand48_r: max_offset_in_ios= 97677778, min_offset_in_ios= 66, entry->size_in_ios= 97677846\r\n\r\nTest results of perf + lrand48_r (w/ enough precondition) are shown in below.\r\nWe got 108KIOPS at NVMe x1, single core, 4K random write.\r\n\r\n[mage]<https://user-images.githubusercontent.com/29906937/40474266-861a3a94-5f79-11e8-8834-5e1bb0fcc48f.png>\r\n\r\n[root@dev16 perf_lrand48_r]# /root/spdk-master/examples/nvme/perf/perf -q 128 -s 4096 -w randwrite -t 600 -c 0x0001 -r 'trtype:PCIe traddr:0000.01.00.0'\r\nStarting SPDK v18.07-pre / DPDK 18.02.1 initialization...\r\n[ DPDK EAL parameters: perf -c 0x0001 --file-prefix=spdk_pid5962 ]\r\nEAL: Multi-process socket /var/run/.spdk_pid5962_unix\r\nEAL: Probing VFIO support...\r\nInitializing NVMe Controllers\r\nEAL: PCI device 0000:01:00.0 on NUMA socket 0\r\nEAL: probe driver: 8086:953 spdk_nvme\r\nAttaching to NVMe Controller at 0000:01:00.0\r\nAttached to NVMe Controller at 0000:01:00.0 [8086:0953]\r\nAssociating INTEL SSDPE2MD400G4 (CVFT7021006H400PGN ) with lcore 0\r\nInitialization complete. Launching workers.\r\nStarting thread on core 0\r\n========================================================\r\nLatency(us)\r\nDevice Information : IOPS MB/s Average min max\r\nINTEL SSDPE2MD400G4 (CVFT7021006H400PGN ) from core 0: 107551.77 420.12 1190.07 4.76 23427.06\r\n========================================================\r\nTotal : 107551.77 420.12 1190.07 4.76 23427.06\r\n\r\n------------------------------------------------\r\n\r\nSo calling lrand48_r x2 on each IO made IOPS lower than calling rand_r.\r\nHowever, the tendency of IOPS by increasing # of cores is more similar to perf + rand_r than fio_plugin, and IOPS is still over than NVMe device spec.\r\nSo we’re now just wondering if improving RNG quality solves this issue.\r\n\r\nBR,\r\nTaisuke Fukuyama\r\n\r\n—\r\nYou are receiving this because you commented.\r\nReply to this email directly, view it on GitHub<https://github.com/spdk/spdk/issues/297#issuecomment-391653019>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABDT6eDHOqtx4JaEHdCcNeWouaaeZoM0ks5t1n-IgaJpZM4T9Iou>.\r\n"}}
{"actor":{"display_login":"jimharris"}}