{"action":"created","issue":{"url":"https://api.github.com/repos/spdk/spdk/issues/264","repository_url":"https://api.github.com/repos/spdk/spdk","labels_url":"https://api.github.com/repos/spdk/spdk/issues/264/labels{/name}","comments_url":"https://api.github.com/repos/spdk/spdk/issues/264/comments","events_url":"https://api.github.com/repos/spdk/spdk/issues/264/events","html_url":"https://github.com/spdk/spdk/issues/264","id":305648195,"number":264,"title":"Nvmf target sees Remote access error while running perf","user":{"login":"rajeshsai810","id":37031402,"avatar_url":"https://avatars3.githubusercontent.com/u/37031402?v=4","gravatar_id":"","url":"https://api.github.com/users/rajeshsai810","html_url":"https://github.com/rajeshsai810","followers_url":"https://api.github.com/users/rajeshsai810/followers","following_url":"https://api.github.com/users/rajeshsai810/following{/other_user}","gists_url":"https://api.github.com/users/rajeshsai810/gists{/gist_id}","starred_url":"https://api.github.com/users/rajeshsai810/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rajeshsai810/subscriptions","organizations_url":"https://api.github.com/users/rajeshsai810/orgs","repos_url":"https://api.github.com/users/rajeshsai810/repos","events_url":"https://api.github.com/users/rajeshsai810/events{/privacy}","received_events_url":"https://api.github.com/users/rajeshsai810/received_events","type":"User","site_admin":false},"labels":[{"id":235898962,"url":"https://api.github.com/repos/spdk/spdk/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-03-15T17:26:40Z","updated_at":"2018-04-19T04:04:59Z","closed_at":null,"author_association":"NONE","body":"I see remote access error after running 8 iterations of perf application. Each iteration would run perf application with queue_depth == 64, size == 131072 and time 300seconds.\r\n\r\nI dont think there is any setup issue or else the test would not run for ~30mins. The target node is seeing remote access error that means some buffer (used by perf application) is either out of protection domain or freed or swapped? How can I know what buffer has caused this issue? Task buffer of perf application? or internal send/recv buffers? I take any buffer gets allocated on hugepages and so they should be pinned.\r\n\r\nMy setup: (Linux <-> Linux)\r\n\r\nLinux: Debian 4.9\r\nRDMA NIC: CX-5\r\nSPDK - v18.01 & master\r\nNVMe disks - 12 1TB disks from Samsung \r\n\r\nBoth initiator & target are connected back-to-back.\r\n\r\nConf file:\r\n~~~\r\n[Nvmf]\r\n  # Set the maximum number of submission and completion queues per session.\r\n  # Setting this to '8', for example, allows for 8 submission and 8 completion queues\r\n  # per session.\r\n  MaxQueuesPerSession 4\r\n\r\n  # Set the maximum number of outstanding I/O per queue.\r\n  #MaxQueueDepth 128\r\n\r\n  # Set the maximum in-capsule data size. Must be a multiple of 16.\r\n  #InCapsuleDataSize 4096\r\n\r\n  # Set the maximum I/O size. Must be a multiple of 4096.\r\n  MaxIOSize 1048576\r\n\r\n  # Set the global acceptor lcore ID, lcores are numbered starting at 0.\r\n  #AcceptorCore r09\r\n\r\n  # Set how often the acceptor polls for incoming connections. The acceptor is also\r\n  # responsible for polling existing connections that have gone idle. 0 means continuously\r\n  # poll. Units in microseconds.\r\n  AcceptorPollRate 10000\r\n~~~\r\nInitiator: \r\n\r\nRun perf application in loop. \r\n~~~\r\n#!/bin/bash\r\ncount=1\r\nwhile true; do\r\necho $count\r\n./perf -q 64 -s 131072 -w write -t 300 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.22.55 trsvcid:4420'\r\nlet count++\r\ndone\r\n\r\n...\r\nnvme_rdma.c:1521:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x557dcf57fca0, qpn: 557, Response Index 1 (5): Work Request Flushed Error\r\nnvme_rdma.c:1521:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x557dcf57fca0, qpn: 557, Response Index 65 (5): Work Request Flushed Error\r\n~~~\r\nTarget side:\r\n~~~\r\nroot@localhost:~/spdk# ./app/nvmf_tgt/nvmf_tgt -c etc/spdk/nvmf_eth0.conf.in\r\n...\r\n...\r\nmlx5: localhost: got completion with error:\r\n00000000 00000000 00000000 00000000\r\n00000000 00000000 00000000 00000000\r\n00000014 00000000 00000000 00000000\r\n00000000 00008813 1000059c 0035e5d3\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649295056 (10): remote access error vendor_err: 136\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649295424 (5): Work Request Flushed Error vendor_err: 249\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649295792 (5): Work Request Flushed Error vendor_err: 249\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649296160 (5): Work Request Flushed Error vendor_err: 249\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649296528 (5): Work Request Flushed Error vendor_err: 249\r\nrdma.c:1802:spdk_nvmf_rdma_poller_poll: *ERROR*: CQ error on CQ 0x7f1304000a60, qp_num: 1436, Request 0x139719649296896 (5): Work Request Flushed Error vendor_err: 249\r\n~~~\r\nLet me know if you need any more information."},"comment":{"url":"https://api.github.com/repos/spdk/spdk/issues/comments/382603746","html_url":"https://github.com/spdk/spdk/issues/264#issuecomment-382603746","issue_url":"https://api.github.com/repos/spdk/spdk/issues/264","id":382603746,"user":{"login":"wanqunintel","id":18566118,"avatar_url":"https://avatars3.githubusercontent.com/u/18566118?v=4","gravatar_id":"","url":"https://api.github.com/users/wanqunintel","html_url":"https://github.com/wanqunintel","followers_url":"https://api.github.com/users/wanqunintel/followers","following_url":"https://api.github.com/users/wanqunintel/following{/other_user}","gists_url":"https://api.github.com/users/wanqunintel/gists{/gist_id}","starred_url":"https://api.github.com/users/wanqunintel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanqunintel/subscriptions","organizations_url":"https://api.github.com/users/wanqunintel/orgs","repos_url":"https://api.github.com/users/wanqunintel/repos","events_url":"https://api.github.com/users/wanqunintel/events{/privacy}","received_events_url":"https://api.github.com/users/wanqunintel/received_events","type":"User","site_admin":false},"created_at":"2018-04-19T04:04:59Z","updated_at":"2018-04-19T04:04:59Z","author_association":"NONE","body":"we are reproducing this issue now and it takes some time, we will update later"}}
{"actor":{"display_login":"wanqunintel"}}