{"action":"created","issue":{"url":"https://api.github.com/repos/spdk/spdk/issues/527","repository_url":"https://api.github.com/repos/spdk/spdk","labels_url":"https://api.github.com/repos/spdk/spdk/issues/527/labels{/name}","comments_url":"https://api.github.com/repos/spdk/spdk/issues/527/comments","events_url":"https://api.github.com/repos/spdk/spdk/issues/527/events","html_url":"https://github.com/spdk/spdk/issues/527","id":386031258,"node_id":"MDU6SXNzdWUzODYwMzEyNTg=","number":527,"title":"Nvme disk Hot remove issue","user":{"login":"cockroach1136","id":29791158,"node_id":"MDQ6VXNlcjI5NzkxMTU4","avatar_url":"https://avatars0.githubusercontent.com/u/29791158?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach1136","html_url":"https://github.com/cockroach1136","followers_url":"https://api.github.com/users/cockroach1136/followers","following_url":"https://api.github.com/users/cockroach1136/following{/other_user}","gists_url":"https://api.github.com/users/cockroach1136/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach1136/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach1136/subscriptions","organizations_url":"https://api.github.com/users/cockroach1136/orgs","repos_url":"https://api.github.com/users/cockroach1136/repos","events_url":"https://api.github.com/users/cockroach1136/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach1136/received_events","type":"User","site_admin":false},"labels":[{"id":235898967,"node_id":"MDU6TGFiZWwyMzU4OTg5Njc=","url":"https://api.github.com/repos/spdk/spdk/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":{"login":"liang2yan","id":29114155,"node_id":"MDQ6VXNlcjI5MTE0MTU1","avatar_url":"https://avatars2.githubusercontent.com/u/29114155?v=4","gravatar_id":"","url":"https://api.github.com/users/liang2yan","html_url":"https://github.com/liang2yan","followers_url":"https://api.github.com/users/liang2yan/followers","following_url":"https://api.github.com/users/liang2yan/following{/other_user}","gists_url":"https://api.github.com/users/liang2yan/gists{/gist_id}","starred_url":"https://api.github.com/users/liang2yan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liang2yan/subscriptions","organizations_url":"https://api.github.com/users/liang2yan/orgs","repos_url":"https://api.github.com/users/liang2yan/repos","events_url":"https://api.github.com/users/liang2yan/events{/privacy}","received_events_url":"https://api.github.com/users/liang2yan/received_events","type":"User","site_admin":false},"assignees":[{"login":"liang2yan","id":29114155,"node_id":"MDQ6VXNlcjI5MTE0MTU1","avatar_url":"https://avatars2.githubusercontent.com/u/29114155?v=4","gravatar_id":"","url":"https://api.github.com/users/liang2yan","html_url":"https://github.com/liang2yan","followers_url":"https://api.github.com/users/liang2yan/followers","following_url":"https://api.github.com/users/liang2yan/following{/other_user}","gists_url":"https://api.github.com/users/liang2yan/gists{/gist_id}","starred_url":"https://api.github.com/users/liang2yan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liang2yan/subscriptions","organizations_url":"https://api.github.com/users/liang2yan/orgs","repos_url":"https://api.github.com/users/liang2yan/repos","events_url":"https://api.github.com/users/liang2yan/events{/privacy}","received_events_url":"https://api.github.com/users/liang2yan/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-11-30T04:17:22Z","updated_at":"2018-12-13T07:39:57Z","closed_at":null,"author_association":"NONE","body":"Hello all,\r\n    Recently we are stuck in the issue of hot remove one disk, the system will crash.\r\nThe phenomenon is strage.\r\nWe constrcut a RAID system of multi-disks and multi-threads\r\nEvery thread open  io_channels to every disk.\r\nSo, if we have 8 disks and 8 spdk reacors,\r\nThere are 8X8=64 io_channels in our system.\r\n\r\nThe hot remove procedure in our system is\r\n(1)master core receive the hot remove cb from hotplug thread\r\n(2)master core send message to every core to stop sending io to the hot removed disk\r\n(3)every core wait for the all IOs is aborted from the io_channel , close io channel , and send message to master core to indicate the IO related to this core is done.\r\n(4) master core collect the message came from every core, start unregisted_bdev\r\n\r\nThe hot remove in low IO loading is OK, But if we full stress the system, there is a strange phenomenon\r\n\r\nHot remove disk A, But the IOs in disk B will be blocked.\r\nWe add some track code in spdk nvme driver, we found that when we **hotremove disk A,** the disk B will receive invalid CID in the nvme complete queue.\r\nThis is the reason why system crash.\r\n\r\nbelow is the message we logged from disk B nvme complete queue \r\n---------------------------------------------------------------------------------------------------\r\nitem 208 SUCCESS (00/00) sqid:6 cid:111 cdw0:0 sqhd:0209 p:0 m:0 dnr:0\r\nitem 209 SUCCESS (00/00) sqid:6 cid:110 cdw0:0 sqhd:0210 p:0 m:0 dnr:0\r\nitem 210 INVALID FIELD (00/02) sqid:6 cid:44979 cdw0:0 sqhd:0211 p:0 m:0 dnr:1\r\nitem 211 SUCCESS (00/00) sqid:6 cid:172 cdw0:0 sqhd:0212 p:1 m:0 dnr:0\r\n----------------------------------------------------------------------------------------------\r\nWe found there is a invalid command  and strange CID in item 210\r\nWe also check the data in submit queue, the data seems correct.\r\n\r\nWe had try to export 8 disks from iscsi individually and stress the system, hot remove one\r\ndisk, it seems fine.\r\n\r\nWe also try the test case in hotplug demo code in spdk source, it seems ok too. \r\n\r\nWe had turn on the enable-asan flag, there are no  error found.\r\n\r\nWe think maybe the problem is related to we use **many IO queues for a disk** ??\r\n \r\nPlease give us advise.\r\nany hint is appreciated\r\nThanks in advance"},"comment":{"url":"https://api.github.com/repos/spdk/spdk/issues/comments/446871422","html_url":"https://github.com/spdk/spdk/issues/527#issuecomment-446871422","issue_url":"https://api.github.com/repos/spdk/spdk/issues/527","id":446871422,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0Njg3MTQyMg==","user":{"login":"cockroach1136","id":29791158,"node_id":"MDQ6VXNlcjI5NzkxMTU4","avatar_url":"https://avatars0.githubusercontent.com/u/29791158?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach1136","html_url":"https://github.com/cockroach1136","followers_url":"https://api.github.com/users/cockroach1136/followers","following_url":"https://api.github.com/users/cockroach1136/following{/other_user}","gists_url":"https://api.github.com/users/cockroach1136/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach1136/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach1136/subscriptions","organizations_url":"https://api.github.com/users/cockroach1136/orgs","repos_url":"https://api.github.com/users/cockroach1136/repos","events_url":"https://api.github.com/users/cockroach1136/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach1136/received_events","type":"User","site_admin":false},"created_at":"2018-12-13T07:39:57Z","updated_at":"2018-12-13T07:39:57Z","author_association":"NONE","body":"Hello Ziye yang,  \r\n        Thank you so much for your attention to this problem\r\n After modified some code of spdk raid, now we can always reproduce the phenomenon. The modified RAID code is here \r\n[bdev_raid.zip](https://github.com/spdk/spdk/files/2674850/bdev_raid.zip)\r\nThe main modification of our code is to response user IO in early stage, before user data written to the disk. Please see the funcion \r\n_raid_bdev_submit_rw_request_my(struct spdk_io_channel *ch, struct spdk_bdev_io *bdev_io)\r\nin my code.\r\n------------------------------------------------------------------------------------------\r\nThe RAID write sequence is ploted in this .jpg\r\n![raid_write_sequence](https://user-images.githubusercontent.com/29791158/49921927-0edd8100-feea-11e8-999a-b8bfa0100c2c.jpg)\r\nWe hope this jpg can help you to understand our writing pattern.\r\n\r\nThe install procedure is as below: \r\n(1) construct 6 disk to a RAID\r\n(2) split this RAID into 16 partitions\r\n(3)export those 16 partitions to iscsi\r\n(4) use fio to send IOs to these iscsi target\r\n(5)suddenly remove one disk, You will see \r\nnvme_pcie.c:2099:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd\r\nnvme_qpair.c: 306:nvme_qpair_print_completion: *NOTICE*: INVALID FIELD (00/02) sqid:9 cid:28494 cdw0:0 sqhd:00fb p:1 m:0 dnr:1\r\nnvme_qpair.c: 306:nvme_qpair_print_completion: *NOTICE*: INVALID FIELD (00/02) sqid:6 cid:5931 cdw0:0 sqhd:00f3 p:0 m:0 dnr:1\r\n\r\nThe IOPS higher, the happen probability is getten higher.\r\nIn our environment, the IOPS can reach 580k IOPS (6 X intel 4510 disk)\r\n\r\nThank you so much for your help.\r\nIf you need any other information to clearify this issue, please tell me.\r\nI will prepare them ASAP.\r\nThe spdk rpc command , fio script and iscsi_tgt config file  is listed below. \r\nrun iscsi_tgt as\r\n\r\n./iscsi_tgt    -c  ./iscsi.conf.agg_raid -m 0xFFFF\r\n----------------\r\nconfig file\r\n[iscsi.conf.agg_raid.zip](https://github.com/spdk/spdk/files/2674926/iscsi.conf.agg_raid.zip)\r\n----------------------\r\nrpc command\r\n\r\n./scripts/rpc.py add_portal_group 0 127.0.0.1:3260;\r\n./scripts/rpc.py add_initiator_group 0 ANY ANY;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme1 -t pcie -a 0000:a2:00.0;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme2 -t pcie -a 0000:a5:00.0;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme3 -t pcie -a 0000:a8:00.0;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme4 -t pcie -a 0000:ab:00.0;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme5 -t pcie -a 0000:96:00.0 ;\r\n./scripts/rpc.py construct_nvme_bdev -b Nvme6 -t pcie -a 0000:99:00.0;\r\n./scripts/rpc.py  construct_raid_bdev -n ra0 -s 64 -r 0 -b \"Nvme1n1 Nvme2n1 Nvme3n1 Nvme4n1 Nvme5n1 Nvme6n1\";\r\n./scripts/rpc.py  construct_split_vbdev    -s 5120  ra0 16;\r\n ./scripts/rpc.py construct_target_node Dev0 Dev0 ra0p0:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev1 Dev1 ra0p1:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev2 Dev2 ra0p2:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev3 Dev3 ra0p3:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev4 Dev4 ra0p4:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev5 Dev5 ra0p5:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev6 Dev6 ra0p6:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev7 Dev7 ra0p7:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev8 Dev8 ra0p8:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev9 Dev9 ra0p9:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev10 Dev10 ra0p10:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev11 Dev11 ra0p11:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev12 Dev12 ra0p12:0 0:0 256;\r\n ./scripts/rpc.py construct_target_node Dev13 Dev13 ra0p13:0 0:0 256;\r\n\r\nfio config file\r\n-----------------------------------------------------------------------------------------\r\n[global]\r\nbs=4096\r\niodepth=32\r\nrw=randwrite\r\nsize=100%\r\nrwmixread=0\r\nnorandommap=1\r\nnumjobs=1\r\ndo_verify=0\r\nverify=null\r\nverify_dump=0\r\nverify_fatal=0\r\ncpus_allowed=\"30-40\"\r\nioengine=libaio\r\ndirect=1\r\nloops=1\r\nthread=1\r\ninvalidate=1\r\nsync=0\r\nrefill_buffers=0\r\nexitall_on_error=1\r\n[job0]\r\nfilename=/dev/sdb\r\n[job1]\r\nfilename=/dev/sdc\r\n[job2]\r\nfilename=/dev/sdd\r\n[job3]\r\nfilename=/dev/sde\r\n[job4]\r\nfilename=/dev/sdf\r\n[job5]\r\nfilename=/dev/sdg\r\n[job6]\r\nfilename=/dev/sdg\r\n[job7]\r\nfilename=/dev/sdh\r\n[job8]\r\nfilename=/dev/sdi\r\n[job9]\r\nfilename=/dev/sdj\r\n[job10]\r\nfilename=/dev/sdk\r\n[job11]\r\nfilename=/dev/sdl\r\n[job12]\r\nfilename=/dev/sdm\r\n[job13]\r\nfilename=/dev/sdn\r\n[job14]\r\nfilename=/dev/sdo\r\n[job15]\r\nfilename=/dev/sdq\r\n\r\n\r\n\r\n\r\n\r\n    \r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"cockroach1136"}}