{"action":"created","issue":{"url":"https://api.github.com/repos/spdk/spdk/issues/331","repository_url":"https://api.github.com/repos/spdk/spdk","labels_url":"https://api.github.com/repos/spdk/spdk/issues/331/labels{/name}","comments_url":"https://api.github.com/repos/spdk/spdk/issues/331/comments","events_url":"https://api.github.com/repos/spdk/spdk/issues/331/events","html_url":"https://github.com/spdk/spdk/issues/331","id":332710506,"node_id":"MDU6SXNzdWUzMzI3MTA1MDY=","number":331,"title":"NVMe-oF target: InCapsuleDataSize 0 does not work","user":{"login":"senthilkumar-v","id":34639974,"node_id":"MDQ6VXNlcjM0NjM5OTc0","avatar_url":"https://avatars0.githubusercontent.com/u/34639974?v=4","gravatar_id":"","url":"https://api.github.com/users/senthilkumar-v","html_url":"https://github.com/senthilkumar-v","followers_url":"https://api.github.com/users/senthilkumar-v/followers","following_url":"https://api.github.com/users/senthilkumar-v/following{/other_user}","gists_url":"https://api.github.com/users/senthilkumar-v/gists{/gist_id}","starred_url":"https://api.github.com/users/senthilkumar-v/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senthilkumar-v/subscriptions","organizations_url":"https://api.github.com/users/senthilkumar-v/orgs","repos_url":"https://api.github.com/users/senthilkumar-v/repos","events_url":"https://api.github.com/users/senthilkumar-v/events{/privacy}","received_events_url":"https://api.github.com/users/senthilkumar-v/received_events","type":"User","site_admin":false},"labels":[{"id":235898962,"node_id":"MDU6TGFiZWwyMzU4OTg5NjI=","url":"https://api.github.com/repos/spdk/spdk/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"xuhuagen","id":33302826,"node_id":"MDQ6VXNlcjMzMzAyODI2","avatar_url":"https://avatars3.githubusercontent.com/u/33302826?v=4","gravatar_id":"","url":"https://api.github.com/users/xuhuagen","html_url":"https://github.com/xuhuagen","followers_url":"https://api.github.com/users/xuhuagen/followers","following_url":"https://api.github.com/users/xuhuagen/following{/other_user}","gists_url":"https://api.github.com/users/xuhuagen/gists{/gist_id}","starred_url":"https://api.github.com/users/xuhuagen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuhuagen/subscriptions","organizations_url":"https://api.github.com/users/xuhuagen/orgs","repos_url":"https://api.github.com/users/xuhuagen/repos","events_url":"https://api.github.com/users/xuhuagen/events{/privacy}","received_events_url":"https://api.github.com/users/xuhuagen/received_events","type":"User","site_admin":false},"assignees":[{"login":"xuhuagen","id":33302826,"node_id":"MDQ6VXNlcjMzMzAyODI2","avatar_url":"https://avatars3.githubusercontent.com/u/33302826?v=4","gravatar_id":"","url":"https://api.github.com/users/xuhuagen","html_url":"https://github.com/xuhuagen","followers_url":"https://api.github.com/users/xuhuagen/followers","following_url":"https://api.github.com/users/xuhuagen/following{/other_user}","gists_url":"https://api.github.com/users/xuhuagen/gists{/gist_id}","starred_url":"https://api.github.com/users/xuhuagen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuhuagen/subscriptions","organizations_url":"https://api.github.com/users/xuhuagen/orgs","repos_url":"https://api.github.com/users/xuhuagen/repos","events_url":"https://api.github.com/users/xuhuagen/events{/privacy}","received_events_url":"https://api.github.com/users/xuhuagen/received_events","type":"User","site_admin":false},{"login":"senthilkumar-v","id":34639974,"node_id":"MDQ6VXNlcjM0NjM5OTc0","avatar_url":"https://avatars0.githubusercontent.com/u/34639974?v=4","gravatar_id":"","url":"https://api.github.com/users/senthilkumar-v","html_url":"https://github.com/senthilkumar-v","followers_url":"https://api.github.com/users/senthilkumar-v/followers","following_url":"https://api.github.com/users/senthilkumar-v/following{/other_user}","gists_url":"https://api.github.com/users/senthilkumar-v/gists{/gist_id}","starred_url":"https://api.github.com/users/senthilkumar-v/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senthilkumar-v/subscriptions","organizations_url":"https://api.github.com/users/senthilkumar-v/orgs","repos_url":"https://api.github.com/users/senthilkumar-v/repos","events_url":"https://api.github.com/users/senthilkumar-v/events{/privacy}","received_events_url":"https://api.github.com/users/senthilkumar-v/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2018-06-15T09:43:20Z","updated_at":"2018-06-20T02:53:09Z","closed_at":null,"author_association":"NONE","body":"\r\nHi,\r\n      I’ve a query regarding Nvmf “InCapsuleDataSize” and its implementation. From the SPDK conf file Nvmf section, the “InCapsuleDataSize” is defined as below,\r\n\r\n### Nvmf Conf. file\r\n[Nvmf]\r\n  ###Set the maximum number of outstanding I/O per queue.\r\n  MaxQueueDepth 128\r\n  ###Set the maximum in-capsule data size. Must be a multiple of 16.\r\n  ###0 is a valid choice.\r\n  InCapsuleDataSize 4096\r\n\r\nPlease note conf file comment says that “0 is a valid choice”. But when it is set to value zero i.e. InCapsuleDataSize 0, connect fails with below error.\r\n“Unable to allocate sufficient memory for RDMA queue.”.\r\n\r\n## Expected Behavior\r\n<!--- Tell us what should happen -->\r\nThe Nvmf connect is expected to pass.\r\n\r\n## Current Behavior\r\n<!--- Tell us what happens instead of the expected behavior -->\r\nThe connect fails with below error.\r\n“Unable to allocate sufficient memory for RDMA queue.”.\r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n\r\n### Probable root-cause\r\nFrom code walk-thru, the error is from below code section,\r\n\r\n[root@vm2 spdkgit]# git describe --tags\r\nv18.04-80-gf750c4e\r\n\r\nstatic int\r\nspdk_nvmf_rdma_qpair_initialize(struct spdk_nvmf_qpair *qpair)\r\n{\r\n…\r\n…\r\n  rqpair->bufs = spdk_dma_zmalloc(**_rqpair->max_queue_depth * rtransport->in_capsule_data_size,_**\r\n          0x1000, NULL);\r\n              if (!rqpair->reqs || !rqpair->recvs || !rqpair->cmds ||\r\n                         !rqpair->cpls || !rqpair->bufs) {\r\n               SPDK_ERRLOG(\"Unable to allocate sufficient memory for RDMA queue.\\n\");\r\n               spdk_nvmf_rdma_qpair_destroy(rqpair);\r\n                return -1;\r\n  }\r\n…\r\n…\r\n}\r\n\r\nThe root cause is in the above highlighted section of code where “rqpair->bufs” becomes zero when “in_capsule_data_size” is set to zero. \r\n\r\nI’ve cross-checked with “NVM Express over Fabrics 1.0” spec and didn’t see anywhere that it mandates to set “InCapsuleDataSize” to a non-zero value always.\r\n\r\nWith my above understanding, there seems to be a bug in above highlighted code where it should ignore “rtransport->in_capsule_data_size” if it is zero.\r\n\r\n### Probable fixes\r\n\r\n**Fix #1:-**\r\n\r\nrqpair->bufs = spdk_dma_zmalloc(rqpair->max_queue_depth * **_( rtransport->in_capsule_data_size ? rtransport->in_capsule_data_size : 1)_**,  0x1000, NULL);\r\n\r\nOR\r\n\r\n**Fix #2:-**\r\n\r\nIn /home/spdkgit/lib/event/subsystems/nvmf/conf.c file,\r\n\r\n  in_capsule_data_size = spdk_conf_section_get_intval(sp, \"InCapsuleDataSize\");\r\n  if (**_in_capsule_data_size > 0_**) {                  **_<====  No more checks for “equal to 0” condition._**\r\n    opts->in_capsule_data_size = in_capsule_data_size;\r\n  **_} else {\r\n    opts->in_capsule_data_size = 1;            <======== For zero & less than zero cases, it sets the value to 1.\r\n}_**\r\n\r\n**_Note-1:- The fix #1 scope is contained within that use case but fix #2 might have global impact compared to fix #1 so please suggest which is better way to fix._**\r\n\r\n## **_Other areas of code to consider_**\r\n\r\nSimilar bug could be in other places of code as shown below,\r\n\r\n1.\tWhile constructing “bufs_mr” as show below,\r\n  rqpair->bufs_mr = ibv_reg_mr(rqpair->cm_id->pd, rqpair->bufs,\r\n             **_rqpair->max_queue_depth * rtransport->in_capsule_data_size,_**\r\n             IBV_ACCESS_LOCAL_WRITE |\r\n             IBV_ACCESS_REMOTE_WRITE);\r\n\r\n**_Note-2:- I’m not sure how it’ll manifest in the field. May be we can apply fix #1(??), if it is a real bug here. Thoughts?_**\r\n\r\n2.\t**_Line 406: rdma_recv->buf = (void *)((uintptr_t)rqpair->bufs + (i * rtransport->in_capsule_data_size));_**\r\n        **_Line 413: rdma_recv->sgl[1].length = rtransport->in_capsule_data_size;_**\r\n\r\n**_Note-3:- I’m not sure whether there is any bug here but thought of highlight it to get the attention of wider audience._** \r\n\r\n3.\tstatic int\r\nspdk_nvmf_rdma_request_parse_sgl(struct spdk_nvmf_rdma_transport *rtransport,\r\n        struct spdk_nvmf_rdma_device *device,\r\n        struct spdk_nvmf_rdma_request *rdma_req)\r\n {\r\n     …\r\n     …\r\n      **_914     uint32_t max_len = rtransport->in_capsule_data_size;_**\r\n    ….\r\n   ….\r\n}\r\n\r\n**_Note-4:- I’m not sure about impact of “max_len” here but highlighting it to get wider attention._**\r\n\r\n4.\tstatic struct spdk_nvmf_transport *\r\n spdk_nvmf_rdma_create(struct spdk_nvmf_tgt *tgt)\r\n {\r\n     …\r\n    …\r\n    **_1141   rtransport->in_capsule_data_size = tgt->opts.in_capsule_data_size;_**\r\n   …\r\n   …\r\n}\r\n\r\n**_Note-5:- Again just highlighting it._**\r\n\r\n**_Let me know your thoughts about the above bug fix proposals (fix1 & fix2) and all 5 notes (Note 1-5)._**\r\n\r\n## Steps to Reproduce\r\n<!--- Provide a link to a live example, or an unambiguous set of steps to -->\r\n<!--- reproduce this bug. Include code to reproduce, if relevant -->\r\n1. Set \"InCapsuleDataSize\" value to 0.\r\n2. Issue connect.\r\n\r\n## Context (Environment including OS version, SPDK version, etc.)\r\n<!--- Providing context helps us come up with a solution that is most useful in the real world -->\r\n[root@vm2 spdkgit]# git describe --tags\r\nv18.04-80-gf750c4e"},"comment":{"url":"https://api.github.com/repos/spdk/spdk/issues/comments/398608051","html_url":"https://github.com/spdk/spdk/issues/331#issuecomment-398608051","issue_url":"https://api.github.com/repos/spdk/spdk/issues/331","id":398608051,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODYwODA1MQ==","user":{"login":"senthilkumar-v","id":34639974,"node_id":"MDQ6VXNlcjM0NjM5OTc0","avatar_url":"https://avatars0.githubusercontent.com/u/34639974?v=4","gravatar_id":"","url":"https://api.github.com/users/senthilkumar-v","html_url":"https://github.com/senthilkumar-v","followers_url":"https://api.github.com/users/senthilkumar-v/followers","following_url":"https://api.github.com/users/senthilkumar-v/following{/other_user}","gists_url":"https://api.github.com/users/senthilkumar-v/gists{/gist_id}","starred_url":"https://api.github.com/users/senthilkumar-v/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senthilkumar-v/subscriptions","organizations_url":"https://api.github.com/users/senthilkumar-v/orgs","repos_url":"https://api.github.com/users/senthilkumar-v/repos","events_url":"https://api.github.com/users/senthilkumar-v/events{/privacy}","received_events_url":"https://api.github.com/users/senthilkumar-v/received_events","type":"User","site_admin":false},"created_at":"2018-06-20T02:53:09Z","updated_at":"2018-06-20T02:53:09Z","author_association":"NONE","body":"Hi @wanqunintel, sure. I'll post it here. "}}
{"actor":{"display_login":"senthilkumar-v"}}