{"action":"created","issue":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/1645","repository_url":"https://api.github.com/repos/raffenet/mpich-migrate-test","labels_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/1645/labels{/name}","comments_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/1645/comments","events_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/1645/events","html_url":"https://github.com/raffenet/mpich-migrate-test/issues/1645","id":168681931,"number":1645,"title":"Can't scale up the number of requests","user":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"assignees":[{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/milestones/14","html_url":"https://github.com/raffenet/mpich-migrate-test/milestones/mpich2-1.5","labels_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/milestones/14/labels","id":1911844,"number":14,"title":"mpich2-1.5","description":"mpich2-1.5 will primarily be a feature release where a large number of new features will be introduced.  Most of the features are based on the upcoming MPI-3 standard, but some other additional features are also being considered.  Below is a wish list of all the features we are considering. Some of these might not make it into the 1.5 release, in which case they will be pushed to a future release.\n\n'''Milestone Major Goals:'''\n\n * '''MPI-3 Draft support'''\n    - ~~Nonblocking collectives [status: complete]~~\n    - ~~Tools interface [status: complete]~~\n    - ~~Mprobe [status: complete]~~\n    - ~~MPI-3 const bindings [status: complete]~~\n    - ~~Neighborhood collectives [status: complete]~~\n\n * '''Other features'''\n    - ~~Build system revamp [status: complete]~~\n    - ~~Support for the Intel MIC architecture [status: complete]~~\n    - ~~BG/Q integration [status: complete]~~","creator":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":56,"state":"closed","created_at":"2016-07-29T14:57:30Z","updated_at":"2016-08-01T16:03:13Z","due_on":"2012-10-08T05:00:00Z","closed_at":"2016-07-29T14:57:30Z"},"comments":8,"created_at":"2016-08-01T16:03:06Z","updated_at":"2016-08-01T16:03:14Z","closed_at":"2016-08-01T16:03:13Z","body":"**Originally by _bobc@us.ibm.com_ on 2012-07-03 10:14:42 -0500**\n\n___\nMPICH has a limit of 256K requests in mpihandlemem.h. I tried to scale it up for the HPCC MPIRandomAccess benchmark.  I am unable to set HANDLE_BLOCK_INDEX_SIZE past 1024 (so 1024 block size  x 1024 index size) despite the comments that it scales to 65536:\n\n```\n/* For indirect, the remainder of the handle has a block and index */\n#define HANDLE_INDIRECT_SHIFT 16\n#define HANDLE_BLOCK(a) (((a)& 0x03FF0000) >> HANDLE_INDIRECT_SHIFT)\n#define HANDLE_BLOCK_INDEX(a) ((a) & 0x0000FFFF)\n\n/* Handle block is between 1 and 1024 *elements* */\n#define HANDLE_BLOCK_SIZE 256\n/* Index size is bewtween 1 and 65536 *elements* */\n#define HANDLE_BLOCK_INDEX_SIZE 1024\n```\n\nThere appears to be come confusion in the code between block and index.  If I trace the macros/constants after init from mpid_isend.h it's stuffing the index into the 0x3FF field not the 0xFFFF field (and thus wrapping at 1024 indices).  \n\nSo when I waitall on the 1M (+512 direct requests) request it's an already completed request.\n\nOutput using smaller block (128)/index sizes (4):\n```\nCreating indirect table with 4 entries\nAdding indirect block 0 of size 128 (obj_size 232)\nrequest=0XEC00007F, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 127, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC00007E, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 126, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC00007D, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 125, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC00007C, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 124, HANDLE_BLOCK_INDEX_SIZE 4\n...\nrequest=0XEC000001, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 1, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC000000, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\nAdding indirect block 1 of size 128 (obj_size 232)\nrequest=0XEC01007F, HANDLE_BLOCK 1, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 127, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC01007E, HANDLE_BLOCK 1, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 126, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC01007D, HANDLE_BLOCK 1, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 125, HANDLE_BLOCK_INDEX_SIZE 4\n```\n\nI found two ways to scale past 1M requests.\n\nFirst (ignoring comments) this works for 2M:\n\n```\n/* Handle block is between 1 and 1024 *elements* */\n#define HANDLE_BLOCK_SIZE 2048\n/* Index size is bewtween 1 and 65536 *elements* */\n#define HANDLE_BLOCK_INDEX_SIZE 1024\n```\n\nor fixing handlemem.c's init and get functions:\n\n```\ndiff --git a/mpich2/src/util/mem/handlemem.c b/mpich2/src/util/mem/handlemem.c\nindex 4bc0060..514662d 100644\n--- a/mpich2/src/util/mem/handlemem.c\n+++ b/mpich2/src/util/mem/handlemem.c\n@@ -212,7 +212,8 @@ static void *MPIU_Handle_indirect_init( void *(**indirect)[],\n        hptr->next = ptr;\n        hptr->handle   = ((unsigned)HANDLE_KIND_INDIRECT << HANDLE_KIND_SHIFT) | \n            (handle_type << HANDLE_MPI_KIND_SHIFT) | \n-           (*indirect_size << HANDLE_INDIRECT_SHIFT) | i;\n+            (i << HANDLE_INDIRECT_SHIFT) | *indirect_size;\n \n         MPIU_HANDLE_VG_LABEL(hptr, obj_size, handle_type, 0);\n     }\n@@ -482,13 +483,13 @@ void *MPIU_Handle_get_ptr_indirect( int handle, MPIU_Object_alloc_t *objmem )\n     }\n \n     /* Find the block */\n-    block_num = HANDLE_BLOCK(handle);\n+    block_num = HANDLE_BLOCK_INDEX(handle);\n     if (block_num >= objmem->indirect_size) {\n        return 0;\n     }\n     \n     /* Find the entry */\n-    index_num = HANDLE_BLOCK_INDEX(handle);\n+    index_num = HANDLE_BLOCK(handle);\n     /* If we could declare the blocks to a known size object, we\n      could do something like \n        return &( (MPID_Info**)*MPIU_Info_mem.indirect)[block_num][index_num];\n```\n\nWhich traced in isend is:\n\n```\nCreating indirect table with 4 entries\nAdding indirect block 0 of size 128 (obj_size 232)\nrequest=0XEC7F0000, HANDLE_BLOCK 127, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC7E0000, HANDLE_BLOCK 126, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC7D0000, HANDLE_BLOCK 125, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\n...\nrequest=0XEC010000, HANDLE_BLOCK 1, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC000000, HANDLE_BLOCK 0, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 0, HANDLE_BLOCK_INDEX_SIZE 4\nAdding indirect block 1 of size 128 (obj_size 232)\nrequest=0XEC7F0001, HANDLE_BLOCK 127, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 1, HANDLE_BLOCK_INDEX_SIZE 4\nrequest=0XEC7E0001, HANDLE_BLOCK 126, HANDLE_BLOCK_SIZE 128 HANDLE_BLOCK_INDEX 1, HANDLE_BLOCK_INDEX_SIZE 4\n```\n"},"comment":{"url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/comments/236625410","html_url":"https://github.com/raffenet/mpich-migrate-test/issues/1645#issuecomment-236625410","issue_url":"https://api.github.com/repos/raffenet/mpich-migrate-test/issues/1645","id":236625410,"user":{"login":"raffenet","id":1410792,"avatar_url":"https://avatars.githubusercontent.com/u/1410792?v=3","gravatar_id":"","url":"https://api.github.com/users/raffenet","html_url":"https://github.com/raffenet","followers_url":"https://api.github.com/users/raffenet/followers","following_url":"https://api.github.com/users/raffenet/following{/other_user}","gists_url":"https://api.github.com/users/raffenet/gists{/gist_id}","starred_url":"https://api.github.com/users/raffenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raffenet/subscriptions","organizations_url":"https://api.github.com/users/raffenet/orgs","repos_url":"https://api.github.com/users/raffenet/repos","events_url":"https://api.github.com/users/raffenet/events{/privacy}","received_events_url":"https://api.github.com/users/raffenet/received_events","type":"User","site_admin":false},"created_at":"2016-08-01T16:03:14Z","updated_at":"2016-08-01T16:03:14Z","body":"**Originally by _bobc@us.ibm.com_ on 2012-08-07 15:09:24 -0500**\n\n___\nThat's what I did too in a local branch - move the 4 bits.  For my testing, I was only using 4M requests.\n```\n+#define HANDLE_NUM_BLOCKS 8192\n+#define HANDLE_NUM_INDICES 512\n```\nWe still need some independent work on BlueGene to decide on our final values."}}
{"actor":{"display_login":"raffenet"}}