{"action":"created","issue":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/2548","repository_url":"https://api.github.com/repos/ofiwg/libfabric","labels_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2548/labels{/name}","comments_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2548/comments","events_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2548/events","html_url":"https://github.com/ofiwg/libfabric/issues/2548","id":192368789,"number":2548,"title":"Extend memory region modes to enable hardware acceleration","user":{"login":"mblockso","id":9088179,"avatar_url":"https://avatars.githubusercontent.com/u/9088179?v=3","gravatar_id":"","url":"https://api.github.com/users/mblockso","html_url":"https://github.com/mblockso","followers_url":"https://api.github.com/users/mblockso/followers","following_url":"https://api.github.com/users/mblockso/following{/other_user}","gists_url":"https://api.github.com/users/mblockso/gists{/gist_id}","starred_url":"https://api.github.com/users/mblockso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mblockso/subscriptions","organizations_url":"https://api.github.com/users/mblockso/orgs","repos_url":"https://api.github.com/users/mblockso/repos","events_url":"https://api.github.com/users/mblockso/events{/privacy}","received_events_url":"https://api.github.com/users/mblockso/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/ofiwg/libfabric/milestones/14","html_url":"https://github.com/ofiwg/libfabric/milestone/14","labels_url":"https://api.github.com/repos/ofiwg/libfabric/milestones/14/labels","id":2029648,"number":14,"title":"release 1.5.0","description":"","creator":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"open_issues":39,"closed_issues":80,"state":"open","created_at":"2016-09-26T18:26:02Z","updated_at":"2016-12-23T00:36:21Z","due_on":null,"closed_at":null},"comments":24,"created_at":"2016-11-29T18:59:26Z","updated_at":"2016-12-23T14:54:52Z","closed_at":null,"body":"Continuing discussion from #2444 \r\n\r\nThe current memory region modes bundle two distinct attributes in their definition. The first is whether the key may be specified by the user or whether the key is generated by the provider and the user is required to exchange these keys with remote processes. The second is whether the remote memory is addressed using an offset or a virtual address.\r\n\r\nThe `FI_MR_SCALABLE` mode is defined as having a key that may be specified by the user and the remote memory is addressed using an offset into the remote memory region.\r\n\r\nThe `FI_MR_BASIC` mode is defined as having a key that is generated by the provider which must be exchanged and the remote memory is addressed using the virtual address of the remote memory.\r\n\r\nWhat needs to be added are the other two combinations of key type and addressing type. This will enable hardware acceleration on certain providers, such as bgq.\r\n\r\nThere are a few ways to enable these memory region modes. We could add a new `mr_addr` enum to the `fi_domain_attr` structure to specify offset- vs virtual- addressing. If the mr addr is unspecified, the value would default to the v1.4 libfabric definition of the mr_mode. In other words, if `FI_MR_SCALABLE` is specified for the `mr_mode` and the `mr_addr` is unspecified, then mr addr would default to \"offset\". Similarly, if `FI_MR_BASIC` is specified then the mr addr would default to \"virtual\".\r\n\r\nAnother option is to extend the `mr_mode` enum to include all combinations. For example:\r\n\r\n    enum fi_mr_mode {\r\n      FI_MR_UNSPEC = 0,\r\n      FI_MR_BASIC = 1,\r\n      FI_MR_SCALABLE = 2,\r\n      FI_MR_BASIC_VIRTUAL = 1,\r\n      FI_MR_SCALABLE_OFFSET = 2,\r\n      FI_MR_BASIC_OFFSET,\r\n      FI_MR_SCALABLE_VIRTUAL\r\n    };\r\n\r\nDuplicating the enum values for FI_MR_BASIC and FI_MR_SCALABLE maintains backward compatibility (assuming that the compiler assigns the values as expected!). Alternatively, providers can check for the v1.4-style mr enum and internally convert the value. In either case, the `FI_MR_SCALABLE` and `FI_MR_BASIC` enum values would remain valid.\r\n\r\n@shefty @sayantansur @hppritcha @pkcoff \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/comments/269001112","html_url":"https://github.com/ofiwg/libfabric/issues/2548#issuecomment-269001112","issue_url":"https://api.github.com/repos/ofiwg/libfabric/issues/2548","id":269001112,"user":{"login":"sdvormwa","id":20797388,"avatar_url":"https://avatars.githubusercontent.com/u/20797388?v=3","gravatar_id":"","url":"https://api.github.com/users/sdvormwa","html_url":"https://github.com/sdvormwa","followers_url":"https://api.github.com/users/sdvormwa/followers","following_url":"https://api.github.com/users/sdvormwa/following{/other_user}","gists_url":"https://api.github.com/users/sdvormwa/gists{/gist_id}","starred_url":"https://api.github.com/users/sdvormwa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdvormwa/subscriptions","organizations_url":"https://api.github.com/users/sdvormwa/orgs","repos_url":"https://api.github.com/users/sdvormwa/repos","events_url":"https://api.github.com/users/sdvormwa/events{/privacy}","received_events_url":"https://api.github.com/users/sdvormwa/received_events","type":"User","site_admin":false},"created_at":"2016-12-23T14:54:52Z","updated_at":"2016-12-23T14:54:52Z","body":"> This reminds me of the way MPI Dynamic windows work. Need to understand GNI better, but if it is indeed a direct 1:1 map with MPI, then an additional call to \"make real pages available\" for FI_MR_SCALABLE sounds reasonable. We should also inspect SHMEM and see if it can provide this additional call.\r\n\r\nI'm not very familiar with MPI Dynamic windows, but given how practical the MPI committee is, my expectation is they noticed that it is really hard for discrete network hardware to automatically update memory regions in response to page table updates since they can't use the processor's page table directly (unless the network vendor controls the processor as well...).  Network vendors could in theory handle this by intercepting page table updates in the OS and updating the region at that point with their OS drivers, but that potentially adds a lot of work to a very hot path in the OS for what is likely to be a rare scenario.  An additional call to \"make real pages available\" as you put it nicely solves the problem without interfering with too many other things by relying on the user to decide when the expense of the update is justified."}}
{"actor":{"display_login":"sdvormwa"}}