{"action":"created","issue":{"url":"https://api.github.com/repos/mercury-hpc/mercury/issues/23","repository_url":"https://api.github.com/repos/mercury-hpc/mercury","labels_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/23/comments","events_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/23/events","html_url":"https://github.com/mercury-hpc/mercury/issues/23","id":95748591,"number":23,"title":"Bulk data optimization","user":{"login":"soumagne","id":5613371,"avatar_url":"https://avatars.githubusercontent.com/u/5613371?v=3","gravatar_id":"","url":"https://api.github.com/users/soumagne","html_url":"https://github.com/soumagne","followers_url":"https://api.github.com/users/soumagne/followers","following_url":"https://api.github.com/users/soumagne/following{/other_user}","gists_url":"https://api.github.com/users/soumagne/gists{/gist_id}","starred_url":"https://api.github.com/users/soumagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumagne/subscriptions","organizations_url":"https://api.github.com/users/soumagne/orgs","repos_url":"https://api.github.com/users/soumagne/repos","events_url":"https://api.github.com/users/soumagne/events{/privacy}","received_events_url":"https://api.github.com/users/soumagne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/feature","name":"feature","color":"006b75"},{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/hg","name":"hg","color":"bfd4f2"},{"url":"https://api.github.com/repos/mercury-hpc/mercury/labels/major","name":"major","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/mercury-hpc/mercury/milestones/7","html_url":"https://github.com/mercury-hpc/mercury/milestones/mercury-0.9.0","labels_url":"https://api.github.com/repos/mercury-hpc/mercury/milestones/7/labels","id":1213618,"number":7,"title":"mercury-0.9.0","description":"### Features:\r\n * Callback API\r\n * Unified client/server code\r\n * Separation of layers:\r\n  * mercury core (low level interface using buffers)\r\n  * mercury (higher level providing encoding/decoding methods)\r\n  * mercury macros (higher level providing code generation and wrappers)","creator":{"login":"soumagne","id":5613371,"avatar_url":"https://avatars.githubusercontent.com/u/5613371?v=3","gravatar_id":"","url":"https://api.github.com/users/soumagne","html_url":"https://github.com/soumagne","followers_url":"https://api.github.com/users/soumagne/followers","following_url":"https://api.github.com/users/soumagne/following{/other_user}","gists_url":"https://api.github.com/users/soumagne/gists{/gist_id}","starred_url":"https://api.github.com/users/soumagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumagne/subscriptions","organizations_url":"https://api.github.com/users/soumagne/orgs","repos_url":"https://api.github.com/users/soumagne/repos","events_url":"https://api.github.com/users/soumagne/events{/privacy}","received_events_url":"https://api.github.com/users/soumagne/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":9,"state":"open","created_at":"2015-07-17T21:24:29Z","updated_at":"2016-02-11T08:05:45Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-07-17T21:27:02Z","updated_at":"2016-02-11T19:25:33Z","closed_at":null,"body":"**Reported by dkimpe on 10/22/13**\r\nFor variable length arguments that could end up being big,\r\nthe user now always has to do the bulk data setup.\r\n\r\nHowever, if the argument ends up being small (for example it would fit in the initial RPC request), a lot of extra work has been done.\r\n\r\nAt the same time, the user is not in a position to determine how much space is available in the eager RPC request (due to unknown encoding/header overhead etc.)\r\n\r\nProposed solution:\r\n- Have the bulk data module determine if a get/put should be used or if the data should be transmitted in eager mode.\r\n- If eager mode: the data gets encoded into the serialized handle transparently. The register/unregister/publish/... calls would do nothing in this case.\r\n- The origin would do the same, and put/get would complete immediately.\r\n\r\nOptimization (not sure that it makes sense though):\r\n- If a function has multiple bulk data arguments, it would be good if eager mode is not used if it would trigger a put/get for the rest of the arguments. \r\n\r\n- To implement this, the encoder routines would need to get access to the number of eager bytes left in the header.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mercury-hpc/mercury/issues/comments/183020939","html_url":"https://github.com/mercury-hpc/mercury/issues/23#issuecomment-183020939","issue_url":"https://api.github.com/repos/mercury-hpc/mercury/issues/23","id":183020939,"user":{"login":"JohnPJenkins","id":8118010,"avatar_url":"https://avatars.githubusercontent.com/u/8118010?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnPJenkins","html_url":"https://github.com/JohnPJenkins","followers_url":"https://api.github.com/users/JohnPJenkins/followers","following_url":"https://api.github.com/users/JohnPJenkins/following{/other_user}","gists_url":"https://api.github.com/users/JohnPJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnPJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnPJenkins/subscriptions","organizations_url":"https://api.github.com/users/JohnPJenkins/orgs","repos_url":"https://api.github.com/users/JohnPJenkins/repos","events_url":"https://api.github.com/users/JohnPJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/JohnPJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-02-11T19:25:33Z","updated_at":"2016-02-11T19:25:33Z","body":":-) \r\n\r\nSo the next question is, do we need to change anything at the API level for eager to happen? In other words, does a bulk transfer done eagerly look the same in user code as it currently does? I'd like to think yes, but there are some subtle issues to consider. For instance, can we guarantee the same behavior w.r.t. triggering the bulk callback? Is satisfying this guarantee as simple as putting the callback onto the completion queue as soon as HG_Bulk_transfer of an eager handle is issued, or do those queue manipulations have to be sourced from NA? Also, are there potential concurrency issues we'd have to consider?"}}