{"action":"created","issue":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/4066","repository_url":"https://api.github.com/repos/ofiwg/libfabric","labels_url":"https://api.github.com/repos/ofiwg/libfabric/issues/4066/labels{/name}","comments_url":"https://api.github.com/repos/ofiwg/libfabric/issues/4066/comments","events_url":"https://api.github.com/repos/ofiwg/libfabric/issues/4066/events","html_url":"https://github.com/ofiwg/libfabric/issues/4066","id":321327558,"number":4066,"title":"util/av: Scalability issue","user":{"login":"gladkovdmitry17","id":11650339,"avatar_url":"https://avatars1.githubusercontent.com/u/11650339?v=4","gravatar_id":"","url":"https://api.github.com/users/gladkovdmitry17","html_url":"https://github.com/gladkovdmitry17","followers_url":"https://api.github.com/users/gladkovdmitry17/followers","following_url":"https://api.github.com/users/gladkovdmitry17/following{/other_user}","gists_url":"https://api.github.com/users/gladkovdmitry17/gists{/gist_id}","starred_url":"https://api.github.com/users/gladkovdmitry17/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gladkovdmitry17/subscriptions","organizations_url":"https://api.github.com/users/gladkovdmitry17/orgs","repos_url":"https://api.github.com/users/gladkovdmitry17/repos","events_url":"https://api.github.com/users/gladkovdmitry17/events{/privacy}","received_events_url":"https://api.github.com/users/gladkovdmitry17/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-08T20:03:45Z","updated_at":"2018-05-08T20:33:30Z","closed_at":null,"author_association":"MEMBER","body":"When running ofi_rxm provider with MPICH/CH4 (and CH3 as well) for overall number of ranks > 1024, it leads to ABORT signal for 1025th rank and so on.\r\nSo, both CH3 and CH4 doesn't set `fi_av_attr::count` value to optimize handling of addresses at the provider level. This is just expected value (and OFI doesn't require to set it) of addresses that will be inserted.\r\nIn this case, util/av uses default value of address limit (1024).\r\n\r\nThe first idea that comes to me is to implement something like libfabric has for utility buffer pool functionality (i.e. having a linked lists of AV tables).\r\nBtw, we can use utility buf pool to generate internal AV entries and use pointer to buf pool entry as a `fi_addr` value that is returned to app. It similar to AV_MAP.\r\n\r\nWhat do you think to add environment variable to increase this value ? Yes, this is weird, but it would be good to have in 1.6.1 to allow customers run RxM at a large scale.\r\nPlease, check this solution in my repo - https://github.com/gladkovdmitry17/libfabric/tree/av. Setting this environment variable fixes scalability issue. If it is OK to have it in 1.6.1 as a workaround, I prepare PR to include this.\r\n\r\nIt seems RxD should have the same issue as soon it uses the util/av functionality\r\n\r\n@shefty @a-ilango @aingerson  @sayantansur @michael-chuvelev "},"comment":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/comments/387533450","html_url":"https://github.com/ofiwg/libfabric/issues/4066#issuecomment-387533450","issue_url":"https://api.github.com/repos/ofiwg/libfabric/issues/4066","id":387533450,"user":{"login":"gladkovdmitry17","id":11650339,"avatar_url":"https://avatars1.githubusercontent.com/u/11650339?v=4","gravatar_id":"","url":"https://api.github.com/users/gladkovdmitry17","html_url":"https://github.com/gladkovdmitry17","followers_url":"https://api.github.com/users/gladkovdmitry17/followers","following_url":"https://api.github.com/users/gladkovdmitry17/following{/other_user}","gists_url":"https://api.github.com/users/gladkovdmitry17/gists{/gist_id}","starred_url":"https://api.github.com/users/gladkovdmitry17/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gladkovdmitry17/subscriptions","organizations_url":"https://api.github.com/users/gladkovdmitry17/orgs","repos_url":"https://api.github.com/users/gladkovdmitry17/repos","events_url":"https://api.github.com/users/gladkovdmitry17/events{/privacy}","received_events_url":"https://api.github.com/users/gladkovdmitry17/received_events","type":"User","site_admin":false},"created_at":"2018-05-08T20:33:30Z","updated_at":"2018-05-08T20:33:30Z","author_association":"MEMBER","body":"@jsquyres Thanks. Hmm, good question.\r\n\r\nIt can be set when initializing OFI netmod without any problems.\r\nBut how much this should value be? `MPI_COMM_WORLD` size?\r\nBut when initializing OFI netmod, we don't know about spawned processes and therefore we don't have a chance to predict the proper `fi_av_attr::count` value"}}
{"actor":{"display_login":"gladkovdmitry17"}}