{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1441","repository_url":"https://api.github.com/repos/Microsoft/LightGBM","labels_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1441/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1441/comments","events_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1441/events","html_url":"https://github.com/Microsoft/LightGBM/issues/1441","id":330980047,"node_id":"MDU6SXNzdWUzMzA5ODAwNDc=","number":1441,"title":"LightGBM does not like NUMA (large performance impact on servers)","user":{"login":"Laurae2","id":9083669,"node_id":"MDQ6VXNlcjkwODM2Njk=","avatar_url":"https://avatars3.githubusercontent.com/u/9083669?v=4","gravatar_id":"","url":"https://api.github.com/users/Laurae2","html_url":"https://github.com/Laurae2","followers_url":"https://api.github.com/users/Laurae2/followers","following_url":"https://api.github.com/users/Laurae2/following{/other_user}","gists_url":"https://api.github.com/users/Laurae2/gists{/gist_id}","starred_url":"https://api.github.com/users/Laurae2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Laurae2/subscriptions","organizations_url":"https://api.github.com/users/Laurae2/orgs","repos_url":"https://api.github.com/users/Laurae2/repos","events_url":"https://api.github.com/users/Laurae2/events{/privacy}","received_events_url":"https://api.github.com/users/Laurae2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"guolinke","id":16040950,"node_id":"MDQ6VXNlcjE2MDQwOTUw","avatar_url":"https://avatars0.githubusercontent.com/u/16040950?v=4","gravatar_id":"","url":"https://api.github.com/users/guolinke","html_url":"https://github.com/guolinke","followers_url":"https://api.github.com/users/guolinke/followers","following_url":"https://api.github.com/users/guolinke/following{/other_user}","gists_url":"https://api.github.com/users/guolinke/gists{/gist_id}","starred_url":"https://api.github.com/users/guolinke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guolinke/subscriptions","organizations_url":"https://api.github.com/users/guolinke/orgs","repos_url":"https://api.github.com/users/guolinke/repos","events_url":"https://api.github.com/users/guolinke/events{/privacy}","received_events_url":"https://api.github.com/users/guolinke/received_events","type":"User","site_admin":false},"assignees":[{"login":"guolinke","id":16040950,"node_id":"MDQ6VXNlcjE2MDQwOTUw","avatar_url":"https://avatars0.githubusercontent.com/u/16040950?v=4","gravatar_id":"","url":"https://api.github.com/users/guolinke","html_url":"https://github.com/guolinke","followers_url":"https://api.github.com/users/guolinke/followers","following_url":"https://api.github.com/users/guolinke/following{/other_user}","gists_url":"https://api.github.com/users/guolinke/gists{/gist_id}","starred_url":"https://api.github.com/users/guolinke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guolinke/subscriptions","organizations_url":"https://api.github.com/users/guolinke/orgs","repos_url":"https://api.github.com/users/guolinke/repos","events_url":"https://api.github.com/users/guolinke/events{/privacy}","received_events_url":"https://api.github.com/users/guolinke/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-06-10T15:56:16Z","updated_at":"2018-06-17T20:00:59Z","closed_at":null,"author_association":"COLLABORATOR","body":"## Environment info\r\n\r\n* Operating System: Windows 8.1 / 10, Windows Server 2012 / 2016, WSL Ubuntu 16.04, Clear Linux (latest)\r\n* CPU: Quad Intel Xeon 6154, 768GB RAM (72 physical cores, all 3.7 GHz, all RAM banks populated evenly)\r\n* C++/Python/R version: R 3.5 / Python 3.6, LightGBM commit 96e7016\r\n\r\nBy default, servers are shipping with NUMA nodes enabled. However, with UMA (Node Interleaving), LightGBM performance increases significantly (sometimes by over 80%+). On dual processor systems, the impact can be over 30%.\r\n\r\nThis issue is not limited to R, includes also Python and CLI.\r\n\r\nI also tested the impact on a 8x Xeon Platinum 8180 (768GB RAM, all RAM banks populated evenly): it was 200%+ slower with Node Interleaving off (NUMA on).\r\n\r\nxgboost is also affected by this issue, but it's less severe than LightGBM.\r\n\r\nRelated issues:\r\n\r\n* Opinion on LightGBM scalability: https://github.com/Microsoft/LightGBM/issues/1225\r\n\r\nUntested:\r\n\r\n* Sub NUMA Cluster (NUMA inside a processor) was untested and would probably result in an even worse performance.\r\n* `numactl` was not used because, first it is not available in Windows (Windows pins data to the correct RAM banks only if affinity is specified), second when using all available threads it does not make sense\r\n\r\n## Reproducible examples\r\n\r\nI used a private dataset (takes 250GB+ RAM), but the issue is easily reproducible using HIGGS dataset: https://archive.ics.uci.edu/ml/datasets/HIGGS\r\n\r\nSee the results on a dual Xeon Gold 6130 and 384GB RAM here: https://public.tableau.com/views/NodeInterleavingv1/Dashboard?:bootstrapWhenNotified=true&:display_count=y&:display_static_image=y&:embed=y&:showVizHome=no&publish=yes\r\n\r\n## Steps to reproduce\r\n\r\n1. Get on a multiprocessor server with full access to BIOS (at least dual Xeon ideally with a \"lot of threads\")\r\n2. In BIOS, turn off Node Interleaving (if enabled)\r\n3. Get the training time on a large dataset using all threads\r\n4. In BIOS, turn on Node Interleaving\r\n5. Get the training time on the same dataset using all threads\r\n6. Compare 2. and 4. timings\r\n\r\nCode from https://github.com/Microsoft/LightGBM/issues/542 could be used through CLI."},"comment":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/comments/397902597","html_url":"https://github.com/Microsoft/LightGBM/issues/1441#issuecomment-397902597","issue_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/1441","id":397902597,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzkwMjU5Nw==","user":{"login":"Laurae2","id":9083669,"node_id":"MDQ6VXNlcjkwODM2Njk=","avatar_url":"https://avatars3.githubusercontent.com/u/9083669?v=4","gravatar_id":"","url":"https://api.github.com/users/Laurae2","html_url":"https://github.com/Laurae2","followers_url":"https://api.github.com/users/Laurae2/followers","following_url":"https://api.github.com/users/Laurae2/following{/other_user}","gists_url":"https://api.github.com/users/Laurae2/gists{/gist_id}","starred_url":"https://api.github.com/users/Laurae2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Laurae2/subscriptions","organizations_url":"https://api.github.com/users/Laurae2/orgs","repos_url":"https://api.github.com/users/Laurae2/repos","events_url":"https://api.github.com/users/Laurae2/events{/privacy}","received_events_url":"https://api.github.com/users/Laurae2/received_events","type":"User","site_admin":false},"created_at":"2018-06-17T20:00:59Z","updated_at":"2018-06-17T20:00:59Z","author_association":"COLLABORATOR","body":"@huanzhang12 Higgs (11M observations, https://archive.ics.uci.edu/ml/datasets/HIGGS), with 2-way interactions (multiplication) of all features.\r\n\r\nIt makes a total of 11M observations x 408 features."}}
{"actor":{"display_login":"Laurae2"}}