{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/13082","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/13082/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/13082/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/13082/events","html_url":"https://github.com/pydata/pandas/pull/13082","id":152999808,"number":13082,"title":"PERF: Sparse ops speedup","user":{"login":"sinhrks","id":1696302,"avatar_url":"https://avatars.githubusercontent.com/u/1696302?v=3","gravatar_id":"","url":"https://api.github.com/users/sinhrks","html_url":"https://github.com/sinhrks","followers_url":"https://api.github.com/users/sinhrks/followers","following_url":"https://api.github.com/users/sinhrks/following{/other_user}","gists_url":"https://api.github.com/users/sinhrks/gists{/gist_id}","starred_url":"https://api.github.com/users/sinhrks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sinhrks/subscriptions","organizations_url":"https://api.github.com/users/sinhrks/orgs","repos_url":"https://api.github.com/users/sinhrks/repos","events_url":"https://api.github.com/users/sinhrks/events{/privacy}","received_events_url":"https://api.github.com/users/sinhrks/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Performance","name":"Performance","color":"a10c02"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Sparse","name":"Sparse","color":"009800"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/40","html_url":"https://github.com/pydata/pandas/milestones/0.18.2","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/40/labels","id":1639795,"number":40,"title":"0.18.2","description":"","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":155,"closed_issues":3,"state":"open","created_at":"2016-03-11T21:24:45Z","updated_at":"2016-05-05T12:03:38Z","due_on":"2016-06-14T04:00:00Z","closed_at":null},"comments":3,"created_at":"2016-05-04T12:32:14Z","updated_at":"2016-05-05T12:03:46Z","closed_at":"2016-05-05T12:03:38Z","pull_request":{"url":"https://api.github.com/repos/pydata/pandas/pulls/13082","html_url":"https://github.com/pydata/pandas/pull/13082","diff_url":"https://github.com/pydata/pandas/pull/13082.diff","patch_url":"https://github.com/pydata/pandas/pull/13082.patch"},"body":" - [x] tests added / passed\r\n - [x] passes ``git diff upstream/master | flake8 --diff``\r\n - [x] whatsnew entry\r\n\r\nFollow-up for #13036. Perf improvements are not significant because the number of previous ``list.append`` call is not so large.\r\n\r\n```\r\nimport numpy as np\r\nimport pandas as pd\r\n\r\nnp.random.seed(1)\r\nN = 1000000\r\na = np.array([np.nan] * N)\r\nb = np.array([np.nan] * N)\r\n\r\nindexer_a = np.unique(np.random.randint(0, N, N / 10))\r\nindexer_b = np.unique(np.random.randint(0, N, N / 10))\r\na[indexer_a] = np.random.randint(0, 100, len(indexer_a))\r\nb[indexer_b] = np.random.randint(0, 100, len(indexer_b))\r\nsa = pd.SparseArray(a)\r\nsb = pd.SparseArray(b)\r\n%timeit a.sp_index.intersect(sb.sp_index0)\r\n\r\n# before\r\n# 100 loops, best of 3: 3.04 ms per loop\r\n\r\n# after\r\n# 100 loops, best of 3: 2.11 ms per loop\r\n```\r\n\r\n```\r\ndef make_sparse_array(length, num_blocks, block_size, fill_value):\r\n    a = np.array([fill_value] * length)\r\n    for block in range(num_blocks):\r\n        i = np.random.randint(0, length)\r\n        a[i:i + block_size] = np.random.randint(0, 100, len(a[i:i + block_size]))\r\n    return pd.SparseArray(a, fill_value=fill_value)\r\n\r\nN = 1000000\r\nB = 10000\r\nBS = 10\r\n\r\na = make_sparse_array(length=N, num_blocks=B,  block_size=BS, fill_value=np.nan) \r\nb = make_sparse_array(length=N, num_blocks=B,  block_size=BS, fill_value=np.nan) \r\n\r\n%timeit a + b\r\n# before\r\n# 10 loops, best of 3: 70.8 ms per loop\r\n\r\n# after\r\n# 10 loops, best of 3: 66 ms per loop\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/217136790","html_url":"https://github.com/pydata/pandas/pull/13082#issuecomment-217136790","issue_url":"https://api.github.com/repos/pydata/pandas/issues/13082","id":217136790,"user":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"created_at":"2016-05-05T12:03:46Z","updated_at":"2016-05-05T12:03:46Z","body":"thanks!\r\n\r\nand hopefully turned off annoying codecov warnings."}}