{"action":"created","issue":{"url":"https://api.github.com/repos/pangeo-data/pangeo/issues/112","repository_url":"https://api.github.com/repos/pangeo-data/pangeo","labels_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/112/comments","events_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/112/events","html_url":"https://github.com/pangeo-data/pangeo/issues/112","id":296256060,"number":112,"title":"Manipulating Zarr metadata with GCSFS is slow","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2018-02-12T03:10:02Z","updated_at":"2018-02-16T18:47:51Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"```python\r\n%%prun\r\n\r\nimport gcsfs\r\ngcsmap = gcsfs.mapping.GCSMap('pangeo-data/newman-met-ensemble')\r\nds = xr.open_zarr(gcsmap)\r\n```\r\n\r\n<details>\r\n\r\n```\r\n         16527714 function calls (16527548 primitive calls) in 11.297 seconds\r\n\r\n   Ordered by: internal time\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n       77    5.832    0.076    9.341    0.121 core.py:439(ls)\r\n 16250415    3.533    0.000    3.533    0.000 {method 'startswith' of 'str' objects}\r\n       46    1.453    0.032    1.453    0.032 {built-in method _openssl.SSL_read}\r\n        7    0.096    0.014    0.098    0.014 core.py:161(<listcomp>)\r\n        1    0.059    0.059    0.082    0.082 core.py:475(<listcomp>)\r\n    15405    0.015    0.000    0.069    0.000 _collections_abc.py:742(__iter__)\r\n    14532    0.014    0.000    0.039    0.000 os.py:664(__getitem__)\r\n      168    0.011    0.000    0.011    0.000 {built-in method posix.stat}\r\n       42    0.011    0.000    0.011    0.000 {built-in method _openssl.SSL_write}\r\n    28502    0.009    0.000    0.009    0.000 {method 'decode' of 'bytes' objects}\r\n       84    0.009    0.000    0.076    0.001 request.py:2458(getproxies_environment)\r\n    28392    0.009    0.000    0.018    0.000 os.py:746(decode)\r\n    14532    0.008    0.000    0.015    0.000 os.py:742(encode)\r\n       42    0.008    0.000    0.008    0.000 {method 'close' of 'select.epoll' objects}\r\n    22817    0.007    0.000    0.011    0.000 {built-in method builtins.isinstance}\r\n    14280    0.007    0.000    0.015    0.000 os.py:687(__iter__)\r\n        7    0.007    0.001    0.013    0.002 core.py:131(<listcomp>)\r\n     9522    0.006    0.000    0.006    0.000 core.py:131(<genexpr>)\r\n    16264    0.006    0.000    0.006    0.000 {method 'encode' of 'str' objects}\r\n    20061    0.004    0.000    0.004    0.000 {method 'lower' of 'str' objects}\r\n     7357    0.003    0.000    0.003    0.000 {method 'split' of 'str' objects}\r\n       42    0.003    0.000    0.003    0.000 {method 'register' of 'select.epoll' objects}\r\n        1    0.003    0.003    0.089    0.089 storage.py:128(_listdir_from_keys)\r\n      800    0.002    0.000    1.459    0.002 {method 'readline' of '_io.BufferedReader' objects}\r\n       88    0.002    0.000    0.002    0.000 {built-in method _openssl.SSL_get_error}\r\n      210    0.002    0.000    0.006    0.000 _collections_abc.py:824(update)\r\n11908/11833    0.002    0.000    0.002    0.000 {built-in method builtins.len}\r\n       42    0.002    0.000    0.006    0.000 feedparser.py:471(_parse_headers)\r\n       42    0.002    0.000    1.519    0.036 connectionpool.py:322(_make_request)\r\n      378    0.002    0.000    0.007    0.000 parse.py:359(urlparse)\r\n     1115    0.002    0.000    0.004    0.000 abc.py:178(__instancecheck__)\r\n       42    0.002    0.000    0.024    0.001 client.py:194(parse_headers)\r\n      420    0.002    0.000    0.004    0.000 message.py:462(get)\r\n       42    0.002    0.000    0.002    0.000 selectors.py:392(__init__)\r\n      420    0.002    0.000    0.003    0.000 parse.py:392(urlsplit)\r\n      294    0.002    0.000    0.003    0.000 client.py:1195(putheader)\r\n  192/185    0.002    0.000    0.002    0.000 indexing.py:508(shape)\r\n       84    0.002    0.000    0.013    0.000 feedparser.py:218(_parsegen)\r\n       42    0.002    0.000    1.552    0.037 connectionpool.py:447(urlopen)\r\n       42    0.002    0.000    1.593    0.038 sessions.py:589(send)\r\n      966    0.002    0.000    0.002    0.000 parse.py:109(_coerce_args)\r\n       42    0.002    0.000    0.003    0.000 client.py:1071(putrequest)\r\n     1126    0.002    0.000    0.002    0.000 {method 'match' of '_sre.SRE_Pattern' objects}\r\n     1442    0.002    0.000    0.002    0.000 _weakrefset.py:70(__contains__)\r\n     1542    0.001    0.000    0.001    0.000 {built-in method builtins.hasattr}\r\n       88    0.001    0.000    0.004    0.000 SSL.py:1417(_raise_ssl_error)\r\n       42    0.001    0.000    0.001    0.000 {method 'poll' of 'select.epoll' objects}\r\n       46    0.001    0.000    1.456    0.032 SSL.py:1598(recv_into)\r\n      714    0.001    0.000    0.003    0.000 _policybase.py:293(header_source_parse)\r\n       42    0.001    0.000    1.485    0.035 client.py:290(begin)\r\n       42    0.001    0.000    0.002    0.000 makefile.py:14(backport_makefile)\r\n       42    0.001    0.000    0.038    0.001 sessions.py:401(prepare_request)\r\n       42    0.001    0.000    0.025    0.001 client.py:1241(_send_request)\r\n      294    0.001    0.000    0.008    0.000 sessions.py:50(merge_setting)\r\n     1176    0.001    0.000    0.001    0.000 structures.py:48(__setitem__)\r\n       42    0.001    0.000    1.581    0.038 adapters.py:388(send)\r\n       72    0.001    0.000    9.343    0.130 core.py:537(info)\r\n       42    0.001    0.000    1.720    0.041 core.py:983(_fetch)\r\n      840    0.001    0.000    0.001    0.000 _collections.py:150(__getitem__)\r\n        1    0.001    0.001   11.298   11.298 <string>:3(<module>)\r\n      294    0.001    0.000    0.003    0.000 utils.py:244(to_key_val_list)\r\n        7    0.001    0.000    0.113    0.016 core.py:135(getem)\r\n       84    0.001    0.000    0.001    0.000 {method 'read' of '_io.StringIO' objects}\r\n       84    0.001    0.000    0.003    0.000 posixpath.py:230(expanduser)\r\n     1934    0.001    0.000    0.001    0.000 mapping.py:79(<genexpr>)\r\n       21    0.001    0.000    0.003    0.000 core.py:1875(normalize_chunks)\r\n      422    0.001    0.000    0.002    0.000 _collections_abc.py:657(get)\r\n      168    0.001    0.000    0.002    0.000 message.py:497(get_all)\r\n       42    0.001    0.000    1.714    0.041 sessions.py:441(request)\r\n      924    0.001    0.000    0.002    0.000 _policybase.py:281(_sanitize_header)\r\n      840    0.001    0.000    0.001    0.000 feedparser.py:78(readline)\r\n       42    0.001    0.000    0.001    0.000 poolmanager.py:57(_default_key_normalizer)\r\n       42    0.001    0.000    0.010    0.000 utils.py:160(get_netrc_auth)\r\n     1008    0.001    0.000    0.001    0.000 structures.py:53(__getitem__)\r\n       42    0.001    0.000    1.719    0.041 core.py:1082(_fetch_range)\r\n       42    0.001    0.000    1.458    0.035 client.py:257(_read_status)\r\n       42    0.001    0.000    1.717    0.041 requests.py:181(request)\r\n       42    0.001    0.000    0.018    0.000 wait.py:9(_wait_for_io_events)\r\n      617    0.001    0.000    0.001    0.000 {built-in method builtins.getattr}\r\n     3498    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\r\n        5    0.001    0.000    0.796    0.159 core.py:1538(_chunk_getitem)\r\n      714    0.001    0.000    0.002    0.000 _collections.py:208(add)\r\n       42    0.001    0.000    0.003    0.000 models.py:588(__init__)\r\n       42    0.001    0.000    0.004    0.000 models.py:433(prepare_headers)\r\n       42    0.001    0.000    0.005    0.000 models.py:347(prepare_url)\r\n       42    0.001    0.000    0.002    0.000 response.py:113(__init__)\r\n       73    0.001    0.000    0.001    0.000 util.py:271(normalize_storage_path)\r\n       84    0.001    0.000    0.003    0.000 cookiejar.py:1657(extract_cookies)\r\n       42    0.001    0.000    0.012    0.000 adapters.py:253(build_response)\r\n       42    0.001    0.000    0.010    0.000 response.py:441(from_httplib)\r\n      126    0.001    0.000    0.002    0.000 cookiejar.py:1239(__init__)\r\n      840    0.001    0.000    0.002    0.000 feedparser.py:128(__next__)\r\n       42    0.001    0.000    6.785    0.162 mapping.py:64(__getitem__)\r\n      168    0.001    0.000    0.006    0.000 structures.py:42(__init__)\r\n       42    0.001    0.000    0.001    0.000 feedparser.py:139(__init__)\r\n       84    0.001    0.000    0.001    0.000 {method 'readlines' of '_io._IOBase' objects}\r\n       42    0.001    0.000    0.015    0.000 SSL.py:1516(send)\r\n       84    0.001    0.000    0.003    0.000 utils.py:698(select_proxy)\r\n        2    0.001    0.000    0.002    0.001 _pslinux.py:1555(memory_info)\r\n       42    0.001    0.000    5.063    0.121 core.py:728(__init__)\r\n       84    0.001    0.000    0.001    0.000 hooks.py:17(default_hooks)\r\n       84    0.001    0.000    0.002    0.000 url.py:132(parse_url)\r\n       25    0.001    0.000    3.954    0.158 core.py:105(__init__)\r\n       42    0.001    0.000    0.005    0.000 response.py:346(read)\r\n      824    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}\r\n       42    0.001    0.000    0.003    0.000 _collections.py:224(extend)\r\n       42    0.001    0.000    0.042    0.001 utils.py:629(should_bypass_proxies)\r\n     1234    0.001    0.000    0.001    0.000 {method 'find' of 'str' objects}\r\n       42    0.001    0.000    1.488    0.035 client.py:1287(getresponse)\r\n       42    0.001    0.000    0.001    0.000 {method 'settimeout' of '_socket.socket' objects}\r\n      714    0.001    0.000    0.001    0.000 {method 'setdefault' of 'collections.OrderedDict' objects}\r\n```\r\n\r\n</details>\r\n\r\nSomething similar to this gets felt by the first read on every machine.  This make the cluster feel like it's hanging for 20s or so.  Somewhat reproducible with the following:\r\n\r\n```python\r\n%%time\r\nimport pickle\r\nb = pickle.dumps(gcsmap)\r\ngcsmap2 = pickle.loads(b)\r\n```\r\n\r\n    Wall time: 111 ms  # decent enough\r\n\r\n```python\r\n%%prun \r\nlen(gcsmap2)\r\n```\r\n\r\n         1230874 function calls (1182219 primitive calls) in 19.923 seconds\r\n\r\n<details>\r\n\r\n```\r\n\r\n   Ordered by: internal time\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n    12768   17.324    0.001   17.324    0.001 {built-in method _openssl.SSL_read}\r\n      109    0.823    0.008    0.829    0.008 decoder.py:345(raw_decode)\r\n        1    0.132    0.132   19.834   19.834 core.py:378(_list_bucket)\r\n    12768    0.103    0.000   17.536    0.001 SSL.py:1598(recv_into)\r\n     9496    0.083    0.000    1.039    0.000 response.py:346(read)\r\n        1    0.065    0.065    0.088    0.088 core.py:475(<listcomp>)\r\n    12876    0.061    0.000    0.091    0.000 SSL.py:1417(_raise_ssl_error)\r\n     9496    0.042    0.000    0.773    0.000 client.py:438(read)\r\n     9496    0.039    0.000    0.676    0.000 {method 'readinto' of '_io.BufferedReader' objects}\r\n    37714    0.039    0.000    0.104    0.000 os.py:664(__getitem__)\r\n    12770    0.038    0.000   17.651    0.001 socket.py:572(readinto)\r\n    39567    0.037    0.000    0.181    0.000 _collections_abc.py:742(__iter__)\r\n     9496    0.032    0.000    0.713    0.000 client.py:470(readinto)\r\n      989    0.030    0.000    1.103    0.001 {method 'join' of 'bytes' objects}\r\n    12768    0.029    0.000   17.565    0.001 pyopenssl.py:278(recv_into)\r\n        2    0.028    0.014    0.028    0.014 {built-in method _socket.getaddrinfo}\r\n     9605    0.027    0.000    0.060    0.000 contextlib.py:85(__exit__)\r\n      432    0.026    0.000    0.026    0.000 {built-in method posix.stat}\r\n    12770    0.025    0.000    0.025    0.000 {method '_checkClosed' of '_io._IOBase' objects}\r\n    74369    0.025    0.000    0.025    0.000 {method 'decode' of 'bytes' objects}\r\n   108491    0.025    0.000    0.025    0.000 {method 'startswith' of 'str' objects}\r\n    10477    0.024    0.000    0.047    0.000 _collections_abc.py:657(get)\r\n    73684    0.023    0.000    0.047    0.000 os.py:746(decode)\r\n      218    0.023    0.000    0.200    0.001 request.py:2458(getproxies_environment)\r\n     9496    0.022    0.000    0.065    0.000 response.py:250(_init_decoder)\r\n    37714    0.022    0.000    0.041    0.000 os.py:742(encode)\r\n   108380    0.021    0.000    0.021    0.000 {method 'get' of 'dict' objects}\r\n    12876    0.021    0.000    0.021    0.000 {built-in method _openssl.SSL_get_error}\r\n      107    0.020    0.000    0.020    0.000 {built-in method time.sleep}\r\n     9605    0.019    0.000    0.025    0.000 contextlib.py:59(__init__)\r\n    37060    0.018    0.000    0.041    0.000 os.py:687(__iter__)\r\n     9605    0.018    0.000    1.066    0.000 response.py:415(stream)\r\n    18992    0.018    0.000    0.027    0.000 response.py:289(_error_catcher)\r\n    12768    0.018    0.000    0.018    0.000 {built-in method allocator}\r\n     9496    0.017    0.000    0.017    0.000 {method 'tobytes' of 'memoryview' objects}\r\n    10694    0.017    0.000    0.022    0.000 _collections.py:150(__getitem__)\r\n      107    0.017    0.000    0.017    0.000 {built-in method _openssl.SSL_write}\r\n    56502    0.016    0.000    0.025    0.000 {built-in method builtins.isinstance}\r\n    41998    0.016    0.000    0.016    0.000 {method 'encode' of 'str' objects}\r\n      107    0.016    0.000    0.016    0.000 models.py:830(text)\r\n    59955    0.015    0.000    0.015    0.000 {method 'lower' of 'str' objects}\r\n     9605    0.013    0.000    0.038    0.000 contextlib.py:157(helper)\r\n    19210    0.013    0.000    0.040    0.000 {built-in method builtins.next}\r\n  48656/1    0.013    0.000   19.924   19.924 {built-in method builtins.len}\r\n    12879    0.012    0.000    0.012    0.000 socket.py:611(readable)\r\n    12770    0.011    0.000    0.022    0.000 {method '_checkReadable' of '_io._IOBase' objects}\r\n      107    0.008    0.000    0.008    0.000 {method 'close' of 'select.epoll' objects}\r\n    12876    0.008    0.000    0.008    0.000 SSL.py:199(raise_if_problem)\r\n    10926    0.008    0.000    0.008    0.000 {built-in method builtins.getattr}\r\n     9605    0.007    0.000    1.073    0.000 models.py:741(generate)\r\n        1    0.007    0.007    0.007    0.007 {built-in method _openssl.SSL_CTX_load_verify_locations}\r\n     9605    0.006    0.000    0.009    0.000 response.py:7(is_fp_closed)\r\n    19317    0.006    0.000    0.006    0.000 client.py:428(isclosed)\r\n     2628    0.006    0.000    0.009    0.000 abc.py:178(__instancecheck__)\r\n     1198    0.006    0.000   17.020    0.014 {method 'readline' of '_io.BufferedReader' objects}\r\n     1102    0.006    0.000    0.010    0.000 parse.py:392(urlsplit)\r\n     9496    0.005    0.000    0.005    0.000 response.py:260(_decode)\r\n     9605    0.005    0.000    0.013    0.000 contextlib.py:79(__enter__)\r\n      993    0.005    0.000    0.020    0.000 parse.py:359(urlparse)\r\n      218    0.005    0.000    0.033    0.000 feedparser.py:218(_parsegen)\r\n     1960    0.005    0.000    0.005    0.000 {method 'match' of '_sre.SRE_Pattern' objects}\r\n     2555    0.005    0.000    0.006    0.000 parse.py:109(_coerce_args)\r\n      109    0.005    0.000    0.064    0.001 client.py:194(parse_headers)\r\n      107    0.005    0.000   19.679    0.184 core.py:340(_call)\r\n      109    0.005    0.000   17.202    0.158 connectionpool.py:322(_make_request)\r\n      109    0.005    0.000    0.012    0.000 feedparser.py:471(_parse_headers)\r\n      549    0.005    0.000    0.012    0.000 _collections_abc.py:824(update)\r\n      109    0.005    0.000   18.461    0.169 sessions.py:589(send)\r\n      109    0.005    0.000   17.266    0.158 connectionpool.py:447(urlopen)\r\n      109    0.004    0.000   17.093    0.157 client.py:290(begin)\r\n      654    0.004    0.000    0.008    0.000 client.py:1195(putheader)\r\n      109    0.004    0.000    0.008    0.000 client.py:1071(putrequest)\r\n     1090    0.004    0.000    0.008    0.000 message.py:462(get)\r\n      107    0.004    0.000    0.004    0.000 selectors.py:392(__init__)\r\n     3393    0.004    0.000    0.004    0.000 _weakrefset.py:70(__contains__)\r\n      109    0.004    0.000   17.021    0.156 client.py:257(_read_status)\r\n      107    0.004    0.000    0.005    0.000 makefile.py:14(backport_makefile)\r\n      292    0.004    0.000    0.005    0.000 utils_perf.py:178(_gc_callback)\r\n      107    0.003    0.000   18.798    0.176 requests.py:181(request)\r\n      109    0.003    0.000   17.343    0.159 adapters.py:388(send)\r\n      763    0.003    0.000    0.022    0.000 sessions.py:50(merge_setting)\r\n      109    0.003    0.000   18.790    0.172 sessions.py:441(request)\r\n     3491    0.003    0.000    0.003    0.000 {built-in method builtins.hasattr}\r\n      109    0.003    0.000    0.108    0.001 sessions.py:401(prepare_request)\r\n      325    0.003    0.000    0.003    0.000 {method 'extend' of 'list' objects}\r\n      109    0.003    0.000    0.061    0.001 client.py:1241(_send_request)\r\n      763    0.003    0.000    0.009    0.000 utils.py:244(to_key_val_list)\r\n      109    0.003    0.000    0.005    0.000 feedparser.py:139(__init__)\r\n      107    0.003    0.000    0.003    0.000 {method 'register' of 'select.epoll' objects}\r\n     2202    0.003    0.000    0.003    0.000 {method 'split' of 'str' objects}\r\n      109    0.003    0.000    0.003    0.000 poolmanager.py:57(_default_key_normalizer)\r\n      109    0.003    0.000    0.025    0.000 models.py:347(prepare_url)\r\n      978    0.002    0.000    0.005    0.000 _policybase.py:293(header_source_parse)\r\n      219    0.002    0.000    0.007    0.000 posixpath.py:230(expanduser)\r\n      109    0.002    0.000    0.009    0.000 models.py:588(__init__)\r\n      109    0.002    0.000    0.023    0.000 utils.py:160(get_netrc_auth)\r\n      327    0.002    0.000    0.004    0.000 cookiejar.py:1239(__init__)\r\n        1    0.002    0.002    0.002    0.002 {built-in method _openssl.SSL_do_handshake}\r\n      109    0.002    0.000    0.031    0.000 adapters.py:253(build_response)\r\n```\r\n\r\n</details>"},"comment":{"url":"https://api.github.com/repos/pangeo-data/pangeo/issues/comments/366324308","html_url":"https://github.com/pangeo-data/pangeo/issues/112#issuecomment-366324308","issue_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/112","id":366324308,"user":{"login":"martindurant","id":6042212,"avatar_url":"https://avatars1.githubusercontent.com/u/6042212?v=4","gravatar_id":"","url":"https://api.github.com/users/martindurant","html_url":"https://github.com/martindurant","followers_url":"https://api.github.com/users/martindurant/followers","following_url":"https://api.github.com/users/martindurant/following{/other_user}","gists_url":"https://api.github.com/users/martindurant/gists{/gist_id}","starred_url":"https://api.github.com/users/martindurant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martindurant/subscriptions","organizations_url":"https://api.github.com/users/martindurant/orgs","repos_url":"https://api.github.com/users/martindurant/repos","events_url":"https://api.github.com/users/martindurant/events{/privacy}","received_events_url":"https://api.github.com/users/martindurant/received_events","type":"User","site_admin":false},"created_at":"2018-02-16T18:47:51Z","updated_at":"2018-02-16T18:47:51Z","author_association":"NONE","body":" > Question: if I ! pip installed something from within my notebook, how persistent is this installation? Do I need to do it again every time my server restarts?\r\n\r\nThis will persist, it actually writes into your site-packages folder within wherever python is installed; this can sometimes cause a problem when installing from conda and pip and setup.py, although in theory they should just clobber one-another."}}
{"actor":{"display_login":"martindurant"}}