{"action":"created","issue":{"url":"https://api.github.com/repos/PCMDI/cmor/issues/403","repository_url":"https://api.github.com/repos/PCMDI/cmor","labels_url":"https://api.github.com/repos/PCMDI/cmor/issues/403/labels{/name}","comments_url":"https://api.github.com/repos/PCMDI/cmor/issues/403/comments","events_url":"https://api.github.com/repos/PCMDI/cmor/issues/403/events","html_url":"https://github.com/PCMDI/cmor/issues/403","id":361527528,"node_id":"MDU6SXNzdWUzNjE1Mjc1Mjg=","number":403,"title":"compression/shuffle defaults for CMOR3","user":{"login":"taylor13","id":4993439,"node_id":"MDQ6VXNlcjQ5OTM0Mzk=","avatar_url":"https://avatars3.githubusercontent.com/u/4993439?v=4","gravatar_id":"","url":"https://api.github.com/users/taylor13","html_url":"https://github.com/taylor13","followers_url":"https://api.github.com/users/taylor13/followers","following_url":"https://api.github.com/users/taylor13/following{/other_user}","gists_url":"https://api.github.com/users/taylor13/gists{/gist_id}","starred_url":"https://api.github.com/users/taylor13/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/taylor13/subscriptions","organizations_url":"https://api.github.com/users/taylor13/orgs","repos_url":"https://api.github.com/users/taylor13/repos","events_url":"https://api.github.com/users/taylor13/events{/privacy}","received_events_url":"https://api.github.com/users/taylor13/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mauzey1","id":42009431,"node_id":"MDQ6VXNlcjQyMDA5NDMx","avatar_url":"https://avatars0.githubusercontent.com/u/42009431?v=4","gravatar_id":"","url":"https://api.github.com/users/mauzey1","html_url":"https://github.com/mauzey1","followers_url":"https://api.github.com/users/mauzey1/followers","following_url":"https://api.github.com/users/mauzey1/following{/other_user}","gists_url":"https://api.github.com/users/mauzey1/gists{/gist_id}","starred_url":"https://api.github.com/users/mauzey1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mauzey1/subscriptions","organizations_url":"https://api.github.com/users/mauzey1/orgs","repos_url":"https://api.github.com/users/mauzey1/repos","events_url":"https://api.github.com/users/mauzey1/events{/privacy}","received_events_url":"https://api.github.com/users/mauzey1/received_events","type":"User","site_admin":false},"assignees":[{"login":"mauzey1","id":42009431,"node_id":"MDQ6VXNlcjQyMDA5NDMx","avatar_url":"https://avatars0.githubusercontent.com/u/42009431?v=4","gravatar_id":"","url":"https://api.github.com/users/mauzey1","html_url":"https://github.com/mauzey1","followers_url":"https://api.github.com/users/mauzey1/followers","following_url":"https://api.github.com/users/mauzey1/following{/other_user}","gists_url":"https://api.github.com/users/mauzey1/gists{/gist_id}","starred_url":"https://api.github.com/users/mauzey1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mauzey1/subscriptions","organizations_url":"https://api.github.com/users/mauzey1/orgs","repos_url":"https://api.github.com/users/mauzey1/repos","events_url":"https://api.github.com/users/mauzey1/events{/privacy}","received_events_url":"https://api.github.com/users/mauzey1/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2018-09-19T00:02:03Z","updated_at":"2018-11-13T19:37:11Z","closed_at":null,"author_association":"NONE","body":"In cmor_reset_variable() (in cmor.c) I find:\r\n```\r\n    cmor_vars[var_id].shuffle = 0;\r\n    cmor_vars[var_id].deflate = 1;\r\n    cmor_vars[var_id].deflate_level = 1;\r\n```\r\nIs this initializing the settings for shuffle, deflate and deflate_level?  And will those settings remain in effect unless the CMOR table specifies different values?  Or do they invariably get ignored?\r\n\r\nIf these are default settings specified for CMOR3, then all data will be compressed by default, correct?"},"comment":{"url":"https://api.github.com/repos/PCMDI/cmor/issues/comments/438407359","html_url":"https://github.com/PCMDI/cmor/issues/403#issuecomment-438407359","issue_url":"https://api.github.com/repos/PCMDI/cmor/issues/403","id":438407359,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODQwNzM1OQ==","user":{"login":"mauzey1","id":42009431,"node_id":"MDQ6VXNlcjQyMDA5NDMx","avatar_url":"https://avatars0.githubusercontent.com/u/42009431?v=4","gravatar_id":"","url":"https://api.github.com/users/mauzey1","html_url":"https://github.com/mauzey1","followers_url":"https://api.github.com/users/mauzey1/followers","following_url":"https://api.github.com/users/mauzey1/following{/other_user}","gists_url":"https://api.github.com/users/mauzey1/gists{/gist_id}","starred_url":"https://api.github.com/users/mauzey1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mauzey1/subscriptions","organizations_url":"https://api.github.com/users/mauzey1/orgs","repos_url":"https://api.github.com/users/mauzey1/repos","events_url":"https://api.github.com/users/mauzey1/events{/privacy}","received_events_url":"https://api.github.com/users/mauzey1/received_events","type":"User","site_admin":false},"created_at":"2018-11-13T19:37:11Z","updated_at":"2018-11-13T19:37:11Z","author_association":"COLLABORATOR","body":"I have done some test using a mask from this file for ocean data:\r\nhttps://vesg.ipsl.upmc.fr/thredds/catalog/esgcet/45/CMIP6.CMIP.IPSL.IPSL-CM6A-LR.1pctCO2.r1i1p1f1.Omon.dpco2.gn.v20180727.html?dataset=CMIP6.CMIP.IPSL.IPSL-CM6A-LR.1pctCO2.r1i1p1f1.Omon.dpco2.gn.v20180727.dpco2_Omon_IPSL-CM6A-LR_1pctCO2_r1i1p1f1_gn_185001-199912.nc\r\n\r\nThe mask removes about 46% of the data from a 332x362 grid.  I made a script that creates data with the same longitude and latitude grid as the above file's mask but with varying numbers of time steps.\r\n\r\nhttps://github.com/mauzey1/cmor/blob/shuffle_testing/Test/shuffle_test_missing_data.py\r\n\r\nHere are some results from that script.  Enabling shuffle causes a 11% decrease in file size with a 13% decrease in run time.\r\n```\r\ngrid dim  \tno shuffle  \t\tshuffle\t\t        size diff %\ttime diff %\r\n332x362x1000 \t 219 mb\t  27.179 s\t 194 mb\t  23.322 s \t-11.099\t\t-13.776\r\n332x362x2000 \t 436 mb\t  53.107 s\t 388 mb\t  45.851 s \t-11.110\t\t-13.646\r\n332x362x5000 \t1090 mb  132.502 s\t 969 mb\t 115.061 s \t-11.117\t\t-13.153\r\n332x362x10000 \t2179 mb  267.695 s\t1936 mb  232.012 s \t-11.120\t\t-13.325\r\n```\r\n\r\nI did however find a case where enabling shuffle would cause an increase in file size.  I used a file that had a land mask that removes 62% of the data if you mask parts that equal 0.0.\r\n\r\nhttps://esgf.nccs.nasa.gov/thredds/catalog/esgcet/47/CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.piControl.r1i1p1f1.fx.sftlf.gn.v20180824.html?dataset=CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.piControl.r1i1p1f1.fx.sftlf.gn.v20180824.sftlf_fx_piControl_GISS-E2-1-G_r1i1p1f1_gn.nc\r\n\r\nhttps://github.com/mauzey1/cmor/blob/7c3f9dce2482528064dab6ee1983186437bf2687/Test/shuffle_test_missing_data2.py\r\n\r\n```\r\ngrid dim        no shuffle              shuffle                 size diff %     time diff %\r\n90x144x1000      18 mb   2.695 s         19 mb   2.412 s        2.721           -10.160\r\n90x144x2000      36 mb   5.138 s         37 mb   4.625 s        2.725           -9.937\r\n90x144x5000      90 mb  12.588 s         92 mb  11.701 s        2.721           -6.988\r\n90x144x10000    179 mb  24.945 s        184 mb  23.566 s        2.721           -5.463\r\n90x144x20000    358 mb  48.430 s        368 mb  45.527 s        2.722           -5.942\r\n```\r\n\r\nAlthough there is a small decrease in run time, the files are 2.7% larger with shuffle than without it."}}
{"actor":{"display_login":"mauzey1"}}