{"action":"created","issue":{"url":"https://api.github.com/repos/Unidata/netcdf-c/issues/489","repository_url":"https://api.github.com/repos/Unidata/netcdf-c","labels_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/489/labels{/name}","comments_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/489/comments","events_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/489/events","html_url":"https://github.com/Unidata/netcdf-c/issues/489","id":259205158,"node_id":"MDU6SXNzdWUyNTkyMDUxNTg=","number":489,"title":"Help with slow open performance with netCDF4 Files","user":{"login":"mathomp4","id":2982494,"node_id":"MDQ6VXNlcjI5ODI0OTQ=","avatar_url":"https://avatars2.githubusercontent.com/u/2982494?v=4","gravatar_id":"","url":"https://api.github.com/users/mathomp4","html_url":"https://github.com/mathomp4","followers_url":"https://api.github.com/users/mathomp4/followers","following_url":"https://api.github.com/users/mathomp4/following{/other_user}","gists_url":"https://api.github.com/users/mathomp4/gists{/gist_id}","starred_url":"https://api.github.com/users/mathomp4/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mathomp4/subscriptions","organizations_url":"https://api.github.com/users/mathomp4/orgs","repos_url":"https://api.github.com/users/mathomp4/repos","events_url":"https://api.github.com/users/mathomp4/events{/privacy}","received_events_url":"https://api.github.com/users/mathomp4/received_events","type":"User","site_admin":false},"labels":[{"id":269141915,"node_id":"MDU6TGFiZWwyNjkxNDE5MTU=","url":"https://api.github.com/repos/Unidata/netcdf-c/labels/area/netcdf4","name":"area/netcdf4","color":"fad8c7","default":false},{"id":845325448,"node_id":"MDU6TGFiZWw4NDUzMjU0NDg=","url":"https://api.github.com/repos/Unidata/netcdf-c/labels/type/performance","name":"type/performance","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"DennisHeimbigner","id":905179,"node_id":"MDQ6VXNlcjkwNTE3OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/905179?v=4","gravatar_id":"","url":"https://api.github.com/users/DennisHeimbigner","html_url":"https://github.com/DennisHeimbigner","followers_url":"https://api.github.com/users/DennisHeimbigner/followers","following_url":"https://api.github.com/users/DennisHeimbigner/following{/other_user}","gists_url":"https://api.github.com/users/DennisHeimbigner/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisHeimbigner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisHeimbigner/subscriptions","organizations_url":"https://api.github.com/users/DennisHeimbigner/orgs","repos_url":"https://api.github.com/users/DennisHeimbigner/repos","events_url":"https://api.github.com/users/DennisHeimbigner/events{/privacy}","received_events_url":"https://api.github.com/users/DennisHeimbigner/received_events","type":"User","site_admin":false},"assignees":[{"login":"DennisHeimbigner","id":905179,"node_id":"MDQ6VXNlcjkwNTE3OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/905179?v=4","gravatar_id":"","url":"https://api.github.com/users/DennisHeimbigner","html_url":"https://github.com/DennisHeimbigner","followers_url":"https://api.github.com/users/DennisHeimbigner/followers","following_url":"https://api.github.com/users/DennisHeimbigner/following{/other_user}","gists_url":"https://api.github.com/users/DennisHeimbigner/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisHeimbigner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisHeimbigner/subscriptions","organizations_url":"https://api.github.com/users/DennisHeimbigner/orgs","repos_url":"https://api.github.com/users/DennisHeimbigner/repos","events_url":"https://api.github.com/users/DennisHeimbigner/events{/privacy}","received_events_url":"https://api.github.com/users/DennisHeimbigner/received_events","type":"User","site_admin":false}],"milestone":null,"comments":42,"created_at":"2017-09-20T15:26:13Z","updated_at":"2018-06-20T13:56:24Z","closed_at":null,"author_association":"NONE","body":"### Environment Information\r\n* What platform are you using? (please provide specific distribution/version in summary)\r\n  * [x] Linux\r\n  * [ ] Windows\r\n  * [ ] OSX\r\n  * [ ] Other\r\n  * [ ] NA\r\n* 32 and/or 64 bit?\r\n  * [ ] 32-bit\r\n  * [x] 64-bit\r\n* What build system are you using?\r\n  * [x] autotools (`configure`)\r\n  * [ ] cmake\r\n* Can you provide a sample netCDF file or `C` code to recreate the issue?\r\n  * [x] Yes (please attach to this issue, thank you!)\r\n  * [ ] No\r\n  * [ ] Not at this time\r\n\r\n### Summary of Issue\r\nI work on a climate model and I/O is slowly becoming a major bottleneck. By default we try to use nc4 files as our main format (a few Fortran binaries remain, but we are trying to rid ourselves of that). We recently did an experiment to see if our slow performance might be due to the fact that many files we write are written with an unlimited time dimension, so when we read them, they are unlimited. So, we fixed the time dimension, no change. We also compress, chunk, etc. the files but still no big changes. Finally, a colleague suggested converting to netCDF Classic, and, hooboy. Muuuuuuch faster.\r\n\r\nEventually I went whole hog. I took our NC4 file, decompressed it, converted it using `nccopy -k ncX MERR.nc4 MERR.ncX` to nc3, nc5, nc6, nc7 and hacked together a simple program to test it out which you can see here:\r\n\r\nhttps://github.com/mathomp4/netcdfopentester/blob/master/testCNetcdf.c\r\n\r\nAll it does is open and close a file 1000 times (might be overkill). If I run this on my test files, I get the following output:\r\n```\r\n(1929)(master) $ ./testCNetcdf.exe\r\nnc3 unlimit dim open Time =  0.820 seconds.\r\nnc4 unlimit dim open Time = 19.870 seconds.\r\nnc5 unlimit dim open Time =  1.020 seconds.\r\nnc6 unlimit dim open Time =  1.130 seconds.\r\nnc7 unlimit dim open Time = 20.950 seconds.\r\nnc4 unlimit dim open with hdf5 Time =  0.250 seconds.\r\n```\r\nI added what I _think_ is the correct HDF5 code to open and close a file, but I mainly program netCDF in Fortran and Python, so even this program is a huge success for me! (Also: why it's ugly.) It seems like a netCDF open and an HDF5 open are ~2 orders of magnitude in speed difference. Is there a way to tune netCDF to open like HDF5? Or do you pay for HDF5's quick opening later?\r\n\r\nOr, assuming I might have got the HDF5 wrong, why does netcdf Classic open so much faster as well?\r\n\r\nNote: The versions of the libraries I'm using are netCDF-C 4.4.1.1 and HDF5 1.8.19. I compiled with Intel 17.0.4 compilers (backed up by GCC 5.3.0). The file is one from the MERRA2 data that can be retrieved (some basic instructions in my README in boring repo mentioned above). "},"comment":{"url":"https://api.github.com/repos/Unidata/netcdf-c/issues/comments/398758489","html_url":"https://github.com/Unidata/netcdf-c/issues/489#issuecomment-398758489","issue_url":"https://api.github.com/repos/Unidata/netcdf-c/issues/489","id":398758489,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODc1ODQ4OQ==","user":{"login":"edhartnett","id":4533212,"node_id":"MDQ6VXNlcjQ1MzMyMTI=","avatar_url":"https://avatars3.githubusercontent.com/u/4533212?v=4","gravatar_id":"","url":"https://api.github.com/users/edhartnett","html_url":"https://github.com/edhartnett","followers_url":"https://api.github.com/users/edhartnett/followers","following_url":"https://api.github.com/users/edhartnett/following{/other_user}","gists_url":"https://api.github.com/users/edhartnett/gists{/gist_id}","starred_url":"https://api.github.com/users/edhartnett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edhartnett/subscriptions","organizations_url":"https://api.github.com/users/edhartnett/orgs","repos_url":"https://api.github.com/users/edhartnett/repos","events_url":"https://api.github.com/users/edhartnett/events{/privacy}","received_events_url":"https://api.github.com/users/edhartnett/received_events","type":"User","site_admin":false},"created_at":"2018-06-20T13:56:24Z","updated_at":"2018-06-20T13:56:24Z","author_association":"CONTRIBUTOR","body":"OK, I think I see a way to do this for vars as well. @DennisHeimbigner if you could avoid major edits to the libsrc4/libhdf5 directories for a few weeks, I can take a stab at this. "}}
{"actor":{"display_login":"edhartnett"}}