{"action":"created","issue":{"url":"https://api.github.com/repos/ornladios/ADIOS/issues/170","repository_url":"https://api.github.com/repos/ornladios/ADIOS","labels_url":"https://api.github.com/repos/ornladios/ADIOS/issues/170/labels{/name}","comments_url":"https://api.github.com/repos/ornladios/ADIOS/issues/170/comments","events_url":"https://api.github.com/repos/ornladios/ADIOS/issues/170/events","html_url":"https://github.com/ornladios/ADIOS/issues/170","id":306044329,"number":170,"title":"Mix of compressed + uncompressed data causing read error","user":{"login":"n01r","id":5416860,"avatar_url":"https://avatars1.githubusercontent.com/u/5416860?v=4","gravatar_id":"","url":"https://api.github.com/users/n01r","html_url":"https://github.com/n01r","followers_url":"https://api.github.com/users/n01r/followers","following_url":"https://api.github.com/users/n01r/following{/other_user}","gists_url":"https://api.github.com/users/n01r/gists{/gist_id}","starred_url":"https://api.github.com/users/n01r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/n01r/subscriptions","organizations_url":"https://api.github.com/users/n01r/orgs","repos_url":"https://api.github.com/users/n01r/repos","events_url":"https://api.github.com/users/n01r/events{/privacy}","received_events_url":"https://api.github.com/users/n01r/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2018-03-16T19:06:02Z","updated_at":"2018-03-27T14:17:56Z","closed_at":null,"author_association":"NONE","body":"I am currently running large simulation jobs with 2400 GPU nodes on a SLURM file system.\r\nI noticed that sometimes I could not read my particle data with the ADIOS NumPy wrappers. That only happens with the simulations that have the most particles in them. \r\n\r\nI am using ADIOS 1.13.0 and `blosc` with `zstd` for compression.\r\nThe full parameters I use for production are: `threshold=2048,shuffle=bit,lvl=1,threads=10,compressor=zstd`.\r\n\r\n```python\r\npath = \"simData_16600.bp\"\r\nf = ad.file(path)\r\n\r\nw = f[\"/data/16600/particles/H_all/weighting\"]\r\nw_unit = w.attrs['unitSI'].value\r\nprint(w.dims[0])\r\n147455991\r\n\r\nprint(w[0])\r\n[Out]: 3.479555831394943e-36\r\nprint(w[100])\r\n[Out]: 2.3418408851396162e-38\r\nprint(w[1000])\r\n[Out]: 0.0\r\n```\r\nand this\r\n```\r\nERROR: Error: Variable (id=36) out of bound 1(the data in dimension 1 to read is 1 \r\nelements from index 1000 but the actual data is [0,828])\r\n```\r\n@psychocoderHPC and I found out that this happens for datasets where there is a mix between compressed particle data and uncompressed particle data.\r\nUnfortunately, the datasets where this happens are quite large (~1-2TB scale).\r\nI also do not know how to reproduce it in smaller scale. Therefore I uploaded the output of `bpdump` and the verbose output of `bpmeta` (links expire on July 14 2018).\r\n\r\n`bpdump` - password: *bpdump*\r\nhttps://owncloud.hzdr.de/index.php/s/MawIFsaGTkSI9R1\r\n\r\n`bpmeta` - password: *bpmeta*\r\nhttps://owncloud.hzdr.de/index.php/s/slZ416U8VAKLq3e\r\n\r\nOne can see from the error that in `data/16600/particles/H_all/weighting` there seems to be a misinterpreted data chunk size.\r\n\r\nLines 98556f in the `bpdump` output show:\r\n```\r\nOffset(985682125)\tPayload Offset(985682241)\tFile Index(2)\tTime Index(1)\tDims (l:g:o): (0:147455991:0)\r\nOffset(1634343392)\tPayload Offset(1634343561)\tFile Index(2)\tTime Index(1)\tDims (l:g:o): (829)\tTransform type: blosc compression (ID = 11)\tPre-transform datatype: real\tPre-transform dims (l:g:o): (184320:147455991:0)\r\n```\r\nThere happens a switch between uncompressed data to compressed data. We suppose that the `(829)` is interpreted as an array size and therefore the error message says that\r\n> `the actual data is [0,828]`\r\n"},"comment":{"url":"https://api.github.com/repos/ornladios/ADIOS/issues/comments/376542101","html_url":"https://github.com/ornladios/ADIOS/issues/170#issuecomment-376542101","issue_url":"https://api.github.com/repos/ornladios/ADIOS/issues/170","id":376542101,"user":{"login":"pnorbert","id":3247149,"avatar_url":"https://avatars1.githubusercontent.com/u/3247149?v=4","gravatar_id":"","url":"https://api.github.com/users/pnorbert","html_url":"https://github.com/pnorbert","followers_url":"https://api.github.com/users/pnorbert/followers","following_url":"https://api.github.com/users/pnorbert/following{/other_user}","gists_url":"https://api.github.com/users/pnorbert/gists{/gist_id}","starred_url":"https://api.github.com/users/pnorbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pnorbert/subscriptions","organizations_url":"https://api.github.com/users/pnorbert/orgs","repos_url":"https://api.github.com/users/pnorbert/repos","events_url":"https://api.github.com/users/pnorbert/events{/privacy}","received_events_url":"https://api.github.com/users/pnorbert/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T14:17:55Z","updated_at":"2018-03-27T14:17:55Z","author_association":"CONTRIBUTOR","body":"I suspect I made a mistake. I only remove them if there are transformed\nblocks (i.e. I think the variable has transformation). But you have whole\nsets where all of them are zero blocks. I will remove them in any case.\nTo confirm this suspicion, can you make a bpdump again?\nThanks\n\nOn Tue, Mar 27, 2018 at 9:42 AM, Marco Garten <notifications@github.com>\nwrote:\n\n> Hmm, reading still doesn't seem to work.\n> I downloaded a new adios-1.13.0 and replaced bpmeta.c with the file from\n> the link @pnorbert <https://github.com/pnorbert> posted and built the\n> code as usual. I made a backup of my old simData_16600.bp and ran bpmeta\n> -z simData_16600.bp again. (Once with and once without verbose output\n> because it's easier to look through the one without)\n>\n> See the logs here:\n>\n> bpmeta_non-verbose_16600.log, pw: *bpmeta*\n> https://owncloud.hzdr.de/index.php/s/rF7d2bMjk1jiJc9\n> bpmeta_verbose_16600.log, pw: *bpmeta*\n> https://owncloud.hzdr.de/index.php/s/aYj0rl3zM3q95He\n>\n> When I try reading the same AdiosVar as before (/data/{}/particles/H_all/\n> weighting) I am getting the same error.\n>\n> ERROR: Error: Variable (id=36) out of bound 1(the data in dimension 1 to read is 147455991 elements from index 0 but the actual data is [0,828])\n>\n> from bpmeta_non-verbose_16600.log\n>\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 19 ZERO, NON-TRANSFORMED blocks and 0 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 19 ZERO, NON-TRANSFORMED blocks and 0 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 2 ZERO, NON-TRANSFORMED blocks and 17 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 15 ZERO, NON-TRANSFORMED blocks and 4 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 19 ZERO, NON-TRANSFORMED blocks and 0 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 19 ZERO, NON-TRANSFORMED blocks and 0 transformed blocks among 19 blocks ***\n> Thread 0: *** Variable /data/16600/particles/H_all/weighting has 19 ZERO, NON-TRANSFORMED blocks and 0 transformed blocks among 19 blocks ***\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/ornladios/ADIOS/issues/170#issuecomment-376529403>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ADGMLSwKy4REQKzxh_mV7XKTK38dOurBks5tikGsgaJpZM4SuS8j>\n> .\n>\n"}}
{"actor":{"display_login":"pnorbert"}}