{"action":"created","issue":{"url":"https://api.github.com/repos/Cyan4973/zstd/issues/304","repository_url":"https://api.github.com/repos/Cyan4973/zstd","labels_url":"https://api.github.com/repos/Cyan4973/zstd/issues/304/labels{/name}","comments_url":"https://api.github.com/repos/Cyan4973/zstd/issues/304/comments","events_url":"https://api.github.com/repos/Cyan4973/zstd/issues/304/events","html_url":"https://github.com/Cyan4973/zstd/issues/304","id":173651704,"number":304,"title":"ZDICT_trainFromBuffer throws \"Error (generic)\" on data with all possible byte values","user":{"login":"borzunov","id":8748943,"avatar_url":"https://avatars.githubusercontent.com/u/8748943?v=3","gravatar_id":"","url":"https://api.github.com/users/borzunov","html_url":"https://github.com/borzunov","followers_url":"https://api.github.com/users/borzunov/followers","following_url":"https://api.github.com/users/borzunov/following{/other_user}","gists_url":"https://api.github.com/users/borzunov/gists{/gist_id}","starred_url":"https://api.github.com/users/borzunov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borzunov/subscriptions","organizations_url":"https://api.github.com/users/borzunov/orgs","repos_url":"https://api.github.com/users/borzunov/repos","events_url":"https://api.github.com/users/borzunov/events{/privacy}","received_events_url":"https://api.github.com/users/borzunov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-28T15:20:00Z","updated_at":"2016-08-30T13:48:13Z","closed_at":null,"body":"I've run ZDICT_trainFromBuffer on some test data and random byte arrays and often got \"Error (generic)\". In case of random byte arrays that are long enough, this error appears almost every time.\r\n\r\nI ended up with building the following small snippet that reproduces the problem on data of different sizes:\r\n```c\r\n#include <stdio.h>\r\n#include <stdlib.h>\r\n#include \"zdict.h\"\r\n\r\n// 1. A smaller example:\r\n#define DICT_SIZE 1000\r\n#define SAMPLE_COUNT 2\r\n#define SAMPLE_SIZE 256\r\n\r\n// 2. A bigger example:\r\n#define DICT_SIZE 112640\r\n#define SAMPLE_COUNT 200\r\n#define SAMPLE_SIZE 3000\r\n\r\nint main()\r\n{\r\n\tunsigned char *samplesBuf = (unsigned char *)malloc(SAMPLE_COUNT * SAMPLE_SIZE);\r\n\tfor (int i = 0; i < SAMPLE_COUNT; i++)\r\n\t\tfor (int j = 0; j < SAMPLE_SIZE; j++)\r\n\t\t\tsamplesBuf[i * SAMPLE_SIZE + j] = (unsigned char)j;\r\n\r\n\tsize_t* sampleSizes = (size_t *)malloc(SAMPLE_COUNT * sizeof(size_t));\r\n\tfor (int i = 0; i < SAMPLE_COUNT; i++)\r\n\t\tsampleSizes[i] = SAMPLE_SIZE;\r\n\r\n\tvoid *dictBuf = malloc(DICT_SIZE);\r\n\r\n\tprintf(\"starting training...\\n\");\r\n\tsize_t code = ZDICT_trainFromBuffer(dictBuf, DICT_SIZE, samplesBuf, sampleSizes, SAMPLE_COUNT);\r\n\tif (ZDICT_isError(code))\r\n\t\tprintf(\"Failure: %s\\n\", ZDICT_getErrorName(code));\r\n\telse\r\n\t\tprintf(\"Success: %zu bytes\\n\", code);\r\n\r\n\tfree(dictBuf);\r\n\tfree(sampleSizes);\r\n\tfree(samplesBuf);\r\n\treturn 0;\r\n}\r\n```\r\n\r\nIn this snippet, the mentioned error is thrown here (and is clearly unexpected): https://github.com/Cyan4973/zstd/blob/master/lib/compress/huf_compress.c#L130 The snippet was compiled in Visual Studio 2015.\r\n\r\nA little experimentation made me think that it may be important for appearance of the bug that source data contain all 256 possible byte values."},"comment":{"url":"https://api.github.com/repos/Cyan4973/zstd/issues/comments/243444076","html_url":"https://github.com/Cyan4973/zstd/issues/304#issuecomment-243444076","issue_url":"https://api.github.com/repos/Cyan4973/zstd/issues/304","id":243444076,"user":{"login":"Cyan4973","id":750081,"avatar_url":"https://avatars.githubusercontent.com/u/750081?v=3","gravatar_id":"","url":"https://api.github.com/users/Cyan4973","html_url":"https://github.com/Cyan4973","followers_url":"https://api.github.com/users/Cyan4973/followers","following_url":"https://api.github.com/users/Cyan4973/following{/other_user}","gists_url":"https://api.github.com/users/Cyan4973/gists{/gist_id}","starred_url":"https://api.github.com/users/Cyan4973/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cyan4973/subscriptions","organizations_url":"https://api.github.com/users/Cyan4973/orgs","repos_url":"https://api.github.com/users/Cyan4973/repos","events_url":"https://api.github.com/users/Cyan4973/events{/privacy}","received_events_url":"https://api.github.com/users/Cyan4973/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T13:48:13Z","updated_at":"2016-08-30T13:48:13Z","body":"Hi @borzunov ,\r\nyou are totally right, this constant can hide an error code.\r\nI'll fix it."}}
{"actor":{"display_login":"Cyan4973"}}