{"action":"created","issue":{"url":"https://api.github.com/repos/datahq/data-cli/issues/154","repository_url":"https://api.github.com/repos/datahq/data-cli","labels_url":"https://api.github.com/repos/datahq/data-cli/issues/154/labels{/name}","comments_url":"https://api.github.com/repos/datahq/data-cli/issues/154/comments","events_url":"https://api.github.com/repos/datahq/data-cli/issues/154/events","html_url":"https://github.com/datahq/data-cli/issues/154","id":251782159,"number":154,"title":"data with different encoding than utf-8 does not work","user":{"login":"Mikanebu","id":19908588,"avatar_url":"https://avatars2.githubusercontent.com/u/19908588?v=4","gravatar_id":"","url":"https://api.github.com/users/Mikanebu","html_url":"https://github.com/Mikanebu","followers_url":"https://api.github.com/users/Mikanebu/followers","following_url":"https://api.github.com/users/Mikanebu/following{/other_user}","gists_url":"https://api.github.com/users/Mikanebu/gists{/gist_id}","starred_url":"https://api.github.com/users/Mikanebu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mikanebu/subscriptions","organizations_url":"https://api.github.com/users/Mikanebu/orgs","repos_url":"https://api.github.com/users/Mikanebu/repos","events_url":"https://api.github.com/users/Mikanebu/events{/privacy}","received_events_url":"https://api.github.com/users/Mikanebu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"zelima","id":15143785,"avatar_url":"https://avatars3.githubusercontent.com/u/15143785?v=4","gravatar_id":"","url":"https://api.github.com/users/zelima","html_url":"https://github.com/zelima","followers_url":"https://api.github.com/users/zelima/followers","following_url":"https://api.github.com/users/zelima/following{/other_user}","gists_url":"https://api.github.com/users/zelima/gists{/gist_id}","starred_url":"https://api.github.com/users/zelima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zelima/subscriptions","organizations_url":"https://api.github.com/users/zelima/orgs","repos_url":"https://api.github.com/users/zelima/repos","events_url":"https://api.github.com/users/zelima/events{/privacy}","received_events_url":"https://api.github.com/users/zelima/received_events","type":"User","site_admin":false},"assignees":[{"login":"zelima","id":15143785,"avatar_url":"https://avatars3.githubusercontent.com/u/15143785?v=4","gravatar_id":"","url":"https://api.github.com/users/zelima","html_url":"https://github.com/zelima","followers_url":"https://api.github.com/users/zelima/followers","following_url":"https://api.github.com/users/zelima/following{/other_user}","gists_url":"https://api.github.com/users/zelima/gists{/gist_id}","starred_url":"https://api.github.com/users/zelima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zelima/subscriptions","organizations_url":"https://api.github.com/users/zelima/orgs","repos_url":"https://api.github.com/users/zelima/repos","events_url":"https://api.github.com/users/zelima/events{/privacy}","received_events_url":"https://api.github.com/users/zelima/received_events","type":"User","site_admin":false},{"login":"Mikanebu","id":19908588,"avatar_url":"https://avatars2.githubusercontent.com/u/19908588?v=4","gravatar_id":"","url":"https://api.github.com/users/Mikanebu","html_url":"https://github.com/Mikanebu","followers_url":"https://api.github.com/users/Mikanebu/followers","following_url":"https://api.github.com/users/Mikanebu/following{/other_user}","gists_url":"https://api.github.com/users/Mikanebu/gists{/gist_id}","starred_url":"https://api.github.com/users/Mikanebu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mikanebu/subscriptions","organizations_url":"https://api.github.com/users/Mikanebu/orgs","repos_url":"https://api.github.com/users/Mikanebu/repos","events_url":"https://api.github.com/users/Mikanebu/events{/privacy}","received_events_url":"https://api.github.com/users/Mikanebu/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/datahq/data-cli/milestones/23","html_url":"https://github.com/datahq/data-cli/milestone/23","labels_url":"https://api.github.com/repos/datahq/data-cli/milestones/23/labels","id":3027806,"number":23,"title":"Sprint - 12 Feb 2018","description":"","creator":{"login":"Mikanebu","id":19908588,"avatar_url":"https://avatars2.githubusercontent.com/u/19908588?v=4","gravatar_id":"","url":"https://api.github.com/users/Mikanebu","html_url":"https://github.com/Mikanebu","followers_url":"https://api.github.com/users/Mikanebu/followers","following_url":"https://api.github.com/users/Mikanebu/following{/other_user}","gists_url":"https://api.github.com/users/Mikanebu/gists{/gist_id}","starred_url":"https://api.github.com/users/Mikanebu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mikanebu/subscriptions","organizations_url":"https://api.github.com/users/Mikanebu/orgs","repos_url":"https://api.github.com/users/Mikanebu/repos","events_url":"https://api.github.com/users/Mikanebu/events{/privacy}","received_events_url":"https://api.github.com/users/Mikanebu/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":1,"state":"open","created_at":"2018-01-11T05:56:54Z","updated_at":"2018-02-05T09:49:50Z","due_on":"2018-02-12T08:00:00Z","closed_at":null},"comments":4,"created_at":"2017-08-21T21:57:27Z","updated_at":"2018-02-05T09:51:22Z","closed_at":"2018-02-05T09:49:50Z","author_association":"COLLABORATOR","body":"Starting from an error report in https://github.com/datahq/frontend/issues/58 of encoding issues in the frontend we have traced issue to a bug in the encoding detection here in the CLI (see 58 issue for details).\r\n\r\nIf data is saved in different encoding that `utf-8`, it will fail.\r\n\r\n## Acceptance criteria \r\n\r\n* [x] we do not push data with strange characters \r\n\r\n## Tasks\r\n\r\n* [x] Create failing test case by adding sample.csv with different encoding(`cyrillic` for example).\r\n* [x] read `iconv` and `iconv-lite` documentation, and try out library [1]\r\n* [x] if does not help, look for other libraries and try to solve [1]\r\n* [x] prepare analysis and discuss whether we need to take care of this issue at all(it is better to ask user to use `utf-8` by default). [1]\r\n\r\n## Analysis\r\n\r\nAs much as I know, after debugging the problem now with encoding in DataHub CLI.\r\n\r\nFirst of all, it is related with `data.js` package, since it is responsible for instantiating File class. \r\nIn `File` class, we have a method called `get encoding` which returns encoding for the given file. \r\n\r\n**Do you have an failing test yet? That is the FIRST thing to do -- actually get this to fail**\r\n\r\n* First step: I replaced `chardet` library with `jscharded` since the second one has more encodings list and it has more stars. \r\n  * I tested with `windows Cyrillic`, `chardet` could not recognized it, when I used `jschardet`, it returns `ascii`\r\n* Second step: I take a look on `stream` method, this is the part where we get our data\r\n  * There are several packages such as `Iconv`, `iconv-lite` and `utf-8` to change encoding to `utf-8`, otherwise, it is not working. \r\n    * Tested with `cyrillic` for example. It could not change the encoding to `utf-8`. \r\nThe best solution would be prompt user to use `utf-8`. "},"comment":{"url":"https://api.github.com/repos/datahq/data-cli/issues/comments/363033558","html_url":"https://github.com/datahq/data-cli/issues/154#issuecomment-363033558","issue_url":"https://api.github.com/repos/datahq/data-cli/issues/154","id":363033558,"user":{"login":"zelima","id":15143785,"avatar_url":"https://avatars3.githubusercontent.com/u/15143785?v=4","gravatar_id":"","url":"https://api.github.com/users/zelima","html_url":"https://github.com/zelima","followers_url":"https://api.github.com/users/zelima/followers","following_url":"https://api.github.com/users/zelima/following{/other_user}","gists_url":"https://api.github.com/users/zelima/gists{/gist_id}","starred_url":"https://api.github.com/users/zelima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zelima/subscriptions","organizations_url":"https://api.github.com/users/zelima/orgs","repos_url":"https://api.github.com/users/zelima/repos","events_url":"https://api.github.com/users/zelima/events{/privacy}","received_events_url":"https://api.github.com/users/zelima/received_events","type":"User","site_admin":false},"created_at":"2018-02-05T09:51:22Z","updated_at":"2018-02-05T09:51:22Z","author_association":"CONTRIBUTOR","body":"FIXED"}}
{"actor":{"display_login":"zelima"}}