{"action":"closed","issue":{"url":"https://api.github.com/repos/sjdirect/abot/issues/120","repository_url":"https://api.github.com/repos/sjdirect/abot","labels_url":"https://api.github.com/repos/sjdirect/abot/issues/120/labels{/name}","comments_url":"https://api.github.com/repos/sjdirect/abot/issues/120/comments","events_url":"https://api.github.com/repos/sjdirect/abot/issues/120/events","html_url":"https://github.com/sjdirect/abot/issues/120","id":141370416,"number":120,"title":"cp1251 encoding is not recognized","user":{"login":"Konard","id":1431904,"avatar_url":"https://avatars.githubusercontent.com/u/1431904?v=3","gravatar_id":"","url":"https://api.github.com/users/Konard","html_url":"https://github.com/Konard","followers_url":"https://api.github.com/users/Konard/followers","following_url":"https://api.github.com/users/Konard/following{/other_user}","gists_url":"https://api.github.com/users/Konard/gists{/gist_id}","starred_url":"https://api.github.com/users/Konard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Konard/subscriptions","organizations_url":"https://api.github.com/users/Konard/orgs","repos_url":"https://api.github.com/users/Konard/repos","events_url":"https://api.github.com/users/Konard/events{/privacy}","received_events_url":"https://api.github.com/users/Konard/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-16T19:00:43Z","updated_at":"2016-03-17T04:28:01Z","closed_at":"2016-03-17T04:28:01Z","body":"Trying to crawl some russian websites. One of them uses `<META http-equiv=Content-Type content=\"text/html; charset=cp1251\">`, but `cp1251` is not recognized by `GetEncoding` function inside `GetContent` method of `WebContentExtractor` (https://github.com/sjdirect/abot/blob/fe779a3c368c4d8db11e1eb2f048fc461d172b82/Abot/Core/WebContentExtractor.cs#L38)\r\n![2016-03-16 21-49-36](https://cloud.githubusercontent.com/assets/1431904/13825000/9ac5317a-ebc1-11e5-85ad-d93247b6ce2d.png)\r\nIf I change manually the value of `cp1251` to `1251` it is still resulting with UTF-8. Only `windows-1251` does the thing:\r\n![2016-03-16 21-53-22](https://cloud.githubusercontent.com/assets/1431904/13825004/a0950f4e-ebc1-11e5-809e-245cbf618bb0.png)\r\n\r\n\r\nSo what is a best way to fix this issue? Looks like the crawler should have some Dictionary or Dictionary-like API for associating source encodings (like `cp1251`) with target encodings (like `windows-1251`). This should work as synonyms dictionary.\r\n\r\nP.S. Source code on screenshots was generated by dotPeek (for some reason I cannot load source code symbols from NuGet service using Visual Studio 2013). The latest NuGet package version (Abot 1.5.1.22) is used."}}