{"action":"opened","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/271","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/271/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/271/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/271/events","html_url":"https://github.com/Norconex/collector-http/issues/271","id":163093941,"number":271,"title":"Unable to download content - charset encoding","user":{"login":"searchquest","id":16082342,"avatar_url":"https://avatars.githubusercontent.com/u/16082342?v=3","gravatar_id":"","url":"https://api.github.com/users/searchquest","html_url":"https://github.com/searchquest","followers_url":"https://api.github.com/users/searchquest/followers","following_url":"https://api.github.com/users/searchquest/following{/other_user}","gists_url":"https://api.github.com/users/searchquest/gists{/gist_id}","starred_url":"https://api.github.com/users/searchquest/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/searchquest/subscriptions","organizations_url":"https://api.github.com/users/searchquest/orgs","repos_url":"https://api.github.com/users/searchquest/repos","events_url":"https://api.github.com/users/searchquest/events{/privacy}","received_events_url":"https://api.github.com/users/searchquest/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-06-30T06:55:42Z","updated_at":"2016-06-30T06:55:42Z","closed_at":null,"body":"We have one legacy site which we want to crawl. We configured and crawled. we started for a small set of URLs and were able to crawl it but there were few pages we were not able to extract the body part.\r\nAfter enabling debuging, we found that this seems to be the issue with charset encoding. The parser treat this as UTF-8 but the html shows that it is ISO-8859-1. In one of the fixed issues, i have seen that handlers should detect charset encoding automatically.\r\nWe are not sure as what could be the reason. I tried looking various post but was not able to figure out as what could be the reason. Please have a look and let me know as what could be the reason.\r\n\r\nBelow is the log snippet:\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Creating new input stream from memory cache.\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Detected encoding: ISO-8859-1\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Creating new input stream from memory cache.\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - OCR parsing is disabled (no path provided).\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Parser \"com.norconex.importer.parser.impl.FallbackParser\" about to parse \"http://test.ss.sw.abcd.se/pci/Download?url=http%3A%2F%2Fpci.al.sw.abcd.se%2Fcgi-bin%2Fparallel%2Falexserv%3FLI%3DEN%2FLZN7010219R9A%26FN%3D7_19080-CNZ222163Uen.B.html%26IX%3DFILTER_LIBTIT\".\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Creating new input stream from memory cache.\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Parser \"class com.norconex.importer.parser.impl.FallbackParser\" did not produce new content for: http://test.ss.sw.abcd.se/pci/Download?url=http%3A%2F%2Fpci.al.sw.abcd.se%2Fcgi-bin%2Fparallel%2Falexserv%3FLI%3DEN%2FLZN7010219R9A%26FN%3D7_19080-CNZ222163Uen.B.html%26IX%3DFILTER_LIBTIT\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Document already parsed, assuming UTF-8 charset: http://test.ss.sw.abcd.se/pci/Download?url=http%3A%2F%2Fpci.al.sw.abcd.se%2Fcgi-bin%2Fparallel%2Falexserv%3FLI%3DEN%2FLZN7010219R9A%26FN%3D7_19080-CNZ222163Uen.B.html%26IX%3DFILTER_LIBTIT\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Transformer \"class com.norconex.importer.handler.transformer.impl.ReduceConsecutivesTransformer\" did not return any content for: http://test.ss.sw.abcd.se/pci/Download?url=http%3A%2F%2Fpci.al.sw.abcd.se%2Fcgi-bin%2Fparallel%2Falexserv%3FLI%3DEN%2FLZN7010219R9A%26FN%3D7_19080-CNZ222163Uen.B.html%26IX%3DFILTER_LIBTIT\r\nNorconex pci: 2016-06-30 09:52:48 INFO -         DOCUMENT_IMPORTED: http://test.ss.sw.abcd.se/pci/Download?url=http%3A%2F%2Fpci.al.sw.abcd.se%2Fcgi-bin%2Fparallel%2Falexserv%3FLI%3DEN%2FLZN7010219R9A%26FN%3D7_19080-CNZ222163Uen.B.html%26IX%3DFILTER_LIBTIT (Subject: com.norconex.importer.response.ImporterResponse@6818f224)\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Creating memory cache from cached stream.\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Creating new input stream from memory cache.\r\nNorconex pci: 2016-06-30 09:52:48 DEBUG - Document checksum from content: d41d8cd98f00b204e9800998ecf8427e\r\n\r\n\r\n&lt;importer&gt;\r\n    &lt;preParseHandlers&gt;                                  \r\n                                &lt;filter \r\n            class=\"com.norconex.importer.handler.filter.impl.RegexMetadataFilter\"\r\n            onMatch=\"exclude\" field=\"document.reference\"&gt;http:\\/\\/test\\.ss\\.sw\\.ericsson\\.se\\/Cpi\\/Page\\?dn=.*&lt;/filter&gt;\r\n                    &lt;filter \r\n            class=\"com.norconex.importer.handler.filter.impl.RegexMetadataFilter\"\r\n            onMatch=\"exclude\" field=\"document.reference\"&gt;http:\\/\\/test\\.ss\\.sw\\.ericsson\\.se\\/Cpi\\/Index\\/.*&lt;/filter&gt;\r\n    &lt;/preParseHandlers&gt;\r\n                &lt;postParseHandlers&gt;\r\n                                                                &lt;tagger class=\"com.norconex.importer.handler.tagger.impl.ConstantTagger\"&gt;\r\n                                                                                &lt;constant name=\"SITE\"&gt;Norconex Test Site&lt;/constant&gt;\r\n                                                                &lt;/tagger&gt;\r\n                   &lt;transformer class=\"com.norconex.importer.handler.transformer.impl.ReduceConsecutivesTransformer\"&gt;\r\n                        &lt;reduce&gt;\\s&lt;/reduce&gt;\r\n                        &lt;reduce&gt;\\n&lt;/reduce&gt;\r\n                        &lt;reduce&gt;\\s\\n&lt;/reduce&gt;\r\n                    &lt;/transformer&gt;\r\n    &lt;/postParseHandlers&gt;\r\n&lt;/importer&gt;\r\n"}}
{"actor":{"display_login":"searchquest"}}