{"action":"created","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/269","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/269/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/269/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/269/events","html_url":"https://github.com/Norconex/collector-http/issues/269","id":162504123,"number":269,"title":"Unexpected REJECTED_FILTER","user":{"login":"nurbardagan","id":13800244,"avatar_url":"https://avatars.githubusercontent.com/u/13800244?v=3","gravatar_id":"","url":"https://api.github.com/users/nurbardagan","html_url":"https://github.com/nurbardagan","followers_url":"https://api.github.com/users/nurbardagan/followers","following_url":"https://api.github.com/users/nurbardagan/following{/other_user}","gists_url":"https://api.github.com/users/nurbardagan/gists{/gist_id}","starred_url":"https://api.github.com/users/nurbardagan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nurbardagan/subscriptions","organizations_url":"https://api.github.com/users/nurbardagan/orgs","repos_url":"https://api.github.com/users/nurbardagan/repos","events_url":"https://api.github.com/users/nurbardagan/events{/privacy}","received_events_url":"https://api.github.com/users/nurbardagan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-06-27T17:23:45Z","updated_at":"2016-06-27T18:42:00Z","closed_at":null,"body":"Hello,\r\nI have follow configuration (file is attached) \r\n[config.txt](https://github.com/Norconex/collector-http/files/335445/config.txt)\r\n\r\nNow, I run the job, and as result see:\r\n\r\nINFO  [CrawlerEventManager]          DOCUMENT_FETCHED: https://en.wikipedia.org/wiki/Tiger\r\n...\r\nINFO  [CrawlerEventManager]            URLS_EXTRACTED: https://en.wikipedia.org/wiki/Tiger\r\nINFO  [CrawlerEventManager]           REJECTED_FILTER: https://en.wikipedia.org/wiki/Tiger\r\n\r\nIn mvstore I have:\r\n\r\nprocessedInvalid\r\nhttps://en.wikipedia.org/wiki/Tiger: https://en.wikipedia.org/wiki/Tiger - REJECTED\r\n\r\n\r\nThis is the code which generate the line from MVStore:\r\n\r\n```\r\n    MVMap<String, ICrawlData> map = s.openMap(\"processedInvalid\");\r\n\r\n    System.out.println(name);\r\n\r\n    for (String key : map.keySet()) {\r\n      ICrawlData data = map.get(key);\r\n      String ref = data.getReference();\r\n      CrawlState state = data.getState();\r\n      System.out.println(key + \": \" + ref + \" - \" + state);\r\n    }\r\n```\r\n\r\nSo why, despite the line:\r\n\r\n```\r\n      <documentFilters>\r\n        <filter class=\"com.norconex.collector.core.filter.impl.RegexReferenceFilter\"\r\n                onMatch=\"include\"\r\n                caseSensitive=\"false\">.*</filter>\r\n      </documentFilters>\r\n```\r\n\r\nit rejected?\r\nI tried with regexp \"tiger\" too."},"comment":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/comments/228836828","html_url":"https://github.com/Norconex/collector-http/issues/269#issuecomment-228836828","issue_url":"https://api.github.com/repos/Norconex/collector-http/issues/269","id":228836828,"user":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars.githubusercontent.com/u/2326274?v=3","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"created_at":"2016-06-27T18:42:00Z","updated_at":"2016-06-27T18:42:00Z","body":"The version is in the filename you downloaded (e.g. norconex-collector-http-2.5.0.zip) , and if you run it from the command prompt, the first few lines will have version information as well:\r\n```\r\n[non-job]: 2016-06-27 14:10:57 INFO - Version: \"Collector\" version is X.X.X.\r\n```\r\nBut I get you are using a recent version if you got it yesterday. :-)\r\n\r\nThe URLs that are processed just fine will be \"committed\" (sent to your committer). In your case, as you are using the FileSystemCommitter, files that were processed properly will end up under the location you specified: ``./tigrsoft/crawledFiles``.   That's fine for testing, but in the end, you probably want to use your own Committer to send crawled files where/how you want them.\r\n\r\nIn your log, if you wan to see a trace of a document that was not rejected and sent for committing, you can look for  something like this in your logs:\r\n```\r\nINFO  [CrawlerEventManager]    DOCUMENT_COMMITTED_ADD: https://en.wikipedia.org/wiki/Tiger\r\n```\r\n\r\nIf you get UNMODIFIED it is because you have not cleared your workdir/crawlstore before running and it remembers documents previously cawled.\r\n\r\nI am afraid I am not fully understanding what is not working for you.  Can you give a concrete use case of what you are getting or not getting that you suspect is not working right? "}}
{"actor":{"display_login":"essiembre"}}