{"action":"created","comment":{"url":"https://api.github.com/repos/rapid7/recog/pulls/comments/88155222","pull_request_review_id":8737449,"id":88155222,"diff_hunk":"@@ -43,7 +47,17 @@ def initialize(xml)\n   # @return [Hash,nil] Keys will be host, service, and os attributes\n   def match(match_string)\n     # match_string.force_encoding('BINARY') if match_string\n-    match_data = @regex.match(match_string)\n+    begin\n+      match_data = @regex.match(match_string)\n+    rescue Encoding::CompatibilityError => e\n+      # 'ASCII-8BIT' encoded responses will be caught and fixed here.\n+      match_data = @regex.match(match_string.force_encoding('UTF-8'))","path":"lib/recog/fingerprint.rb","position":22,"original_position":22,"commit_id":"8df8fec9dfa293fa109e8ac91176c88bdf65d21b","original_commit_id":"8df8fec9dfa293fa109e8ac91176c88bdf65d21b","user":{"login":"tsellers-r7","id":20306699,"avatar_url":"https://avatars.githubusercontent.com/u/20306699?v=3","gravatar_id":"","url":"https://api.github.com/users/tsellers-r7","html_url":"https://github.com/tsellers-r7","followers_url":"https://api.github.com/users/tsellers-r7/followers","following_url":"https://api.github.com/users/tsellers-r7/following{/other_user}","gists_url":"https://api.github.com/users/tsellers-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/tsellers-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsellers-r7/subscriptions","organizations_url":"https://api.github.com/users/tsellers-r7/orgs","repos_url":"https://api.github.com/users/tsellers-r7/repos","events_url":"https://api.github.com/users/tsellers-r7/events{/privacy}","received_events_url":"https://api.github.com/users/tsellers-r7/received_events","type":"User","site_admin":false},"body":"Handled, but my debugging code is still in place for the moment.  Pretty much everything I tested with today was ASCII-8BIT but the only time this was a problem was when (I guess?) the regex was trying to automatically change it to UTF-8.   I would just encode everything as UTF-8 but I'm not sure about the performance impacts yet.","created_at":"2016-11-16T01:56:52Z","updated_at":"2016-11-16T01:56:53Z","html_url":"https://github.com/rapid7/recog/pull/114#discussion_r88155222","pull_request_url":"https://api.github.com/repos/rapid7/recog/pulls/114","_links":{"self":{"href":"https://api.github.com/repos/rapid7/recog/pulls/comments/88155222"},"html":{"href":"https://github.com/rapid7/recog/pull/114#discussion_r88155222"},"pull_request":{"href":"https://api.github.com/repos/rapid7/recog/pulls/114"}}},"pull_request":{"url":"https://api.github.com/repos/rapid7/recog/pulls/114","id":93179034,"html_url":"https://github.com/rapid7/recog/pull/114","diff_url":"https://github.com/rapid7/recog/pull/114.diff","patch_url":"https://github.com/rapid7/recog/pull/114.patch","issue_url":"https://api.github.com/repos/rapid7/recog/issues/114","number":114,"state":"open","locked":false,"title":"[WIP] Add support for matching against all XML databases","user":{"login":"tsellers-r7","id":20306699,"avatar_url":"https://avatars.githubusercontent.com/u/20306699?v=3","gravatar_id":"","url":"https://api.github.com/users/tsellers-r7","html_url":"https://github.com/tsellers-r7","followers_url":"https://api.github.com/users/tsellers-r7/followers","following_url":"https://api.github.com/users/tsellers-r7/following{/other_user}","gists_url":"https://api.github.com/users/tsellers-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/tsellers-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsellers-r7/subscriptions","organizations_url":"https://api.github.com/users/tsellers-r7/orgs","repos_url":"https://api.github.com/users/tsellers-r7/repos","events_url":"https://api.github.com/users/tsellers-r7/events{/privacy}","received_events_url":"https://api.github.com/users/tsellers-r7/received_events","type":"User","site_admin":false},"body":"The goal of this PR is to enable the using Recog to search **all** fingerprint databases for a match for a given string.  To do this effectively Recog must support filtering of the results in order to tailor them to the caller's needs.   It must allow prioritization of the databases to ensure that those containing the highest quality matches are processed first.  These changes **must** not impact the behavior of existing functions or materially change the results that they currently provide.\r\n\r\nThe changes here were written with an eye towards existing use cases.  They **should** be able to drop in w/o breakage for any environment where the XML XSD has not been modified or isn't being overwritten by a build process.\r\n\r\n###  Code Changes\r\n\r\n* Add `Nizer#match_all_db` and `Nizer#multi_match_all_db` to support matching a string \r\n  against multiple databases.\r\n  * Allow filtering both of the above searches by providing a Hash of filter options\r\n    * `:match_key` - Same as the `match_key` used in the `Nizer#match` search. This\r\n                      value is set at the top level `fingerprints` element in an XML fingerprints DB file. If \r\n                      this value isn't set then it will be the name of the XML file without the extension.\r\n                      This value in the XML file and it's related auto-population were in the existing code.\r\n\r\n    * `:datbase_type` - Allows filtering by fingerprint type such as `service`, `util.os`, etc with the goal of\r\n                      limiting undesirable matches.  An example of this would be matching against an \r\n                      `util.os` fingerprint database just because it has `Windows 2008` in the name. This\r\n                      is a **NEW** value set at the top level `fingerprints` element in an XML fingerprints\r\n                      DB file.\r\n\r\n    * `:protocol` - Allows filtering by protocol such as `smtp`, `ftp`, etc.  This is a **NEW**\r\n                      value set at the top level `fingerprints` element in an XML fingerprints DB\r\n                      file **but** can be over written by specifying a `service.protocol` element in an \r\n                      individual fingerprint match definition.\r\n\r\n  * When `Nizer#match_all_db` or `Nizer#multi_match_all_db` are called they will sort the DBManager object's databases based on the priority specified in the XML fingerprint file. This will happen even if the DBManager object already existed and was created by `Nizer#match` or similar where only one database is loaded.\r\n\r\n* Add `Nizer#load_db` which allows using `Nizer` to load a specific XML fingerprint file or directory of files.  Previously it would, via DBManager, only load the databases in the `xml` directory under the `Recog` directory in use.  It will automatically sort the databases when they are loaded by `Nizer#load_db`.\r\n\r\n* Add `Nizer#display_db_order` which outputs the currently loaded fingerprints in the order in which they will be queried. It also displays their priority.  This is useful for debugging issues related to matching across multiple databases.\r\n\r\n* Modify `DBManager#load_databases` so as to permit loading an individual file.  Previously it would attempt to load all XML files in the specified directory. Now it will detect if the specified path is a file or directory and handle as appropriate.\r\n\r\n* `Fingerprint#initialize` now supports passing in **optional** parameters `match_key` and `protocol` so the fingerprint definitions will contain this information.  By doing it this way it provides this information to any applications that are using the fingerprinting but not doing so via `Recog::Nizer`.  `recog_match` is an example of this.\r\n\r\n*`rspec` tests have been written to handle new code and some tests for existing code were expanded.\r\n\r\n### XML Changes\r\n\r\n* Added support for specifying a `priority` at the top level `fingerprints` of a database. This \r\n   is used by `Nizer#match_all_db` and `Nizer#multi_match_all_db` to determine the order in\r\n   which databases will be used to process a banner string. The values are current intended to\r\n   be between 0 and 100 with the lowest number having priority.  If an XML database doesn't\r\n   have a priority value set then it will be set to the value of `DEFAULT_FP_DB_PRIORITY` as \r\n   defined in `Recog::DB`. This value is currently 100.  I've set the values on certain current\r\n  databases based on the likelihood that the db will provide a quality match. If databases were\r\n  thought to have similar quality then in some cases priority was given to a database that \r\n  covered a more common service. The lowest priority value currently set is 10.  The intent was\r\n  to allow users to always have room to create/modify databases that will have priority over existing\r\n  databases by setting #'s below 10 and to set catch-all/low preference databases by setting #'s\r\n  above 100.\r\n\r\n  **Obviously these values will be the subject of discussion and will be tuned over time.**\r\n\r\n* Add support for specifying a `protocol` at the top level `fingerprints` of a database and at the individual fingerprint level.  Fingerprint match level protocol identifiers take precedent over file/db level identifiers. Protocol can be used as an **optional** filter when calling `Nizer#match_all_db` or `Nizer#multi_match_all_db`.  It **will** be shown in the results.\r\n  * Top level XML example:\r\n    ```xml\r\n    <fingerprints matches=\"ldap.search_result\" protocol=\"ldap\">\r\n    ```\r\n  * Within an individual fingerprint\r\n    ```xml\r\n    <param pos=\"0\" name=\"service.protocol\" value=\"ldap\"/>\r\n    ```\r\n* Added support for specifying a `type` at the top level `fingerprints` of a database. This is used\r\n  to indicate the type of fingerprint the database contains.  Examples of this would be `service`\r\n   and `os`.  This was required as there are certain databases that are only used to identify operating systems based on OS name strings.  This has\r\n  has the potential to create false positives when matching across all databases.  Type can be\r\n  used as an **optional** filter when calling `Nizer#match_all_db` or `Nizer#multi_match_all_db`. \r\n\r\n* Updated `xml/fingerprints.xsd` to support the new `fingerprints` attributes for protocol, type, and priority.\r\n\r\n### Results output changes\r\n\r\nAll output, even that which doesn't call new methods, will contain values for `fingerprint_db` and `service.protocol` when a match is made and if the data is populated in the source XML.  The logic in `Recog::DB` will ensure that `fingerprint_db` is always populated.\r\n\r\n```bash\r\necho 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match xml/ssh_banners.xml -\r\n\r\nMATCH: {\r\n  \"matched\"=>\"OpenSSH running on Ubuntu 14.04\", \r\n  \"service.version\"=>\"6.6p1\", \r\n  \"openssh.comment\"=>\"Ubuntu-2ubuntu1\", \r\n  \"service.vendor\"=>\"OpenBSD\", \r\n  \"service.family\"=>\"OpenSSH\", \r\n  \"service.product\"=>\"OpenSSH\", \r\n  \"os.vendor\"=>\"Ubuntu\", \r\n  \"os.device\"=>\"General\", \"os.family\"=>\"Linux\", \r\n  \"os.product\"=>\"Linux\", \"os.version\"=>\"14.04\", \r\n  \"service.protocol\"=>\"ssh\", \r\n  \"fingerprint_db\"=>\"ssh.banner\", \r\n  \"data\"=>\"OpenSSH_6.6p1 Ubuntu-2ubuntu1\"}\r\n```","created_at":"2016-11-10T16:05:04Z","updated_at":"2016-11-16T01:56:53Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ff37abc2a2b483f1c4ea088d089c539df4918499","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/rapid7/recog/pulls/114/commits","review_comments_url":"https://api.github.com/repos/rapid7/recog/pulls/114/comments","review_comment_url":"https://api.github.com/repos/rapid7/recog/pulls/comments{/number}","comments_url":"https://api.github.com/repos/rapid7/recog/issues/114/comments","statuses_url":"https://api.github.com/repos/rapid7/recog/statuses/8df8fec9dfa293fa109e8ac91176c88bdf65d21b","head":{"label":"tsellers-r7:recog_all_xmls","ref":"recog_all_xmls","sha":"8df8fec9dfa293fa109e8ac91176c88bdf65d21b","user":{"login":"tsellers-r7","id":20306699,"avatar_url":"https://avatars.githubusercontent.com/u/20306699?v=3","gravatar_id":"","url":"https://api.github.com/users/tsellers-r7","html_url":"https://github.com/tsellers-r7","followers_url":"https://api.github.com/users/tsellers-r7/followers","following_url":"https://api.github.com/users/tsellers-r7/following{/other_user}","gists_url":"https://api.github.com/users/tsellers-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/tsellers-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsellers-r7/subscriptions","organizations_url":"https://api.github.com/users/tsellers-r7/orgs","repos_url":"https://api.github.com/users/tsellers-r7/repos","events_url":"https://api.github.com/users/tsellers-r7/events{/privacy}","received_events_url":"https://api.github.com/users/tsellers-r7/received_events","type":"User","site_admin":false},"repo":{"id":65576537,"name":"recog","full_name":"tsellers-r7/recog","owner":{"login":"tsellers-r7","id":20306699,"avatar_url":"https://avatars.githubusercontent.com/u/20306699?v=3","gravatar_id":"","url":"https://api.github.com/users/tsellers-r7","html_url":"https://github.com/tsellers-r7","followers_url":"https://api.github.com/users/tsellers-r7/followers","following_url":"https://api.github.com/users/tsellers-r7/following{/other_user}","gists_url":"https://api.github.com/users/tsellers-r7/gists{/gist_id}","starred_url":"https://api.github.com/users/tsellers-r7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsellers-r7/subscriptions","organizations_url":"https://api.github.com/users/tsellers-r7/orgs","repos_url":"https://api.github.com/users/tsellers-r7/repos","events_url":"https://api.github.com/users/tsellers-r7/events{/privacy}","received_events_url":"https://api.github.com/users/tsellers-r7/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tsellers-r7/recog","description":"Pattern recognition for hosts, services, and content","fork":true,"url":"https://api.github.com/repos/tsellers-r7/recog","forks_url":"https://api.github.com/repos/tsellers-r7/recog/forks","keys_url":"https://api.github.com/repos/tsellers-r7/recog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tsellers-r7/recog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tsellers-r7/recog/teams","hooks_url":"https://api.github.com/repos/tsellers-r7/recog/hooks","issue_events_url":"https://api.github.com/repos/tsellers-r7/recog/issues/events{/number}","events_url":"https://api.github.com/repos/tsellers-r7/recog/events","assignees_url":"https://api.github.com/repos/tsellers-r7/recog/assignees{/user}","branches_url":"https://api.github.com/repos/tsellers-r7/recog/branches{/branch}","tags_url":"https://api.github.com/repos/tsellers-r7/recog/tags","blobs_url":"https://api.github.com/repos/tsellers-r7/recog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tsellers-r7/recog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tsellers-r7/recog/git/refs{/sha}","trees_url":"https://api.github.com/repos/tsellers-r7/recog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tsellers-r7/recog/statuses/{sha}","languages_url":"https://api.github.com/repos/tsellers-r7/recog/languages","stargazers_url":"https://api.github.com/repos/tsellers-r7/recog/stargazers","contributors_url":"https://api.github.com/repos/tsellers-r7/recog/contributors","subscribers_url":"https://api.github.com/repos/tsellers-r7/recog/subscribers","subscription_url":"https://api.github.com/repos/tsellers-r7/recog/subscription","commits_url":"https://api.github.com/repos/tsellers-r7/recog/commits{/sha}","git_commits_url":"https://api.github.com/repos/tsellers-r7/recog/git/commits{/sha}","comments_url":"https://api.github.com/repos/tsellers-r7/recog/comments{/number}","issue_comment_url":"https://api.github.com/repos/tsellers-r7/recog/issues/comments{/number}","contents_url":"https://api.github.com/repos/tsellers-r7/recog/contents/{+path}","compare_url":"https://api.github.com/repos/tsellers-r7/recog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tsellers-r7/recog/merges","archive_url":"https://api.github.com/repos/tsellers-r7/recog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tsellers-r7/recog/downloads","issues_url":"https://api.github.com/repos/tsellers-r7/recog/issues{/number}","pulls_url":"https://api.github.com/repos/tsellers-r7/recog/pulls{/number}","milestones_url":"https://api.github.com/repos/tsellers-r7/recog/milestones{/number}","notifications_url":"https://api.github.com/repos/tsellers-r7/recog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tsellers-r7/recog/labels{/name}","releases_url":"https://api.github.com/repos/tsellers-r7/recog/releases{/id}","deployments_url":"https://api.github.com/repos/tsellers-r7/recog/deployments","created_at":"2016-08-12T18:52:05Z","updated_at":"2016-08-12T18:52:06Z","pushed_at":"2016-11-14T22:57:10Z","git_url":"git://github.com/tsellers-r7/recog.git","ssh_url":"git@github.com:tsellers-r7/recog.git","clone_url":"https://github.com/tsellers-r7/recog.git","svn_url":"https://github.com/tsellers-r7/recog","homepage":"","size":1256,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"rapid7:master","ref":"master","sha":"62c8001887f7ae9fe3d1df0433aebff56cc4aa86","user":{"login":"rapid7","id":1013671,"avatar_url":"https://avatars.githubusercontent.com/u/1013671?v=3","gravatar_id":"","url":"https://api.github.com/users/rapid7","html_url":"https://github.com/rapid7","followers_url":"https://api.github.com/users/rapid7/followers","following_url":"https://api.github.com/users/rapid7/following{/other_user}","gists_url":"https://api.github.com/users/rapid7/gists{/gist_id}","starred_url":"https://api.github.com/users/rapid7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rapid7/subscriptions","organizations_url":"https://api.github.com/users/rapid7/orgs","repos_url":"https://api.github.com/users/rapid7/repos","events_url":"https://api.github.com/users/rapid7/events{/privacy}","received_events_url":"https://api.github.com/users/rapid7/received_events","type":"Organization","site_admin":false},"repo":{"id":15459617,"name":"recog","full_name":"rapid7/recog","owner":{"login":"rapid7","id":1013671,"avatar_url":"https://avatars.githubusercontent.com/u/1013671?v=3","gravatar_id":"","url":"https://api.github.com/users/rapid7","html_url":"https://github.com/rapid7","followers_url":"https://api.github.com/users/rapid7/followers","following_url":"https://api.github.com/users/rapid7/following{/other_user}","gists_url":"https://api.github.com/users/rapid7/gists{/gist_id}","starred_url":"https://api.github.com/users/rapid7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rapid7/subscriptions","organizations_url":"https://api.github.com/users/rapid7/orgs","repos_url":"https://api.github.com/users/rapid7/repos","events_url":"https://api.github.com/users/rapid7/events{/privacy}","received_events_url":"https://api.github.com/users/rapid7/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/rapid7/recog","description":"Pattern recognition for hosts, services, and content","fork":false,"url":"https://api.github.com/repos/rapid7/recog","forks_url":"https://api.github.com/repos/rapid7/recog/forks","keys_url":"https://api.github.com/repos/rapid7/recog/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rapid7/recog/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rapid7/recog/teams","hooks_url":"https://api.github.com/repos/rapid7/recog/hooks","issue_events_url":"https://api.github.com/repos/rapid7/recog/issues/events{/number}","events_url":"https://api.github.com/repos/rapid7/recog/events","assignees_url":"https://api.github.com/repos/rapid7/recog/assignees{/user}","branches_url":"https://api.github.com/repos/rapid7/recog/branches{/branch}","tags_url":"https://api.github.com/repos/rapid7/recog/tags","blobs_url":"https://api.github.com/repos/rapid7/recog/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rapid7/recog/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rapid7/recog/git/refs{/sha}","trees_url":"https://api.github.com/repos/rapid7/recog/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rapid7/recog/statuses/{sha}","languages_url":"https://api.github.com/repos/rapid7/recog/languages","stargazers_url":"https://api.github.com/repos/rapid7/recog/stargazers","contributors_url":"https://api.github.com/repos/rapid7/recog/contributors","subscribers_url":"https://api.github.com/repos/rapid7/recog/subscribers","subscription_url":"https://api.github.com/repos/rapid7/recog/subscription","commits_url":"https://api.github.com/repos/rapid7/recog/commits{/sha}","git_commits_url":"https://api.github.com/repos/rapid7/recog/git/commits{/sha}","comments_url":"https://api.github.com/repos/rapid7/recog/comments{/number}","issue_comment_url":"https://api.github.com/repos/rapid7/recog/issues/comments{/number}","contents_url":"https://api.github.com/repos/rapid7/recog/contents/{+path}","compare_url":"https://api.github.com/repos/rapid7/recog/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rapid7/recog/merges","archive_url":"https://api.github.com/repos/rapid7/recog/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rapid7/recog/downloads","issues_url":"https://api.github.com/repos/rapid7/recog/issues{/number}","pulls_url":"https://api.github.com/repos/rapid7/recog/pulls{/number}","milestones_url":"https://api.github.com/repos/rapid7/recog/milestones{/number}","notifications_url":"https://api.github.com/repos/rapid7/recog/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rapid7/recog/labels{/name}","releases_url":"https://api.github.com/repos/rapid7/recog/releases{/id}","deployments_url":"https://api.github.com/repos/rapid7/recog/deployments","created_at":"2013-12-26T20:16:05Z","updated_at":"2016-11-10T21:23:51Z","pushed_at":"2016-11-14T22:57:10Z","git_url":"git://github.com/rapid7/recog.git","ssh_url":"git@github.com:rapid7/recog.git","clone_url":"https://github.com/rapid7/recog.git","svn_url":"https://github.com/rapid7/recog","homepage":"","size":1584,"stargazers_count":98,"watchers_count":98,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":41,"mirror_url":null,"open_issues_count":7,"forks":41,"open_issues":7,"watchers":98,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/rapid7/recog/pulls/114"},"html":{"href":"https://github.com/rapid7/recog/pull/114"},"issue":{"href":"https://api.github.com/repos/rapid7/recog/issues/114"},"comments":{"href":"https://api.github.com/repos/rapid7/recog/issues/114/comments"},"review_comments":{"href":"https://api.github.com/repos/rapid7/recog/pulls/114/comments"},"review_comment":{"href":"https://api.github.com/repos/rapid7/recog/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/rapid7/recog/pulls/114/commits"},"statuses":{"href":"https://api.github.com/repos/rapid7/recog/statuses/8df8fec9dfa293fa109e8ac91176c88bdf65d21b"}}}}
{"actor":{"display_login":"tsellers-r7"}}