{"action":"created","comment":{"url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/comments/78467625","id":78467625,"diff_hunk":"@@ -0,0 +1,199 @@\n+(ns puppetlabs.puppetdb.queue\n+  (:import [java.nio.charset StandardCharsets]\n+           [java.io InputStreamReader BufferedReader InputStream]\n+           [java.util TreeMap HashMap]\n+           [java.nio.file Files LinkOption]\n+           [java.nio.file.attribute FileAttribute])\n+  (:require [clojure.string :as str :refer [re-quote-replacement]]\n+            [puppetlabs.stockpile.queue :as stock]\n+            [clj-time.coerce :as tcoerce]\n+            [puppetlabs.puppetdb.cheshire :as json]\n+            [puppetlabs.puppetdb.command.constants :as constants]\n+            [metrics.timers :refer [timer time!]]\n+            [metrics.counters :refer [inc!]]\n+            [clj-time.core :as time]\n+            [puppetlabs.kitchensink.core :as kitchensink]\n+            [slingshot.slingshot :refer [throw+]]\n+            [clojure.core.async :as async]\n+            [clojure.core.async.impl.protocols :as async-protos]\n+            [puppetlabs.puppetdb.nio :refer [get-path]]))\n+\n+(def metadata-command-names\n+  (vals constants/command-names))\n+\n+(defn stream->json [^InputStream stream]\n+  (try\n+    (-> stream\n+        (InputStreamReader. StandardCharsets/UTF_8)\n+        BufferedReader.\n+        (json/parse-stream true))\n+    (catch Exception e\n+      (throw+ {:kind ::parse-error} e \"Error parsing command\"))))\n+\n+(defn metadata-str [received command version certname]\n+  (format \"%d_%s_%d_%s.json\"\n+          (tcoerce/to-long received) command version certname))\n+\n+(defn- metadata-rx [valid-commands]","path":"src/puppetlabs/puppetdb/queue.clj","position":37,"original_position":37,"commit_id":"7c0b89d29ec81c026b76816de76587ab0b6a5250","original_commit_id":"7c0b89d29ec81c026b76816de76587ab0b6a5250","user":{"login":"rbrw","id":10120698,"avatar_url":"https://avatars.githubusercontent.com/u/10120698?v=3","gravatar_id":"","url":"https://api.github.com/users/rbrw","html_url":"https://github.com/rbrw","followers_url":"https://api.github.com/users/rbrw/followers","following_url":"https://api.github.com/users/rbrw/following{/other_user}","gists_url":"https://api.github.com/users/rbrw/gists{/gist_id}","starred_url":"https://api.github.com/users/rbrw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rbrw/subscriptions","organizations_url":"https://api.github.com/users/rbrw/orgs","repos_url":"https://api.github.com/users/rbrw/repos","events_url":"https://api.github.com/users/rbrw/events{/privacy}","received_events_url":"https://api.github.com/users/rbrw/received_events","type":"User","site_admin":false},"body":"Heh, well I'm sure you know who did it; I probably did that because if felt like internal machinery, but it's certainly not critical.  ","created_at":"2016-09-12T22:34:18Z","updated_at":"2016-09-12T22:34:18Z","html_url":"https://github.com/puppetlabs/puppetdb/pull/2061#discussion_r78467625","pull_request_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061","_links":{"self":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/comments/78467625"},"html":{"href":"https://github.com/puppetlabs/puppetdb/pull/2061#discussion_r78467625"},"pull_request":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061"}}},"pull_request":{"url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061","id":84892846,"html_url":"https://github.com/puppetlabs/puppetdb/pull/2061","diff_url":"https://github.com/puppetlabs/puppetdb/pull/2061.diff","patch_url":"https://github.com/puppetlabs/puppetdb/pull/2061.patch","issue_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/2061","number":2061,"state":"open","locked":false,"title":"Migrate from ActiveMQ to Stockpile","user":{"login":"senior","id":215910,"avatar_url":"https://avatars.githubusercontent.com/u/215910?v=3","gravatar_id":"","url":"https://api.github.com/users/senior","html_url":"https://github.com/senior","followers_url":"https://api.github.com/users/senior/followers","following_url":"https://api.github.com/users/senior/following{/other_user}","gists_url":"https://api.github.com/users/senior/gists{/gist_id}","starred_url":"https://api.github.com/users/senior/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senior/subscriptions","organizations_url":"https://api.github.com/users/senior/orgs","repos_url":"https://api.github.com/users/senior/repos","events_url":"https://api.github.com/users/senior/events{/privacy}","received_events_url":"https://api.github.com/users/senior/received_events","type":"User","site_admin":false},"body":"This PR merges the cmd-proc-refactor branch into master and moves from ActiveMQ to Stockpile. Some parts of this PR have been reviewed in prior PRs (before going into cmd-proc-refactor) and others have not. The code that has been reviewed, often had a caveat of, \"this is good enough for now\" or \"this will change in the next set of commits\" etc. So this review is intended to cover those situations.\r\n\r\nIn particular, I'm looking for review comments to be in one of the three categories.\r\n\r\n1. Things that are obviously bad/wrong/inconsistent and that need to be fixed/reworked/cleaned up before we merge into master.\r\n\r\n2. Significant changes/refactoring that could/should be done soon but after we merge to master (before we cut a release). An example of this is merging the mq_listener and command namespaces (ticketed at [PDB-3038](https://tickets.puppetlabs.com/browse/PDB-3038)).\r\n\r\n3. Significant changes/refactoring that are enabled by this change, but not critical to get in before we release. We'll ticket these and put them in the backlog.\r\n\r\nBelow are the changes we know need to be made before releasing, but were planning to work on after merging to master\r\n\r\n- We currently store commands with a filename <integer>_<received_time>_<command>_<version>_<certname>.json where command is our normal command names (i.e. \"replace catalog\"). We should have something more concise and without spaces [PDB-2982](https://tickets.puppetlabs.com/browse/PDB-2982)\r\n\r\n- Deprecate old ActiveMQ and DLO related config that is no longer relevant [PDB-2948](https://tickets.puppetlabs.com/browse/PDB-2948)\r\n\r\n- Ensure the new stockpile code is HUPable [PDB-2983](https://tickets.puppetlabs.com/browse/PDB-2983)\r\n\r\n- Ensure that in the unlikely case that we have a hugely long certname, that we gracefully handle filenames that are too long [PDB-3039](https://tickets.puppetlabs.com/browse/PDB-3039)\r\n\r\n- Merge the mq_listener and command namespaces [PDB-3038](https://tickets.puppetlabs.com/browse/PDB-3038)\r\n\r\nThere are several performance enhancements we're planning to make before a release, but those should be independent of this set of core functionality.","created_at":"2016-09-12T15:32:32Z","updated_at":"2016-09-12T22:34:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"009ea3414b1b425be741f6bf5d0a0e91e54fb9ce","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061/commits","review_comments_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061/comments","review_comment_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/2061/comments","statuses_url":"https://api.github.com/repos/puppetlabs/puppetdb/statuses/7c0b89d29ec81c026b76816de76587ab0b6a5250","head":{"label":"puppetlabs:cmd-proc-refactor","ref":"cmd-proc-refactor","sha":"7c0b89d29ec81c026b76816de76587ab0b6a5250","user":{"login":"puppetlabs","id":234268,"avatar_url":"https://avatars.githubusercontent.com/u/234268?v=3","gravatar_id":"","url":"https://api.github.com/users/puppetlabs","html_url":"https://github.com/puppetlabs","followers_url":"https://api.github.com/users/puppetlabs/followers","following_url":"https://api.github.com/users/puppetlabs/following{/other_user}","gists_url":"https://api.github.com/users/puppetlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/puppetlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puppetlabs/subscriptions","organizations_url":"https://api.github.com/users/puppetlabs/orgs","repos_url":"https://api.github.com/users/puppetlabs/repos","events_url":"https://api.github.com/users/puppetlabs/events{/privacy}","received_events_url":"https://api.github.com/users/puppetlabs/received_events","type":"Organization","site_admin":false},"repo":{"id":2193032,"name":"puppetdb","full_name":"puppetlabs/puppetdb","owner":{"login":"puppetlabs","id":234268,"avatar_url":"https://avatars.githubusercontent.com/u/234268?v=3","gravatar_id":"","url":"https://api.github.com/users/puppetlabs","html_url":"https://github.com/puppetlabs","followers_url":"https://api.github.com/users/puppetlabs/followers","following_url":"https://api.github.com/users/puppetlabs/following{/other_user}","gists_url":"https://api.github.com/users/puppetlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/puppetlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puppetlabs/subscriptions","organizations_url":"https://api.github.com/users/puppetlabs/orgs","repos_url":"https://api.github.com/users/puppetlabs/repos","events_url":"https://api.github.com/users/puppetlabs/events{/privacy}","received_events_url":"https://api.github.com/users/puppetlabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/puppetlabs/puppetdb","description":"Centralized Puppet Storage","fork":false,"url":"https://api.github.com/repos/puppetlabs/puppetdb","forks_url":"https://api.github.com/repos/puppetlabs/puppetdb/forks","keys_url":"https://api.github.com/repos/puppetlabs/puppetdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/puppetlabs/puppetdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/puppetlabs/puppetdb/teams","hooks_url":"https://api.github.com/repos/puppetlabs/puppetdb/hooks","issue_events_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/events{/number}","events_url":"https://api.github.com/repos/puppetlabs/puppetdb/events","assignees_url":"https://api.github.com/repos/puppetlabs/puppetdb/assignees{/user}","branches_url":"https://api.github.com/repos/puppetlabs/puppetdb/branches{/branch}","tags_url":"https://api.github.com/repos/puppetlabs/puppetdb/tags","blobs_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/puppetlabs/puppetdb/statuses/{sha}","languages_url":"https://api.github.com/repos/puppetlabs/puppetdb/languages","stargazers_url":"https://api.github.com/repos/puppetlabs/puppetdb/stargazers","contributors_url":"https://api.github.com/repos/puppetlabs/puppetdb/contributors","subscribers_url":"https://api.github.com/repos/puppetlabs/puppetdb/subscribers","subscription_url":"https://api.github.com/repos/puppetlabs/puppetdb/subscription","commits_url":"https://api.github.com/repos/puppetlabs/puppetdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/puppetlabs/puppetdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/puppetlabs/puppetdb/contents/{+path}","compare_url":"https://api.github.com/repos/puppetlabs/puppetdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/puppetlabs/puppetdb/merges","archive_url":"https://api.github.com/repos/puppetlabs/puppetdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/puppetlabs/puppetdb/downloads","issues_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues{/number}","pulls_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls{/number}","milestones_url":"https://api.github.com/repos/puppetlabs/puppetdb/milestones{/number}","notifications_url":"https://api.github.com/repos/puppetlabs/puppetdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/puppetlabs/puppetdb/labels{/name}","releases_url":"https://api.github.com/repos/puppetlabs/puppetdb/releases{/id}","deployments_url":"https://api.github.com/repos/puppetlabs/puppetdb/deployments","created_at":"2011-08-11T18:54:46Z","updated_at":"2016-09-06T10:16:10Z","pushed_at":"2016-09-12T21:13:00Z","git_url":"git://github.com/puppetlabs/puppetdb.git","ssh_url":"git@github.com:puppetlabs/puppetdb.git","clone_url":"https://github.com/puppetlabs/puppetdb.git","svn_url":"https://github.com/puppetlabs/puppetdb","homepage":"http://docs.puppetlabs.com/puppetdb","size":20307,"stargazers_count":193,"watchers_count":193,"language":"Clojure","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":161,"mirror_url":null,"open_issues_count":7,"forks":161,"open_issues":7,"watchers":193,"default_branch":"master"}},"base":{"label":"puppetlabs:master","ref":"master","sha":"2150bb18433fd4116e0b8fe05af162171c296ddb","user":{"login":"puppetlabs","id":234268,"avatar_url":"https://avatars.githubusercontent.com/u/234268?v=3","gravatar_id":"","url":"https://api.github.com/users/puppetlabs","html_url":"https://github.com/puppetlabs","followers_url":"https://api.github.com/users/puppetlabs/followers","following_url":"https://api.github.com/users/puppetlabs/following{/other_user}","gists_url":"https://api.github.com/users/puppetlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/puppetlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puppetlabs/subscriptions","organizations_url":"https://api.github.com/users/puppetlabs/orgs","repos_url":"https://api.github.com/users/puppetlabs/repos","events_url":"https://api.github.com/users/puppetlabs/events{/privacy}","received_events_url":"https://api.github.com/users/puppetlabs/received_events","type":"Organization","site_admin":false},"repo":{"id":2193032,"name":"puppetdb","full_name":"puppetlabs/puppetdb","owner":{"login":"puppetlabs","id":234268,"avatar_url":"https://avatars.githubusercontent.com/u/234268?v=3","gravatar_id":"","url":"https://api.github.com/users/puppetlabs","html_url":"https://github.com/puppetlabs","followers_url":"https://api.github.com/users/puppetlabs/followers","following_url":"https://api.github.com/users/puppetlabs/following{/other_user}","gists_url":"https://api.github.com/users/puppetlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/puppetlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puppetlabs/subscriptions","organizations_url":"https://api.github.com/users/puppetlabs/orgs","repos_url":"https://api.github.com/users/puppetlabs/repos","events_url":"https://api.github.com/users/puppetlabs/events{/privacy}","received_events_url":"https://api.github.com/users/puppetlabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/puppetlabs/puppetdb","description":"Centralized Puppet Storage","fork":false,"url":"https://api.github.com/repos/puppetlabs/puppetdb","forks_url":"https://api.github.com/repos/puppetlabs/puppetdb/forks","keys_url":"https://api.github.com/repos/puppetlabs/puppetdb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/puppetlabs/puppetdb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/puppetlabs/puppetdb/teams","hooks_url":"https://api.github.com/repos/puppetlabs/puppetdb/hooks","issue_events_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/events{/number}","events_url":"https://api.github.com/repos/puppetlabs/puppetdb/events","assignees_url":"https://api.github.com/repos/puppetlabs/puppetdb/assignees{/user}","branches_url":"https://api.github.com/repos/puppetlabs/puppetdb/branches{/branch}","tags_url":"https://api.github.com/repos/puppetlabs/puppetdb/tags","blobs_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/refs{/sha}","trees_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/puppetlabs/puppetdb/statuses/{sha}","languages_url":"https://api.github.com/repos/puppetlabs/puppetdb/languages","stargazers_url":"https://api.github.com/repos/puppetlabs/puppetdb/stargazers","contributors_url":"https://api.github.com/repos/puppetlabs/puppetdb/contributors","subscribers_url":"https://api.github.com/repos/puppetlabs/puppetdb/subscribers","subscription_url":"https://api.github.com/repos/puppetlabs/puppetdb/subscription","commits_url":"https://api.github.com/repos/puppetlabs/puppetdb/commits{/sha}","git_commits_url":"https://api.github.com/repos/puppetlabs/puppetdb/git/commits{/sha}","comments_url":"https://api.github.com/repos/puppetlabs/puppetdb/comments{/number}","issue_comment_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues/comments{/number}","contents_url":"https://api.github.com/repos/puppetlabs/puppetdb/contents/{+path}","compare_url":"https://api.github.com/repos/puppetlabs/puppetdb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/puppetlabs/puppetdb/merges","archive_url":"https://api.github.com/repos/puppetlabs/puppetdb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/puppetlabs/puppetdb/downloads","issues_url":"https://api.github.com/repos/puppetlabs/puppetdb/issues{/number}","pulls_url":"https://api.github.com/repos/puppetlabs/puppetdb/pulls{/number}","milestones_url":"https://api.github.com/repos/puppetlabs/puppetdb/milestones{/number}","notifications_url":"https://api.github.com/repos/puppetlabs/puppetdb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/puppetlabs/puppetdb/labels{/name}","releases_url":"https://api.github.com/repos/puppetlabs/puppetdb/releases{/id}","deployments_url":"https://api.github.com/repos/puppetlabs/puppetdb/deployments","created_at":"2011-08-11T18:54:46Z","updated_at":"2016-09-06T10:16:10Z","pushed_at":"2016-09-12T21:13:00Z","git_url":"git://github.com/puppetlabs/puppetdb.git","ssh_url":"git@github.com:puppetlabs/puppetdb.git","clone_url":"https://github.com/puppetlabs/puppetdb.git","svn_url":"https://github.com/puppetlabs/puppetdb","homepage":"http://docs.puppetlabs.com/puppetdb","size":20307,"stargazers_count":193,"watchers_count":193,"language":"Clojure","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":161,"mirror_url":null,"open_issues_count":7,"forks":161,"open_issues":7,"watchers":193,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061"},"html":{"href":"https://github.com/puppetlabs/puppetdb/pull/2061"},"issue":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/issues/2061"},"comments":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/issues/2061/comments"},"review_comments":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061/comments"},"review_comment":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/pulls/2061/commits"},"statuses":{"href":"https://api.github.com/repos/puppetlabs/puppetdb/statuses/7c0b89d29ec81c026b76816de76587ab0b6a5250"}}}}
{"actor":{"display_login":"rbrw"}}