{"action":"created","comment":{"url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/comments/273177347","pull_request_review_id":224009190,"id":273177347,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI3MzE3NzM0Nw==","diff_hunk":"@@ -0,0 +1,39 @@\n+(ns org.purefn.kurosawa.config.env\n+  \"Load configuration from environment variables.\"\n+  (:require [clojure.string :as str]\n+            [org.purefn.kurosawa.config.parse :as parse]))\n+\n+(defn- munge-name\n+  [s]\n+  (str/join\n+   \"-\"\n+   (-> (str/lower-case s)\n+       (str/split #\"_\")\n+       (rest))))\n+\n+(defn- env-variables*\n+  []\n+  (->> (System/getenv)\n+       (map (juxt (comp str/lower-case key)\n+                  (comp munge-name key)\n+                  val))\n+       (group-by (comp first #(str/split % #\"_\") first))\n+       (into {})))\n+\n+(def env-variables (memoize env-variables*))\n+\n+(defn fetch-config","path":"core/src/org/purefn/kurosawa/config/env.clj","position":25,"original_position":25,"commit_id":"e27d7cb90aa925792e7f7035db2e85eb4f4e2172","original_commit_id":"e27d7cb90aa925792e7f7035db2e85eb4f4e2172","user":{"login":"kocubinski","id":1236532,"node_id":"MDQ6VXNlcjEyMzY1MzI=","avatar_url":"https://avatars1.githubusercontent.com/u/1236532?v=4","gravatar_id":"","url":"https://api.github.com/users/kocubinski","html_url":"https://github.com/kocubinski","followers_url":"https://api.github.com/users/kocubinski/followers","following_url":"https://api.github.com/users/kocubinski/following{/other_user}","gists_url":"https://api.github.com/users/kocubinski/gists{/gist_id}","starred_url":"https://api.github.com/users/kocubinski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kocubinski/subscriptions","organizations_url":"https://api.github.com/users/kocubinski/orgs","repos_url":"https://api.github.com/users/kocubinski/repos","events_url":"https://api.github.com/users/kocubinski/events{/privacy}","received_events_url":"https://api.github.com/users/kocubinski/received_events","type":"User","site_admin":false},"body":"In general I agree, with the caveat that a component should be able to coerce an input map of config into the structure it expects, then validate it against a spec.  Otherwise a whole lot of copy-paste goes on between deployments, jobs, streams apps, dev namespaces, etc.\r\n\r\nOf course you can always wire up configuration manually, but all of the import sources outlined here will emit a config map of the same structure, 2-levels deep, with string, kebab-cased, keys.\r\n\r\nI'm also still trying to get to a place where something like\r\n```(create-system [:big-domain-component])``` is possible, injected with all dependencies and config from the current environment.  So wherever config lands should support that too, ultimately.","created_at":"2019-04-08T18:18:06Z","updated_at":"2019-04-08T18:18:07Z","html_url":"https://github.com/PureFnOrg/kurosawa/pull/11#discussion_r273177347","pull_request_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/comments/273177347"},"html":{"href":"https://github.com/PureFnOrg/kurosawa/pull/11#discussion_r273177347"},"pull_request":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11"}},"in_reply_to_id":273093854},"pull_request":{"url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11","id":268045360,"node_id":"MDExOlB1bGxSZXF1ZXN0MjY4MDQ1MzYw","html_url":"https://github.com/PureFnOrg/kurosawa/pull/11","diff_url":"https://github.com/PureFnOrg/kurosawa/pull/11.diff","patch_url":"https://github.com/PureFnOrg/kurosawa/pull/11.patch","issue_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/11","number":11,"state":"open","locked":false,"title":"Configuration improvements","user":{"login":"kocubinski","id":1236532,"node_id":"MDQ6VXNlcjEyMzY1MzI=","avatar_url":"https://avatars1.githubusercontent.com/u/1236532?v=4","gravatar_id":"","url":"https://api.github.com/users/kocubinski","html_url":"https://github.com/kocubinski","followers_url":"https://api.github.com/users/kocubinski/followers","following_url":"https://api.github.com/users/kocubinski/following{/other_user}","gists_url":"https://api.github.com/users/kocubinski/gists{/gist_id}","starred_url":"https://api.github.com/users/kocubinski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kocubinski/subscriptions","organizations_url":"https://api.github.com/users/kocubinski/orgs","repos_url":"https://api.github.com/users/kocubinski/repos","events_url":"https://api.github.com/users/kocubinski/events{/privacy}","received_events_url":"https://api.github.com/users/kocubinski/received_events","type":"User","site_admin":false},"body":"Ladders' current convention for upholding the [Config Principle](https://12factor.net/config) of the 12-factor app is to store config/secrets on the filesystem.  Configuration is loaded by each component at application startup time.  This approach has worked, but has some drawbacks:\r\n\r\n- Manually mounting ConfigMaps/Secrets in Kubernetes is a tedious, error-prone task when creating new deployments and jobs\r\n- Sensitive information may be sitting in plaintext on the file system during local development, and the confusing `KUBERNETES_SERVICE_HOST` environment variable must be set for this to work.  Otherwise the engineer must manually inject all necessary config/secrets themselves, leading to (more often than not) secrets in source control.\r\n\r\nThis pull request introduces a more flexible way to inject config/secrets into an application or library from a variety of sources, without breaking our existing contract.  The file system implementation is maintained as a last resort, and to keep backward compatibility.  Configuration is fetched from (in order of precedence):\r\n\r\n1) Environment Variables\r\n2) AWS SSM Parameter Store\r\n3) The file system.\r\n\r\nThe first source to return to a map of any size is used.\r\n\r\n Environment variables are more or the less the standard approach, but still require some additional machinery to bootstrap.  For this reason the SSM store is next, and will likely be the de facto source for Ladders in the near term for a few reasons:\r\n\r\n- We use AWS as our cloud Provider, so arranging engineers' access to parameters is easy.\r\n- Permission for AWS is already inherited from the environment on a local machine and an EC2 node.\r\n- No additional config/secrets mounting are needed in Kubernetes deploy YAMLs.  The mechanism to ship config is the same both locally and in the cloud.\r\n- `lein repl` should _just work_, as long as AWS credentials are present.\r\n\r\n\r\nThank you to @ashea-ladders for the introduction to the Parameter Store, and @MerelyAPseudonym for the chat about this design!","created_at":"2019-04-06T16:12:52Z","updated_at":"2019-04-08T18:18:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"54c45cdd5e48a9f2152f24844f3be08c103e87b3","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11/commits","review_comments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11/comments","review_comment_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/11/comments","statuses_url":"https://api.github.com/repos/PureFnOrg/kurosawa/statuses/e27d7cb90aa925792e7f7035db2e85eb4f4e2172","head":{"label":"PureFnOrg:config","ref":"config","sha":"e27d7cb90aa925792e7f7035db2e85eb4f4e2172","user":{"login":"PureFnOrg","id":26098233,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDk4MjMz","avatar_url":"https://avatars1.githubusercontent.com/u/26098233?v=4","gravatar_id":"","url":"https://api.github.com/users/PureFnOrg","html_url":"https://github.com/PureFnOrg","followers_url":"https://api.github.com/users/PureFnOrg/followers","following_url":"https://api.github.com/users/PureFnOrg/following{/other_user}","gists_url":"https://api.github.com/users/PureFnOrg/gists{/gist_id}","starred_url":"https://api.github.com/users/PureFnOrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PureFnOrg/subscriptions","organizations_url":"https://api.github.com/users/PureFnOrg/orgs","repos_url":"https://api.github.com/users/PureFnOrg/repos","events_url":"https://api.github.com/users/PureFnOrg/events{/privacy}","received_events_url":"https://api.github.com/users/PureFnOrg/received_events","type":"Organization","site_admin":false},"repo":{"id":128401724,"node_id":"MDEwOlJlcG9zaXRvcnkxMjg0MDE3MjQ=","name":"kurosawa","full_name":"PureFnOrg/kurosawa","private":false,"owner":{"login":"PureFnOrg","id":26098233,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDk4MjMz","avatar_url":"https://avatars1.githubusercontent.com/u/26098233?v=4","gravatar_id":"","url":"https://api.github.com/users/PureFnOrg","html_url":"https://github.com/PureFnOrg","followers_url":"https://api.github.com/users/PureFnOrg/followers","following_url":"https://api.github.com/users/PureFnOrg/following{/other_user}","gists_url":"https://api.github.com/users/PureFnOrg/gists{/gist_id}","starred_url":"https://api.github.com/users/PureFnOrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PureFnOrg/subscriptions","organizations_url":"https://api.github.com/users/PureFnOrg/orgs","repos_url":"https://api.github.com/users/PureFnOrg/repos","events_url":"https://api.github.com/users/PureFnOrg/events{/privacy}","received_events_url":"https://api.github.com/users/PureFnOrg/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/PureFnOrg/kurosawa","description":"Kurosawa base library","fork":false,"url":"https://api.github.com/repos/PureFnOrg/kurosawa","forks_url":"https://api.github.com/repos/PureFnOrg/kurosawa/forks","keys_url":"https://api.github.com/repos/PureFnOrg/kurosawa/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PureFnOrg/kurosawa/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PureFnOrg/kurosawa/teams","hooks_url":"https://api.github.com/repos/PureFnOrg/kurosawa/hooks","issue_events_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/events{/number}","events_url":"https://api.github.com/repos/PureFnOrg/kurosawa/events","assignees_url":"https://api.github.com/repos/PureFnOrg/kurosawa/assignees{/user}","branches_url":"https://api.github.com/repos/PureFnOrg/kurosawa/branches{/branch}","tags_url":"https://api.github.com/repos/PureFnOrg/kurosawa/tags","blobs_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/refs{/sha}","trees_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PureFnOrg/kurosawa/statuses/{sha}","languages_url":"https://api.github.com/repos/PureFnOrg/kurosawa/languages","stargazers_url":"https://api.github.com/repos/PureFnOrg/kurosawa/stargazers","contributors_url":"https://api.github.com/repos/PureFnOrg/kurosawa/contributors","subscribers_url":"https://api.github.com/repos/PureFnOrg/kurosawa/subscribers","subscription_url":"https://api.github.com/repos/PureFnOrg/kurosawa/subscription","commits_url":"https://api.github.com/repos/PureFnOrg/kurosawa/commits{/sha}","git_commits_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/commits{/sha}","comments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/comments{/number}","issue_comment_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/comments{/number}","contents_url":"https://api.github.com/repos/PureFnOrg/kurosawa/contents/{+path}","compare_url":"https://api.github.com/repos/PureFnOrg/kurosawa/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PureFnOrg/kurosawa/merges","archive_url":"https://api.github.com/repos/PureFnOrg/kurosawa/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PureFnOrg/kurosawa/downloads","issues_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues{/number}","pulls_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls{/number}","milestones_url":"https://api.github.com/repos/PureFnOrg/kurosawa/milestones{/number}","notifications_url":"https://api.github.com/repos/PureFnOrg/kurosawa/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PureFnOrg/kurosawa/labels{/name}","releases_url":"https://api.github.com/repos/PureFnOrg/kurosawa/releases{/id}","deployments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/deployments","created_at":"2018-04-06T13:57:08Z","updated_at":"2019-01-07T20:10:13Z","pushed_at":"2019-04-08T14:51:58Z","git_url":"git://github.com/PureFnOrg/kurosawa.git","ssh_url":"git@github.com:PureFnOrg/kurosawa.git","clone_url":"https://github.com/PureFnOrg/kurosawa.git","svn_url":"https://github.com/PureFnOrg/kurosawa","homepage":null,"size":106,"stargazers_count":2,"watchers_count":2,"language":"Clojure","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":3,"license":null,"forks":2,"open_issues":3,"watchers":2,"default_branch":"master"}},"base":{"label":"PureFnOrg:master","ref":"master","sha":"6705ed8ab6f38feef9d849ed9389ca17a86744e7","user":{"login":"PureFnOrg","id":26098233,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDk4MjMz","avatar_url":"https://avatars1.githubusercontent.com/u/26098233?v=4","gravatar_id":"","url":"https://api.github.com/users/PureFnOrg","html_url":"https://github.com/PureFnOrg","followers_url":"https://api.github.com/users/PureFnOrg/followers","following_url":"https://api.github.com/users/PureFnOrg/following{/other_user}","gists_url":"https://api.github.com/users/PureFnOrg/gists{/gist_id}","starred_url":"https://api.github.com/users/PureFnOrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PureFnOrg/subscriptions","organizations_url":"https://api.github.com/users/PureFnOrg/orgs","repos_url":"https://api.github.com/users/PureFnOrg/repos","events_url":"https://api.github.com/users/PureFnOrg/events{/privacy}","received_events_url":"https://api.github.com/users/PureFnOrg/received_events","type":"Organization","site_admin":false},"repo":{"id":128401724,"node_id":"MDEwOlJlcG9zaXRvcnkxMjg0MDE3MjQ=","name":"kurosawa","full_name":"PureFnOrg/kurosawa","private":false,"owner":{"login":"PureFnOrg","id":26098233,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MDk4MjMz","avatar_url":"https://avatars1.githubusercontent.com/u/26098233?v=4","gravatar_id":"","url":"https://api.github.com/users/PureFnOrg","html_url":"https://github.com/PureFnOrg","followers_url":"https://api.github.com/users/PureFnOrg/followers","following_url":"https://api.github.com/users/PureFnOrg/following{/other_user}","gists_url":"https://api.github.com/users/PureFnOrg/gists{/gist_id}","starred_url":"https://api.github.com/users/PureFnOrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PureFnOrg/subscriptions","organizations_url":"https://api.github.com/users/PureFnOrg/orgs","repos_url":"https://api.github.com/users/PureFnOrg/repos","events_url":"https://api.github.com/users/PureFnOrg/events{/privacy}","received_events_url":"https://api.github.com/users/PureFnOrg/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/PureFnOrg/kurosawa","description":"Kurosawa base library","fork":false,"url":"https://api.github.com/repos/PureFnOrg/kurosawa","forks_url":"https://api.github.com/repos/PureFnOrg/kurosawa/forks","keys_url":"https://api.github.com/repos/PureFnOrg/kurosawa/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PureFnOrg/kurosawa/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PureFnOrg/kurosawa/teams","hooks_url":"https://api.github.com/repos/PureFnOrg/kurosawa/hooks","issue_events_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/events{/number}","events_url":"https://api.github.com/repos/PureFnOrg/kurosawa/events","assignees_url":"https://api.github.com/repos/PureFnOrg/kurosawa/assignees{/user}","branches_url":"https://api.github.com/repos/PureFnOrg/kurosawa/branches{/branch}","tags_url":"https://api.github.com/repos/PureFnOrg/kurosawa/tags","blobs_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/refs{/sha}","trees_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PureFnOrg/kurosawa/statuses/{sha}","languages_url":"https://api.github.com/repos/PureFnOrg/kurosawa/languages","stargazers_url":"https://api.github.com/repos/PureFnOrg/kurosawa/stargazers","contributors_url":"https://api.github.com/repos/PureFnOrg/kurosawa/contributors","subscribers_url":"https://api.github.com/repos/PureFnOrg/kurosawa/subscribers","subscription_url":"https://api.github.com/repos/PureFnOrg/kurosawa/subscription","commits_url":"https://api.github.com/repos/PureFnOrg/kurosawa/commits{/sha}","git_commits_url":"https://api.github.com/repos/PureFnOrg/kurosawa/git/commits{/sha}","comments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/comments{/number}","issue_comment_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/comments{/number}","contents_url":"https://api.github.com/repos/PureFnOrg/kurosawa/contents/{+path}","compare_url":"https://api.github.com/repos/PureFnOrg/kurosawa/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PureFnOrg/kurosawa/merges","archive_url":"https://api.github.com/repos/PureFnOrg/kurosawa/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PureFnOrg/kurosawa/downloads","issues_url":"https://api.github.com/repos/PureFnOrg/kurosawa/issues{/number}","pulls_url":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls{/number}","milestones_url":"https://api.github.com/repos/PureFnOrg/kurosawa/milestones{/number}","notifications_url":"https://api.github.com/repos/PureFnOrg/kurosawa/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PureFnOrg/kurosawa/labels{/name}","releases_url":"https://api.github.com/repos/PureFnOrg/kurosawa/releases{/id}","deployments_url":"https://api.github.com/repos/PureFnOrg/kurosawa/deployments","created_at":"2018-04-06T13:57:08Z","updated_at":"2019-01-07T20:10:13Z","pushed_at":"2019-04-08T14:51:58Z","git_url":"git://github.com/PureFnOrg/kurosawa.git","ssh_url":"git@github.com:PureFnOrg/kurosawa.git","clone_url":"https://github.com/PureFnOrg/kurosawa.git","svn_url":"https://github.com/PureFnOrg/kurosawa","homepage":null,"size":106,"stargazers_count":2,"watchers_count":2,"language":"Clojure","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":3,"license":null,"forks":2,"open_issues":3,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11"},"html":{"href":"https://github.com/PureFnOrg/kurosawa/pull/11"},"issue":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/11"},"comments":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/issues/11/comments"},"review_comments":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11/comments"},"review_comment":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/pulls/11/commits"},"statuses":{"href":"https://api.github.com/repos/PureFnOrg/kurosawa/statuses/e27d7cb90aa925792e7f7035db2e85eb4f4e2172"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"kocubinski"}}