{"action":"created","comment":{"url":"https://api.github.com/repos/Woorank/structured-logging/pulls/comments/61059674","id":61059674,"diff_hunk":"@@ -0,0 +1,289 @@\n+# Log Schema\n+\n+## Introduction\n+\n+This is a proposal for a fixed log schema for all our web applications.\n+Both the client request side and the server response side will be logged.\n+Each log object contains the request, the response and the error objects; all optional.\n+\n+## Caveats\n+\n+- Some errors like Sequelize validation errors contains lists of errors\n+(Jan and I think that in this case we should just pick the first error in that list)\n+\n+- Technically only the custom x-woorank-id header is needed to link a log object created by an internal request made by a client to the corresponding server log object. But if you want to aggregate or query by pairs of (app.name, request.headers.x-woorank-app-name) you'd need to make a query similar to a join which isn't possible in ElasticSearch.\n+(For this a propose that we duplicate the app information of the log object inside the x-worrank-* headers.)\n+\n+- It's difficult to create a fixed schema when there is no fixed enforced header list.\n+(To remdiate to this issue, I think we should log the standard HTTP headers as well as some defacto standard ones like x-forwarded-for and filter the the ones that we agree are not interesting like the date header.)\n+\n+- For security purposes I would remove all headers that containt secrets or leak information like authorization (bearer tokens), cookie / cookie2 (sessions), ETag (cache state), referrer / referer (leaks internal URLs), x-csrf-token (CSRF token).\n+(If we still want this information we could compromise by only logging the length of the value for example or if it contains comas / semi-colons that indicate list values in HTTP headers)\n+\n+- The body would be too big to log.\n+(Same as the previous bullet point, we could log the length of the body so as to not blindly trust the content-length header.)\n+\n+- There are a some headers that have duplicate meanings or have a corresponding x-* header.\n+(It might be interesting to then merge them together.)\n+\n+## Proposed Schema\n+\n+This is the first draft for the log schema.\n+The header list contains all the standard HTTP headers as well as the most used non-standard ones.\n+I also added the proposed x-woorank-* headers.\n+The final list of headers we log will be drastically shorter since a lot of these are not interesting or non-standard / legacy.\n+\n+```json\n+{\n+\t// Note: Unique identifier and primary key\n+\t\"id\": \"string\",\n+\t// Note: Date of the current day in DD-MM-YYYY format\n+\t\"type\": \"string\",\n+\t// Note: Logstash index name\n+\t\"index\": \"string\",\n+\t// Note: Exact insertion date with second precisions\n+\t\"timestamp\": \"date\",\n+\n+\t// Note: Host's IPv4 address\n+\t// (Not sure how to get this information?)\n+\t\"host\": \"string\",\n+\t// Note: Information on the Docker container running\n+\t\"container\": {\n+\t\t\"id\": \"string\",\n+\t\t\"name\": \"string\",\n+\t},\n+\t// Note: Information on the Docker image running\n+\t\"image\": {\n+\t\t\"id\": \"string\",\n+\t\t\"name\": \"string\",\n+\t},\n+\t// Note: Process id (Starts at 1 then is increments in Docker I think)\n+\t\"pid\": \"number\",\n+\n+\t// Note: Application related information\n+\t\"app\": {\n+\t\t// Note: Application name (as in package.json)\n+\t\t\"name\": \"string\",\n+\t\t// Note: Application version (as in package.json)\n+\t\t\"version\": \"string\",\n+\t\t// Note: CircleCI / Docker build number (passed by Dockerfile ENV variables)\n+\t\t\"build\": \"string\",\n+\t\t// Note: Docker command with which the application was launched with\n+\t\t\"command\": \"string\",\n+\t\t// Note: Environement with which the application was launched with (i.e: test or prod not the NODE_ENV)\n+\t\t\"environement\": \"string\",\n+\t},\n+\n+\t// Note: Log level of this message\n+\t// (Decide on format / standard levels for each type of event)\n+\t// e.g: info, warning, error ...etc.\n+\t\"level\": \"number\",\n+\t\"event\": \"string\",\n+\n+\t\"httpVersion\": \"string\",\n+\n+\t\"request\": {\n+\t\t// Note: If the log originates from the client that made the request\n+\t\t\"isOrigin\": \"boolean\",\n+\t\t// Note: If the request was made from and to an internal service\n+\t\t\"isInternal\": \"boolean\",\n+\t\t// Note: IPv4 address of the client making the request\n+\t\t\"address\": \"string\",\n+\t\t// Note: TCP port of the client making the request\n+\t\t\"port\": \"number\", // Note: Not really useful imo\n+\t\t\"method\": \"string\",\n+\t\t\"url\": {\n+\t\t\t\"protocol\": \"string\",\n+\t\t\t\"auth\": \"string\", // Note: Probably insecure to log this\n+\t\t\t\"hostname\": \"string\",\n+\t\t\t\"port\": \"number\",\n+\t\t\t\"pathname\": \"string\",\n+\t\t\t\"query\": \"string\", // Note: We have the choice between a string or an object","path":"doc/log-schema.md","position":101,"original_position":101,"commit_id":"39c393829efa3379c7a3724836e5b33b68fb4dde","original_commit_id":"39c393829efa3379c7a3724836e5b33b68fb4dde","user":{"login":"Janpot","id":2109932,"avatar_url":"https://avatars.githubusercontent.com/u/2109932?v=3","gravatar_id":"","url":"https://api.github.com/users/Janpot","html_url":"https://github.com/Janpot","followers_url":"https://api.github.com/users/Janpot/followers","following_url":"https://api.github.com/users/Janpot/following{/other_user}","gists_url":"https://api.github.com/users/Janpot/gists{/gist_id}","starred_url":"https://api.github.com/users/Janpot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Janpot/subscriptions","organizations_url":"https://api.github.com/users/Janpot/orgs","repos_url":"https://api.github.com/users/Janpot/repos","events_url":"https://api.github.com/users/Janpot/events{/privacy}","received_events_url":"https://api.github.com/users/Janpot/received_events","type":"User","site_admin":false},"body":"object will generate way topo many logs","created_at":"2016-04-26T10:00:58Z","updated_at":"2016-04-26T10:00:58Z","html_url":"https://github.com/Woorank/structured-logging/pull/7#discussion_r61059674","pull_request_url":"https://api.github.com/repos/Woorank/structured-logging/pulls/7","_links":{"self":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/comments/61059674"},"html":{"href":"https://github.com/Woorank/structured-logging/pull/7#discussion_r61059674"},"pull_request":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/7"}}},"pull_request":{"url":"https://api.github.com/repos/Woorank/structured-logging/pulls/7","id":67851648,"html_url":"https://github.com/Woorank/structured-logging/pull/7","diff_url":"https://github.com/Woorank/structured-logging/pull/7.diff","patch_url":"https://github.com/Woorank/structured-logging/pull/7.patch","issue_url":"https://api.github.com/repos/Woorank/structured-logging/issues/7","number":7,"state":"open","locked":false,"title":"First log schema draft.","user":{"login":"Mickael-van-der-Beek","id":4224802,"avatar_url":"https://avatars.githubusercontent.com/u/4224802?v=3","gravatar_id":"","url":"https://api.github.com/users/Mickael-van-der-Beek","html_url":"https://github.com/Mickael-van-der-Beek","followers_url":"https://api.github.com/users/Mickael-van-der-Beek/followers","following_url":"https://api.github.com/users/Mickael-van-der-Beek/following{/other_user}","gists_url":"https://api.github.com/users/Mickael-van-der-Beek/gists{/gist_id}","starred_url":"https://api.github.com/users/Mickael-van-der-Beek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mickael-van-der-Beek/subscriptions","organizations_url":"https://api.github.com/users/Mickael-van-der-Beek/orgs","repos_url":"https://api.github.com/users/Mickael-van-der-Beek/repos","events_url":"https://api.github.com/users/Mickael-van-der-Beek/events{/privacy}","received_events_url":"https://api.github.com/users/Mickael-van-der-Beek/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-04-26T09:17:05Z","updated_at":"2016-04-26T10:00:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ab84de6c45fa93fa2922e4fea1e3274cc9c57f39","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Woorank/structured-logging/pulls/7/commits","review_comments_url":"https://api.github.com/repos/Woorank/structured-logging/pulls/7/comments","review_comment_url":"https://api.github.com/repos/Woorank/structured-logging/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Woorank/structured-logging/issues/7/comments","statuses_url":"https://api.github.com/repos/Woorank/structured-logging/statuses/39c393829efa3379c7a3724836e5b33b68fb4dde","head":{"label":"Woorank:log-schema","ref":"log-schema","sha":"39c393829efa3379c7a3724836e5b33b68fb4dde","user":{"login":"Woorank","id":2657775,"avatar_url":"https://avatars.githubusercontent.com/u/2657775?v=3","gravatar_id":"","url":"https://api.github.com/users/Woorank","html_url":"https://github.com/Woorank","followers_url":"https://api.github.com/users/Woorank/followers","following_url":"https://api.github.com/users/Woorank/following{/other_user}","gists_url":"https://api.github.com/users/Woorank/gists{/gist_id}","starred_url":"https://api.github.com/users/Woorank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Woorank/subscriptions","organizations_url":"https://api.github.com/users/Woorank/orgs","repos_url":"https://api.github.com/users/Woorank/repos","events_url":"https://api.github.com/users/Woorank/events{/privacy}","received_events_url":"https://api.github.com/users/Woorank/received_events","type":"Organization","site_admin":false},"repo":{"id":51521418,"name":"structured-logging","full_name":"Woorank/structured-logging","owner":{"login":"Woorank","id":2657775,"avatar_url":"https://avatars.githubusercontent.com/u/2657775?v=3","gravatar_id":"","url":"https://api.github.com/users/Woorank","html_url":"https://github.com/Woorank","followers_url":"https://api.github.com/users/Woorank/followers","following_url":"https://api.github.com/users/Woorank/following{/other_user}","gists_url":"https://api.github.com/users/Woorank/gists{/gist_id}","starred_url":"https://api.github.com/users/Woorank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Woorank/subscriptions","organizations_url":"https://api.github.com/users/Woorank/orgs","repos_url":"https://api.github.com/users/Woorank/repos","events_url":"https://api.github.com/users/Woorank/events{/privacy}","received_events_url":"https://api.github.com/users/Woorank/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Woorank/structured-logging","description":"Initialises a list of serialisers used to format logger input.","fork":false,"url":"https://api.github.com/repos/Woorank/structured-logging","forks_url":"https://api.github.com/repos/Woorank/structured-logging/forks","keys_url":"https://api.github.com/repos/Woorank/structured-logging/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Woorank/structured-logging/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Woorank/structured-logging/teams","hooks_url":"https://api.github.com/repos/Woorank/structured-logging/hooks","issue_events_url":"https://api.github.com/repos/Woorank/structured-logging/issues/events{/number}","events_url":"https://api.github.com/repos/Woorank/structured-logging/events","assignees_url":"https://api.github.com/repos/Woorank/structured-logging/assignees{/user}","branches_url":"https://api.github.com/repos/Woorank/structured-logging/branches{/branch}","tags_url":"https://api.github.com/repos/Woorank/structured-logging/tags","blobs_url":"https://api.github.com/repos/Woorank/structured-logging/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Woorank/structured-logging/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Woorank/structured-logging/git/refs{/sha}","trees_url":"https://api.github.com/repos/Woorank/structured-logging/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Woorank/structured-logging/statuses/{sha}","languages_url":"https://api.github.com/repos/Woorank/structured-logging/languages","stargazers_url":"https://api.github.com/repos/Woorank/structured-logging/stargazers","contributors_url":"https://api.github.com/repos/Woorank/structured-logging/contributors","subscribers_url":"https://api.github.com/repos/Woorank/structured-logging/subscribers","subscription_url":"https://api.github.com/repos/Woorank/structured-logging/subscription","commits_url":"https://api.github.com/repos/Woorank/structured-logging/commits{/sha}","git_commits_url":"https://api.github.com/repos/Woorank/structured-logging/git/commits{/sha}","comments_url":"https://api.github.com/repos/Woorank/structured-logging/comments{/number}","issue_comment_url":"https://api.github.com/repos/Woorank/structured-logging/issues/comments{/number}","contents_url":"https://api.github.com/repos/Woorank/structured-logging/contents/{+path}","compare_url":"https://api.github.com/repos/Woorank/structured-logging/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Woorank/structured-logging/merges","archive_url":"https://api.github.com/repos/Woorank/structured-logging/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Woorank/structured-logging/downloads","issues_url":"https://api.github.com/repos/Woorank/structured-logging/issues{/number}","pulls_url":"https://api.github.com/repos/Woorank/structured-logging/pulls{/number}","milestones_url":"https://api.github.com/repos/Woorank/structured-logging/milestones{/number}","notifications_url":"https://api.github.com/repos/Woorank/structured-logging/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Woorank/structured-logging/labels{/name}","releases_url":"https://api.github.com/repos/Woorank/structured-logging/releases{/id}","deployments_url":"https://api.github.com/repos/Woorank/structured-logging/deployments","created_at":"2016-02-11T14:52:48Z","updated_at":"2016-02-11T19:04:31Z","pushed_at":"2016-04-26T09:17:05Z","git_url":"git://github.com/Woorank/structured-logging.git","ssh_url":"git@github.com:Woorank/structured-logging.git","clone_url":"https://github.com/Woorank/structured-logging.git","svn_url":"https://github.com/Woorank/structured-logging","homepage":null,"size":27,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"Woorank:master","ref":"master","sha":"51f13632dd79303bbc3a512d561f32a8124784f7","user":{"login":"Woorank","id":2657775,"avatar_url":"https://avatars.githubusercontent.com/u/2657775?v=3","gravatar_id":"","url":"https://api.github.com/users/Woorank","html_url":"https://github.com/Woorank","followers_url":"https://api.github.com/users/Woorank/followers","following_url":"https://api.github.com/users/Woorank/following{/other_user}","gists_url":"https://api.github.com/users/Woorank/gists{/gist_id}","starred_url":"https://api.github.com/users/Woorank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Woorank/subscriptions","organizations_url":"https://api.github.com/users/Woorank/orgs","repos_url":"https://api.github.com/users/Woorank/repos","events_url":"https://api.github.com/users/Woorank/events{/privacy}","received_events_url":"https://api.github.com/users/Woorank/received_events","type":"Organization","site_admin":false},"repo":{"id":51521418,"name":"structured-logging","full_name":"Woorank/structured-logging","owner":{"login":"Woorank","id":2657775,"avatar_url":"https://avatars.githubusercontent.com/u/2657775?v=3","gravatar_id":"","url":"https://api.github.com/users/Woorank","html_url":"https://github.com/Woorank","followers_url":"https://api.github.com/users/Woorank/followers","following_url":"https://api.github.com/users/Woorank/following{/other_user}","gists_url":"https://api.github.com/users/Woorank/gists{/gist_id}","starred_url":"https://api.github.com/users/Woorank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Woorank/subscriptions","organizations_url":"https://api.github.com/users/Woorank/orgs","repos_url":"https://api.github.com/users/Woorank/repos","events_url":"https://api.github.com/users/Woorank/events{/privacy}","received_events_url":"https://api.github.com/users/Woorank/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Woorank/structured-logging","description":"Initialises a list of serialisers used to format logger input.","fork":false,"url":"https://api.github.com/repos/Woorank/structured-logging","forks_url":"https://api.github.com/repos/Woorank/structured-logging/forks","keys_url":"https://api.github.com/repos/Woorank/structured-logging/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Woorank/structured-logging/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Woorank/structured-logging/teams","hooks_url":"https://api.github.com/repos/Woorank/structured-logging/hooks","issue_events_url":"https://api.github.com/repos/Woorank/structured-logging/issues/events{/number}","events_url":"https://api.github.com/repos/Woorank/structured-logging/events","assignees_url":"https://api.github.com/repos/Woorank/structured-logging/assignees{/user}","branches_url":"https://api.github.com/repos/Woorank/structured-logging/branches{/branch}","tags_url":"https://api.github.com/repos/Woorank/structured-logging/tags","blobs_url":"https://api.github.com/repos/Woorank/structured-logging/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Woorank/structured-logging/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Woorank/structured-logging/git/refs{/sha}","trees_url":"https://api.github.com/repos/Woorank/structured-logging/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Woorank/structured-logging/statuses/{sha}","languages_url":"https://api.github.com/repos/Woorank/structured-logging/languages","stargazers_url":"https://api.github.com/repos/Woorank/structured-logging/stargazers","contributors_url":"https://api.github.com/repos/Woorank/structured-logging/contributors","subscribers_url":"https://api.github.com/repos/Woorank/structured-logging/subscribers","subscription_url":"https://api.github.com/repos/Woorank/structured-logging/subscription","commits_url":"https://api.github.com/repos/Woorank/structured-logging/commits{/sha}","git_commits_url":"https://api.github.com/repos/Woorank/structured-logging/git/commits{/sha}","comments_url":"https://api.github.com/repos/Woorank/structured-logging/comments{/number}","issue_comment_url":"https://api.github.com/repos/Woorank/structured-logging/issues/comments{/number}","contents_url":"https://api.github.com/repos/Woorank/structured-logging/contents/{+path}","compare_url":"https://api.github.com/repos/Woorank/structured-logging/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Woorank/structured-logging/merges","archive_url":"https://api.github.com/repos/Woorank/structured-logging/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Woorank/structured-logging/downloads","issues_url":"https://api.github.com/repos/Woorank/structured-logging/issues{/number}","pulls_url":"https://api.github.com/repos/Woorank/structured-logging/pulls{/number}","milestones_url":"https://api.github.com/repos/Woorank/structured-logging/milestones{/number}","notifications_url":"https://api.github.com/repos/Woorank/structured-logging/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Woorank/structured-logging/labels{/name}","releases_url":"https://api.github.com/repos/Woorank/structured-logging/releases{/id}","deployments_url":"https://api.github.com/repos/Woorank/structured-logging/deployments","created_at":"2016-02-11T14:52:48Z","updated_at":"2016-02-11T19:04:31Z","pushed_at":"2016-04-26T09:17:05Z","git_url":"git://github.com/Woorank/structured-logging.git","ssh_url":"git@github.com:Woorank/structured-logging.git","clone_url":"https://github.com/Woorank/structured-logging.git","svn_url":"https://github.com/Woorank/structured-logging","homepage":null,"size":27,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/7"},"html":{"href":"https://github.com/Woorank/structured-logging/pull/7"},"issue":{"href":"https://api.github.com/repos/Woorank/structured-logging/issues/7"},"comments":{"href":"https://api.github.com/repos/Woorank/structured-logging/issues/7/comments"},"review_comments":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/7/comments"},"review_comment":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Woorank/structured-logging/pulls/7/commits"},"statuses":{"href":"https://api.github.com/repos/Woorank/structured-logging/statuses/39c393829efa3379c7a3724836e5b33b68fb4dde"}}}}