{"action":"created","issue":{"url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2500","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2500/labels{/name}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2500/comments","events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2500/events","html_url":"https://github.com/ClusterHQ/flocker/issues/2500","id":127805524,"number":2500,"title":"Reduce overly noisy logging of services","user":{"login":"sgnn7","id":175811,"avatar_url":"https://avatars.githubusercontent.com/u/175811?v=3","gravatar_id":"","url":"https://api.github.com/users/sgnn7","html_url":"https://github.com/sgnn7","followers_url":"https://api.github.com/users/sgnn7/followers","following_url":"https://api.github.com/users/sgnn7/following{/other_user}","gists_url":"https://api.github.com/users/sgnn7/gists{/gist_id}","starred_url":"https://api.github.com/users/sgnn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sgnn7/subscriptions","organizations_url":"https://api.github.com/users/sgnn7/orgs","repos_url":"https://api.github.com/users/sgnn7/repos","events_url":"https://api.github.com/users/sgnn7/events{/privacy}","received_events_url":"https://api.github.com/users/sgnn7/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ClusterHQ/flocker/labels/kind:%20feature-request","name":"kind: feature-request","color":"006b75"},{"url":"https://api.github.com/repos/ClusterHQ/flocker/labels/kind:%20feedback","name":"kind: feedback","color":"bfdadc"},{"url":"https://api.github.com/repos/ClusterHQ/flocker/labels/kind:%20using%20flocker","name":"kind: using flocker","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-01-20T23:02:40Z","updated_at":"2016-01-21T16:08:49Z","closed_at":null,"body":"All services seem to create an exorbitant amount of log output on systemd machines that literally eats up almost any amount of space on a machine/VM that it's installed on:\r\n- flocker-* services have `--journald` appended to them\r\n- They all are really chatty - at least one message per service per second\r\n- `/etc/systemd/journald.conf` has `ForwardToSyslog=True` by default so it sends everything to `/var/log/syslog`\r\n- `/var/log/syslog` is rotated daily and 7 are saved by default\r\n- `/var/log/kern.log` is weekly and 4 are saved default\r\n- We get about >1GB of logs per day (roughly) on a working cluster with no apps running and exponentially more when there's Flocker problems\r\n- Within a few days, we run out of space on slave/master node\r\n\r\nOptions:\r\n- Squelch Flocker services with removal of `--journald` and/or addition of `StandardOutput=null` in unit files (https://github.com/ClusterHQ/flocker/tree/1.9.0/admin/package-files/systemd and https://github.com/ClusterHQ/flocker/blob/1.9.0/admin/package-files/docker-plugin/systemd/flocker-docker-plugin.service). \r\n- Fix Eliot to not log these by default\r\n- Fix Flocker service messages to not print as many debug messages but only real informational messages."},"comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments/173619200","html_url":"https://github.com/ClusterHQ/flocker/issues/2500#issuecomment-173619200","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2500","id":173619200,"user":{"login":"sgnn7","id":175811,"avatar_url":"https://avatars.githubusercontent.com/u/175811?v=3","gravatar_id":"","url":"https://api.github.com/users/sgnn7","html_url":"https://github.com/sgnn7","followers_url":"https://api.github.com/users/sgnn7/followers","following_url":"https://api.github.com/users/sgnn7/following{/other_user}","gists_url":"https://api.github.com/users/sgnn7/gists{/gist_id}","starred_url":"https://api.github.com/users/sgnn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sgnn7/subscriptions","organizations_url":"https://api.github.com/users/sgnn7/orgs","repos_url":"https://api.github.com/users/sgnn7/repos","events_url":"https://api.github.com/users/sgnn7/events{/privacy}","received_events_url":"https://api.github.com/users/sgnn7/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T16:08:49Z","updated_at":"2016-01-21T16:08:49Z","body":"@itamarst It's Ubuntu 15.10 (systemd) and *all* of the services are pretty noisy - I had to manually squelch/override these units in `/etc/systemd/system` in my deployment: flocker-control, flocker-docker-plugin, flocker-dataset-agent, and flocker-container-agent. \r\n\r\nEven if you disable things from going to syslog, I think kern.log still gets this stuff and will consume a lot of space, just on a smaller scale and might get log-rotated before it eats up the allocated space. Either way, seeing each service report \"success\" json each second on every machine is a bit superfluous and on the \"debug\" output side of things imo since signal-to-noise ratio is pretty low."}}