{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24677","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24677/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24677/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24677/events","html_url":"https://github.com/kubernetes/kubernetes/issues/24677","id":150424721,"number":24677,"title":"Kubernetes logging, journalD, fluentD, and Splunk, oh my!","user":{"login":"dcowden","id":1297923,"avatar_url":"https://avatars.githubusercontent.com/u/1297923?v=3","gravatar_id":"","url":"https://api.github.com/users/dcowden","html_url":"https://github.com/dcowden","followers_url":"https://api.github.com/users/dcowden/followers","following_url":"https://api.github.com/users/dcowden/following{/other_user}","gists_url":"https://api.github.com/users/dcowden/gists{/gist_id}","starred_url":"https://api.github.com/users/dcowden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcowden/subscriptions","organizations_url":"https://api.github.com/users/dcowden/orgs","repos_url":"https://api.github.com/users/dcowden/repos","events_url":"https://api.github.com/users/dcowden/events{/privacy}","received_events_url":"https://api.github.com/users/dcowden/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/logging","name":"area/logging","color":"fef2c0"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2016-04-22T17:13:34Z","updated_at":"2016-04-30T14:15:48Z","closed_at":null,"body":"In order to run k8s in our environment, I need to get logging information into Splunk.  I need to pick up node logs AND application logs. I know at this point there is no drop-in solution, so I know i'll need to hack something together.  \r\n\r\nWhat I'm seeking in this issue is not a 'here's the solution', though that would be great of course. \r\n\r\nI'm seeking advice on which of several solutions I've come up with are most likely to work, given how Kubernetes is evolving.  \r\n\r\nResearch and references\r\n------------------------------------\r\n\r\nIssue https://github.com/kubernetes/kubernetes/issues/1071  seems to have resulted in the current ability to choose 'elasticsearch' as a provider in kube-up.sh\r\n\r\nIssue  https://github.com/kubernetes/kubernetes/issues/17183  seems to indicate that the future is not yet clear.\r\n\r\nIssue https://github.com/kubernetes/kubernetes/issues/23782 contemplates some limitations of the current approach also\r\n\r\nIssue https://github.com/kubernetes/kubernetes/issues/21285 provides a proxy to send data to AWS instead of ES\r\n\r\nBased on many sources, it seems clear that systemd/journalctl is the future of logging in *nix. latest versions of RHEL/Centos/Fedora  and Ubuntu have moved this way.\r\n\r\nWhile i'm of course willing to hack, i'd really rather avoid needing to re-build the k8s images if possible.\r\n\r\nOption 1 -- k8s->fluentd-->splunk\r\n-----------------------------------------------\r\nI could use fluentd, and then use a fluentd splunk plugin.  If I use kube-up.sh with KUBERNETES_LOGGING_DESTINATION='elasticsearch', I get a fluentd/ES setup.  There are plugins for fluentd that can forward content on to splunk.\r\n\r\nBut i do not know how i would configure k8s to NOT install elasticsearch as a part of startup. I also think this represents an extra layer of forwarding i'd rather avoid.   With this solution, I think i would need to send all application output to STDOUT/STDERR, since this is how k8s currently gathers stuff.  This solution does not use  journald, which causes me to think this option is a doomed-to-change-in-the-near-future.   Which leads me to option 2\r\n\r\nOption 2-- k8s->journald, docker->journald\r\n-----------------------------------------------------------\r\nI could forward all logs ( from the nodes and pods too ) to journalD using the docker journald log driver, and then capture data out of the journald logs and send to splunk from there.  Honestly this seems like the 'right' solution.  Why re-invent logging capture?  If not already true, any strong security setup is going to require use of centralized logging capture,  and that will need to be based on journald. \r\n\r\nThere are two problems with this, though: \r\n   1. doing it this way will not allow kubernetes metadata to be included in the log stream, and \r\n   2. I believe this will break kubectl logs, because it relies on the docker json logs.\r\n\r\nOption 3 -- docker -> k8s -> journald\r\n----------------------------------------------------\r\nLogging directly from docker to journald means there isnt a chance to add k8s meta.  What might be the coolest option is if k8s provided a docker logging driver that then proxies the data and sends it to journald.  That way, as a system administrator, i just need to go to one place to get all my logs-- the journald service on each node.  But i still get the k8s metadata too.\r\n\r\nI have no reasonable ability to execute this option though, it would be a big change. But i'd be willing to help if this is a good way to do it.\r\n\r\n\r\nOption 4-- splunk docker log driver\r\n-------------------------------------------------\r\n\r\nSplunk has an experimental log driver I could use. But it doesnt allow k8s to see the data, or to enrich it ( IE, it will certainly break k8s logs ). This might work as a workaround, but it is not appealing.\r\n\r\nhttp://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/\r\n\r\nClosing Thoughts\r\n-------------------------------\r\n\r\nIf option 1 is the best, I could use some ideas about how i could get K8s to start up fluentD, but NOT start elasticsearch.\r\n\r\nIf option 2 is best, I could use some help pointing me in the right direction about how to avoid breaking kubernetes when i configure the docker daemons on the nodes to send logs to journald instead of the json logs.\r\n\r\nif option 3 is best, I need some pointers on how I could contribute.\r\n\r\nThanks for insights you can offer.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/215968351","html_url":"https://github.com/kubernetes/kubernetes/issues/24677#issuecomment-215968351","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24677","id":215968351,"user":{"login":"dcowden","id":1297923,"avatar_url":"https://avatars.githubusercontent.com/u/1297923?v=3","gravatar_id":"","url":"https://api.github.com/users/dcowden","html_url":"https://github.com/dcowden","followers_url":"https://api.github.com/users/dcowden/followers","following_url":"https://api.github.com/users/dcowden/following{/other_user}","gists_url":"https://api.github.com/users/dcowden/gists{/gist_id}","starred_url":"https://api.github.com/users/dcowden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcowden/subscriptions","organizations_url":"https://api.github.com/users/dcowden/orgs","repos_url":"https://api.github.com/users/dcowden/repos","events_url":"https://api.github.com/users/dcowden/events{/privacy}","received_events_url":"https://api.github.com/users/dcowden/received_events","type":"User","site_admin":false},"created_at":"2016-04-30T14:15:48Z","updated_at":"2016-04-30T14:15:48Z","body":"Ok I was thinking about this more, and I have a nagging 'detail' that I am worried could become a big deal.\r\n\r\n### The problem: some apps generate messages that cannot be enriched without parsing them\r\n\r\nConsider this JVM gc logging format:\r\n\r\n`2015-05-26T14:45:59.690-0200: 172.829: [GC (Allocation Failure) 172.829: [DefNew: 629120K->629120K(629120K), 0.0000372 secs]172.829: [Tenured: 1203359K->755802K(1398144K), 0.1855567 secs] 1832479K->755802K(2027264K), [Metaspace: 6741K->6741K(1056768K)], 0.1856954 secs] [Times: user=0.18 sys=0.00, real=0.18 secs]`\r\n\r\nIt is non-trivial to convert this into a sane, fielded message. Tools like fluentd and splunk can do it, with some work. We've agreed that we do NOT want k8s to get into the parsing businesses.\r\n\r\n### The current state  is broken, but its not kubernetes' fault.\r\n\r\nToday, docker journald and json drivers encapsulate this type of message into a json message.\r\n\r\n```\r\n{\r\n     containerid:  'a7b82c9sd3f92202'\r\n     time: '2015-05-26T14:45:59.690'\r\n     pid : '20122'\r\n     message: '2015-05-26T14:45:59.690-0200: 172.829: [GC (Allocation Failure) 172.829: [DefNew: 629120K->629120K(629120K), 0.0000372 secs]172.829: [Tenured: 1203359K->755802K(1398144K), 0.1855567 secs] 1832479K->755802K(2027264K), [Metaspace: 6741K->6741K(1056768K)], 0.1856954 secs] [Times: user=0.18 sys=0.00, real=0.18 secs]'\r\n\r\n     other-fields...\r\n}\r\n```\r\n\r\nBoth fluentd and splunk will have a hard time with this type of message. It is now an 'enveloped' format. Getting the top-layer, json fields out of it is easy, but getting those AND then dealing with the structure of the internal message is not easy at all.  \r\n\r\nCurrently most of our applications are splunk optimized, so we have messages that look like this:\r\n\r\n`2015-05-26T14:45:59.690  class=MyJavaClass priority=2 user=joe url=http:/my/url  message`\r\n\r\nAnd then end up enveloped like this:\r\n\r\n```\r\n{\r\n     containerid:  'a7b82c9sd3f92202'\r\n     time: '2015-05-26T14:45:59.690'\r\n     pid : '20122'\r\n     message: '`2015-05-26T14:45:59.690  class=MyJavaClass priority=2 user=joe url=http:/my/url  message`'\r\n}\r\n```\r\n\r\nK8S isn't making things worse, and could easily add fields into these encapsulated messages since the outer message is json. But the situation still stinks.\r\n\r\n### How can we collect the logs, while also letting k8s and docker add fields along the way?\r\n\r\n1. **Make all apps use json**  Obviously, this wont work for lots of cases, like the JVM GC logs\r\n\r\n2. **Stick with the envelope format**, like things are working today. Then write fluentd and splunk parsers that understand this format.  That'd be quite a pain, but i guess its somehow do-able. Offhand i have no idea how to do it in splunk though.\r\n\r\n3. **Send logs to fluentd before docker/k8s gets them**  FLuentd gets it first, and can parse it and produce a nice json message. Unfortunately, this goes against our architectural desire to avoid sending messages directly to fluentd, due to reliability considerations\r\n\r\n4. **Have k8s do just enough parsing to detect if an envelope is necessary**   If the message already looks like json, then the k8s fields could be added in easily. Otherwise, the fields could be added onto the end of the message in kv format, which would be fairly easy for either splunk or fluentd to handle downstream. ( much easier than a json with odd format inside of one field ). \r\n\r\nOf these options, I recommend option 4. It stinks to have k8s doing some <limited> log parsing, but that's better than accepting weird problems when the fluentd daemon goes down, or accepting that its really hard to parse logs downstream. \r\n\r\nI think eventually option 1 is the right answer in the long term. I think nearly everyone in the logging community is in favor of json log messages. Option 4 would allow people/systems to transition to json over time.\r\n\r\nI'm open to other options. Sorry for the long post.\r\n\r\n\r\n"}}