{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/4560","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/4560/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/4560/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/4560/events","html_url":"https://github.com/prometheus/prometheus/issues/4560","id":355049763,"node_id":"MDU6SXNzdWUzNTUwNDk3NjM=","number":4560,"title":"Prometheus sends packages with different Content-Type in HTTP header ","user":{"login":"waldauf","id":24603883,"node_id":"MDQ6VXNlcjI0NjAzODgz","avatar_url":"https://avatars3.githubusercontent.com/u/24603883?v=4","gravatar_id":"","url":"https://api.github.com/users/waldauf","html_url":"https://github.com/waldauf","followers_url":"https://api.github.com/users/waldauf/followers","following_url":"https://api.github.com/users/waldauf/following{/other_user}","gists_url":"https://api.github.com/users/waldauf/gists{/gist_id}","starred_url":"https://api.github.com/users/waldauf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/waldauf/subscriptions","organizations_url":"https://api.github.com/users/waldauf/orgs","repos_url":"https://api.github.com/users/waldauf/repos","events_url":"https://api.github.com/users/waldauf/events{/privacy}","received_events_url":"https://api.github.com/users/waldauf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-29T08:21:40Z","updated_at":"2018-08-29T09:51:02Z","closed_at":null,"author_association":"NONE","body":"## Proposal\r\nGlobaly Promehteus supports [*remote_write*](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_write%3E) to InfluxDB. If I use direct write into InfluxDB everything works great. Scheme:\r\n```\r\n[Prometheus: remote_write into InfluxDB] ---> [InfluxDB]\r\n```\r\nI captured sent packages by Prometheus to InfluxDB by `tcpdump`. All packages are sent wiht *Context-Type: text/plain* in HTTP header. \r\n\r\nExample of one captured package:\r\n```\r\n# tcpdump -i ens160 -A -s 0 -ttttnnvvS 'dst port 8086 and src k8s.sw.lab and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'\r\n16:02:10.326345 IP k8sm.sw.lab.46838 > tick.sw.lab.8086: Flags [P.], seq 863959075:863961110, ack 855948242, win 1413, options [nop,nop,TS val 1641754224 ecr 91267025], length 2035\r\nE..'jA@.?..D\r\n...\r\n.......3~.#3.......De.....\r\na..p.p..POST /write?consistency=any&db=telegraf_k8s HTTP/1.1\r\nHost: 10.22.20.17:8086\r\nUser-Agent: telegraf\r\nTransfer-Encoding: chunked\r\nContent-Type: text/plain; charset=utf-8\r\nAccept-Encoding: gzip\r\n\r\n72a\r\ndocker_container_mem,annotation.io.kubernetes.container.hash=5e4e5a97,... total_pgfault=83005582i 1535464922000000000\r\n```\r\n<br><br>\r\nIn my case is not possible to write directly into InfluxDB because I need to save metrics into Kafka first. That's why I included stream processor [Benthos](https://github.com/Jeffail/benthos) (tried [Telegraf](https://github.com/influxdata/telegraf) too) in the flow - between Prometheus and Kafka. \r\n\r\nScheme:\r\n```\r\n[Prometheus: remote_write into InfluxDB] ---> [http_server input - Benthos - kafka output] ---> [Kafka] ---> [Telegraf] ---> [InfluxDB]\r\n```\r\nBut all packages sent by Prometheus have binary content and it's not possible to work with them. In HTTP header is set *Content-Type: application/x-protobuf*:\r\n```\r\n# tcpdump -i ens160 -A -s 0 -ttttnnvvS 'dst port 4197 and src k8s.sw.lab and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'\r\nHost: 10.22.20.15:4197\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 6891\r\nContent-Encoding: snappy\r\nContent-Type: application/x-protobuf\r\nX-Prometheus-Remote-Write-Version: 0.1.0\r\n\r\n....L\r\n..\r\n!\r\n.__name__..container_tasks_state\r\n \r\n.beta_kubernetes_io_arch..amd64\r\n.\r\n.bF\".0os..linux\r\n.\r\n..[.m....POD\r\n.\r\n\r\ndatacenter..k8s-lab\r\n..\r\n.id.}/kubepods/besteffort/pod3e4f7e20-aac9-11e8-9ec2-0050568fadab/97cf5928643b576f686de352160d7b02e42054087f3d265771ecb452bbd71e3b\r\n#\r\n.image..k8s.gcr.io/pause-amd64:3.1\r\n.\r\n.instance..k8sm\r\n \r\n.job..kube)-H-nodes-cadvisor\r\n.\r\n.6J..host%. .k8sm\r\nq\r\n...<ik8s_POD_prom01-...etheus-alertmanager-744764c4d9-t9dns_monitoring_3e4f7e2r+.._0\r\n.\r\n  .s.space.\r\n.@.\r\n;\r\n.pod)../......\r\n.\r\n.EAL..running........,\r\n..y..y..y..y..y..y..y..y..y.ry...Qy..sleep2z................................stopped...............................r........uni.O  .ruptibleY......  ..  ..  .53f829-aj. .q513fa45ca21b62368c3abedc421baf05494f4369697afe4d586f16f587fa2047\r\n#\r\n.image..k8s.gcr.io/pause-amd64:3.1\r\n~\r\n~\r\n```\r\n<br>\r\nFor me it looks like Prometheus decides which *Content-Type* choose and use.  That's why I'm not possible to include stream processor between Prometheus and InfluxDB.\r\n<br>\r\n\r\nCan I ask you for help how to set Prometheus to keep *Content-Type: text/plain*?\r\n\r\n## Environment\r\n\r\n* System information: Linux 4.4.0-116-generic x86_64\r\n* Prometheus version: 2.2.1\r\n* Prometheus configuration file: If you think that config file is relevant I'll send it.\r\n* Logs: If you think that log file is relevant I'll send it."},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/416894645","html_url":"https://github.com/prometheus/prometheus/issues/4560#issuecomment-416894645","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/4560","id":416894645,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjg5NDY0NQ==","user":{"login":"brian-brazil","id":7115638,"node_id":"MDQ6VXNlcjcxMTU2Mzg=","avatar_url":"https://avatars0.githubusercontent.com/u/7115638?v=4","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"created_at":"2018-08-29T09:51:02Z","updated_at":"2018-08-29T09:51:02Z","author_association":"MEMBER","body":"Remote write doesn't use gzip or text/plain, that's some other packet you've captured. All remote write is via snappy-compressed protobufs, and if you're using remote write then https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage/example_write_adapter will help you get going."}}
{"actor":{"display_login":"brian-brazil"}}