<p>I want to be able to capture (log) (at least some of) <code>envoy</code>'s HTTP headers on my <code>istio</code> service mesh.</p>
<p>I have gone through <code>envoy</code>'s <a href="https://www.envoyproxy.io/docs/envoy/latest/start/quick-start/run-envoy#debugging-envoy" rel="noreferrer">docs</a>, and in the log levels' section, it does not mention any header-specific information.</p>
<p>Currently, my <code>istio-proxy</code> log is like this (this is from a <code>stern</code> output):</p>
<pre><code>mysvc-69c46fbc75-d9v8j istio-proxy {&quot;bytes_sent&quot;:&quot;124&quot;,&quot;upstream_cluster&quot;:&quot;inbound|80|http|mysvc.default.svc.cluster.local&quot;,&quot;downstream_remote_address&quot;:&quot;10.11.11.1:0&quot;,&quot;authority&quot;:&quot;some.url.com&quot;,&quot;path&quot;:&quot;/health?source=dd_cluster_agent&quot;,&quot;protocol&quot;:&quot;HTTP/1.1&quot;,&quot;upstream_service_time&quot;:&quot;1&quot;,&quot;upstream_local_address&quot;:&quot;127.0.0.1:40406&quot;,&quot;duration&quot;:&quot;2&quot;,&quot;upstream_transport_failure_reason&quot;:&quot;-&quot;,&quot;route_name&quot;:&quot;default&quot;,&quot;downstream_local_address&quot;:&quot;10.11.32.32:20000&quot;,&quot;user_agent&quot;:&quot;Datadog Agent/7.24.0&quot;,&quot;response_code&quot;:&quot;200&quot;,&quot;response_flags&quot;:&quot;-&quot;,&quot;start_time&quot;:&quot;2021-01-17T18:54:57.449Z&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;request_id&quot;:&quot;61ae63c7-aa10-911b-9562-939kdhd49ddhj&quot;,&quot;upstream_host&quot;:&quot;127.0.0.1:20000&quot;,&quot;x_forwarded_for&quot;:&quot;10.16.32.1&quot;,&quot;requested_server_name&quot;:&quot;outbound_.80_.mysvc_.faros.default.svc.cluster.local&quot;,&quot;bytes_received&quot;:&quot;0&quot;,&quot;istio_policy_status&quot;:&quot;-&quot;}
</code></pre>
<p>Is there a way to log <code>http</code> headers? (ideally <strong>some</strong> of them, to keep the logging cost under control)</p>
<p><strong>edit1</strong> following advice in the comments, I checked my <code>istio-operator</code> resource and I see that access logging seems to be enabled</p>
<pre><code>    meshConfig:
      accessLogEncoding: JSON
      accessLogFile: /dev/stdout
</code></pre>
<p><strong>edit2</strong> I have also tried the following:</p>
<pre><code>curl -i -H &quot;Custom-Header: application/json&quot; https://my.url.net
</code></pre>
<p>but in the logs of the <code>istio-ingressgateway</code> I don't see my custom header</p>
<pre><code>istio-ingressgateway-58f69d8696-rmpwn istio-proxy {&quot;user_agent&quot;:&quot;curl/7.64.1&quot;,&quot;response_code&quot;:&quot;200&quot;,&quot;response_flags&quot;:&quot;-&quot;,&quot;start_time&quot;:&quot;2021-01-18T19:02:48.645Z&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;request_id&quot;:&quot;8e32c93c-484d-9c56-9489-8c5392793d97&quot;,&quot;upstream_host&quot;:&quot;10.16.32.55:20000&quot;,&quot;x_forwarded_for&quot;:&quot;10.16.32.1&quot;,&quot;requested_server_name&quot;:&quot;my.url.net&quot;,&quot;bytes_received&quot;:&quot;0&quot;,&quot;istio_policy_status&quot;:&quot;-&quot;,&quot;bytes_sent&quot;:&quot;124&quot;,&quot;upstream_cluster&quot;:&quot;outbound|80||mysvc.default.svc.cluster.local&quot;,&quot;downstream_remote_address&quot;:&quot;10.16.32.1:52804&quot;,&quot;authority&quot;:&quot;my.url.net&quot;,&quot;path&quot;:&quot;/&quot;,&quot;protocol&quot;:&quot;HTTP/2&quot;,&quot;upstream_service_time&quot;:&quot;9&quot;,&quot;upstream_local_address&quot;:&quot;10.16.32.17:49826&quot;,&quot;duration&quot;:&quot;10&quot;,&quot;upstream_transport_failure_reason&quot;:&quot;-&quot;,&quot;route_name&quot;:&quot;-&quot;,&quot;downstream_local_address&quot;:&quot;10.16.32.17:8443&quot;}
</code></pre>

pkaramol
<p>I think I've succesfully made a reproduction of your issue and I was able to print <code>MY_CUSTOM_HEADER</code> in the ingress gateway logs.</p>
<p>There is a part of my istio ingress gateway logs.</p>
<pre><code>[2021-01-20T08:26:18.587Z] pkarambol GET /productpage HTTP/1.1 200
</code></pre>
<p>I've used below curl command:</p>
<pre><code>curl -v -H &quot;MY_CUSTOM_HEADER: pkarambol&quot; xx.xxx.xx.xxx/productpage
</code></pre>
<hr />
<p>To make that happen you have to change the <a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage?fbclid=IwAR0xaS-EtEniJfbpGWf-GN_o75aLSpyn5FI2Uojp9YPWQD4PjQqs9ObqavQ#default-format-string" rel="noreferrer">default format</a> of the logs. As mentioned in the <a href="https://istio.io/latest/docs/tasks/observability/logs/access-log/#enable-envoy-s-access-logging" rel="noreferrer">documentation</a>, you can use <code>meshConfig.accessLogFormat</code> to change that.</p>
<hr />
<p>There is an example of <a href="https://istio.io/latest/docs/setup/install/operator/" rel="noreferrer">Istio Operator</a> I've used.</p>
<p><code>%REQ(MY_CUSTOM_HEADER)%</code> is a part responsible for displaying the custom header.</p>
<pre><code>apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogFormat: &quot;[%START_TIME%] %REQ(MY_CUSTOM_HEADER)% %REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% %REQ(X-FORWARDED-FOR)% %REQ(USER-AGENT)%
%REQ(X-REQUEST-ID)% %REQ(:AUTHORITY)% %UPSTREAM_HOST%\n&quot;
</code></pre>
