<h3>Background</h3>
<p>We're hosting a proprietary web application, which we like to customize rudimentary to our CD (mainly colors). Since the application doesn't support this and I don't have access to the source, I'd like to create a custom css stylesheet and include it in the app by manipulating it's ingress (= injecting css/stylesheets into the DOM).</p>
<h2><code>http_sub_module</code> of the Nginx Ingress Controller for Kubernetes</h2>
<p>The <a href="http://nginx.org/en/docs/http/ngx_http_sub_module.html" rel="nofollow noreferrer"><code>http_sub_module</code></a> seems similar to <a href="https://httpd.apache.org/docs/2.4/mod/mod_substitute.html" rel="nofollow noreferrer">Apaches <code>mod_substitute</code></a>. When I exec <code>nginx -V</code> inside the nginx ingress pod, there is <code>--with-http_sub_module</code> listed in the configure arguments, so it must be avaliable in the currently deployed 1.19.2 version.</p>
<p>I found <a href="https://serverfault.com/questions/648135/change-nginx-response-body-with-reverse-proxy">a similar question</a> using <code>subs_filter</code> (instead of <code>sub_filter</code>). It seems that <a href="https://docs.nginx.com/nginx/admin-guide/dynamic-modules/http-substitutions-filter/" rel="nofollow noreferrer">the one with s is from nginx plus</a>, also documented <a href="https://www.nginx.com/products/nginx/modules/http-substitutions-filter/" rel="nofollow noreferrer">here</a>. All linked examples there use <code>subs_</code>, where the <a href="http://nginx.org/en/docs/http/ngx_http_sub_module.html" rel="nofollow noreferrer">regular community documentation</a> uses <code>sub_filter</code>. Both of them worked without an error, I guess the plus one is an alias if no plus subscription is avaliable.</p>
<p>Since <code>sub_filter</code> doesn't seem to work, I tried both of them without success:</p>
<pre><code>annotations:
  kubernetes.io/ingress.class: &quot;nginx&quot;
  nginx.ingress.kubernetes.io/rewrite-target: /
  nginx.ingress.kubernetes.io/configuration-snippet: |
    #subs_filter_once on;
    #subs_filter_types 'text/html';
    # subs_filter 'en' 'de';
    # subs_filter '&lt;/body&gt;' '&lt;h1&gt;TEST&lt;/h1&gt;&lt;/body&gt;';
    sub_filter '&lt;/body&gt;' '&lt;h1&gt;TEST&lt;/h1&gt;&lt;/body&gt;';
</code></pre>
<h2>Further things I've checked/tried</h2>
<p>The configuration snippet <em>is</em> applied. I looked into the nginx ingress pod, the server vhost block for my application has the <code>sub_filter</code>/<code>subs_filter</code> directive inside the <code>/</code> location block.</p>
<p>The default type for filtering is <code>sub_filter_types text/html</code>. The website returns <code>Content-Type: text/html</code> so it should match. To be sure, I also set <code>sub_filter_types &quot;text/html&quot;;</code> in the snippet. And even <code>*</code> which matches any mime type according to the docs, both without any difference.</p>
<p>Doesn't this module work with <code>mod_proxy</code> used by k8s? I can't imagine that since the module is relatively old and I see no reason why it shouldn't work when nginx acts as reverse proxy, since it needs to have access to the HTML header/body too.</p>

<p>Found out that the problem was something completely different: The application supports gzip, so this compression was enabled. But the <code>http_sub_module</code> doesn't support gzip, as <a href="https://github.com/yaoweibin/ngx_http_substitutions_filter_module/blob/master/README" rel="nofollow noreferrer">documented here</a>. It only works with plain text reasponses. If the response is compressed, the module just do nothing. This explains why it doesn't work and there was no error.</p>
<p>Luckily, the compression could be easily disabled without any modifications on the application by setting the following header:</p>
<pre><code>proxy_set_header Accept-Encoding &quot;&quot;;
</code></pre>
<p>If this is added to the <code>nginx.ingress.kubernetes.io/configuration-snippet</code> section, it accepts only plain text and every application which is compilant to the specification will respect that.</p>
<p>But it doesn't mean that we can't use any compression. The Gzip filter can be used from the ingress to the user, it's only not supported from the ingress to the application.</p>
<h2>Full working example annotation snippet</h2>
<pre><code>  ingres: 
    # ...
    annotations:
      kubernetes.io/ingress.class: &quot;nginx&quot;
      # http://nginx.org/en/docs/http/ngx_http_sub_module.html
      nginx.ingress.kubernetes.io/configuration-snippet: |
        sub_filter &quot;&lt;/body&gt;&quot; &quot;&lt;style&gt;body{ color: red !important; }&lt;/style&gt;&lt;/body&gt;&quot;;
        # The http_sub_module doesn't support compression from the ingress to the backend application
        proxy_set_header Accept-Encoding &quot;&quot;;
</code></pre>
<p>This would apply the style block before the closing body tag like this:</p>
<pre><code>&lt;style&gt;body{ color: red !important; }&lt;/style&gt;&lt;/body&gt;&lt;/html&gt;
</code></pre>
<p>For productive usage, we can include a link to a custom css file here, which overrides the applications design to our needs.</p>
