title: ProxyRemote leads to "AH00898: Error during SSL Handshake with remote server ...", works on other server
body: <h1>Goal</h1>
  <p>Set up a Virtual Host as Reverse Proxy, which also acts as Forward Proxy to another &quot;Remote&quot; Proxy for a specific URL pattern.</p>
  <h1>Issue</h1>
  <p>I have 2 servers (actually 2 separate machines), both having the same config but <strong>only one server is able to forward the requests</strong>.</p>
  <p>I've searched the whole web and did even more experiments than described below (but seem irrelevant to mention here), so I'm very very thankful for any idea/experiment you come up with!</p>
  <h1>Config</h1>
  <h2>Dump</h2>
  <p>Acc. to <code>apache2ctl -DDUMP_CONFIG | grep -vE &quot;^[ ]*#[ ]*[0-9]+:$&quot; &gt; apache_dump.conf</code> the apache config of both servers is <strong>identical</strong>.</p>
  <h2>Virtual Host</h2>
  <pre><code>&lt;VirtualHost *:80&gt;
  
  [...]
  
  SSLProxyEngine on
  ProxyRemote &quot;https://booking-service.com/&quot; &quot;http://remote-proxy:3128&quot;
  
  &lt;Location /booking&gt;
      ProxyPass https://booking-service.com/api
      ProxyPassReverse https://booking-service.com/api
      ProxyPreserveHost Off
      RequestHeader set X-Api-Key &quot;...&quot;
      RequestHeader unset Cookie
      RequestHeader unset Authorization
  &lt;/Location&gt;
  
  &lt;/VirtualHost&gt;
  </code></pre>
  <h2>Modules</h2>
  <p>Here an excerpt of activated modules which IMHO might be relevant:</p>
  <pre><code>[...]
  http_module (static)
  [...]
  ssl_module (shared)
  [...]
  proxy_module (shared)
  proxy_http_module (shared)
  proxy_ftp_module (shared)
  proxy_ajp_module (shared)
  proxy_wstunnel_module (shared)
  proxy_balancer_module (shared)
  [...]
  </code></pre>
  <h1>Regular Error Logs</h1>
  <p>(IPs and host names obfuscated)</p>
  <pre><code>[proxy:trace2] [pid 21616:tid 140692231767808] proxy_util.c(3016): HTTPS: fam 2 socket created to connect to booking-service.com
  [proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(3050): AH02824: HTTPS: connection established with 192.18.191.131:3128 (booking-service.com)
  [proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2677): AH00948: CONNECT: sending the CONNECT request for booking-service.com:443 to the remote proxy 192.18.191.131:3128 (remote-proxy.net)
  [proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2731): AH00949: send_http_connect: response from the forward proxy: HTTP/1.1 200 Connection established\r\n\r\n
  [proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(3218): AH00962: HTTPS: connection complete to 192.18.191.131:3128 (remote-proxy.net)
  [proxy:error] [pid 21616:tid 140692231767808] (20014)Internal error (specific information not available): [client 111.222.33.444:20435] AH01084: pass request body failed to 192.18.191.131:3128 (remote-proxy.net)
  [proxy:error] [pid 21616:tid 140692231767808] [client 111.222.33.444:20435] AH00898: Error during SSL Handshake with remote server returned by /booking/test-request
  [proxy_http:error] [pid 21616:tid 140692231767808] [client 111.222.33.444:20435] AH01097: pass request body failed to 192.18.191.131:3128 (remote-proxy.net) from 111.222.33.444 ()
  [proxy:debug] [pid 21616:tid 140692231767808] proxy_util.c(2334): AH00943: HTTPS: has released connection for (booking-service.com)
  </code></pre>
  <h1>Error Logs With ssl:trace7</h1>
  <p>I get the following for both web servers:</p>
  <pre><code>[ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2180): [remote 192.18.191.131:3128] OpenSSL: Handshake: start
  [...]
  [ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: before/connect initialization
  [ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: write 517/517 bytes to BIO#7fb05c009ba0 [mem: 7fb05c011070] (BIO dump follows)
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 01 02 00 01 00 01-fc 03 03 c5 c2 b9 30 65  ..............0e |
  [...]
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0140: 03 00 0f 00 01 01 00 15-00 bb                    ..........       |
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0517 - &lt;SPACES/NULS&gt;
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: SSLv2/v3 write client hello A
  [ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 7/7 bytes from BIO#7fb05c00cc70 [mem: 7fb05c0165d0] (BIO dump follows)
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 03 00 41 02                                ....A.           |
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0007 - &lt;SPACES/NULS&gt;
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 63/63 bytes from BIO#7fb05c00cc70 [mem: 7fb05c0165da] (BIO dump follows)
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 00 3d 03 03 f8 86 f8 5b-c5 71 0e 3f d6 fb 37 1d  .=.....[.q.?..7. |
  [...]
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0030: 00 00 00 00 0b 00 04 03-00 01 02 00 23           ............#    |
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2179): [remote 192.18.191.131:3128] | 0063 - &lt;SPACES/NULS&gt;
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace3] [pid 25298:tid 140395937773312] ssl_engine_kernel.c(2189): [remote 192.18.191.131:3128] OpenSSL: Loop: unknown state
  [ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 5/5 bytes from BIO#7fb05c00cc70 [mem: 7fb05c026da3] (BIO dump follows)
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 16 03 03 0d ce                                   .....            |
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace4] [pid 25298:tid 140395937773312] ssl_engine_io.c(2214): [remote 192.18.191.131:3128] OpenSSL: read 3534/3534 bytes from BIO#7fb05c00cc70 [mem: 7fb05c026da8] (BIO dump follows)
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2137): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0000: 0b 00 0d ca 00 0d c7 00-07 12 30 82 07 0e 30 82  ..........0...0. |
  [...]
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2175): [remote 192.18.191.131:3128] | 0dc0: ca 5b e0 d5 f6 6c 23 9d-20 29 55 cd 3a c5        .[...l#. )U.:.   |
  [ssl:trace7] [pid 25298:tid 140395937773312] ssl_engine_io.c(2181): [remote 192.18.191.131:3128] +-------------------------------------------------------------------------+
  </code></pre>
  <p>The <strong>BAD</strong> web server does abruptly stop here, there is no &quot;Certificate Verification&quot; and no &quot;Handshake: done&quot;, i.e. no further <code>ssl:...</code> log entries related to this client request.</p>
  <p>In contrast, the <strong>GOOD</strong> web server does the following:</p>
  <pre><code>[ssl:debug] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(1738): [remote 192.18.191.131:3128] AH02275: Certificate Verification, depth 1, CRL checking mode: none (0) [subject: CN=...]
  [ssl:debug] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(1738): [remote 192.18.191.131:3128] AH02275: Certificate Verification, depth 0, CRL checking mode: none (0) [subject: CN=...]
  [...]
  [ssl:trace3] [pid 9659:tid 140475999237888] ssl_engine_kernel.c(2184): [remote 192.18.191.131:3128] OpenSSL: Handshake: done
  [...]
  </code></pre>
  <h1>Failed Experiments</h1>
  <p>What I tried so far:</p>
  <ul>
  <li>Adding the following settings (even if the other web server works w/o them, i know... I'm desperate :D):</li>
  </ul>
  <pre><code>SSLProxyVerify none
  SSLProxyCheckPeerCN off
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off
  SSLProxyProtocol all -SSLv2 -SSLv3 -TLSv1
  SSLProxyCACertificateFile /etc/ssl/certs/&lt;the-ca-cert&gt;.crt [afaik should be considered anyway b/c in /etc/ssl/certs]
  </code></pre>
  <ul>
  <li>Restarting the apache2.service</li>
  <li>Restarting the whole linux machine</li>
  <li>Request with <strong>curl: works</strong>!
  <ul>
  <li><code>curl --request POST 'https://booking-service.com/api/test-request' --header 'Content-Type: application/json' --header 'X-Api-Key: ...' --proxy 'http://remote-proxy.net:3128' --data '@/tmp/request-body.txt' -iv</code></li>
  </ul>
  </li>
  <li>Debug with openssl: looks good &amp; is the same for both servers
  <ul>
  <li><code>openssl-1_1 s_client -connect booking-service.com -proxy remote-proxy.net:3128 -state -debug</code></li>
  </ul>
  </li>
  </ul>
  <h1>App Versions (identical on both servers)</h1>
  <ul>
  <li>Linux: <code>lsb_release -a</code>:</li>
  </ul>
  <pre><code>LSB Version:    n/a
  Distributor ID: SUSE
  Description:    SUSE Linux Enterprise Server 12 SP5
  Release:        12.5
  Codename:       n/a
  </code></pre>
  <ul>
  <li>Apache: <code>httpd -v</code>:</li>
  </ul>
  <pre><code>Server version: Apache/2.4.38 (Linux/SUSE)
  Server built:   2019-02-08 01:59:10.000000000 +0000
  </code></pre>
  <ul>
  <li>OpenSSL: <code>openssl version</code>:</li>
  </ul>
  <pre><code>OpenSSL 1.0.2p-fips  14 Aug 2018
  </code></pre>