title: How to debug a HTTPS proxy failure in wget?
body: <p>(This is a <a href="https://stackoverflow.com/questions/41837847/how-to-get-https-endpoints-to-proxy-successfully-in-wiremock">continuation of this question</a>, in which I solved one problem and found another).</p>
  
  <p>I am fetching a HTTPS website using wget via a WireMock proxy. Here is my fetch command pointing to a demo secure site:</p>
  
  <pre><code>wget -e use_proxy=yes -e https_proxy=localhost:8100 \
      https://www.rottentomatoes.com/
  </code></pre>
  
  <p>Here is my proxy set-up:</p>
  
  <pre><code>java -jar wiremock-standalone-2.5.1.jar \
      --port 8081 --https-port 8100 \
      --proxy-all https://www.rottentomatoes.com/ \
      --record-mappings \
      --root-dir ./proxy-cache \
      --verbose
  </code></pre>
  
  <p>The WireMock on-screen logs say this:</p>
  
  <pre><code>2017-03-27 12:08:09.066 Verbose logging enabled
  SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  SLF4J: Defaulting to no-operation (NOP) logger implementation
  SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
  2017-03-27 12:08:09.827 Verbose logging enabled
  2017-03-27 12:08:09.892 Recording mappings to ./proxy-cache/mappings
   /$$      /$$ /$$                     /$$      /$$                     /$$      
  | $$  /$ | $$|__/                    | $$$    /$$$                    | $$      
  | $$ /$$$| $$ /$$  /$$$$$$   /$$$$$$ | $$$$  /$$$$  /$$$$$$   /$$$$$$$| $$   /$$
  | $$/$$ $$ $$| $$ /$$__  $$ /$$__  $$| $$ $$/$$ $$ /$$__  $$ /$$_____/| $$  /$$/
  | $$$$_  $$$$| $$| $$  \__/| $$$$$$$$| $$  $$$| $$| $$  \ $$| $$      | $$$$$$/ 
  | $$$/ \  $$$| $$| $$      | $$_____/| $$\  $ | $$| $$  | $$| $$      | $$_  $$ 
  | $$/   \  $$| $$| $$      |  $$$$$$$| $$ \/  | $$|  $$$$$$/|  $$$$$$$| $$ \  $$
  |__/     \__/|__/|__/       \_______/|__/     |__/ \______/  \_______/|__/  \__/
  
  port:                         8081
  https-port:                   8100
  https-keystore:               jar:file:(removed)/wiremock-standalone-2.5.1.jar!/keystore
  proxy-all:                    https://www.rottentomatoes.com/
  preserve-host-header:         false
  enable-browser-proxying:      false
  record-mappings:              true
  match-headers:                []
  no-request-journal:           false
  verbose:                      true
  </code></pre>
  
  <p>The result is:</p>
  
  <pre><code>--2017-03-27 12:08:25--  https://www.rottentomatoes.com/
  Resolving localhost (localhost)... 127.0.0.1
  Connecting to localhost (localhost)|127.0.0.1|:8100... connected.
  Failed reading proxy response: Success
  Retrying.
  
  --2017-03-27 12:08:26--  (try: 2)  https://www.rottentomatoes.com/
  Connecting to localhost (localhost)|127.0.0.1|:8100... connected.
  Failed reading proxy response: Success
  Retrying.
  
  ^C
  </code></pre>
  
  <p>As you can see, the fetch fails, automatically retries, and needs to be cancelled to end.</p>
  
  <p>I have tried <code>--preserve-host-header</code> in the WireMock command (<a href="http://wiremock.org/docs/running-standalone/" rel="nofollow noreferrer">standalone docs here</a>) but the result is the same.</p>
  
  <p>I wonder if the proxy is failing internally because it needs to be pointed to a valid HTTPS cert store? That said, I would expect the running proxy to be outputting something (even errors) but it is appearing not to be handling the call at all. An equivalent HTTP call works fine.</p>
  
  <p>Is there something I can do to see why Wget is failing? The error message is not very helpful, and as far as I know I cannot make it more verbose (verbose is on by default in wget).</p>
  
  <p>This behaviour is the same across Alpine 3.4 (in a Docker container) and running on my Ubuntu 14.04 VM. It's also the same across WireMock 2.4.1, 2.5.0 and 2.5.1.</p>
  
  <h2>Trying certificates</h2>
  
  <p>I have tried switching my browser's (Firefox) HTTPS proxy settings to point to WireMock, and it fails on the website I'm trying to fetch due to a bad certificate. Interestingly WireMock does not output anything to stdout, even though it looks like Firefox has contacted the remote server.</p>
  
  <p>I wondered if the built-in keystore in Wiremock was out of date, or empty, so learning how to specify a "real" cert store was probably the next thing worth trying. I used <a href="https://docs.oracle.com/cd/E35976_01/server.740/es_admin/src/tadm_ssl_convert_pem_to_jks.html" rel="nofollow noreferrer">these instructions</a> to convert a <a href="https://curl.haxx.se/ca/cacert.pem" rel="nofollow noreferrer">browser cert file</a> to JKS format, and this made no difference, either to Wget or Firefox.</p>
  
  <p>I note that my newly created keystore is 955 bytes, whereas the original PEM cert file is ~260K, so it is clear that not all certs have been added (maybe it just added the first one?). FWIW I used this command:</p>
  
  <pre><code>keytool -import -v -trustcacerts -alias endeca-ca \
      -file cacert.pem -keystore truststore.ks
  </code></pre>
  
  <h2>Confirming proxy is being reached</h2>
  
  <p>I have added the <code>-verbose</code> and <code>-verbose:jni</code> switches to the <code>java</code> call, to prove that something is happening when the HTTPS proxy is required. There is a veritable essay printed out when I run the Wget command, so I am confident the HTTPS proxy is being hit. Wget also fetches fine in HTTP mode.</p>
  
  <h2>How to log what is happening in a standalone Jar file?</h2>
  
  <p>I am at the stage where I could try all sorts of things blindly, and I think I need to get some intelligence from the Java system to see <em>why</em> it is failing first. My guess is that Wiremock is the problem rather than Wget.</p>
  
  <p>I have found an undocumented feature in Wiremock, <code>--print-all-network-traffic</code>, which offers this:</p>
  
  <pre><code>2017-03-27 17:36:51.287 Opened Socket[addr=/127.0.0.1,port=54140,localport=8100]
  2017-03-27 17:36:51.397 Incoming bytes: CONNECT www.rottentomatoes.com:443 HTTP/1.1
  User-Agent: Wget/1.15 (linux-gnu)
  Host: www.rottentomatoes.com:443
  
  
  2017-03-27 17:36:51.398 Closed Socket[addr=/127.0.0.1,port=54140,localport=8100]
  2017-03-27 17:36:51.399 Closed Socket[addr=/127.0.0.1,port=54140,localport=8100]
  2017-03-27 17:36:52.400 Opened Socket[addr=/127.0.0.1,port=54142,localport=8100]
  2017-03-27 17:36:52.483 Incoming bytes: CONNECT www.rottentomatoes.com:443 HTTP/1.1
  User-Agent: Wget/1.15 (linux-gnu)
  Host: www.rottentomatoes.com:443
  </code></pre>
  
  <p>The second section repeats for as often as wget retries, but there is still nothing much useful here. I want to know <em>why</em> it is failing.</p>
  
  <p>Are there logging parameters I can add to <code>java -jar</code>, or is there is system-wide error log for Java I can consult? I have installed VisualVM, but the various outputs do not seem very relevant. I expect I would be most interested in exceptions?</p>