title: php curl localhost is slow when making concurrent requests
body: <p>I have an interesting issue which I am not sure what the root cause is. I have a server and two virtual hosts A and B with ports running on 80 and 81 respectively. I have written a simple PHP code on A which looks like this:</p>
  
  <pre><code>&lt;?php
  
  echo "from A server\n";
  </code></pre>
  
  <p>And another simple PHP code on B:</p>
  
  <pre><code>&lt;?php
  
  echo "B server:\n";
  
  // create curl resource 
  $ch = curl_init(); 
  
  // set url 
  curl_setopt($ch, CURLOPT_URL, "localhost:81/a.php"); 
  
  //return the transfer as a string 
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
  
  // $output contains the output string 
  $output = curl_exec($ch); 
  
  // close curl resource to free up system resources 
  curl_close($ch);
  
  echo $output;
  </code></pre>
  
  <p>When making concurrent requests using <code>ab</code>, I get the following results:</p>
  
  <pre><code>ab -n 10 -c 5 http://192.168.10.173/b.php
  This is ApacheBench, Version 2.3 &lt;$Revision: 1706008 $&gt;
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/
  
  Benchmarking 192.168.10.173 (be patient).....done
  
  
  Server Software:        nginx/1.10.0
  Server Hostname:        192.168.10.173
  Server Port:            80
  
  Document Path:          /b.php
  Document Length:        26 bytes
  
  Concurrency Level:      5
  Time taken for tests:   2.680 seconds
  Complete requests:      10
  Failed requests:        0
  Total transferred:      1720 bytes
  HTML transferred:       260 bytes
  Requests per second:    3.73 [#/sec] (mean)
  Time per request:       1340.197 [ms] (mean)
  Time per request:       268.039 [ms] (mean, across all concurrent requests)
  Transfer rate:          0.63 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.1      0       1
  Processing:     2 1339 1408.8   2676    2676
  Waiting:        2 1339 1408.6   2676    2676
  Total:          3 1340 1408.8   2676    2677
  
  Percentage of the requests served within a certain time (ms)
    50%   2676
    66%   2676
    75%   2676
    80%   2676
    90%   2677
    95%   2677
    98%   2677
    99%   2677
   100%   2677 (longest request)
  </code></pre>
  
  <p>But making 1000 requests with concurrency level 1 is extremely fast:</p>
  
  <pre><code>$ ab -n 1000 -c 1 http://192.168.10.173/b.php
  This is ApacheBench, Version 2.3 &lt;$Revision: 1706008 $&gt;
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/
  
  Benchmarking 192.168.10.173 (be patient)
  Completed 100 requests
  Completed 200 requests
  Completed 300 requests
  Completed 400 requests
  Completed 500 requests
  Completed 600 requests
  Completed 700 requests
  Completed 800 requests
  Completed 900 requests
  Completed 1000 requests
  Finished 1000 requests
  
  
  Server Software:        nginx/1.10.0
  Server Hostname:        192.168.10.173
  Server Port:            80
  
  Document Path:          /b.php
  Document Length:        26 bytes
  
  Concurrency Level:      1
  Time taken for tests:   1.659 seconds
  Complete requests:      1000
  Failed requests:        0
  Total transferred:      172000 bytes
  HTML transferred:       26000 bytes
  Requests per second:    602.86 [#/sec] (mean)
  Time per request:       1.659 [ms] (mean)
  Time per request:       1.659 [ms] (mean, across all concurrent requests)
  Transfer rate:          101.26 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.1      0       1
  Processing:     1    1  10.3      1     201
  Waiting:        1    1  10.3      1     201
  Total:          1    2  10.3      1     201
  
  Percentage of the requests served within a certain time (ms)
    50%      1
    66%      1
    75%      1
    80%      1
    90%      1
    95%      1
    98%      1
    99%      2
   100%    201 (longest request)
  </code></pre>
  
  <p>Can anyone explain why this happened? I really want to know the root cause. Is it an issue of curl? It doesn't feel like an network bottleneck or open file issue since the concurrency is just 5. By the way, I also try the same thing with guzzlehttp, but the result is the same. I use ab on my laptop and the server is in the same local network. Plus, it  certainly has nothing to do with network bandwidth because requests between host A and B are done at localhost.</p>
  
  <hr>
  
  <p>I have modified the code so that testing is more flexible:</p>
  
  <pre><code>&lt;?php
  
  require 'vendor/autoload.php';
  
  use GuzzleHttp\Client;
  
  $opt = 1;
  $url = 'http://localhost:81/a.php';
  
  switch ($opt) {
      case 1:
          // create curl resource
          $ch = curl_init();
  
          // set url
          curl_setopt($ch, CURLOPT_URL, $url);
  
          //return the transfer as a string
          curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  
          // $output contains the output string
          $output = curl_exec($ch);
  
          curl_close($ch);
  
          echo $output;
          break;
      case 2:
          $client = new Client();
          $response = $client-&gt;request('GET', $url);
          echo $response-&gt;getBody();
          break;
      case 3:
          echo file_get_contents($url);
          break;
      default:
          echo "no opt";
  }
  
  echo "app server:\n";
  </code></pre>
  
  <p>I try the file_get_contents, but there is no obvious differences when switching to file_get_contents. When the concurrency is 1, all methods are good. But they all start downgrading when concurrency increases.</p>
  
  <hr>
  
  <p>I think I find something related to this issue so I just post another question <a href="https://unix.stackexchange.com/questions/311380/concurrent-curl-could-not-resolve-host">concurrent curl could not resolve host</a>. This might be the root cause but I don't have any answer yet.</p>
  
  <hr>
  
  <p>After trying for so long, I think this is definitely related to name resolving. And here is the php script that can execute at concurrent level 500</p>
  
  <pre><code>&lt;?php
  
  require 'vendor/autoload.php';
  
  use GuzzleHttp\Client;
  
  $opt = 1;
  $url = 'http://localhost:81/a.php';
  
  switch ($opt) {
      case 1:
          // create curl resource
          $ch = curl_init();
  
          // set url
          curl_setopt($ch, CURLOPT_URL, $url);
  
          //return the transfer as a string
          curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
  
          curl_setopt($ch, CURLOPT_PROXY, 'localhost');
  
          // $output contains the output string
          $output = curl_exec($ch);
  
          curl_close($ch);
  
          echo $output;
          break;
      case 2:
          $client = new Client();
          $response = $client-&gt;request('GET', $url, ['proxy' =&gt; 'localhost']);
          echo $response-&gt;getBody();
          break;
      case 3:
          echo file_get_contents($url);
          break;
      default:
          echo "no opt";
  }
  
  echo "app server:\n";
  </code></pre>
  
  <p>What really matters are <code>curl_setopt($ch, CURLOPT_PROXY, 'localhost');</code> and <code>$response = $client-&gt;request('GET', $url, ['proxy' =&gt; 'localhost']);</code>. It tells curl to use localhost as proxy.</p>
  
  <p>And here is the result of ab test</p>
  
  <pre><code>ab -n 1000 -c 500 http://192.168.10.173/b.php
  This is ApacheBench, Version 2.3 &lt;$Revision: 1528965 $&gt;
  Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
  Licensed to The Apache Software Foundation, http://www.apache.org/
  
  Benchmarking 192.168.10.173 (be patient)
  Completed 100 requests
  Completed 200 requests
  Completed 300 requests
  Completed 400 requests
  Completed 500 requests
  Completed 600 requests
  Completed 700 requests
  Completed 800 requests
  Completed 900 requests
  Completed 1000 requests
  Finished 1000 requests
  
  
  Server Software:        nginx/1.10.0
  Server Hostname:        192.168.10.173
  Server Port:            80
  
  Document Path:          /b.php
  Document Length:        182 bytes
  
  Concurrency Level:      500
  Time taken for tests:   0.251 seconds
  Complete requests:      1000
  Failed requests:        184
     (Connect: 0, Receive: 0, Length: 184, Exceptions: 0)
  Non-2xx responses:      816
  Total transferred:      308960 bytes
  HTML transferred:       150720 bytes
  Requests per second:    3985.59 [#/sec] (mean)
  Time per request:       125.452 [ms] (mean)
  Time per request:       0.251 [ms] (mean, across all concurrent requests)
  Transfer rate:          1202.53 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    6   4.9      5      14
  Processing:     9   38  42.8     22     212
  Waiting:        8   38  42.9     22     212
  Total:         11   44  44.4     31     214
  
  Percentage of the requests served within a certain time (ms)
    50%     31
    66%     37
    75%     37
    80%     38
    90%    122
    95%    135
    98%    207
    99%    211
   100%    214 (longest request)
  </code></pre>
  
  <p>But still why name resolving failed at concurrency level 5 when not using localhost as proxy?</p>
  
  <hr>
  
  <p>The virtual host setting is very simple and clean, and almost everything is in default configuration. I do not use iptables on this server, neither do I config anything special.</p>
  
  <pre><code>server {
      listen 81 default_server;
      listen [::]:81 default_server;
  
      root /var/www/html;
  
      index index.html index.htm index.nginx-debian.html;
  
      server_name _;
  
      location / {
          try_files $uri $uri/ =404;
      }
  
      location ~ \.php$ {
          include snippets/fastcgi-php.conf;
          fastcgi_pass unix:/run/php/php7.0-fpm.sock;
      }
  }
  </code></pre>
  
  <hr>
  
  <p>Find something interesting! If you do another ab test right after the first one in about 3 seconds. The second ab test pretty quick.</p>
  
  <p><strong>Without using localhost as proxy</strong></p>
  
  <pre><code>ab -n 10 -c 5 http://192.168.10.173/b.php &lt;-- This takes 2.8 seconds to finish.
  ab -n 10 -c 5 http://192.168.10.173/b.php &lt;-- This takes 0.008 seconds only.
  </code></pre>
  
  <p><strong>Using localhost as proxy</strong></p>
  
  <pre><code>ab -n 10 -c 5 http://192.168.10.173/b.php &lt;-- This takes 0.006 seconds.
  ab -n 10 -c 5 http://192.168.10.173/b.php &lt;-- This takes 0.006 seconds.
  </code></pre>
  
  <p>I think it still means that the issue is <strong>name resolving</strong>. But Why?</p>
  
  <hr>
  
  <p>Assumption: nginx is not listening to localhost:81</p>
  
  <p>I tried adding <code>listen 127.0.0.1:81;</code> to nginx, and it shows no effect.</p>
  
  <p>Find myself making some mistakes about using curl proxy, <strong>that does not work!</strong> Update other details later.</p>
  
  <hr>
  
  <p>Solved, not related to proxy, or anything. The root cause is <code>pm.start_servers</code> in php-fpm's <code>www.conf</code>.</p>