title: Node.js request module getting ETIMEDOUT and ESOCKETTIMEDOUT
body: <p>I'm crawling a lot of links with the <a href="https://www.npmjs.com/package/request" rel="noreferrer">request</a> module in parallel with combination of the <a href="https://www.npmjs.com/package/async" rel="noreferrer">async</a> module.
  <br/>I'm noticing alot of <code>ETIMEDOUT</code> and <code>ESOCKETTIMEDOUT</code> errors although the links are reachable and respond fairly quickly using chrome.</p>
  
  <p>I've limit the <code>maxSockets</code> to 2 and the <code>timeout</code> to 10000 in the request options.
  I'm using <code>async.filterLimit()</code> with a limit of 2 to even cut down the parallelism to 2 request each time.
  So I have 2 sockets, 2 request, and a timeout of 10 seconds to wait for headers response from the server yet I get these errors.</p>
  
  <p>Here;s request configuration I use:</p>
  
  <pre><code>{
      ...
      pool: {
          maxSockets: 2
      },
      timeout: 10000
      ,
      time: true
      ...
  }
  </code></pre>
  
  <p>Here's the snippet of code I use to fecth links:</p>
  
  <pre><code>var self = this;
  async.filterLimit(resources, 2, function(resource, callback) {
      request({
          uri: resource.uri
      }, function (error, response, body) {
          if (!error &amp;&amp; response.statusCode === 200) {
              ...
          } else {
              self.emit('error', resource, error);
          }
          callback(...);
      })
  }, function(result) {
      callback(null, result);
  });
  </code></pre>
  
  <p>I listened to the error event and I see whenever the error code is <code>ETIMEDOUT</code> the connect object is either true/false so sometimes it's a connection timeout and sometimes it's not (according to request docs)</p>
  
  <p><strong>UPDATE:</strong>
  I decided to boost up the <code>maxSockets</code> to <code>Infinity</code> so no connection will be hangup due to lack of available sockets:</p>
  
  <pre><code>pool: {
      maxSockets: Infinity
  }
  </code></pre>
  
  <p>In-order to control the bandwidth I implemented a <code>requestLoop</code> method that handle the request with a <code>maxAttemps</code> and <code>retryDelay</code> parameters to control the requests:</p>
  
  <pre><code>async.filterLimit(resources, 10, function(resource, callback) {
      self.requestLoop({
          uri: resource.uri
      }, 100, 5000, function (error, response, body) {
              var fetched = false;
              if (!error) {
                  ...
              } else {
                  ....
              }
              callback(...);
          });
  }, function(result) {
      callback(null, result);
  });
  </code></pre>
  
  <p>Implementation of requestLoop:</p>
  
  <pre><code>requestLoop = function(options, attemptsLeft, retryDelay, callback, lastError) {
      var self = this;
      if (attemptsLeft &lt;= 0) {
          callback((lastError != null ? lastError : new Error('...')));
      } else {
          request(options, function (error, response, body) {
              var recoverableErrors = ['ESOCKETTIMEDOUT', 'ETIMEDOUT', 'ECONNRESET', 'ECONNREFUSED'];
              var e;
              if ((error &amp;&amp; _.contains(recoverableErrors, error.code)) || (response &amp;&amp; (500 &lt;= response.statusCode &amp;&amp; response.statusCode &lt; 600))) {
                  e = error ? new Error('...');
                  e.code = error ? error.code : response.statusCode;
                  setTimeout((function () {
                      self.requestLoop(options, --attemptsLeft, retryDelay, callback, e);
                  }), retryDelay);
              } else if (!error &amp;&amp; (200 &lt;= response.statusCode &amp;&amp; response.statusCode &lt; 300)) {
                  callback(null, response, body);
              } else if (error) {
                  e = new Error('...');
                  e.code = error.code;
                  callback(e);
              } else {
                  e = new Error('...');
                  e.code = response.statusCode;
                  callback(e);
              }
          });
      }
  };
  </code></pre>
  
  <p>So this to sum it up:
  - Boosted <code>maxSockets</code> to <code>Infinity</code> to try overcome timeout error of sockets connection
  - Implemnted <code>requestLoop</code> method to control failed request and <code>maxAttemps</code> as well as <code>retryDelay</code> of such requests
  - Also there's maxium number of concurrent request set by the number passed to <code>async.filterLimit</code></p>
  
  <p>I want to note that I've also played with the settings of everything here in-order to get errors free crawling but so far attempts failed as-well.</p>
  
  <p>Still looking for help about solving this problem.</p>
  
  <p><strong>UPDATE2:</strong>
  I've decided to drop async.filterLimit and make my own limit mechanism.
  I just have 3 variables to help me achieve this:<br/>
  <code>pendingRequests</code> - a request array which will hold all requests (will explain later)
  <code>activeRequests</code> - number of active requests
  <code>maxConcurrentRequests</code> - number of maximum allowed concurrent requests</p>
  
  <p>into the pendingRequests array, i push a complex object containing a reference to the requestLoop function as well as arguments array containing the arguments to be passed to the loop function:</p>
  
  <pre><code>self.pendingRequests.push({
      "arguments": [{
          uri: resource.uri.toString()
      }, self.maxAttempts, function (error, response, body) {
          if (!error) {
              if (self.policyChecker.isMimeTypeAllowed((response.headers['content-type'] || '').split(';')[0]) &amp;&amp;
                  self.policyChecker.isFileSizeAllowed(body)) {
                  self.totalBytesFetched += body.length;
                  resource.content = self.decodeBuffer(body, response.headers["content-type"] || '', resource);
                  callback(null, resource);
              } else {
                  self.fetchedUris.splice(self.fetchedUris.indexOf(resource.uri.toString()), 1);
                  callback(new Error('Fetch failed because a mime-type is not allowed or file size is bigger than permited'));
              }
          } else {
              self.fetchedUris.splice(self.fetchedUris.indexOf(resource.uri.toString()), 1);
              callback(error);
          }
          self.activeRequests--;
          self.runRequest();
      }],
      "function": self.requestLoop
  });
  self.runRequest();
  </code></pre>
  
  <p>You'' notice the call to <code>runRequest()</code> at the end.
  This function job is to manage the requests and fire requests when it can while keeping the maximum <code>activeRequests</code> under the limit of <code>maxConcurrentRequests</code>:</p>
  
  <pre><code>var self = this;
  process.nextTick(function() {
      var next;
      if (!self.pendingRequests.length || self.activeRequests &gt;= self.maxConcurrentRequests) {
          return;
      }
      self.activeRequests++;
      next = self.pendingRequests.shift();
      next["function"].apply(self, next["arguments"]);
      self.runRequest();
  });
  </code></pre>
  
  <p>This should solve any Timeouts errors, through my testings tho, I've still noticed some timeouts in specific websites I've tested this on. I can't be 100% sure about this, but I'm thinking it's due to the nature of the website backing http-server limiting a user requests to a maximum by doing an ip-checking and as a result returning some HTTP 400 messages to prevent a possible 'attack' on the server.</p>