title: How Can I Effectively 'Max Out' Concurrent HTTP Requests?
body: <p>I'm currently trying a bit of an experiment with Go. Here's what I'm attempting to do:</p>
  
  <p>I've got a REST API service running, and I'd like to query a specific URL over and over again in as many Goroutines as possible, to see how performant these responses are (by viewing my REST API server logs). I'd like to send off a total of 1 million HTTP requests before quitting the program -- executing as many concurrently as my computer will allow.</p>
  
  <p>I'm aware that there are tools which are meant to do this, but I'm primarily interested in how to maximize my HTTP concurrency in Go with goroutines.</p>
  
  <p>Here's my code:</p>
  
  <pre><code>package main
  
  import (
      "fmt"
      "net/http"
      "runtime"
      "time"
  )
  
  func main() {
      runtime.GOMAXPROCS(runtime.NumCPU())
      transport := &amp;http.Transport{}
  
      for i := 0; i &lt; 1000000; i++ {
          go func() {
              req, _ := http.NewRequest("GET", "http://myapi.com", nil)
              req.Header.Set("User-Agent", "custom-agent")
              req.SetBasicAuth("xxx", "xxx")
              resp, err := transport.RoundTrip(req)
              if err != nil {
                  panic("HTTP request failed.")
              }
              defer resp.Body.Close()
  
              if resp.StatusCode != 302 {
                  panic("Unexpected response returned.")
              }
  
              location := resp.Header.Get("Location")
              if location == "" {
                  panic("No location header returned.")
              }
              fmt.Println("Location Header Value:", location)
          }()
      }
  
      time.Sleep(60 * time.Second)
  }
  </code></pre>
  
  <p>What I'm expecting this code to do is:</p>
  
  <ul>
  <li>Start 1,000,000 goroutines, each one making HTTP requests to my API service.</li>
  <li>Run the goroutines concurrently across all of my CPUs (since I used the runtime package to increase the <code>GOMAXPROCS</code> setting).</li>
  </ul>
  
  <p>What happens, however, is that I get the following errors (too many to paste, so I'm only including a bit of the output):</p>
  
  <pre><code>goroutine 16680 [IO wait]:
  net.runtime_pollWait(0xcb1d878, 0x77, 0x0)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/netpoll.goc:116 +0x6a
  net.(*pollDesc).Wait(0xc212a86ca0, 0x77, 0x55d0c0, 0x24)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/fd_poll_runtime.go:81 +0x34
  net.(*pollDesc).WaitWrite(0xc212a86ca0, 0x24, 0x55d0c0)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/fd_poll_runtime.go:90 +0x30
  net.(*netFD).connect(0xc212a86c40, 0x0, 0x0, 0xb4c97e8, 0xc212a84500, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/fd_unix.go:86 +0x166
  net.(*netFD).dial(0xc212a86c40, 0xb4c87d8, 0x0, 0xb4c87d8, 0xc212a878d0, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/sock_posix.go:121 +0x2fd
  net.socket(0x2402c0, 0x3, 0x2, 0x1, 0x0, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/sock_posix.go:91 +0x40b
  net.internetSocket(0x2402c0, 0x3, 0xb4c87d8, 0x0, 0xb4c87d8, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/ipsock_posix.go:136 +0x161
  net.dialTCP(0x2402c0, 0x3, 0x0, 0xc212a878d0, 0x0, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/tcpsock_posix.go:155 +0xef
  net.dialSingle(0x2402c0, 0x3, 0xc210d161e0, 0x15, 0x0, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/dial.go:225 +0x3d8
  net.func·015(0x0, 0x0, 0x0, 0x2402c0, 0x3, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/dial.go:158 +0xde
  net.dial(0x2402c0, 0x3, 0xb4c8748, 0xc212a878d0, 0xafbbcd8, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/fd_unix.go:40 +0x45
  net.(*Dialer).Dial(0xafbbd78, 0x2402c0, 0x3, 0xc210d161e0, 0x15, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/dial.go:165 +0x3e0
  net.Dial(0x2402c0, 0x3, 0xc210d161e0, 0x15, 0x0, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/dial.go:138 +0x75
  net/http.(*Transport).dial(0xc210057280, 0x2402c0, 0x3, 0xc210d161e0, 0x15, ...)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/transport.go:401 +0xd4
  net/http.(*Transport).dialConn(0xc210057280, 0xc2112efa80, 0x0, 0x0, 0x0)
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/transport.go:444 +0x6e
  net/http.func·014()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/transport.go:419 +0x3e
  created by net/http.(*Transport).getConn
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/transport.go:421 +0x11a
  </code></pre>
  
  <p>I'm running this script on a Mac OSX 10.9.2 laptop with 16GB of RAM and a 2.6GHz Intel Core i5 processor.</p>
  
  <p>What can I do to 'flood' my laptop with as many concurrent HTTP requests as possible?</p>