title: How can I call many URLs from a list asynchronously
body: <p>I have a few hundred thousand URLs that I need to call. These are calls to an application server which will process them and write a status code to a table. I do not need to wait for a response (success/fail), only that the server got the request. I also want to be able to specify how many concurrent jobs can be running at once as I haven't worked out how many concurrent requests tomcat can handle.</p>
  
  <p>Here's what I've got so far, basically taken from someone's else's attempt to do something similar, just not with url calls. The text file contains each url on its own line. The url looks like this:</p>
  
  <pre><code>http://webserver:8080/app/mwo/services/create?server=ServerName&amp;e1user=admin&amp;newMWONum=123456&amp;sourceMWONum=0&amp;tagNum=33-A-1B
  </code></pre>
  
  <p>And the code:</p>
  
  <pre><code>$maxConcurrentJobs = 10
  $content = Get-Content -Path "C:\Temp\urls.txt"
  
  foreach ($url in $content) {
      $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
      if ($running.Count -le $maxConcurrentJobs) {
          Start-Job {
               Invoke-WebRequest -UseBasicParsing -Uri $using:url
          }
      } else {
           $running | Wait-Job -Any
      }
      Get-Job | Receive-Job
  }
  </code></pre>
  
  <p>The problems I'm having is that it is giving 2 errors per "job" and I'm not sure why. When I dump the url array $content it looks fine and when I run my Invoke-WebRequest one by one they work without error.</p>
  
  <pre><code>126    Job126          BackgroundJob   Running       True            localhost            ...                
  Invalid URI: The hostname could not be parsed.
      + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
      + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
     d
      + PSComputerName        : localhost
  
  Invalid URI: The hostname could not be parsed.
      + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException
      + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodComman 
     d
      + PSComputerName        : localhost
  </code></pre>
  
  <p>Any help or alternative implementations would be appreciated. I'm open to not using powershell, but I'm limited to Windows 7 Desktops or Windows 2008 R2 servers, and I'd probably be running the final script on the server itself using localhost in the url to cut down on network delays.</p>