title: Multiple HttpClients with proxies, trying to achieve maximum download speed
body: <p>I need to use proxies to download a forum. The problem with my code is that it takes only 10% of my internet bandwidth. Also I have read that I need to use a single <code>HttpClient</code> instance, but with multiple proxies I don't know how to do it. Changing <code>MaxDegreeOfParallelism</code> doesn't change anything.</p>
  <pre><code>public static IAsyncEnumerable&lt;IFetchResult&gt; FetchInParallelAsync(
      this IEnumerable&lt;Url&gt; urls, FetchContext context)
  {
      var fetchBlcock = new TransformBlock&lt;Url, IFetchResult&gt;(
          transform: url =&gt; url.FetchAsync(context), 
          dataflowBlockOptions: new ExecutionDataflowBlockOptions 
          {
              MaxDegreeOfParallelism = 128
          }
      );
      foreach(var url in urls)
          fetchBlcock.Post(url);
  
      fetchBlcock.Complete();
      var result = fetchBlcock.ToAsyncEnumerable();
      return result;
  }
  </code></pre>
  <p>Every call to <code>FetchAsync</code> will create or reuse a <code>HttpClient</code> with a <code>WebProxy</code>.</p>
  <pre><code>public static async Task&lt;IFetchResult&gt; FetchAsync(this Url url, FetchContext context)
  {
      var httpClient = context.ProxyPool.Rent();
      var result = await url.FetchAsync(httpClient, context.Observer, context.Delay,
          context.isReloadWithCookie);
      context.ProxyPool.Return(httpClient);
      return result;
  }
  
  public HttpClient Rent() 
  {
      lock(_lockObject)
      {
          if (_uninitiliazedDatacenterProxiesAddresses.Count != 0)
          {
              var proxyAddress = _uninitiliazedDatacenterProxiesAddresses.Pop();
              return proxyAddress.GetWebProxy(DataCenterProxiesCredentials).GetHttpClient();
          }
          return _proxiesQueue.Dequeue();
      }
  }
  </code></pre>
  <p>I am a novice at software developing, but the task of downloading using hundreds or thousands of proxies asynchronously looks like a trivial task that many should have been faced with and found a correct way to do it. So far I was unable to find any solutions to my problem on the internet. Any thoughts of how to achieve maximum download speed?</p>