title: web crawling in a multi threaded console app
body: <p>New to multi-threded apps.</p>
  
  <p>I am trying to create a console app to check a given list of IP addresses (intranet). Each web page for any given IP address contains some stats, displayed in an html table, that I need to collect.
  I can do this in a single thread: set up the request/response sequence, get the page content and parse it. </p>
  
  <p>What I am struggling with right now is to make this multi-threaded since I have to deal with 4000 IP addresses and single thread would take some time. I have the list of IPs in a list or array of strings; do you know how I can set up the threads?</p>
  
  <p>Assuming I have a function that processes the response, say, "ProcessResponse(string s)", and want to start with 10 threads, can I start with something like:</p>
  
  <pre><code>public class PASSServer
  {
      private string _ip;
      public string IPAddress
      {
          get;
          set;
      }
  
      public PASSServer()
      {
      }
  }
  static void Main(string[] args)
  {
          int iNumThreads = 3;
          Thread[] threads = new Thread[iNumThreads];
  
          string[] sIPs = { "192.168.10.20", "192.168.10.21", "192.168.10.22" };
  
          for (int i = 0; i &lt; threads.Length; i++)
          {
              ParameterizedThreadStart start = new ParameterizedThreadStart(Start);
              threads[i] = new Thread(start);
              PASSServer pserver = new PASSServer();
              pserver.IPAddress = sIPs[i];
              threads[i].Start(pserver);
          }
          Console.WriteLine("DONE");
          Console.ReadKey();
      }
      static void Start(object info)
      {
          PASSServer pserver = (PASSServer)info;
          crawl(pserver.IPAddress);
      }
      private static void crawl(string sUrl)
      {
          PASSData cData = new PASSData();
          string sRequestUrl = "http://" + sUrl.Trim() + "/cgi-bin/sysstat?";
          string sEncodingType = "utf-8";
          HttpWebRequest request = (HttpWebRequest)WebRequest.Create(sRequestUrl);
          request.KeepAlive = true;
          request.Timeout = 15 * 1000;
  
          System.Net.HttpWebResponse response = (HttpWebResponse)request.GetResponse();
          string sStatus = ((HttpWebResponse)response).StatusDescription;
          sEncodingType = GetEncodingType(response);
          System.IO.StreamReader reader = new System.IO.StreamReader(response.GetResponseStream(), Encoding.GetEncoding(sEncodingType));
  
          // Read the content.
          string responseFromServer = reader.ReadToEnd();
          Console.WriteLine(responseFromServer);
      }
  </code></pre>
  
  <p>Any help is greatly appreciated. </p>
  
  <p>I have not used multi threading but googled the subject and got some ideas just not sure how best to set up my scenario.</p>