title: Parsing individual lines in a robots.txt file with C#
body: <p>Working on an application to parse robots.txt. I wrote myself a method that pulled the the file from a webserver, and threw the ouput into a textbox. I would like the output to display a single line of text for every line thats in the file, just as it would appear if you were looking at the robots.txt normally, however the ouput in my textbox is all of the lines of text without carriage returns or line breaks. So I thought I'd be crafty, make a string[] for all the lines, make a foreach loop and all would be well. Alas that did not work, so then I thought I would try System.Enviornment.Newline, still not working. Here's the code as it sounds now....how can I change this so I get all the individual lines of robots.txt as opposed to a bunch of text cobbled together?</p>
  
  <pre><code>public void getRobots()
  {
      WebClient wClient = new WebClient();
      string url = String.Format("http://{0}/robots.txt", urlBox.Text);
  
      try
      {
          Stream data = wClient.OpenRead(url);
          StreamReader read = new StreamReader(data);
          string[] lines = new string[] { read.ReadToEnd() };
  
          foreach (string line in lines)
          {
              textBox1.AppendText(line + System.Environment.NewLine);
          }
      }
      catch (WebException ex)
      {
          MessageBox.Show(ex.Message, null, MessageBoxButtons.OK);
      }
  }
  </code></pre>