title: Effective way to get urls of files and folders
body: <p>Let's say that I can get certain txt files from a website via a url that looks something like something.com/folderIndex/folderIndex_fileIndex.txt and I don't know the exact number of files, but I want to get the URLs of all the files in all the folders. What would be the optimal way to do this ? Currently I am doing the following. </p>
  
  <pre><code>public async void GetValidFolderCount()
          {
              bool valid = true;
              int folderIndex = 0;
              using (var client = new HttpClient())
              {
                  while (valid)
                  {
                      folderIndex++;
                      string url = String.Format("{0}/{1}", rootUrl, folderIndex);
                      var result = await client.GetAsync(new Uri(url));
                      if (result.StatusCode != HttpStatusCode.OK)
                      {
                          valid = false;
                          break;
                      }
                      FolderCount++;
                      bool res =  await GetValidFileCount(folderIndex, url);
                  }
              }
              GetComplete = true;
          }
  </code></pre>
  
  <p>btw, I had to use <code>Task&lt;bool&gt;</code> here because without it the number of url's would be different every time.</p>
  
  <pre><code>public async Task&lt;bool&gt; GetValidFileCount(int folderIndex, string baseUrl)
      {
          bool valid = true;
          int fileIndex = 0;
          using (var client = new HttpClient())
          {
              while (valid)
              {
                  fileIndex++;
                  string fileName = folderIndex + "_" + ((fileIndex &lt; 10) ? ("0" + fileIndex) : fileIndex.ToString()) + fileExtension;
                  string fileAddress = String.Format("{0}/{1}", baseUrl,fileName);
                  try
                  {
                      var result = await client.GetAsync(new Uri(fileAddress));
                      if (result.StatusCode != HttpStatusCode.OK)
                      {
                          valid = false;
                          break;
                      }
                  }
                  catch
                  {
                      continue;
                  }
                  UrlRepository.UrlList.Add(fileAddress);
              }
          }
          return true;
  
      }
  </code></pre>
  
  <p>This approach takes up to 2 minutes,  I am pretty sure it could not be considered effective, however the overall number of items in the end is about 1900, so I am not really sure.</p>
  
  <p>I also considered the following approach : parse the HTML and find the anchors, but it did not seem like a good idea.</p>
  
  <p>UPD: The overall goal is to download all the files, but I have to implement a progress bar that would show the number of files I downloaded and number of files left ,so I thought that getting the url's beforehand is a good idea as I get the number of files and I got the url's. </p>