title: How to find files in website directory?
body: <p>I'm creating a web crawler. I'm ganna give it an URL and it will scan through the directory and sub directories for .html files. I've been looking at two alternatives:</p>
  
  <ol>
  <li><p><code>scandir($url)</code>. This works on local files but not on http sites. Is this because of file permissions? I'm guessing it shouldn't work since it would be dangerous for everyone to have access to your website files.</p></li>
  <li><p>Searching for links and following them. I can do file_get_contents on the index file, find links and then follow them to their .html files. </p></li>
  </ol>
  
  <p>Do any of these 2 work or is there a third alternative?</p>