title: How can I check if page has noindex?
body: <p>My intention is to fetch the link with PHP and maybe with <a href="http://simplehtmldom.sourceforge.net" rel="nofollow noreferrer">Simple PHP DOM parser</a> (or something similar) parse the content and look for H1-H6 tags. But prior to that I would need to find out if the page is being indexed at all.</p>
  
  <p>Other than parsing the content and searching for <code>&lt;meta name="robots" content="noindex"&gt;</code> or similar, is there a way I could check if a page is set to noindex also in robots.txt?</p>