DOC: Disallowing search engines in the development docs
- [X] closes #26924
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=h1) Report
> Merging [#26931](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d39a6de66803b2a13d6bbb8aa98c56e90f95526c?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26931/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26931      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50716    50716              
==========================================
- Hits        46593    46589       -4     
- Misses       4123     4127       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.45% <ø> (ø)` | :arrow_up: |
| #single | `41.1% <ø> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26931/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26931/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.88% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=footer). Last update [d39a6de...dbd3764](https://codecov.io/gh/pandas-dev/pandas/pull/26931?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 sure. note sure how we can tell if this works? The file has been created successfully, that worked: https://dev.pandas.io/robots.txt

I think Google has tools where they give you information on how they crawl your site, but not much idea how that works, and not really looking forward to start working in SEO now. :)

I think we can simply wait some days until Google updates its data, and then we shouldn't see all the results here: https://www.google.com/search?q=site%3Adev.pandas.io