PERF: don't materialize arrays on checking in groupby
```
    before     after       ratio
  [a6fcec6c] [6edcdc0d]
-   15.41ms     9.48ms      0.62  groupby.groupby_size.time_groupby_size

```

this is more dramatic as the size increases as this should be pretty much a constant time operation (the actual size computation itself).
cc @wesm @cpcloud  # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=h1) Report
> Merging [#16413](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ee54722fbf43fdfdafb25282ff84f88fd7426334?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16413/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16413      +/-   ##
==========================================
+ Coverage   90.41%   90.42%   +0.01%     
==========================================
  Files         161      161              
  Lines       51024    51023       -1     
==========================================
+ Hits        46133    46138       +5     
+ Misses       4891     4885       -6
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.26% <ø> (+0.01%)` | :arrow_up: |
| #single | `40.17% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.74% <ø> (-0.01%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.39% <0%> (+0.33%)` | :arrow_up: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `96.02% <0%> (+0.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=footer). Last update [ee54722...400e14a](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=h1) Report
> Merging [#16413](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ee54722fbf43fdfdafb25282ff84f88fd7426334?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16413/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16413      +/-   ##
==========================================
+ Coverage   90.41%   90.42%   +0.01%     
==========================================
  Files         161      161              
  Lines       51024    51023       -1     
==========================================
+ Hits        46133    46138       +5     
+ Misses       4891     4885       -6
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.26% <ø> (+0.01%)` | :arrow_up: |
| #single | `40.17% <ø> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.74% <ø> (-0.01%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.39% <0%> (+0.33%)` | :arrow_up: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `96.02% <0%> (+0.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=footer). Last update [ee54722...400e14a](https://codecov.io/gh/pandas-dev/pandas/pull/16413?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
