REF: use cached inferred_type when calling lib.infer_dtype(index)
Among other things, this will let us avoid a couple of ugly calls in Series

```
        if isinstance(key, Index):
            key_type = key.inferred_type
        else:
            key_type = lib.infer_dtype(key, skipna=False)
```

Cleanup of nearby EA-handling code
lgtm. can you merge master and ping on green. ping