PERF: improves performance in remove_unused_categories
on master:

``` python
In [1]: np.random.seed(2718281)

In [2]: n = 1 << 20

In [3]: ix = tm.makeCategoricalIndex(n, n // 10)

In [4]: %timeit ix.remove_unused_categories()
1 loops, best of 3: 527 ms per loop
```

on branch:

``` python
In [4]: %timeit ix.remove_unused_categories()
1 loops, best of 3: 216 ms per loop
```

merged via https://github.com/pydata/pandas/commit/7031e5f11155f6e8f6a8f04afb6159455bd62036

thanks you sir!
