BUG: Fix Series nunique groupby with object dtype
closes #11077

cc @behzadnouri 

or you want to use do a factorize?
 ``` diff
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index f34fd6e..39be706 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2565,13 +2565,21 @@ class SeriesGroupBy(GroupBy):
         ids, _, _ = self.grouper.group_info
         val = self.obj.get_values()

-        sorter = np.lexsort((val, ids))
+        try:
+            sorter = np.lexsort((val, ids))
+        except TypeError:
+            val, _ = algos.factorize(val, sort=False)
+            sorter = np.lexsort((val, ids))
+            isnull = lambda a: a == -1
+        else:
+            isnull = com.isnull
+
         ids, val = ids[sorter], val[sorter]

         # group boundries are where group ids change
         # unique observations are where sorted values change
-        idx = com._ensure_int64(np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]])
-        inc = com._ensure_int64(np.r_[1, val[1:] != val[:-1]])
+        idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
+        inc = np.r_[1, val[1:] != val[:-1]]

         # 1st item of each group is a new unique observation
         mask = isnull(val)
```
 @behzadnouri If you submit that as a PR to my fork, I'll happily merge it in
 the error says `TypeError: merge sort not available for item 0` even though numpy _does_ have merge sort for objects (both `np.argsort` and `np.sort`). so it may be a bug on numpy side. so, it is better to implement try catch in case future numpy releases do implement lexsort for this case

once, values are factorized, original values are not needed and it is more efficient to work with integer factors, as long as `isnull` function is adjusted accordingly. 
 @jreback getting a somewhat strange travis failure here that seems unrelated to my change: https://travis-ci.org/cpcloud/pandas/jobs/80036566
 any ideas what that might be?
 hmm someone else reported that as well
try restarting it and see
 this is passing on my travis ci fork
 couple comments, squash and merge ok
 @cpcloud can you update (e.g. revert those 2 lines). ok to merge on green after that.
 ty sir!
