Aggregate a ndarray with structured dtype or recarray
    and include original data in the result.

    Take aggregate of data set on specified columns, then add the resulting 
    rows back into data set to make a composite object containing both original 
    non-aggregate data rows as well as the aggregate rows.

    First read comments for :func:`tabular.spreadsheet.aggregate`.

    This function returns a numpy ndarray, with the number of rows equaling::

            len(Data) + len(A)

    where `A` is the the result of::

            Data.aggregate(On,AggFuncDict)

    `A` represents the aggregate rows; the other rows were the original data 
    rows.

    This function supports _multiple_ aggregation, meaning that one can first 
    aggregate on one set of factors, then repeat aggregation on the result for 
    another set of factors, without the results of the first aggregation 
    interfering the second.  To achieve this, the method adds two new columns:

    *   a column called "__aggregates__" specifying on which factors the rows 
        that are aggregate rows were aggregated.  Rows added by aggregating on 
        factor `A` (a column in the original data set) will have `A` in the 
        "__aggregates__" column.  When multiple factors `A1`, `A2` , ... are 
        aggregated on, the notation is a comma-separated list:  `A1,A2,...`.  
        This way, when you call `aggregate_in` again, the function only 
        aggregates on the columns that have the empty char '' in their 
        "__aggregates__" column.

    *   a column called '__color__', specifying Gray-Scale colors for 
        aggregated rows that will be used by the Data Environment system 
        browser for colorizing the  data.   When there are multiple levels of 
        aggregation, the coarser aggregate groups (e.g. on fewer factors) get 
        darker gray color then those on finer aggregate groups (e.g. more 
        factors).

    Implemented by the tabarray method 
    :func:`tabular.tab.tabarray.aggregate_in`.

    **Parameters**

            **Data** :  numpy ndarray with structured dtype or recarray

                    The data set to aggregate in.

            **On** :  list of  strings, optional

                    List of column names in `X`.

            **AggFuncDict** :  dictionary, optional

                    Dictionary where

                    *   keys are some (all) column names of `X` that are NOT in 
                        `On`

                    *   values are functions that can be applied to lists or
                        numpy arrays.

                    This specifies how to aggregate the factors _not_ listed in
                    `On`, e.g. the so-called `Off` columns.

            **AggFunc** :  function, optional

                    Function that can be applied to lists or numpy arrays,
                    specifying how to aggregate factors not listed in either 
                    `On` or the keys of `AggFuncDict`, e.g. a "default"
                    aggregation function for the `Off` columns not explicitly
                    listed in `AggFuncDict`.

            **interspersed** :  boolean, optional

                    *   If `True`, aggregate rows are interleaved with the data 
                        of which they are aggregates.

                    *   If `False`, all aggregate rows placed at the end of the 
                        array.

    **Returns**

            **agg** :  numpy ndarray with structured dtype

                    Composite aggregated data set plus original data set.

    **See also:**

            :func:`tabular.spreadsheet.aggregate`

def aggregate_in(Data, On=None, AggFuncDict=None, AggFunc=None, AggList=None,
                  interspersed=True):
    """
    Aggregate a ndarray with structured dtype or recarray
    and include original data in the result.

    Take aggregate of data set on specified columns, then add the resulting 
    rows back into data set to make a composite object containing both original 
    non-aggregate data rows as well as the aggregate rows.

    First read comments for :func:`tabular.spreadsheet.aggregate`.

    This function returns a numpy ndarray, with the number of rows equaling::

            len(Data) + len(A)

    where `A` is the the result of::

            Data.aggregate(On,AggFuncDict)

    `A` represents the aggregate rows; the other rows were the original data 
    rows.

    This function supports _multiple_ aggregation, meaning that one can first 
    aggregate on one set of factors, then repeat aggregation on the result for 
    another set of factors, without the results of the first aggregation 
    interfering the second.  To achieve this, the method adds two new columns:

    *   a column called "__aggregates__" specifying on which factors the rows 
        that are aggregate rows were aggregated.  Rows added by aggregating on 
        factor `A` (a column in the original data set) will have `A` in the 
        "__aggregates__" column.  When multiple factors `A1`, `A2` , ... are 
        aggregated on, the notation is a comma-separated list:  `A1,A2,...`.  
        This way, when you call `aggregate_in` again, the function only 
        aggregates on the columns that have the empty char '' in their 
        "__aggregates__" column.

    *   a column called '__color__', specifying Gray-Scale colors for 
        aggregated rows that will be used by the Data Environment system 
        browser for colorizing the  data.   When there are multiple levels of 
        aggregation, the coarser aggregate groups (e.g. on fewer factors) get 
        darker gray color then those on finer aggregate groups (e.g. more 
        factors).

    Implemented by the tabarray method 
    :func:`tabular.tab.tabarray.aggregate_in`.

    **Parameters**

            **Data** :  numpy ndarray with structured dtype or recarray

                    The data set to aggregate in.

            **On** :  list of  strings, optional

                    List of column names in `X`.

            **AggFuncDict** :  dictionary, optional

                    Dictionary where

                    *   keys are some (all) column names of `X` that are NOT in 
                        `On`

                    *   values are functions that can be applied to lists or
                        numpy arrays.

                    This specifies how to aggregate the factors _not_ listed in
                    `On`, e.g. the so-called `Off` columns.

            **AggFunc** :  function, optional

                    Function that can be applied to lists or numpy arrays,
                    specifying how to aggregate factors not listed in either 
                    `On` or the keys of `AggFuncDict`, e.g. a "default"
                    aggregation function for the `Off` columns not explicitly
                    listed in `AggFuncDict`.

            **interspersed** :  boolean, optional

                    *   If `True`, aggregate rows are interleaved with the data 
                        of which they are aggregates.

                    *   If `False`, all aggregate rows placed at the end of the 
                        array.

    **Returns**

            **agg** :  numpy ndarray with structured dtype

                    Composite aggregated data set plus original data set.

    **See also:**

            :func:`tabular.spreadsheet.aggregate`

    """

    # See if there's an '__aggregates__ column'.  
    # If so, strip off all those that are nontrivial.

    Data = deletecols(Data,'__color__')
    if '__aggregates__' in Data.dtype.names:
        X = Data[Data['__aggregates__'] == ''][:]
        OldAggregates = Data[Data['__aggregates__'] != ''][:]
        AggVars = utils.uniqify(utils.listunion([x.split(',') for x in 
                                OldAggregates['__aggregates__']]))
    else:
        X = Data
        OldAggregates = Data[0:0]
        AggVars = []

    if On == None:
        On = []

    NewAggregates = aggregate(X, On, AggFuncDict=AggFuncDict, 
                AggFunc=AggFunc, AggList=AggList, KeepOthers=True)
    on = ','.join(On)
    NewAggregates = addcols(NewAggregates,   
                            utils.fromarrays([[on]*len(NewAggregates)], 
                            type=np.ndarray, names=['__aggregates__']))
    AggVars = utils.uniqify(AggVars + On)
    Aggregates = rowstack([OldAggregates,NewAggregates],mode='nulls')

    ANLen = np.array([len(x.split(',')) for x in Aggregates['__aggregates__']])
    U = np.array(utils.uniqify(ANLen)); U.sort()
    [A,B] = fast.equalspairs(ANLen,U)
    Grays = np.array(grayspec(len(U)))
    AggColor = utils.fromarrays([Grays[A]], type=np.ndarray, 
           names = ['__color__'])

    Aggregates = addcols(Aggregates,AggColor)

    if not interspersed or len(AggVars) == 0:
        return rowstack([X,Aggregates],mode='nulls')
    else:
        s = ANLen.argsort()
        Aggregates = Aggregates[s[range(len(Aggregates) - 1, -1, -1)]]
        X.sort(order = AggVars)
        Diffs = np.append(np.append([0], 1 + (X[AggVars][1:] != 
                                    X[AggVars][:-1]).nonzero()[0]), [len(X)])
        DiffAtts = ([[t for t in AggVars if X[t][Diffs[i]] != X[t][Diffs[i+1]]] 
                      for i in range(len(Diffs) - 2)] 
                     if len(Diffs) > 2 else []) + [AggVars]

        HH = {}
        for l in utils.uniqify(Aggregates['__aggregates__']):
            Avars = l.split(',')
            HH[l] = fast.recarrayequalspairs(X[Avars][Diffs[:-1]], 
                                             Aggregates[Avars])

        Order = []
        for i in range(len(Diffs)-1):
            Order.extend(range(Diffs[i], Diffs[i+1]))

            Get = []
            for l in HH.keys():
                Get += [len(X) + j for j in 
                        HH[l][2][range(HH[l][0][i], HH[l][1][i])] if 
                        len(set(DiffAtts[i]).intersection(
                        Aggregates['__aggregates__'][j].split(','))) > 0 and 
                        set(Aggregates['__aggregates__'][j].split(',')) == 
                        set(l.split(','))]

            Order.extend(Get)

        return rowstack([X, Aggregates], mode='nulls')[Order]