Aggregate a ndarray with structured dtype (or recarray) on columns for 
    given functions.

    Aggregate a numpy recarray (or tabular tabarray) on a set of specified 
    factors, using specified aggregation functions.

    Intuitively, this function will aggregate the dataset `X` on a set of 
    columns, whose names are listed in `On`, so that the resulting aggregate 
    data set has one record for each unique tuples of values in those columns.

    The more factors listed in `On` argument, the "finer" is the aggregation, 
    the fewer factors, the "coarser" the aggregation.  For example, if::

            On = 'A'

    the resulting tabarray will have one record for each unique value of a in 
    X['A'], while if On = ['A', 'B'] then the resulting tabarray will have 
    one record for each unique (a, b) pair in X[['A', 'B']]

    The `AggFunc` argument is a function that specifies how to aggregate the 
    factors _not_ listed in `On`, e.g. the so-called `Off` columns.  For 
    example.  For instance, if On = ['A', 'B'] and `C` is a third column, then ::

            AggFunc = numpy.mean

    will result in a tabarray containing a `C` column whose values are the 
    average of the values from the original `C` columns corresponding to each 
    unique (a, b) pair. 
    
    If you want to specify a different aggreagtion method for each `Off` column,
    use `AggFuncDict` instead of AggFunc.  `AggFuncDict` is a dictionary of
    functions whose keys are column names.  AggFuncDict[C] will be applied to 
    the C column, AggFuncDict[D] to the D column, etc.   AggFunc and AggFuncDict
    can be used simultaneously, with the elements of AggFuncDict overriding 
    AggFunc for the specified columns. 
    
    Using either AggFunc or AggFuncDict, the resulting tabarray has the same 
    columns as the original tabarray.  Sometimes you want to specify the ability
    to create new aggregate columns not corresponding to one specific column in the 
    original tabarray, and taking data from several.  To achieve this, use the
    AggList argument.   AggList is a list of three-element lists of the form:
         (name, func, col_names)
    where `name` specifies the resulting column name in the aggregated tabarray,
    `func` specifies the aggregation function, and `col_names` specifies the 
    list of columns names from the original tabarray that will be needed to 
    compute the aggregate values.  (That is, for each unique tuple `t` in the `On` 
    columns, the subarray of X[col_names] for which X[On] == t is passed to 
    `func`.)
        
    If an `Off` column is _not_ provided as a key in `AggFuncDict`, a default 
    aggregator function will be used:  the sum function for numerical columns, 
    concatenation for string columns.

    Implemented by the tabarray method 
    :func:`tabular.tab.tabarray.aggregate`.

    **Parameters**

            **X** :  numpy ndarray with structured dtype or recarray

                    The data set to aggregate.

            **On** :  string or list of  strings, optional

                    List of column names in `X`.

            **AggFuncDict** :  dictionary, optional

                    Dictionary where

                    *   keys are some (all) column names of `X` that are NOT
                        in `On`

                    *   values are functions that can be applied to lists or
                        numpy arrays.

                    This specifies how to aggregate the factors _not_ listed in
                    `On`, e.g. the so-called `Off` columns.

            **AggFunc** :  function, optional

                    Function that can be applied to lists or numpy arrays,
                    specifying how to aggregate factors not listed in either
                    `On` or the keys of `AggFuncDict`, e.g. a "default"
                    aggregation function for the `Off` columns not explicitly
                    listed in `AggFuncDict`.
                    
            **AggList** :  list, optional

                    List of tuples 

            **returnsort** :        Boolean, optional

                    If `returnsort == True`, then return a list of indices
                    describing how `X` was sorted as a result of aggregation.
                    Default value is `False`.

    **Returns**

            **agg** :  numpy ndarray with structured dtype

                    Aggregated data set.

            **index_array** :  numpy ndarray (int, 1D)

                    Returned only if `returnsort == True`.  List of indices
                    describing how `X` was sorted as a result of aggregation.

    **See also:**

            :func:`tabular.spreadsheet.aggregate_in`

def aggregate(X, On=None, AggFuncDict=None, AggFunc=None,
              AggList=None, returnsort=False, KeepOthers=True,
              keyfuncdict=None):
    """
    Aggregate a ndarray with structured dtype (or recarray) on columns for 
    given functions.

    Aggregate a numpy recarray (or tabular tabarray) on a set of specified 
    factors, using specified aggregation functions.

    Intuitively, this function will aggregate the dataset `X` on a set of 
    columns, whose names are listed in `On`, so that the resulting aggregate 
    data set has one record for each unique tuples of values in those columns.

    The more factors listed in `On` argument, the "finer" is the aggregation, 
    the fewer factors, the "coarser" the aggregation.  For example, if::

            On = 'A'

    the resulting tabarray will have one record for each unique value of a in 
    X['A'], while if On = ['A', 'B'] then the resulting tabarray will have 
    one record for each unique (a, b) pair in X[['A', 'B']]

    The `AggFunc` argument is a function that specifies how to aggregate the 
    factors _not_ listed in `On`, e.g. the so-called `Off` columns.  For 
    example.  For instance, if On = ['A', 'B'] and `C` is a third column, then ::

            AggFunc = numpy.mean

    will result in a tabarray containing a `C` column whose values are the 
    average of the values from the original `C` columns corresponding to each 
    unique (a, b) pair. 
    
    If you want to specify a different aggreagtion method for each `Off` column,
    use `AggFuncDict` instead of AggFunc.  `AggFuncDict` is a dictionary of
    functions whose keys are column names.  AggFuncDict[C] will be applied to 
    the C column, AggFuncDict[D] to the D column, etc.   AggFunc and AggFuncDict
    can be used simultaneously, with the elements of AggFuncDict overriding 
    AggFunc for the specified columns. 
    
    Using either AggFunc or AggFuncDict, the resulting tabarray has the same 
    columns as the original tabarray.  Sometimes you want to specify the ability
    to create new aggregate columns not corresponding to one specific column in the 
    original tabarray, and taking data from several.  To achieve this, use the
    AggList argument.   AggList is a list of three-element lists of the form:
         (name, func, col_names)
    where `name` specifies the resulting column name in the aggregated tabarray,
    `func` specifies the aggregation function, and `col_names` specifies the 
    list of columns names from the original tabarray that will be needed to 
    compute the aggregate values.  (That is, for each unique tuple `t` in the `On` 
    columns, the subarray of X[col_names] for which X[On] == t is passed to 
    `func`.)
        
    If an `Off` column is _not_ provided as a key in `AggFuncDict`, a default 
    aggregator function will be used:  the sum function for numerical columns, 
    concatenation for string columns.

    Implemented by the tabarray method 
    :func:`tabular.tab.tabarray.aggregate`.

    **Parameters**

            **X** :  numpy ndarray with structured dtype or recarray

                    The data set to aggregate.

            **On** :  string or list of  strings, optional

                    List of column names in `X`.

            **AggFuncDict** :  dictionary, optional

                    Dictionary where

                    *   keys are some (all) column names of `X` that are NOT
                        in `On`

                    *   values are functions that can be applied to lists or
                        numpy arrays.

                    This specifies how to aggregate the factors _not_ listed in
                    `On`, e.g. the so-called `Off` columns.

            **AggFunc** :  function, optional

                    Function that can be applied to lists or numpy arrays,
                    specifying how to aggregate factors not listed in either
                    `On` or the keys of `AggFuncDict`, e.g. a "default"
                    aggregation function for the `Off` columns not explicitly
                    listed in `AggFuncDict`.
                    
            **AggList** :  list, optional

                    List of tuples 

            **returnsort** :        Boolean, optional

                    If `returnsort == True`, then return a list of indices
                    describing how `X` was sorted as a result of aggregation.
                    Default value is `False`.

    **Returns**

            **agg** :  numpy ndarray with structured dtype

                    Aggregated data set.

            **index_array** :  numpy ndarray (int, 1D)

                    Returned only if `returnsort == True`.  List of indices
                    describing how `X` was sorted as a result of aggregation.

    **See also:**

            :func:`tabular.spreadsheet.aggregate_in`


    """
    
    names = X.dtype.names
    if len(X) == 0:
        if returnsort:
            return [X,None]
        else:
            return X

    if On == None:
        On = []
    elif isinstance(On,str):
        On = On.split(',')

    assert all([o in names for o in On]), \
           ("Axes " + str([o for o in On if o not in names]) + 
            " can't be  found.")
   
    if AggList is None:
        AggList = []
    
    if AggFuncDict:
        AggList += AggFuncDict.items()
        
    for (i,x) in enumerate(AggList):
        if utils.is_string_like(x):
            AggList[i] = (x,)
        elif isinstance(x,tuple):
            assert 1 <= len(x) <= 3
            assert isinstance(x[0],str)
            if len(x) == 2 and isinstance(x[1],tuple):
                assert len(x[1]) == 2
                AggList[i] = (x[0],) + x[1]
        else:
            raise ValueError, 'bork'

    Names = [x[0] for x in AggList]
    assert Names == utils.uniqify(Names)
        
    if KeepOthers:
        AggList = [(x,) for x in X.dtype.names if x not in Names + On] + AggList
   
    DefaultChoices = {'string':[], 'sum':[], 'first':[]}

    for (i,v) in enumerate(AggList):
        if len(v) == 1:
            assert v[0] in X.dtype.names
            if AggFunc:
                AggList[i] = v + (AggFunc,v[0])
            else:
                AggList[i] = v + (DefaultChooser(X,v[0], DefaultChoices),v[0])
        elif len(v) == 2:
            if isftype(v[1]):
                assert v[0] in X.dtype.names
                AggList[i] = v + (v[0],)
            elif utils.is_string_like(v[1]):
                if AggFunc:
                    _a = v[1] in X.dtype.names
                    _b = isinstance(v[1],list) and set(v[1]) <= set(X.dtype.names)
                    assert _a or _b
                    AggList[i] = (v[0], AggFunc, v[1])
                else:
                    assert v[1] in X.dtype.names
                    AggList[i] = (v[0],
                                  DefaultChooser(X,v[1],
                                  DefaultChoices),
                                  v[1])
            else:
                raise ValueError,'No specific of name for column.'
        elif len(v) == 3:
            if utils.is_string_like(v[2]):
                assert isftype(v[1]) and v[2] in X.dtype.names
            else:
                assert isftype(v[1]) and \
                (isinstance(v[2],list) and \
                set(v[2]) <= set(X.dtype.names))   

    if len(DefaultChoices['sum']) > 0:
        print('No aggregation function provided for', DefaultChoices['sum'], 
              'so assuming "sum" by default.')
    if len(DefaultChoices['string']) > 0:
        print('No aggregation function provided for', DefaultChoices['string'], 
              'so assuming string concatenation by default.')
    if len(DefaultChoices['first']) > 0:
        print('No aggregation function provided for', DefaultChoices['first'], 
              'and neither summing nor concatenation works, so choosing ' 
              'first value by default.')

    return strictaggregate(X, On, AggList, returnsort, keyfuncdict)