Implements pivoting on numpy ndarrays (with structured dtype) or recarrays.

    See http://en.wikipedia.org/wiki/Pivot_table for information about pivot 
    tables.

    Returns `X` pivoted on (a,b) with `a` as the row axis and `b` values as the 
    column axis.

    So-called "nontrivial columns relative to `b`" in `X` are added as 
    color-grouped sets of columns, and "trivial columns relative to `b`" are 
    also retained as cross-grouped sets of columns if they are listed in `Keep` 
    argument.

    Note that a column `c` in `X` is "trivial relative to `b`" if for all rows 
    i, X[c][i] can be determined from X[b][i], e.g the elements in X[c] are in 
    many-to-any correspondence with the values in X[b].

    The function will raise an exception if the list of pairs of value in 
    X[[a,b]] is not the product of the individual columns values, e.g.::

            X[[a,b]] == set(X[a]) x set(X[b])

    in some ordering.

    Implemented by the tabarray method :func:`tabular.tab.tabarray.pivot`

    **Parameters**

            **X** :  numpy ndarray with structured dtype or recarray

                    The  data set to pivot.

            **a** : string

                    Column name in `X`.

            **b** : string

                    Another column name in `X`.

            **Keep** :  list of strings, optional

                    List of other columns names in `X`.

            **NullVals** :  optional

                    Dictionary mapping column names in `X` other than `a` or 
                    `b` to appropriate null values for their types.

                    If `None`, then the null values defined by the `nullvalue`
                    function are used, see
                    :func:`tabular.spreadsheet.nullvalue`.

            **prefix** :  string, optional

                    Prefix to add to `coloring` keys corresponding to 
                    cross-grouped "trivial columns relative to `b`".  Default 
                    value is an underscore, '_'.

    **Returns**

            **ptable** :  numpy ndarray with structured dtype

                    The resulting pivot table.

            **coloring** :  dictionary

                    Dictionary whose keys are strings and corresponding values 
                    are lists of column names (e.g. strings).

                    There are two groups of keys:

                    *   So-called "nontrivial columns relative to `b`" in `X`.  
                        These correspond to columns in::

                                    set(`X.dtype.names`) - set([a, b])

                    *   Cross-grouped "trivial columns relative to `b`".  The 
                        `prefix` is used to distinguish these.

                    The `coloring` parameter is used by the the tabarray pivot 
                    method, :func:`tabular.tab.tabarray.pivot`.

                    See :func:`tabular.tab.tabarray.__new__` for more
                    information about coloring.

def pivot(X, a, b, Keep=None, NullVals=None, order = None, prefix='_'):
    '''
    Implements pivoting on numpy ndarrays (with structured dtype) or recarrays.

    See http://en.wikipedia.org/wiki/Pivot_table for information about pivot 
    tables.

    Returns `X` pivoted on (a,b) with `a` as the row axis and `b` values as the 
    column axis.

    So-called "nontrivial columns relative to `b`" in `X` are added as 
    color-grouped sets of columns, and "trivial columns relative to `b`" are 
    also retained as cross-grouped sets of columns if they are listed in `Keep` 
    argument.

    Note that a column `c` in `X` is "trivial relative to `b`" if for all rows 
    i, X[c][i] can be determined from X[b][i], e.g the elements in X[c] are in 
    many-to-any correspondence with the values in X[b].

    The function will raise an exception if the list of pairs of value in 
    X[[a,b]] is not the product of the individual columns values, e.g.::

            X[[a,b]] == set(X[a]) x set(X[b])

    in some ordering.

    Implemented by the tabarray method :func:`tabular.tab.tabarray.pivot`

    **Parameters**

            **X** :  numpy ndarray with structured dtype or recarray

                    The  data set to pivot.

            **a** : string

                    Column name in `X`.

            **b** : string

                    Another column name in `X`.

            **Keep** :  list of strings, optional

                    List of other columns names in `X`.

            **NullVals** :  optional

                    Dictionary mapping column names in `X` other than `a` or 
                    `b` to appropriate null values for their types.

                    If `None`, then the null values defined by the `nullvalue`
                    function are used, see
                    :func:`tabular.spreadsheet.nullvalue`.

            **prefix** :  string, optional

                    Prefix to add to `coloring` keys corresponding to 
                    cross-grouped "trivial columns relative to `b`".  Default 
                    value is an underscore, '_'.

    **Returns**

            **ptable** :  numpy ndarray with structured dtype

                    The resulting pivot table.

            **coloring** :  dictionary

                    Dictionary whose keys are strings and corresponding values 
                    are lists of column names (e.g. strings).

                    There are two groups of keys:

                    *   So-called "nontrivial columns relative to `b`" in `X`.  
                        These correspond to columns in::

                                    set(`X.dtype.names`) - set([a, b])

                    *   Cross-grouped "trivial columns relative to `b`".  The 
                        `prefix` is used to distinguish these.

                    The `coloring` parameter is used by the the tabarray pivot 
                    method, :func:`tabular.tab.tabarray.pivot`.

                    See :func:`tabular.tab.tabarray.__new__` for more
                    information about coloring.

    '''

    othernames = [o for o in X.dtype.names if o not in [a,b]]

    for c in [a,b]:
        assert c in X.dtype.names, 'Column ' + c + ' not found.'

    [D,s] = fast.recarrayuniqify(X[[a,b]])
    unique_ab = X[[a,b]][s[D.nonzero()[0]]]
    assert len(X) == len(unique_ab) , \
           ('Pairs of values in columns', a, 'and', b, 'must be unique.')

    [D,s] = fast.arrayuniqify(X[a])
    unique_a = X[a][s[D.nonzero()[0]]]
    [D,s] = fast.arrayuniqify(X[b])
    unique_b = X[b][s[D.nonzero()[0]]]

    Da = len(unique_a)
    Db = len(unique_b)

    if len(X) != Da * Db:
        if list(X.dtype.names).index(a) < list(X.dtype.names).index(b):
            n1 = a ; f1 = unique_a; n2 = b ; f2 = unique_b
        else:
            n1 = b ; f1 = unique_b; n2 = a ; f2 = unique_a
            
        dtype = np.dtype([(n1,f1.dtype.descr[0][1]),(n2,f2.dtype.descr[0][1])])
        allvalues = utils.fromarrays([np.repeat(f1,
              len(f2)),
              np.tile(f2,len(f1))],
              np.ndarray,
              dtype=dtype)
        
        
        missingvalues = allvalues[np.invert(fast.recarrayisin(allvalues,
                                      X[[a,b]]))]

        if NullVals == None:
           NullVals = {}
        
        if not isinstance(NullVals,dict):
            if hasattr(NullVals,'__call__'):
                NullVals = dict([(o,NullVals(o)) for o in othernames])
            else:
                NullVals = dict([(o,NullVals) for o in othernames])

        nullvals = utils.fromrecords([[NullVals[o] if o in NullVals.keys() 
                            else utils.DEFAULT_NULLVALUE(X[o][0]) for o in 
                            othernames]], type=np.ndarray, names=othernames)
        
        nullarray = nullvals.repeat(len(missingvalues))
        Y = colstack([missingvalues, nullarray])
        Y = Y.astype(np.dtype([(o,
                  X.dtype[o].descr[0][1]) for o in Y.dtype.names]))
        X = rowstack([X, Y])

    X.sort(order = [a,b])

    Bvals = X[b][:Db]
    bnames = [str(bv).replace(' ','') for bv in Bvals]

    assert (len(set(othernames).intersection(bnames)) == 0 and 
            a not in bnames), ('Processed values of column', b, 
                               'musn\'t intersect with other column names.')

    acol = X[a][::Db]

    Cols = [acol]
    names = [a]
    Trivials = []
    NonTrivials = []
    for c in othernames:
        Z = X[c].reshape((Da,Db))

        if all([len(set(Z[:,i])) == 1 for i in range(Z.shape[1])]):
            Trivials.append(c)
        else:
            NonTrivials.append(c)
        Cols += [Z[:,i] for i in range(Z.shape[1])]
        names += [bn + '_' + c for bn in bnames]
    
    if order is not None:
        ordering = [names.index(ord) for ord in order]
        Cols = [Cols[i] for i in ordering]
        names = [names[i] for i in ordering]
    dtype = np.dtype([(n,c.dtype.descr[0][1]) for (n,c) in zip(names,Cols)])
    D = utils.fromarrays(Cols,type=np.ndarray,dtype=dtype)

    coloring = {}
    if Keep != None:
        Trivials = set(Trivials).intersection(Keep)
        for c in Trivials:
            X.sort(order=[c])
            cvals = np.array(utils.uniqify(X[c]))
            [AA,BB] = fast.equalspairs(cvals,X[c])

            for (i,cc) in enumerate(cvals):
                blist = [str(bv).replace(' ', '') for bv in Bvals if bv in 
                         X[b][AA[i]:BB[i]]]
                coloring[str(cc)] = [a] + [bn + '_' + d for bn in blist for d 
                                           in NonTrivials]
                for d in NonTrivials:
                    coloring[str(cc) + '_' + d] = [a] + blist

    for c in NonTrivials:
        coloring[c] = [a] + [bn + '_' + c for bn in bnames]
    for bn in bnames:
        coloring[prefix + bn] = [a] + [bn + '_' + c for c in NonTrivials]

    return [D, coloring]