Take the 2-minute tour ×

How do I normalize a multiindex dataframe?

Let's say I have the dataframe:

d = pd.DataFrame([["a",1,3],["a",2,2],["b",4,4],["b",5,8]], 

how do I calculate the normalized values for each "name"?

I know how to normalize a basic dataframe:

d = (d-d.mean(axis=0))/data.std(axis=0, ddof=1)

but I'm not able to apply this on each "name" group of my dataframe

SO the result I want is:

name, value1, value2
a     -0.5     0.5
a      0.5    -0.5
b     -0.5    -1
b      0.5     1

I tried groupby and a multiindex data frame but probably I'm not doing it in the right way

share|improve this question

2 Answers 2

up vote 3 down vote accepted

Normalizing by group is one of the examples in the groupby documentation. But it doesn't do exactly what you seem to want here.

In [2]: d.groupby('name').transform(lambda x: (x-x.mean())/x.std(ddof=1))
     value1    value2
0 -0.707107  0.707107
1  0.707107 -0.707107
2 -0.707107 -0.707107
3  0.707107  0.707107

Your desired result suggests that you actually want to normalize the values in each name group with reference to the elements in value1 and value2. For something like that, you can apply a function to each group individually, and reassemble the result.

In [3]: def normalize(group):                                                      
    mean = group.values.ravel().mean()
    std = group.values.ravel().std(ddof=1)
    return group.applymap(lambda x: (x - mean)/std)

In [4]: pd.concat([normalize(group) for _, group in d.set_index('name').groupby(level=0)])
        value1    value2
a    -1.224745  1.224745
a     0.000000  0.000000
b    -0.660338 -0.660338
b    -0.132068  1.452744
share|improve this answer
Apparently flatten makes a copy of the array. ravel() works the same, but without the copy. –  TomAugspurger Jul 16 '13 at 13:52
I did not know that one. Thanks! –  Dan Allan Jul 16 '13 at 13:52
Thank you both! sorry I cannot upvote yet due my low reputation! I'm starting to have a better vision on how to work with pandas noW! –  user1883737 Jul 16 '13 at 14:17

Are you sure the result you gave is the correct one? I'm assuming you want to normalize value1 and value2 separately. If that's not correct, let me know.

#  Easier with `name` as the index.

In [65]: d = d.set_index('name')

In [66]: d
      value1  value2
a          1       3
a          2       2
b          4       4
b          5       8

In [68]: means = g.mean()

In [69]: stds = g.std()

In [70]: means
      value1  value2
a        1.5     2.5
b        4.5     6.0

In [71]: stds
        value1    value2
a     0.707107  0.707107
b     0.707107  2.828427

In [76]: g.transform(lambda x: (x - means) / stds)
        value1    value2
a    -0.707107  0.707107
a     0.707107 -0.707107
a          NaN       NaN
b          NaN       NaN
b    -0.707107 -0.707107
b     0.707107  0.707107

# Get rid of the nans

In [77]: g.transform(lambda x: (x - means) / stds).dropna()
        value1    value2
a    -0.707107  0.707107
a     0.707107 -0.707107
b    -0.707107 -0.707107
b     0.707107  0.707107
share|improve this answer
yes sorry you are correct, I normalized it in the wrong way! Thanks a ton, for future reference I think you forgot g = d.groupby("name") –  user1883737 Jul 16 '13 at 14:09
Yep, you're right. FWIW, I think Dan's answer is cleaner. –  TomAugspurger Jul 16 '13 at 14:19

Your Answer


