We present a method for automatically learning a set of discriminatory facial components for face recognition. The algorithm performs an iterative growing of components starting with small initial components located around pre-selected points in the face. The direction of growing is determined by the gradient of the cross-validation error of the component classifiers. In experiments we analyze how the shape of the components and their discriminatory power changes across different individuals and views.