In daily life, we frequently encounter actions performed by other people. Here we aimed to examine the key categories and features underlying the organization of a wide range of actions in three behavioral experiments (N= 378 participants). In Experiment1, we used a multi-arrangement task of 100 different actions. Inverse multidimensional scaling and hierarchical clustering revealed 11 action categories, includingLocomotion,Communication, andAggressive actions. In Experiment2, we used a feature-listing paradigm to obtain a wide range of action features that were subsequently reduced to 59 key features and used in a rating study (Experiment3). A direct comparison of the feature ratings obtained in Experiment3between actions belonging to the categories identified in Experiment1revealed a number of features that appear to be critical for the distinction between these categories, e.g., the featuresHarmandNoisefor the categoryAggressive actions, and the featuresTargeting a personandContact with othersfor the categoryInteraction. Finally, we found that a part of the category-based organization is explained by a combination of weighted features, whereas a significant proportion of variability remained unexplained, suggesting that there are additional sources of information that contribute to the categorization of observed actions. The characterization of action categories and their associated features serves as an important extension of previous studies examining the cognitive structure of actions. Moreover, our results may serve as the basis for future behavioral, neuroimaging and computational modeling studies.Supplementary InformationThe online version contains supplementary material available at 10.3758/s13428-022-01894-5.