Peripherally presented objects are often more difficult to identify when located in cluttered visual environments than when presented in isolation, a phenomenon known as visual crowding. Crowding tends to be stronger when target and nearby flanking elements are composed of similar sets of features. This study investigates the extent to which target–flanker orientation and/or color similarity determines luminance and orientation performance across different tasks under identical stimulus conditions. Targets were near-vertical Gabor patches defined by modulating only the green component of the RGB display. Subjects performed both target luminance and orientation discrimination tasks in separate blocks while both flanker hue (green or red flankers) and orientation (vertical or horizontal flankers) were manipulated as a function of target–flanker separation. We find strong evidence for a double dissociation between task and the specific set of features by which target–flanker similarity is defined. Whereas luminance judgments were highly contingent upon target–flanker hue similarity, orientation judgments showed the inverse pattern, largely contingent upon flanker orientation. The magnitude of this double dissociation decreased with target–flanker separation, at a rate predicted by Bouma's law. This specific pattern of performance provides strong support for the idea that crowding operates independently for the most part within orientation and color domains. That luminance judgments are constrained by target–flanker flanker hue similarity and, to a far lesser extent, by target–flanker orientation similarity suggests that the neural mechanisms responsible for mediating perceived luminance are principally linked to those mediating stimulus hue independent of those mediating stimulus orientation.