When multiple stimuli are simultaneously presented, they compete against each other to be represented in the capacity-limited visual system. This competition increases as stimulus heterogeneity increases. Given that selective attention is a way to resolve this competition, it has been known that the effect of attention on task performance is magnified as the level of competition increases due to increased stimulus heterogeneity. While previous studies showed that stimulus heterogeneity in a task-irrelevant dimension affects task performance, it remains unknown how this kind of stimulus heterogeneity interacts with visual attention and stimulus-driven competition. Here, we found that the process of searching for a target stimulus among non-targets became inefficient as stimulus heterogeneity in a task-irrelevant dimension increased. The results also showed that the magnitude of the attentional cuing effect could be affected by increased heterogeneity. However, this modulation was dependent on the type of varied feature or task demand. We suggest that increased stimulus heterogeneity in a task-irrelevant dimension would increase stimulus-driven competition, which impoverishes the quality of stimulus representations.