Title: Representational Task Bias in Zero-shot Recognition at Scale. Abstract: Research from the last year has demonstrated that vision-language pre-training at scale from incidental supervision on the Internet can result in representations with clear advantages over traditional supervised training for many computer vision tasks. We conduct an in-depth exploration of the CLIP model, and find that the interface that language creates to these learned representations -- by the same token as enabling zero-shot application for many tasks -- leads the model to solve tasks that may not have been intended by the user in realistic scenarios. We call the inherent uncertainty of which task a user intends to solve in zero-shot recognition \textit{task ambiguity}. To evaluate task ambiguity, we construct a dataset of images where each image has labels for multiple semantic recognition tasks. We demonstrate that the representation produced for a given image tends to be strongly biased towards a particular task over others; in other words, they exhibit \textit{task bias}. Moreover, which task a particular image will be biased towards is unpredictable, with little consistency across images. Our results show that we can learn visual prompts to serve as effective conditioning mechanisms for which task is desired, and can even improve performance for the task when used outside the context of evaluating task ambiguity. 