plus millions of parameters can easily lead to over-fitting, we found it is still hard to train a CNN naively with random image-text/tag pairs. For example, most Flickr tags correspond to meta information and specific locations, which usually results in extremely high intra-tag variation. One possibility is to use commercial text-based image search engine to increase diversity in the training data. But if thousands of query strings are used some of them might not correspond to a visualizable concept and some of the query strings might be too fine grained (random names of a person or abstract concepts). These non-visualizable concepts and fine-grained categories incur unexpected noise during the training process[^1]. One can use specifically designed techniques [@chen2013neil; @divvala2014learning] and loss layers [@sukhbaatar2014learning; @reed2014training] to alleviate some of these problems. But these approaches are based on estimating the empirical noise distribution which is non-trivial. Learning the noise distribution is non-trivial since it is heavily dependent on the representation, and weak features (HOG or when the network is being trained from scratch) often lead to incorrect estimates. On the other hand, for many basic categories commonly used in the vision community, the top results returned by Google image search are pretty 