easy examples first, and later during fine-tuning, the error can back-propagate through the graph and get properly regularized. The outline of our approach is shown in Figure \[fig:outline\].

Initial Network
---------------

As noted above, common categories used in vision nowadays are well-studied and search engines give relatively clean results. Therefore, instead of using random noun phrases, we obtained three lists of categories from ImageNet Challenge [@russakovsky2014imagenet], SUN database [@xiao2010sun] and NEIL knowledge base [@chen2013neil]. ImageNet syn-sets are transformed to its surface forms by just taking the first explanation, with most of them focusing on object categories. To better assist querying and reducing noise, we remove the suffix (usually correspond to attributes, indoor/outdoor) of the SUN categories. Since NEIL is designed to query search engines, its list is comprehensive and favorable, we collected the list for objects and attributes and removed the duplicate queries with ImageNet. The category names are directly used to query Google for images. Apart from removing unreadable images, no pre-processing is performed. This leave us with $\sim$600 images for each query. All the images are then fed directly into the CNN as training data.

For fair comparison, we use the same architecture (besides the output layer) as 