400 million images search
https://laion.ai/blog/laion-400-open-dataset/
LAION-400-MILLION OPEN DATASET | LAION
2021-08-08
None
<div><div><p>We present LAION-400M: 400M English (image, text) pairs - see also our <a href="https://arxiv.org/abs/2111.02114">Data Centric AI NeurIPS Workshop 2021 paper</a></p>
<h2>Concept and Content</h2>
<p>The LAION-400M dataset is entirely openly, freely accessible.</p>
<p><strong>WARNING</strong>: be aware that this large-scale dataset is non-curated. It was built for research purposes to enable testing model training on larger scale for broad researcher and other interested communities, and is <strong>not</strong> meant for any real-world production or application.</p>
<p>We have filtered all images and texts in the LAION-400M dataset with OpenAI‘s <a href="https://openai.com/blog/clip/">CLIP</a> by calculating the cosine similarity between the text and image embeddings and dropping those with a similarity below 0.3. The threshold of 0.3 had been determined through human evaluations and seemed to be a good heuristic for estimating semantic image-text-content matching.</p>
<p>The image-text-pairs have been extracted from the <a href="https://commoncrawl.org/">Common Crawl</a> web data dump and are from random web pages crawled between 2014 and 2021.</p>
<h3>Download Information</h3>
<h6>UPDATE 16 dec 2021</h6>
<p>While <strong>the eye</strong> experiences technical difficulties, we provide an alternate download server for this dataset at this link: <a href="http://deploy.laion.ai/8f83b608504d46bb81708ec86e912220/">laion400m at deploy.laion.ai</a></p>
<h6>Original information</h6>
<p>You can find</p>
<ul>
<li>The CLIP image embeddings (NumPy files)</li>
<li>The parquet files</li>
<li>KNN index of image embeddings</li>
</ul>
<p>To download from <strong>the eye</strong>, run this command</p>
<p><code>aria2c "https://the-eye.eu/public/AI/cah/laion400m-met-release.torrent"</code></p>
<p>You may want to use the <code>–show-files</code> and <code>–select-file</code> options to download only some files.</p>
<p>You can also find the files in <a href="https://the-eye.eu/public/AI/cah/laion400m-met-release/">laion400m-met-release</a></p>
<p>Some more significant knn indices are present in <a href="https://the-eye.eu/public/AI/cah/laion400m-indexes/">laion400m-indexes</a>. We advise using the 128GB ones.</p>
<p>The parquet files in Kaggle: <a href="https://www.kaggle.com/romainbeaumont/laion400m">laion400m on Kaggle</a></p>
<p>After downloading the metadata as indicated above, you can run <a href="https://github.com/rom1504/laion-prepro/blob/main/laion400m/download_images/download_images.sh">this command</a> to download the images and generate the webdataset files (command using <a href="https://github.com/rom1504/img2dataset">img2dataset</a> )</p>
<h3>LAION-400M Dataset Statistics</h3>
<p>The LAION-400M and future even bigger ones are, in fact, datasets of datasets. For instance, we can filter it out by image sizes into smaller datasets like this:</p>
<pre><code>Number of unique samples 413M
Number with height or width &gt;= 1024 26M
Number with height and width &gt;= 1024 9.6M
Number with height or width &gt;= 512 112M
Number with height and width &gt;= 512 67M
Number with height or width &gt;= 256 268M
Number with height and width &gt;= 256 211M
</code></pre>
<p>By using the KNN index, we can extract specialized datasets by domains of interest. They are (or will be) sufficient in size to train technical domain models.</p>
<p>Also, use <a href="https://rom1504.github.io/clip-retrieval/">https://rom1504.github.io/clip-retrieval/</a> for simple visualisation of the dataset. There you can search among the dataset using CLIP and a knn index.</p>
<h3>Disclaimer &amp; Content Warning</h3>
<p>Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, <strong>non-curated</strong> set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with <strong>caution</strong>. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.</p>
<p>There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.</p>
<p>Using KNN clustering should make it easy to further deduplicate by image content.</p>
<h3>LAION-400M Open Dataset structure</h3>
<p>We produced the dataset in several formats to address the various use cases:</p>
<ul>
<li>a 50GB url+caption metadata dataset in parquet files. We can use the metadata to compute statistics and redownload part of the dataset</li>
<li>a 10TB webdataset with 256×256 images, captions and metadata. It is a full version of the dataset that can be used directly for training (this one is for internal use, you need to redownload images yourself due to licensing issues)</li>
<li>a 1TB set of the 400M text and image clip embeddings, useful to rebuild new knn indices</li>
<li>pairs of 16G, 32G, 64G and 128G knn indices (running in the web demo)</li>
</ul>
<h4>URL and caption metadata dataset.</h4>
<p>We provide 32 parquet files of size around 1GB (total 50GB) with the image URLs, the associated texts and additional metadata in the following format:</p>
<blockquote>
<p>SAMPLE_ID | URL | TEXT | LICENSE | NSFW | similarity | WIDTH | HEIGHT</p>
</blockquote>
<p>where</p>
<ul>
<li><strong>SAMPLE_ID</strong>: A unique identifier</li>
<li><strong>LICENSE</strong>: Where we found a Creative Commons License in the image data, we named it here like, e.g. “creativecommons.org/licenses/by-nc-sa/3.0/” – otherwise you’ll find it here a “?”</li>
<li><strong>NSFW</strong>: we used CLIP to estimate if the image has NSFW content. The estimation has been pretty conservative, reducing false negatives at the cost of more false positives. Possible values are “UNLIKELY”, “UNSURE” and “NSFW”.</li>
<li><strong>similarity</strong>: Value of the cosine similarity between the text and image embedding</li>
<li>WIDTH and HEIGHT: image size as the image was embedded. We downsized originals that were larger than 4K to 4K.</li>
</ul>
<p>This metadata dataset purpose is to download the images for the whole dataset or a subset of it by supplying it to the very efficient <a href="https://github.com/rom1504/img2dataset">img2dataset</a> tool.</p>
<h4>10 TB webdataset with images and captions</h4>
<p>By running the img2dataset tool, we can download a 10TB webdataset. It will resize all images at 256×256 resolution, will append the corresponding caption and will generate a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and related parquet files containing the same metadata</p>
<ul>
<li>00000.tar of size 270MB containing at most 10k samples
<ul>
<li>0.jpg</li>
<li>0.txt containing the caption</li>
<li>0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW</li>
</ul>
</li>
<li>00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files</li>
</ul>
<p>The 400M dataset will therefore have 41455 tar and 41455 parquet files. This dataset purpose is to train multimodal models like CLIP or DALL-E.</p>
<h4>1TB of clip embeddings</h4>
<p>The clip embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. Each NPY file is 1GB, and each parquet file is 150MB. There are a total of 400 such files. The embeddings purpose is to compute statistics on the dataset, for example, using clustering or knn indices.</p>
<h4>Two small 6GB knn indices</h4>
<p>We provide two 6GB knn indices built using the <a href="https://github.com/criteo/autofaiss">autofaiss</a>. We can use them to compute a subset of the dataset and, more generally, to search among it efficiently. See the search <a href="https://rom1504.github.io/clip-retrieval/">web demo</a> of it. We can use the CLIP filter tool along with this index to produce subsets using search terms efficiently. We also provide two 16GB knn indices of higher quality.</p>
<h3>What can we do with the LAION-400M dataset?</h3>
<p>Vision and language modelling has been taking off in 2021. Here are some pointers about what this kind of image + text datasets unlocks and why it seems interesting:</p>
<ul>
<li>Six months ago, OpenAI released two blog posts and papers, <a href="https://openai.com/blog/clip/">CLIP</a> and <a href="https://openai.com/blog/dall-e/">DALL-E</a>. Both models rely on a large amount of (text, image) pairs. They used an unreleased 400M pairs dataset.
<ul>
<li>CLIP is a model that computes how related are a text and an image. It makes it possible to build large text to image search, and it makes it possible to create that kind of crazy text to image art <a href="https://ml.berkeley.edu/blog/blog/clip-art/">clip-art</a>. They released a small and medium version of the model but no training code.</li>
<li>DALL-E is a model that directly generates images from texts. As can be seen from the blog post, it achieves awe-inspiring results that could directly impact the world for anything that needs drawing and illustrations. OpenAI did not release any model, even through an API</li>
</ul>
</li>
</ul>
<p>Since then, various researchers have organised several efforts to replicate DALL-E. People gathered initially around this excellent DALLE replication repository <a href="https://github.com/lucidrains/DALLE-pytorch">DALLE-PyTorch</a> with some fantastic results visible in the readme. More recently, as part of huggingface events, new developments have been achieved (see <a href="https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini--Vmlldzo4NjIxODA">DALLE-mini report</a> ), and an online demo is now available at <a href="https://huggingface.co/spaces/flax-community/dalle-mini">DALLE-mini demo.</a></p>
<p>The replication effort is still far from achieving the same performance as the original DALLE, and it seems possible to go even further. Some people also want to make a better CLIP to produce even better-generated art.</p>
<p>A large part of the results that we can achieve with such models is thanks to a large amount of data. Before LAION-400M, the largest open dataset for (image, text) pairs are in the order of 10M (see <a href="https://github.com/robvanvolt/DALLE-datasets">DALLE-datasets</a> ), which is enough to train exciting models but not enough to reach the best performance. Having a public dataset with hundreds of millions of pairs will help build these image+text models.</p>
<h3>Analysis of the LAION-400M data</h3>
<p>We annotated 3456 samples of the dataset and got the following results:</p>
<ul>
<li>Correct positive NSFW: 4</li>
<li>Correct negative NSFW: 3371</li>
<li>False-positive NSFW: 73</li>
<li>False-negative NSFW: 8</li>
<li>Bad captions: 3 (0.09 %)</li>
</ul>
<p>The matching is excellent, thanks to CLIP. We could improve the NSFW automatic tagging in the future; however, the NSFW total rate is low enough (less than 1%) to make this not an issue.</p>
<h2>Technical Details</h2>
<p>The dataset acquisition has into two significant parts:</p>
<ol>
<li>a distributed processing of the vast (many PBs) Common Crawl datasets, which produces a collection of matching URL and caption</li>
<li>a single node much lighter post-processing of the data that anyone can run in a few days and which produces the final dataset</li>
</ol>
<h3>1. Distributed processing of Common Crawl</h3>
<p>We acquire the raw web data for the creation of our dataset from Common Crawl. Common Crawl is a non-profit organisation dedicated to providing a copy of the internet to internet researchers, companies, and individuals at no cost for research and analysis. They regularly release dumps of HTML-like data parsed from billions of public websites found <a href="https://commoncrawl.org/the-data/get-started/">on the Common Crawl website</a>. To create image-text pairs, we parse through the data from Common Crawl and parse out all HTML IMG tags containing an <a href="https://en.wikipedia.org/wiki/Alt_attribute">alt text attribute</a>. Common Crawl provides its data in several formats. For our purpose, we chose to use the data in the WAT format. The WAT files contain only the metadata of the crawled sites, which includes all links and IMG tags contained in the website. Parsing only this metadata is much faster than parsing the whole HTML text (provided in the WARC format).</p>
<h4>Downloading original images</h4>
<p>We download the raw images from the URLs we parsed from Common Crawl with asynchronous requests using the libraries <a href="https://github.com/python-trio/trio">Trio</a> and <a href="https://github.com/theelous3/asks">Asks</a>. They allow us to go multithreading for a single CPU. Usually, a home internet link will be exhausted by a single or two CPUs. A data centre node can scale up benefits from guaranteed internet speed with a multiprocessing pool much faster than a single CPU node. At this time, we were able to use 50 cores with a full, secured 1Gbps connection to the public internet. This bandwidth must be available to the downloading node, not shared among many nodes or apps. We have optimised the script for speed while mitigating various errors we encountered. Usually, to satisfy a high-end demanding node such as above, we must take additional steps to provide DNS caching capabilities. We found that the knot-resolver ran with two processes and configured with caching option can solve this problem.</p>
<h4>Filtering out unsuitable image-text pairs</h4>
<p>After downloading the WAT files from Common Crawl, we filter the samples in the following steps:</p>
<ol>
<li>We dropped all samples with less than five character alt text length</li>
<li>We dropped all samples with less than 5 KB image size</li>
<li>We use continuously updated bloom filters to drop samples that are already in our dataset. The bloom filters deduplicate by concatenating the URL and the alt text.</li>
<li>We use continuously updated bloom filters to drop samples from URLs that had timed out previously and therefore seem unreachable (or at least not reachable in an efficient way)</li>
<li>We use OpenAI’s CLIP model (the ‘<em>ViT-B-32</em>‘ version) to compute the image and alt text embeddings. Then we calculate the cosine similarity of both embedding vectors and drop all samples with a similarity below 0.3. We chose this threshold after trying different values and using human evaluations of how well the texts fit the images. Lower values like 0.28 or 0.29 also seemed okay in many cases, but after further inspections, we decided to choose the conservative value of 0.3.</li>
<li>We use the CLIP embeddings of the images to estimate if their contents contain NSFW content. We do this by calculating CLIP embeddings for a list of image categories like, e.g. “selfie”, “illustration”, or “landscape”, which also contains categories that indicate NSFW content like “porn” and “sex”.</li>
<li>Then we compute the cosine similarities between the embedding image we are currently filtering and each of these category keywords. If the category with the highest similarity and the keyword with the second-highest similarity belong both to NSFW keywords, we tag the sample as “NSFW”. If only one of them belongs to an NSFW keyword, we categorise the sample as “UNSURE”. If both keywords with the highest similarities are not NSFW, we tag the sample as “UNLIKELY”.</li>
<li>In the next step, we look at all samples with either the “NSFW” or “UNSURE” tag and drop those with any keywords in their text related to kids, teens, or other semantically related content.</li>
<li>In step 8, we repeat the procedure of computing the cosine similarities from step 6 with the difference that we now use category texts that indicate contents semantically related to kids and teens on a CLIP embedding level. If either the highest similarity or the second-highest similarity between a sample’s image embedding and a text of the precomputed categories belongs to a text that indicates content related to under-aged persons, we drop this sample.</li>
<li>Finally, we repeat the procedure from step 8 with texts semantically related to animal categories like e.g. “animal”, “bird”, etc.</li>
</ol>
<p>We perform these rigorous filtering steps for NSFW with potentially illegal content because we cannot guarantee that the contents of Common Crawl are free of such. We feel obligated to try our best to filter out such content. Inspections of samples filtered out by steps 7 to 9 have shown that our filtering procedure is very conservative and produces many false positives (samples it drops, which are not problematic). This process is okay because the number of potential samples waiting for us to crawl is vast.</p>
<h4>System Architecture</h4>
<p>To orchestrate the interactions of the many crawling scripts (called <em>workers</em>) in our project, we use a server that keeps track of processed WAT files and of which worker gets which unprocessed WAT. We call this orchestrating server a <em>tracker</em>. Its functions are offering jobs to both download workers and inference workers, confirming cleanup requests from the DL staging server, maintaining ACLs for the Bloom server, and some more. We also employ several staging servers as buffers for jobs on their way to the storage location. The staging servers continuously update filters in the central bloom server where we use RedisBloom for high-performance reasons.</p>
<h4>Workflow</h4>
<p>During the evolution of our crawling project, we applied two different workflows:</p>
<h5>Workflow 1 (<em>“Hybrid”</em> – workers)</h5>
<p>This worker performs all computation steps during one job and then submits the result to the staging server. It then queues the results for release to the storage area.</p>
<h5>Workflow 2 (<em>“CPU – GPU – 2 stages”</em> – workflow)</h5>
<p>We soon discovered that the best way to utilise resources is to split the workload into CPU + networking tasks (downloading steps) and GPU tasks (CLIP inference steps). Hence, the 2 stage approach uses “CPU workers” to download images, create image-text pairs, and save the intermediate result to a staging server. Then “GPU workers” pick up jobs, concatenate a number of them to group around 20000 pairs per final result file. The 2 stage workflow proved to be most efficient, with speeds up to 25 million pairs added to the dataset per day when using 100 CPU workers with one core and one GPU worker employing an NVidia RTX 3090 graphic card utilising all 16 lanes of PCIe bus. The GPU node also needs about CPU 24 threads to keep up with the GPU processing capacity.</p>
<h4>Removing abuse alerts</h4>
<p>During downloading, we encountered abuse alerts from manual and automated tools that protect websites. After some learning curve, we reduced most of the issues by employing these mitigation techniques:</p>
<ul>
<li>By far, the most efficient one was to use centralised bloom filters that eliminate requests going to the duplicate URLs over and over. Of course, the efficiency of these filters dramatically depends on how fast they are updated and used by the workers. By definition, having multiple downloading workers performing jobs in parallel makes them prone to overlap requests to the same URL even if the bloom filters are up to date at the beginning of the job.</li>
<li>Therefore the second technique significantly reduced the problem of parallel workers via randomising the jobs at the tracker server level. While executing jobs in sequence (with the oldest WAT files from 2013), we discovered that adjacent jobs were overlapping considerably. When we randomised jobs, we saw a dramatic decrease in such overlapping.</li>
</ul>
<h4>Who ran this?</h4>
<p>We want to thank :</p>
<ul>
<li>the <a href="https://laion.ai/#team">LAION folks</a>, via so many worker nodes everywhere in the cloud</li>
<li><a href="https://www.reddit.com/r/DataHoarder/comments/oyta8q/crawlinghome_help_build_the_worlds_largest/">the data hoarders</a> Reddit community</li>
<li><a href="https://the-eye.eu/">the-eye</a> community</li>
<li>as well as all our friends and relatives that did not know what they were helping with</li>
</ul>
<p>for running the workers to produce this vast dataset in a few months.</p>
<h3>2. Post-processing of the dataset</h3>
<p>Once the distributed pipeline has run, resulting in a sizeable caption+url dataset, it’s time to package it in the best way. The objective of this second pipeline is to produce a version of the dataset that is easy to use for multimodal training. For this, we built tools that anyone can run out of a collection of caption+url. The exact command line to run is available in <a href="https://github.com/rom1504/cah-prepro">cah-prepro</a> (which uses mainly <a href="https://github.com/rom1504/img2dataset">img2dataset</a> and <a href="https://github.com/rom1504/clip-retrieval">clip-retrieval</a> )</p>
<h4>Pyspark preprocessing of the CSV files</h4>
<p>After a fast run of a script to <a href="https://github.com/rom1504/cah-prepro/tree/main/download_csv">download the CSV files,</a> the first step of this post-processing pipeline is to do deduplication by url+caption. The first pipeline does some partial deduplication using a bloom filter, but it is approximate, and some duplicates remain. Doing that pyspark post-processing also makes it possible to reduce the number of metadata files from hundred of thousands to 32 parquet files of size 1.7GB. See this <a href="https://github.com/rom1504/cah-prepro/blob/main/deduplicate/cah_stats_spark.py">deduplication script there</a>. Pyspark would be an excellent way to do any further filtering, and we <a href="https://github.com/rom1504/cah-prepro/blob/main/deduplicate/compute_more_stats.py">provide</a> an example to compute some statistics. The resulting output is 32 parquet files containing columns such as URL, text, NSFW described at the beginning of the post.</p>
<h4>Img2dataset</h4>
<p>Once this set of 50GB parquet files has is ready, we can use the <a href="https://github.com/rom1504/img2dataset">img2dataset</a> tool to download, resize and store the images and captions as <a href="https://github.com/webdataset/webdataset">webdataset</a>. This tool can download 100M images in 20h in a single node (1Gbps 32GB of ram 16 i7 cores), so anyone can run this for the whole dataset or a smaller subset. The format this tool outputs is a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and corresponding parquet files containing the same metadata</p>
<ul>
<li>00000.tar of size 270MB containing at most 10k samples
<ul>
<li>0.jpg</li>
<li>0.txt containing the caption</li>
<li>0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW</li>
</ul>
</li>
<li>00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files</li>
</ul>
<p>The size of the tars of 270MB is when using the options of img2dataset indicated there <a href="https://github.com/rom1504/cah-prepro/blob/main/download_images/download_images.sh">download_images.sh</a> (resizing all images to 256×256 with padding for maximum file uniformity and avoid losing information). If using different options, you may have larger or smaller tar files.</p>
<h4>Clip retrieval and autofaiss</h4>
<p>Finally, the tar dataset aims to compute and package clip embeddings and compute a KNN index over the clip embeddings. The <a href="https://github.com/rom1504/clip-retrieval/">clip-retrieval</a> tool makes it fast to compute 100M embeddings per 20h with a single 3080 GPU, so it’s possible to rerun this part on the whole dataset or a subset at a low cost. The embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. NPY files are 1GB in size, and parquet files are 150MB. There are a total of 400 such files. These embeddings help build text and an image knn index using the <a href="https://github.com/criteo/autofaiss">autofaiss</a> tool, making it possible to produce a quantised index of an arbitrary file. The chosen index type is 6GB, so it’s cheap for anyone to load and run fast (10ms) queries over the whole dataset. We also generated another kind of index of size 16GB. Thanks to memory mapping, it’s also possible to load it at no ram usage. A simple <a href="https://rom1504.github.io/clip-retrieval/">web demo</a> shows the results.</p>
<p></p>
<h3>License</h3>
<p>We distribute the metadata dataset (the parquet files) under the most open <a href="https://creativecommons.org/licenses/by/4.0/">Creative Common CC-BY 4.0</a> license, which poses no particular restriction. The images are under their copyright.</p>
<h2>Contributing</h2>
<p>You can contribute to the project to help us release the following dataset sizes at 1 billion pairs, 2 billion pairs and so on.</p>
<p>Choose one or more methods that suit you or your company:</p>
<ol>
<li>donate either <a href="https://laion.ai/laion-400-open-dataset/#">cash</a> or <a href="https://laion.ai/how-to-donate-computing-time/">computing time</a>. We also launched a <a href="https://gogetfunding.com/help-us-build-the-worlds-largest-open-billion-scale-image-text-dataset-perfect-for-training-dall-e-clip-other-multimodal-models/">Go Get Funding campaign</a>.</li>
<li>participate in the development effort</li>
<li>spread the word. At best, use the dataset, get nice results and mention it in your papers</li>
</ol>
<p>Useful links:</p>
<ul>
<li>Dataset progress <a href="http://crawling.at/">Crawling@Home Dashboard</a> and <a href="http://crawling.at/leaderboard">leaderboard</a></li>
<li>Reddit <a href="https://www.reddit.com/r/DataHoarder/comments/oyta8q/crawlinghome_help_build_the_worlds_largest/?utm_source=share&amp;utm_medium=web2x&amp;context=3">post</a></li>
<li>DALLE-PyTorch <a href="https://discord.gg/mVcgxMPD7e">Discord server</a></li>
<li>DALLE-PyTorch <a href="https://github.com/lucidrains/DALLE-pytorch">GitHub Repository</a></li>
</ul>
<h3>Sponsors</h3>
<p>We made it so far due to the generosity of these donors:</p>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="http://doodlebot.ai/">doodlebot.ai</a></td>
<td><a href="https://gentec.ro/">Gentec Data</a></td>
<td><a href="http://the-eye.eu/">the-eye.eu</a></td>
</tr>
</tbody>
</table>
</div>||||I|||| LAION

 ProjectsTeamBlogAboutFAQPrivacy PolicyDataset RequestsImpressum
 Open Menu

 Projects

 Team

 Blog

 About

 FAQ

 Privacy Policy

 Dataset Requests

 Impressum

 Close Menu

 LAION-400-MILLION OPEN DATASET

 by: Christoph Schuhmann, 8 Aug, 2021

 We present LAION-400M: 400M English (image, text) pairs - see also our Data Centric AI NeurIPS Workshop 2021 paper

 Concept and Content

 The LAION-400M dataset is entirely openly, freely accessible.

 WARNING: be aware that this large-scale dataset is non-curated. It was built for research purposes to enable testing model training on larger scale for broad researcher and other interested communities, and is not meant for any real-world production or application.

 We have filtered all images and texts in the LAION-400M dataset with OpenAI‘s CLIP by calculating the cosine similarity between the text and image embeddings and dropping those with a similarity below 0.3. The threshold of 0.3 had been determined through human evaluations and seemed to be a good heuristic for estimating semantic image-text-content matching.

 The image-text-pairs have been extracted from the Common Crawl web data dump and are from random web pages crawled between 2014 and 2021.

 Download Information

 UPDATE 16 dec 2021

 While the eye experiences technical difficulties, we provide an alternate download server for this dataset at this link: laion400m at deploy.laion.ai

 Original information

 You can find

 * The CLIP image embeddings (NumPy files)
 * The parquet files
 * KNN index of image embeddings

 To download from the eye, run this command

 aria2c "https://the-eye.eu/public/AI/cah/laion400m-met-release.torrent"

 You may want to use the –show-files and –select-file options to download only some files.

 You can also find the files in laion400m-met-release

 Some more significant knn indices are present in laion400m-indexes. We advise using the 128GB ones.

 The parquet files in Kaggle: laion400m on Kaggle

 After downloading the metadata as indicated above, you can run this command to download the images and generate the webdataset files (command using img2dataset )

 LAION-400M Dataset Statistics

 The LAION-400M and future even bigger ones are, in fact, datasets of datasets. For instance, we can filter it out by image sizes into smaller datasets like this:

 Number of unique samples 413M
 Number with height or width >= 1024 26M
 Number with height and width >= 1024 9.6M
 Number with height or width >= 512 112M
 Number with height and width >= 512 67M
 Number with height or width >= 256 268M
 Number with height and width >= 256 211M
 

 By using the KNN index, we can extract specialized datasets by domains of interest. They are (or will be) sufficient in size to train technical domain models.

 Also, use https://rom1504.github.io/clip-retrieval/ for simple visualisation of the dataset. There you can search among the dataset using CLIP and a knn index.

 Disclaimer & Content Warning

 Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.

 There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.

 Using KNN clustering should make it easy to further deduplicate by image content.

 LAION-400M Open Dataset structure

 We produced the dataset in several formats to address the various use cases:

 * a 50GB url+caption metadata dataset in parquet files. We can use the metadata to compute statistics and redownload part of the dataset
 * a 10TB webdataset with 256×256 images, captions and metadata. It is a full version of the dataset that can be used directly for training (this one is for internal use, you need to redownload images yourself due to licensing issues)
 * a 1TB set of the 400M text and image clip embeddings, useful to rebuild new knn indices
 * pairs of 16G, 32G, 64G and 128G knn indices (running in the web demo)

 URL and caption metadata dataset.

 We provide 32 parquet files of size around 1GB (total 50GB) with the image URLs, the associated texts and additional metadata in the following format:

 SAMPLE_ID | URL | TEXT | LICENSE | NSFW | similarity | WIDTH | HEIGHT

 where

 * SAMPLE_ID: A unique identifier
 * LICENSE: Where we found a Creative Commons License in the image data, we named it here like, e.g. “creativecommons.org/licenses/by-nc-sa/3.0/” – otherwise you’ll find it here a “?”
 * NSFW: we used CLIP to estimate if the image has NSFW content. The estimation has been pretty conservative, reducing false negatives at the cost of more false positives. Possible values are “UNLIKELY”, “UNSURE” and “NSFW”.
 * similarity: Value of the cosine similarity between the text and image embedding
 * WIDTH and HEIGHT: image size as the image was embedded. We downsized originals that were larger than 4K to 4K.

 This metadata dataset purpose is to download the images for the whole dataset or a subset of it by supplying it to the very efficient img2dataset tool.

 10 TB webdataset with images and captions

 By running the img2dataset tool, we can download a 10TB webdataset. It will resize all images at 256×256 resolution, will append the corresponding caption and will generate a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and related parquet files containing the same metadata

 * 00000.tar of size 270MB containing at most 10k samples
 + 0.jpg
 + 0.txt containing the caption
 + 0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW
 * 00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files

 The 400M dataset will therefore have 41455 tar and 41455 parquet files. This dataset purpose is to train multimodal models like CLIP or DALL-E.

 1TB of clip embeddings

 The clip embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. Each NPY file is 1GB, and each parquet file is 150MB. There are a total of 400 such files. The embeddings purpose is to compute statistics on the dataset, for example, using clustering or knn indices.

 Two small 6GB knn indices

 We provide two 6GB knn indices built using the autofaiss. We can use them to compute a subset of the dataset and, more generally, to search among it efficiently. See the search web demo of it. We can use the CLIP filter tool along with this index to produce subsets using search terms efficiently. We also provide two 16GB knn indices of higher quality.

 What can we do with the LAION-400M dataset?

 Vision and language modelling has been taking off in 2021. Here are some pointers about what this kind of image + text datasets unlocks and why it seems interesting:

 * Six months ago, OpenAI released two blog posts and papers, CLIP and DALL-E. Both models rely on a large amount of (text, image) pairs. They used an unreleased 400M pairs dataset.
 + CLIP is a model that computes how related are a text and an image. It makes it possible to build large text to image search, and it makes it possible to create that kind of crazy text to image art clip-art. They released a small and medium version of the model but no training code.
 + DALL-E is a model that directly generates images from texts. As can be seen from the blog post, it achieves awe-inspiring results that could directly impact the world for anything that needs drawing and illustrations. OpenAI did not release any model, even through an API

 Since then, various researchers have organised several efforts to replicate DALL-E. People gathered initially around this excellent DALLE replication repository DALLE-PyTorch with some fantastic results visible in the readme. More recently, as part of huggingface events, new developments have been achieved (see DALLE-mini report ), and an online demo is now available at DALLE-mini demo.

 The replication effort is still far from achieving the same performance as the original DALLE, and it seems possible to go even further. Some people also want to make a better CLIP to produce even better-generated art.

 A large part of the results that we can achieve with such models is thanks to a large amount of data. Before LAION-400M, the largest open dataset for (image, text) pairs are in the order of 10M (see DALLE-datasets ), which is enough to train exciting models but not enough to reach the best performance. Having a public dataset with hundreds of millions of pairs will help build these image+text models.

 Analysis of the LAION-400M data

 We annotated 3456 samples of the dataset and got the following results:

 * Correct positive NSFW: 4
 * Correct negative NSFW: 3371
 * False-positive NSFW: 73
 * False-negative NSFW: 8
 * Bad captions: 3 (0.09 %)

 The matching is excellent, thanks to CLIP. We could improve the NSFW automatic tagging in the future; however, the NSFW total rate is low enough (less than 1%) to make this not an issue.

 Technical Details

 The dataset acquisition has into two significant parts:

 1. a distributed processing of the vast (many PBs) Common Crawl datasets, which produces a collection of matching URL and caption
 2. a single node much lighter post-processing of the data that anyone can run in a few days and which produces the final dataset

 1. Distributed processing of Common Crawl

 We acquire the raw web data for the creation of our dataset from Common Crawl. Common Crawl is a non-profit organisation dedicated to providing a copy of the internet to internet researchers, companies, and individuals at no cost for research and analysis. They regularly release dumps of HTML-like data parsed from billions of public websites found on the Common Crawl website. To create image-text pairs, we parse through the data from Common Crawl and parse out all HTML IMG tags containing an alt text attribute. Common Crawl provides its data in several formats. For our purpose, we chose to use the data in the WAT format. The WAT files contain only the metadata of the crawled sites, which includes all links and IMG tags contained in the website. Parsing only this metadata is much faster than parsing the whole HTML text (provided in the WARC format).

 Downloading original images

 We download the raw images from the URLs we parsed from Common Crawl with asynchronous requests using the libraries Trio and Asks. They allow us to go multithreading for a single CPU. Usually, a home internet link will be exhausted by a single or two CPUs. A data centre node can scale up benefits from guaranteed internet speed with a multiprocessing pool much faster than a single CPU node. At this time, we were able to use 50 cores with a full, secured 1Gbps connection to the public internet. This bandwidth must be available to the downloading node, not shared among many nodes or apps. We have optimised the script for speed while mitigating various errors we encountered. Usually, to satisfy a high-end demanding node such as above, we must take additional steps to provide DNS caching capabilities. We found that the knot-resolver ran with two processes and configured with caching option can solve this problem.

 Filtering out unsuitable image-text pairs

 After downloading the WAT files from Common Crawl, we filter the samples in the following steps:

 1. We dropped all samples with less than five character alt text length
 2. We dropped all samples with less than 5 KB image size
 3. We use continuously updated bloom filters to drop samples that are already in our dataset. The bloom filters deduplicate by concatenating the URL and the alt text.
 4. We use continuously updated bloom filters to drop samples from URLs that had timed out previously and therefore seem unreachable (or at least not reachable in an efficient way)
 5. We use OpenAI’s CLIP model (the ‘ViT-B-32‘ version) to compute the image and alt text embeddings. Then we calculate the cosine similarity of both embedding vectors and drop all samples with a similarity below 0.3. We chose this threshold after trying different values and using human evaluations of how well the texts fit the images. Lower values like 0.28 or 0.29 also seemed okay in many cases, but after further inspections, we decided to choose the conservative value of 0.3.
 6. We use the CLIP embeddings of the images to estimate if their contents contain NSFW content. We do this by calculating CLIP embeddings for a list of image categories like, e.g. “selfie”, “illustration”, or “landscape”, which also contains categories that indicate NSFW content like “porn” and “sex”.
 7. Then we compute the cosine similarities between the embedding image we are currently filtering and each of these category keywords. If the category with the highest similarity and the keyword with the second-highest similarity belong both to NSFW keywords, we tag the sample as “NSFW”. If only one of them belongs to an NSFW keyword, we categorise the sample as “UNSURE”. If both keywords with the highest similarities are not NSFW, we tag the sample as “UNLIKELY”.
 8. In the next step, we look at all samples with either the “NSFW” or “UNSURE” tag and drop those with any keywords in their text related to kids, teens, or other semantically related content.
 9. In step 8, we repeat the procedure of computing the cosine similarities from step 6 with the difference that we now use category texts that indicate contents semantically related to kids and teens on a CLIP embedding level. If either the highest similarity or the second-highest similarity between a sample’s image embedding and a text of the precomputed categories belongs to a text that indicates content related to under-aged persons, we drop this sample.
 10. Finally, we repeat the procedure from step 8 with texts semantically related to animal categories like e.g. “animal”, “bird”, etc.

 We perform these rigorous filtering steps for NSFW with potentially illegal content because we cannot guarantee that the contents of Common Crawl are free of such. We feel obligated to try our best to filter out such content. Inspections of samples filtered out by steps 7 to 9 have shown that our filtering procedure is very conservative and produces many false positives (samples it drops, which are not problematic). This process is okay because the number of potential samples waiting for us to crawl is vast.

 System Architecture

 To orchestrate the interactions of the many crawling scripts (called workers) in our project, we use a server that keeps track of processed WAT files and of which worker gets which unprocessed WAT. We call this orchestrating server a tracker. Its functions are offering jobs to both download workers and inference workers, confirming cleanup requests from the DL staging server, maintaining ACLs for the Bloom server, and some more. We also employ several staging servers as buffers for jobs on their way to the storage location. The staging servers continuously update filters in the central bloom server where we use RedisBloom for high-performance reasons.

 Workflow

 During the evolution of our crawling project, we applied two different workflows:

 Workflow 1 (“Hybrid” – workers)

 This worker performs all computation steps during one job and then submits the result to the staging server. It then queues the results for release to the storage area.

 Workflow 2 (“CPU – GPU – 2 stages” – workflow)

 We soon discovered that the best way to utilise resources is to split the workload into CPU + networking tasks (downloading steps) and GPU tasks (CLIP inference steps). Hence, the 2 stage approach uses “CPU workers” to download images, create image-text pairs, and save the intermediate result to a staging server. Then “GPU workers” pick up jobs, concatenate a number of them to group around 20000 pairs per final result file. The 2 stage workflow proved to be most efficient, with speeds up to 25 million pairs added to the dataset per day when using 100 CPU workers with one core and one GPU worker employing an NVidia RTX 3090 graphic card utilising all 16 lanes of PCIe bus. The GPU node also needs about CPU 24 threads to keep up with the GPU processing capacity.

 Removing abuse alerts

 During downloading, we encountered abuse alerts from manual and automated tools that protect websites. After some learning curve, we reduced most of the issues by employing these mitigation techniques:

 * By far, the most efficient one was to use centralised bloom filters that eliminate requests going to the duplicate URLs over and over. Of course, the efficiency of these filters dramatically depends on how fast they are updated and used by the workers. By definition, having multiple downloading workers performing jobs in parallel makes them prone to overlap requests to the same URL even if the bloom filters are up to date at the beginning of the job.
 * Therefore the second technique significantly reduced the problem of parallel workers via randomising the jobs at the tracker server level. While executing jobs in sequence (with the oldest WAT files from 2013), we discovered that adjacent jobs were overlapping considerably. When we randomised jobs, we saw a dramatic decrease in such overlapping.

 Who ran this?

 We want to thank :

 * the LAION folks, via so many worker nodes everywhere in the cloud
 * the data hoarders Reddit community
 * the-eye community
 * as well as all our friends and relatives that did not know what they were helping with

 for running the workers to produce this vast dataset in a few months.

 2. Post-processing of the dataset

 Once the distributed pipeline has run, resulting in a sizeable caption+url dataset, it’s time to package it in the best way. The objective of this second pipeline is to produce a version of the dataset that is easy to use for multimodal training. For this, we built tools that anyone can run out of a collection of caption+url. The exact command line to run is available in cah-prepro (which uses mainly img2dataset and clip-retrieval )

 Pyspark preprocessing of the CSV files

 After a fast run of a script to download the CSV files, the first step of this post-processing pipeline is to do deduplication by url+caption. The first pipeline does some partial deduplication using a bloom filter, but it is approximate, and some duplicates remain. Doing that pyspark post-processing also makes it possible to reduce the number of metadata files from hundred of thousands to 32 parquet files of size 1.7GB. See this deduplication script there. Pyspark would be an excellent way to do any further filtering, and we provide an example to compute some statistics. The resulting output is 32 parquet files containing columns such as URL, text, NSFW described at the beginning of the post.

 Img2dataset

 Once this set of 50GB parquet files has is ready, we can use the img2dataset tool to download, resize and store the images and captions as webdataset. This tool can download 100M images in 20h in a single node (1Gbps 32GB of ram 16 i7 cores), so anyone can run this for the whole dataset or a smaller subset. The format this tool outputs is a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and corresponding parquet files containing the same metadata

 * 00000.tar of size 270MB containing at most 10k samples
 + 0.jpg
 + 0.txt containing the caption
 + 0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW
 * 00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files

 The size of the tars of 270MB is when using the options of img2dataset indicated there download_images.sh (resizing all images to 256×256 with padding for maximum file uniformity and avoid losing information). If using different options, you may have larger or smaller tar files.

 Clip retrieval and autofaiss

 Finally, the tar dataset aims to compute and package clip embeddings and compute a KNN index over the clip embeddings. The clip-retrieval tool makes it fast to compute 100M embeddings per 20h with a single 3080 GPU, so it’s possible to rerun this part on the whole dataset or a subset at a low cost. The embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. NPY files are 1GB in size, and parquet files are 150MB. There are a total of 400 such files. These embeddings help build text and an image knn index using the autofaiss tool, making it possible to produce a quantised index of an arbitrary file. The chosen index type is 6GB, so it’s cheap for anyone to load and run fast (10ms) queries over the whole dataset. We also generated another kind of index of size 16GB. Thanks to memory mapping, it’s also possible to load it at no ram usage. A simple web demo shows the results.

 License

 We distribute the metadata dataset (the parquet files) under the most open Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.

 Contributing

 You can contribute to the project to help us release the following dataset sizes at 1 billion pairs, 2 billion pairs and so on.

 Choose one or more methods that suit you or your company:

 1. donate either cash or computing time. We also launched a Go Get Funding campaign.
 2. participate in the development effort
 3. spread the word. At best, use the dataset, get nice results and mention it in your papers

 Useful links:

 * Dataset progress Crawling@Home Dashboard and leaderboard
 * Reddit post
 * DALLE-PyTorch Discord server
 * DALLE-PyTorch GitHub Repository


 Sponsors

 We made it so far due to the generosity of these donors:

 
 doodlebot.ai Gentec Data the-eye.eu
 
We present LAION-400M: 400M English (image, text) pairs - see also our Data Centric AI NeurIPS Workshop 2021 paper 
Concept and Content
The LAION-400M dataset is entirely openly, freely accessible.
 WARNING: be aware that this large-scale dataset is non-curated. It was built for research purposes to enable testing model training on larger scale for broad researcher and other interested communities, and is not meant for any real-world production or application.
We have filtered all images and texts in the LAION-400M dataset with OpenAI‘s CLIP by calculating the cosine similarity between the text and image embeddings and dropping those with a similarity below 0.3. The threshold of 0.3 had been determined through human evaluations and seemed to be a good heuristic for estimating semantic image-text-content matching.
The image-text-pairs have been extracted from the Common Crawl web data dump and are from random web pages crawled between 2014 and 2021.
Download Information
UPDATE 16 dec 2021
While the eye experiences technical difficulties, we provide an alternate download server for this dataset at this link: laion400m at deploy.laion.ai 
Original information
You can find
The CLIP image embeddings (NumPy files)
The parquet files
KNN index of image embeddings
To download from the eye, run this command
 aria2c "https://the-eye.eu/public/AI/cah/laion400m-met-release.torrent" 
You may want to use the –show-files and –select-file options to download only some files.
You can also find the files in laion400m-met-release 
Some more significant knn indices are present in laion400m-indexes. We advise using the 128GB ones.
The parquet files in Kaggle: laion400m on Kaggle 
After downloading the metadata as indicated above, you can run this command to download the images and generate the webdataset files (command using img2dataset )
LAION-400M Dataset Statistics
The LAION-400M and future even bigger ones are, in fact, datasets of datasets. For instance, we can filter it out by image sizes into smaller datasets like this:
 Number of unique samples 413M
Number with height or width &gt;= 1024 26M
Number with height and width &gt;= 1024 9.6M
Number with height or width &gt;= 512 112M
Number with height and width &gt;= 512 67M
Number with height or width &gt;= 256 268M
Number with height and width &gt;= 256 211M
 
By using the KNN index, we can extract specialized datasets by domains of interest. They are (or will be) sufficient in size to train technical domain models.
Also, use https://rom1504.github.io/clip-retrieval/ for simple visualisation of the dataset. There you can search among the dataset using CLIP and a knn index.
Disclaimer &amp; Content Warning
Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.
There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.
Using KNN clustering should make it easy to further deduplicate by image content.
LAION-400M Open Dataset structure
We produced the dataset in several formats to address the various use cases:
a 50GB url+caption metadata dataset in parquet files. We can use the metadata to compute statistics and redownload part of the dataset
a 10TB webdataset with 256×256 images, captions and metadata. It is a full version of the dataset that can be used directly for training (this one is for internal use, you need to redownload images yourself due to licensing issues)
a 1TB set of the 400M text and image clip embeddings, useful to rebuild new knn indices
pairs of 16G, 32G, 64G and 128G knn indices (running in the web demo)
URL and caption metadata dataset.
We provide 32 parquet files of size around 1GB (total 50GB) with the image URLs, the associated texts and additional metadata in the following format:
SAMPLE_ID | URL | TEXT | LICENSE | NSFW | similarity | WIDTH | HEIGHT
where
 SAMPLE_ID: A unique identifier
 LICENSE: Where we found a Creative Commons License in the image data, we named it here like, e.g. “creativecommons.org/licenses/by-nc-sa/3.0/” – otherwise you’ll find it here a “?”
 NSFW: we used CLIP to estimate if the image has NSFW content. The estimation has been pretty conservative, reducing false negatives at the cost of more false positives. Possible values are “UNLIKELY”, “UNSURE” and “NSFW”.
 similarity: Value of the cosine similarity between the text and image embedding
WIDTH and HEIGHT: image size as the image was embedded. We downsized originals that were larger than 4K to 4K.
This metadata dataset purpose is to download the images for the whole dataset or a subset of it by supplying it to the very efficient img2dataset tool.
10 TB webdataset with images and captions
By running the img2dataset tool, we can download a 10TB webdataset. It will resize all images at 256×256 resolution, will append the corresponding caption and will generate a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and related parquet files containing the same metadata
00000.tar of size 270MB containing at most 10k samples
0.jpg
0.txt containing the caption
0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW
00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files
The 400M dataset will therefore have 41455 tar and 41455 parquet files. This dataset purpose is to train multimodal models like CLIP or DALL-E.
1TB of clip embeddings
The clip embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. Each NPY file is 1GB, and each parquet file is 150MB. There are a total of 400 such files. The embeddings purpose is to compute statistics on the dataset, for example, using clustering or knn indices.
Two small 6GB knn indices
We provide two 6GB knn indices built using the autofaiss. We can use them to compute a subset of the dataset and, more generally, to search among it efficiently. See the search web demo of it. We can use the CLIP filter tool along with this index to produce subsets using search terms efficiently. We also provide two 16GB knn indices of higher quality.
What can we do with the LAION-400M dataset?
Vision and language modelling has been taking off in 2021. Here are some pointers about what this kind of image + text datasets unlocks and why it seems interesting:
Six months ago, OpenAI released two blog posts and papers, CLIP and DALL-E. Both models rely on a large amount of (text, image) pairs. They used an unreleased 400M pairs dataset.
CLIP is a model that computes how related are a text and an image. It makes it possible to build large text to image search, and it makes it possible to create that kind of crazy text to image art clip-art. They released a small and medium version of the model but no training code.
DALL-E is a model that directly generates images from texts. As can be seen from the blog post, it achieves awe-inspiring results that could directly impact the world for anything that needs drawing and illustrations. OpenAI did not release any model, even through an API
Since then, various researchers have organised several efforts to replicate DALL-E. People gathered initially around this excellent DALLE replication repository DALLE-PyTorch with some fantastic results visible in the readme. More recently, as part of huggingface events, new developments have been achieved (see DALLE-mini report ), and an online demo is now available at DALLE-mini demo. 
The replication effort is still far from achieving the same performance as the original DALLE, and it seems possible to go even further. Some people also want to make a better CLIP to produce even better-generated art.
A large part of the results that we can achieve with such models is thanks to a large amount of data. Before LAION-400M, the largest open dataset for (image, text) pairs are in the order of 10M (see DALLE-datasets ), which is enough to train exciting models but not enough to reach the best performance. Having a public dataset with hundreds of millions of pairs will help build these image+text models.
Analysis of the LAION-400M data
We annotated 3456 samples of the dataset and got the following results:
Correct positive NSFW: 4
Correct negative NSFW: 3371
False-positive NSFW: 73
False-negative NSFW: 8
Bad captions: 3 (0.09 %)
The matching is excellent, thanks to CLIP. We could improve the NSFW automatic tagging in the future; however, the NSFW total rate is low enough (less than 1%) to make this not an issue.
Technical Details
The dataset acquisition has into two significant parts:
a distributed processing of the vast (many PBs) Common Crawl datasets, which produces a collection of matching URL and caption
a single node much lighter post-processing of the data that anyone can run in a few days and which produces the final dataset
1. Distributed processing of Common Crawl
We acquire the raw web data for the creation of our dataset from Common Crawl. Common Crawl is a non-profit organisation dedicated to providing a copy of the internet to internet researchers, companies, and individuals at no cost for research and analysis. They regularly release dumps of HTML-like data parsed from billions of public websites found on the Common Crawl website. To create image-text pairs, we parse through the data from Common Crawl and parse out all HTML IMG tags containing an alt text attribute. Common Crawl provides its data in several formats. For our purpose, we chose to use the data in the WAT format. The WAT files contain only the metadata of the crawled sites, which includes all links and IMG tags contained in the website. Parsing only this metadata is much faster than parsing the whole HTML text (provided in the WARC format).
Downloading original images
We download the raw images from the URLs we parsed from Common Crawl with asynchronous requests using the libraries Trio and Asks. They allow us to go multithreading for a single CPU. Usually, a home internet link will be exhausted by a single or two CPUs. A data centre node can scale up benefits from guaranteed internet speed with a multiprocessing pool much faster than a single CPU node. At this time, we were able to use 50 cores with a full, secured 1Gbps connection to the public internet. This bandwidth must be available to the downloading node, not shared among many nodes or apps. We have optimised the script for speed while mitigating various errors we encountered. Usually, to satisfy a high-end demanding node such as above, we must take additional steps to provide DNS caching capabilities. We found that the knot-resolver ran with two processes and configured with caching option can solve this problem.
Filtering out unsuitable image-text pairs
After downloading the WAT files from Common Crawl, we filter the samples in the following steps:
We dropped all samples with less than five character alt text length
We dropped all samples with less than 5 KB image size
We use continuously updated bloom filters to drop samples that are already in our dataset. The bloom filters deduplicate by concatenating the URL and the alt text.
We use continuously updated bloom filters to drop samples from URLs that had timed out previously and therefore seem unreachable (or at least not reachable in an efficient way)
We use OpenAI’s CLIP model (the ‘ViT-B-32‘ version) to compute the image and alt text embeddings. Then we calculate the cosine similarity of both embedding vectors and drop all samples with a similarity below 0.3. We chose this threshold after trying different values and using human evaluations of how well the texts fit the images. Lower values like 0.28 or 0.29 also seemed okay in many cases, but after further inspections, we decided to choose the conservative value of 0.3.
We use the CLIP embeddings of the images to estimate if their contents contain NSFW content. We do this by calculating CLIP embeddings for a list of image categories like, e.g. “selfie”, “illustration”, or “landscape”, which also contains categories that indicate NSFW content like “porn” and “sex”.
Then we compute the cosine similarities between the embedding image we are currently filtering and each of these category keywords. If the category with the highest similarity and the keyword with the second-highest similarity belong both to NSFW keywords, we tag the sample as “NSFW”. If only one of them belongs to an NSFW keyword, we categorise the sample as “UNSURE”. If both keywords with the highest similarities are not NSFW, we tag the sample as “UNLIKELY”.
In the next step, we look at all samples with either the “NSFW” or “UNSURE” tag and drop those with any keywords in their text related to kids, teens, or other semantically related content.
In step 8, we repeat the procedure of computing the cosine similarities from step 6 with the difference that we now use category texts that indicate contents semantically related to kids and teens on a CLIP embedding level. If either the highest similarity or the second-highest similarity between a sample’s image embedding and a text of the precomputed categories belongs to a text that indicates content related to under-aged persons, we drop this sample.
Finally, we repeat the procedure from step 8 with texts semantically related to animal categories like e.g. “animal”, “bird”, etc.
We perform these rigorous filtering steps for NSFW with potentially illegal content because we cannot guarantee that the contents of Common Crawl are free of such. We feel obligated to try our best to filter out such content. Inspections of samples filtered out by steps 7 to 9 have shown that our filtering procedure is very conservative and produces many false positives (samples it drops, which are not problematic). This process is okay because the number of potential samples waiting for us to crawl is vast.
System Architecture
To orchestrate the interactions of the many crawling scripts (called workers) in our project, we use a server that keeps track of processed WAT files and of which worker gets which unprocessed WAT. We call this orchestrating server a tracker. Its functions are offering jobs to both download workers and inference workers, confirming cleanup requests from the DL staging server, maintaining ACLs for the Bloom server, and some more. We also employ several staging servers as buffers for jobs on their way to the storage location. The staging servers continuously update filters in the central bloom server where we use RedisBloom for high-performance reasons.
Workflow
During the evolution of our crawling project, we applied two different workflows:
Workflow 1 (“Hybrid” – workers)
This worker performs all computation steps during one job and then submits the result to the staging server. It then queues the results for release to the storage area.
Workflow 2 (“CPU – GPU – 2 stages” – workflow)
We soon discovered that the best way to utilise resources is to split the workload into CPU + networking tasks (downloading steps) and GPU tasks (CLIP inference steps). Hence, the 2 stage approach uses “CPU workers” to download images, create image-text pairs, and save the intermediate result to a staging server. Then “GPU workers” pick up jobs, concatenate a number of them to group around 20000 pairs per final result file. The 2 stage workflow proved to be most efficient, with speeds up to 25 million pairs added to the dataset per day when using 100 CPU workers with one core and one GPU worker employing an NVidia RTX 3090 graphic card utilising all 16 lanes of PCIe bus. The GPU node also needs about CPU 24 threads to keep up with the GPU processing capacity.
Removing abuse alerts
During downloading, we encountered abuse alerts from manual and automated tools that protect websites. After some learning curve, we reduced most of the issues by employing these mitigation techniques:
By far, the most efficient one was to use centralised bloom filters that eliminate requests going to the duplicate URLs over and over. Of course, the efficiency of these filters dramatically depends on how fast they are updated and used by the workers. By definition, having multiple downloading workers performing jobs in parallel makes them prone to overlap requests to the same URL even if the bloom filters are up to date at the beginning of the job.
Therefore the second technique significantly reduced the problem of parallel workers via randomising the jobs at the tracker server level. While executing jobs in sequence (with the oldest WAT files from 2013), we discovered that adjacent jobs were overlapping considerably. When we randomised jobs, we saw a dramatic decrease in such overlapping.
Who ran this?
We want to thank :
the LAION folks, via so many worker nodes everywhere in the cloud
 the data hoarders Reddit community
 the-eye community
as well as all our friends and relatives that did not know what they were helping with
for running the workers to produce this vast dataset in a few months.
2. Post-processing of the dataset
Once the distributed pipeline has run, resulting in a sizeable caption+url dataset, it’s time to package it in the best way. The objective of this second pipeline is to produce a version of the dataset that is easy to use for multimodal training. For this, we built tools that anyone can run out of a collection of caption+url. The exact command line to run is available in cah-prepro (which uses mainly img2dataset and clip-retrieval )
Pyspark preprocessing of the CSV files
After a fast run of a script to download the CSV files, the first step of this post-processing pipeline is to do deduplication by url+caption. The first pipeline does some partial deduplication using a bloom filter, but it is approximate, and some duplicates remain. Doing that pyspark post-processing also makes it possible to reduce the number of metadata files from hundred of thousands to 32 parquet files of size 1.7GB. See this deduplication script there. Pyspark would be an excellent way to do any further filtering, and we provide an example to compute some statistics. The resulting output is 32 parquet files containing columns such as URL, text, NSFW described at the beginning of the post.
Img2dataset
Once this set of 50GB parquet files has is ready, we can use the img2dataset tool to download, resize and store the images and captions as webdataset. This tool can download 100M images in 20h in a single node (1Gbps 32GB of ram 16 i7 cores), so anyone can run this for the whole dataset or a smaller subset. The format this tool outputs is a collection of tar files (that dataset format is called webdataset) containing images, captions, and metadata and corresponding parquet files containing the same metadata
00000.tar of size 270MB containing at most 10k samples
0.jpg
0.txt containing the caption
0.json containing metadata such as the URL, the original width, the EXIF data, whether the image is NSFW
00000.parquet of size 1.6MB containing the same metadata as the JSON file. Useful to compute statistics without reading all the tar files
The size of the tars of 270MB is when using the options of img2dataset indicated there download_images.sh (resizing all images to 256×256 with padding for maximum file uniformity and avoid losing information). If using different options, you may have larger or smaller tar files.
Clip retrieval and autofaiss
Finally, the tar dataset aims to compute and package clip embeddings and compute a KNN index over the clip embeddings. The clip-retrieval tool makes it fast to compute 100M embeddings per 20h with a single 3080 GPU, so it’s possible to rerun this part on the whole dataset or a subset at a low cost. The embeddings are stored in NPY files next to parquet files in the same order. Since this dataset is much smaller than image one, each NPY file stores 1M samples. NPY files are 1GB in size, and parquet files are 150MB. There are a total of 400 such files. These embeddings help build text and an image knn index using the autofaiss tool, making it possible to produce a quantised index of an arbitrary file. The chosen index type is 6GB, so it’s cheap for anyone to load and run fast (10ms) queries over the whole dataset. We also generated another kind of index of size 16GB. Thanks to memory mapping, it’s also possible to load it at no ram usage. A simple web demo shows the results.
 
License
We distribute the metadata dataset (the parquet files) under the most open Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.
Contributing
You can contribute to the project to help us release the following dataset sizes at 1 billion pairs, 2 billion pairs and so on.
Choose one or more methods that suit you or your company:
donate either cash or computing time. We also launched a Go Get Funding campaign.
participate in the development effort
spread the word. At best, use the dataset, get nice results and mention it in your papers
Useful links:
Dataset progress Crawling@Home Dashboard and leaderboard 
Reddit post 
DALLE-PyTorch Discord server 
DALLE-PyTorch GitHub Repository 
Sponsors
We made it so far due to the generosity of these donors:
 
 
 
 doodlebot.ai 
 Gentec Data 
 the-eye.eu
LAION-400-MILLION OPEN DATASET | LAION
https://laion.ai/blog/laion-400-open-dataset/
https://arxiv.org/abs/2111.02114
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
2021-11-03
Schuhmann; Christoph; Vencu; Richard; Beaumont; Romain; Kaczmarczyk; Robert; Mullis; Clayton; Katta; Aarush; Coombes; Theo; Jitsev; Jenia; Komatsuzaki; Aran
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2111.02114">Download PDF</a></p><blockquote>
 Abstract: Multi-modal language-vision models trained on hundreds of millions of
image-text pairs (e.g. CLIP, DALL-E) gained a recent surge, showing remarkable
capability to perform zero- or few-shot learning and transfer even in absence
of per-sample labels on target image data. Despite this trend, to date there
has been no publicly available datasets of sufficient scale for training such
models from scratch. To address this issue, in a community effort we build and
release for public LAION-400M, a dataset with CLIP-filtered 400 million
image-text pairs, their CLIP embeddings and kNN indices that allow efficient
similarity search.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Aran Komatsuzaki [<a href="/show-email/26085e3b/2111.02114">view email</a>]
 <br /><strong>[v1]</strong>
Wed, 3 Nov 2021 10:16:39 UTC (2,473 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2111.02114

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computer Vision and Pattern Recognition

 arXiv:2111.02114 (cs)
 [Submitted on 3 Nov 2021]

 Title: LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

 Authors: Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, Aran Komatsuzaki
 Download PDF
 Abstract: Multi-modal language-vision models trained on hundreds of millions of image-text pairs (e.g. CLIP, DALL-E) gained a recent surge, showing remarkable capability to perform zero- or few-shot learning and transfer even in absence of per-sample labels on target image data. Despite this trend, to date there has been no publicly available datasets of sufficient scale for training such models from scratch. To address this issue, in a community effort we build and release for public LAION-400M, a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.
 Comments: Short version. Accepted at Data Centric AI NeurIPS Workshop 2021 
 Subjects: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL); Machine Learning (cs.LG)
 Cite as: arXiv:2111.02114 [cs.CV]  
 (or arXiv:2111.02114v1 [cs.CV] for this version) 
 https://doi.org/10.48550/arXiv.2111.02114  
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Aran Komatsuzaki [view email]
 [v1] Wed, 3 Nov 2021 10:16:39 UTC (2,473 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 Current browse context:
 cs.CV
 < prev | next >
 new | recent | 2111
 Change to browse by:
 cs
 cs.CL
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Jenia Jitsev
 Aran Komatsuzaki
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Christoph Schuhmann 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Richard Vencu richard.vencu@gentec.ro 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Romain Beaumont 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Robert Kaczmarczyk robert.kaczmarczyk@tum.de 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Clayton Mullis claymullis@fastmail.com 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Aarush Katta Laion 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Theo Coombes theocoombes06@gmail.com 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Jenia Jitsev j.jitsev@fz-juelich.de 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
Aran Komatsuzaki akomatsuzaki3@gatech.edu 
LAION Gentec Data
LAION
LAION Juelich Supercomputing Center (JSC) Research Center Juelich (FZJ)
LAION Georgia Institute of Technology EleutherAI
LAION Technical University of Munich
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Multi-modal language-vision models trained on hundreds of millions of image-text pairs (e.g. CLIP, DALL-E) gained a recent surge, showing remarkable capability to perform zero-or few-shot learning and transfer even in absence of per-sample labels on target image data. Despite this trend, to date there has been no publicly available datasets of sufficient scale for training such models from scratch. To address this issue, in a community effort we build and release for public LAION-400M, a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search. 1
Introduction
Multi-modal language-vision models demonstrated recently strong transfer capability to novel datasets in absense of per-sample labels [1,2,3]. This capability requires sufficiently large model and data scale during pre-training. Increasing data scale alone can often improve model performance [4]. When increasing model and compute budget scale in addition, scaling laws suggest further increase in generalization and transfer performance if not bottlenecked by the data scale [5,6,7,8]. There is a plethora of recent works that have built massive datasets in order to optimally scale up various models [9,1,2,3]. However, these massive datasets have rarely been released for various reasons. Gao et. al. recently released The Pile, an openly-available 800GB text dataset [10], in an attempt to loosely mimic the dataset used for GPT-3. The largest publicly known image-text paired datasets range from 400 million to around a billion, but none of them has been released. Figure 1: Sample images retrieved from the queries "blue cat" or "cat with blue eyes" in the web demo To address this issue, we build and release LAION-400M, a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices. We describe the procedure to create the dataset and demonstrate successful training of DALL-E architecture. Having sufficiently large scale, the dataset opens venues for research on multi-modal language-vision models to broad community.
Dataset and Methods
Overview of LAION-400M. We officially release the following packages under LAION-400M project:
• 400 million pairs of image URL and the corresponding metadata • 400 million pairs of CLIP image embedding and the corresponding text • Several sets of kNN indices that enable quick search in the dataset • img2dataset library that enables efficient crawling and processing of hundreds of millions of images and their metadata from a list of URLs with minimal resources • Web demo of image-text search on LAION-400M ( Fig. 1) 2 As for the pairs of image URL and metadata, we provide parquet files that consist of the following attributes for each pair: sample ID, URL, type of Creative Commons license (if applicable), NSFW tag (detected with CLIP), cosine similarity score between the text and image embedding and height and width of the image. We found less than 1% of images were detected as NSFW, which can be filtered out by an user with NSFW tag. Acquisition. The acquisition follows the flowchart of Fig. 2 and can be split into two major components:
• Distributed processing of petabyte-scale Common Crawl dataset, which produces a collection of matching URLs and captions.
• Single node post-processing of the data, which is much lighter and can be run in a few days, producing the final dataset.
Distributed processing of Common Crawl
To create image-text pairs, we parse through WAT files from Common Crawl and parse out all HTML IMG tags containing an alt-text attribute. We download the raw images from the parsed URLs with asynchronous requests using Trio and Asks libraries.
Filtering out unsuitable image-text pairs
After downloading the WAT files from Common Crawl, we apply the following filtering conditions:
• All samples with less than 5 character alt-text length or less than 5 KB image size are dropped.
• Duplicate removal is performed with bloom filter based on URL and alt-text.
• We use CLIP to compute embeddings of the image and alt-text. Then we compute the cosine similarity of both embeddings and drop all samples with cosine similarity below 0.3. This threshold was selected based on human inspections.
• We use the CLIP embeddings of images and texts to filter out illegal contents.  
img2dataset
We developed img2dataset library to comfortably download from a given set of URLs, resize and store the images and captions in the webdataset format. 3 This allows to download 100 million images from our list of URLs in 20 hours with a single node (1Gbps connection speed, 32GB of RAM, an i7 CPU with 16 cores), which allows anyone to obtain the whole dataset or a smaller subset.
Analysis & Results
Web demo and similarity search. A web demo was created to allow an user to search images and texts based on a query image or text using the CLIP embeddings of the input and our precomputed kNN indices. It demonstrates the diversity of images and captions that can be found in LAION-400M as well as high semantic relevance (Fig. 1).
Tab. 1 shows the distribution of image sizes of LAION-400M. Given the abundance of high-resolution images, one can produce subsets of images for training various customized models, and also choose image resolution that is suitable for purpose of particular training.
Training DALL-E model. We ran DALLE-pytorch [11], an open-source replication of DALL-E [2], to assess the dataset's capability to train a text-to-image model. The VQGAN [12] pretrained on ImageNet is used to encode image tokens. For generation, we use CLIP ViT-B/16 [1] to rank the top 8 of 128 total samples per caption. Despite only seeing a subset of approximately 7.2 million images for a single epoch, we observe fast convergence across a variety of categories. Samples generated from the model show sufficient quality and provide evidence for successful training progress (Fig. 3).
Conclusion
By releasing an openly available dataset that contains 400 million image-text pairs, we have closed the gap to proprietary large scale datasets that were necessary to train state-of-the-art language-vision models such as DALL-E and CLIP. As proof of concept, we demonstrated that a subset of our dataset can be used to train a DALL-E model, producing samples of sufficient quality. The dataset opens the road for large-scale training and research of language-vision models, that were previously restricted to those having access to proprietary large datasets, to the broad community.
Figure 2 :
2Acquisition workflow
Figure 3 :
3DALL-E Experiments. (Left) Generated samples from a DALL-E model trained with 7.2M randomly picked LAION-400M samples on 1 RTX 2070 Super (8 GB VRAM) for 1 epoch (Right) DALL-E runs with Conceptual Captions 3M (green), Conceptual Captions 12M (orange) and a 3M subset of LAION-400M (grey)
Project page: https://laion.ai/laion-400-open-dataset/ Preprint. Under review.
https://rom1504.github.io/clip-retrieval/
https://github.com/rom1504/img2dataset
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, arXiv:2103.00020Learning Transferable Visual Models From Natural Language Supervision. arXiv e-prints. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv e-prints, page arXiv:2103.00020, February 2021.
. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever, arXiv:2102.12092Zero-Shot Text-to-Image Generation. arXiv e-printsAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. arXiv e-prints, page arXiv:2102.12092, February 2021.
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, Tom Duerig, arXiv:2102.05918Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv e-prints, page arXiv:2102.05918, February 2021.
Aran Komatsuzaki, arXiv:1906.06669One Epoch Is All You Need. arXiv e-prints. Aran Komatsuzaki. One Epoch Is All You Need. arXiv e-prints, page arXiv:1906.06669, Jun 2019.
Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, arXiv:2001.08361Scaling Laws for Neural Language Models. arXiv e-prints. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv e-prints, page arXiv:2001.08361, Jan 2020.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, arXiv:2010.14701John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. arXiv e-prints. Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, DanielTom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. arXiv e-prints, page arXiv:2010.14701, October 2020.
Big transfer (bit): General visual representation learning. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby, Computer Vision -ECCV 2020. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael FrahmChamSpringer International PublishingAlexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020, pages 491-507, Cham, 2020. Springer International Publishing.
Scaling vision transformers. Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, arXiv:2106.04560arXiv preprintXiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint arXiv:2106.04560, 2021.
. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel MTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
Jeffrey Ziegler, Clemens Wu, Christopher Winter, Mark Hesse, Eric Chen, Mateusz Sigler, Scott Litwin, Benjamin Gray, Jack Chess, Christopher Clark, Sam Berner, Alec Mccand Lish, Radford, arXiv:2005.14165Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv e-prints. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCand lish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv e-prints, page arXiv:2005.14165, May 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, Connor Leahy, arXiv:2101.00027The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv e-prints. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Ja- son Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv e-prints, page arXiv:2101.00027, December 2020.
Dall-e in pytorch: A text to image transformer. Phil Wang, Phil Wang. Dall-e in pytorch: A text to image transformer, 2021.
Taming transformers for high-resolution image synthesis. Patrick Esser, Robin Rombach, Björn Ommer, Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
https://arxiv.org/abs/2111.02114
https://paperswithcode.com/dataset/laion-400m
Papers with Code - LAION-400M Dataset
2019-01-01
None
<div><div>
 <p><strong>LAION-400M</strong> is a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.</p>
<h4>⚠️ Disclaimer &amp; Content Warning (from the authors)</h4>
<p><em>Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.</em></p>
<p><em>There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.</em></p>
<p><em>Using KNN clustering should make it easy to further deduplicate by image content.</em></p>
 
 </div>||||I|||| * 
 * Browse State-of-the-Art
 * Datasets
 * Methods
 * More
 Newsletter RC2022
 About Trends Portals Libraries
 * 
 * 
 * Sign In

 Subscribe to the PwC Newsletter

 ×
 Stay informed on the latest trending ML papers with code, research developments, libraries, methods, and datasets.

 Read previous issues
 Subscribe

 Join the community

 ×
 You need to log in to edit.
 You can create a new account if you don't have one.

 Or, discuss a change on Slack.

 Edit Dataset

 ×
 Name: *
 Full name (optional):
 Description (Markdown and $\LaTeX$ enabled): *
 **LAION-400M** is a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search. #### ⚠️ Disclaimer & Content Warning (from the authors) *Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.* *There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.* *Using KNN clustering should make it easy to further deduplicate by image content.*
 Homepage URL (optional):
 Paper where the dataset was introduced:
 --------- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
 Introduction date:
 Dataset license:
 URL to full license terms:
 Image
 Currently
 datasets/969267f7-ca38-49cb-b843-08c1490b88e5.png Clear
 Change
 ---
 Save

 Edit Dataset Tasks

 ×
 Add or remove tasks:
 Image-to-Text Retrieval Semantic Image-Text Similarity Video Generation

 Some tasks are inferred based on the benchmarks list.

 Save

 Add a Data Loader

 ×
 Code Repository URL: *
 [Optional] URL to documentation for this dataset:
 Supported frameworks:
 TensorFlow
 PyTorch
 JAX
 Submit

 Remove a Data Loader

 ×

 Edit Dataset Modalities

 ×
 Add or remove modalities:
 Images Texts
 Save

 Edit Dataset Languages

 ×
 Add or remove languages:
 Save

 Edit Dataset Variants

 ×
 Add or remove variants:
 LAION-400M

 The benchmarks section lists all benchmarks using a given dataset or any of its variants. We use variants to distinguish between results evaluated on slightly different versions of the same dataset. For example, ImageNet 32⨉32 and ImageNet 64⨉64 are variants of the ImageNet dataset.

 Save

 Add a new evaluation result row

 ×
 Paper title: *
 ---------
 Dataset or its variant: *
 --------- LAION-400M
 Task: *
 ---------
 Model name: *
 Metric name: *
 ---------
 Higher is better (for the metric)
 Metric value: *
 Uses extra training data
 Data evaluated on
 Submit
 Texts
 Edit

 LAION-400M

 Introduced by Schuhmann et al. in LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

 LAION-400M is a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.

 ⚠️ Disclaimer & Content Warning (from the authors)

 Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering.

 There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated.

 Using KNN clustering should make it easy to further deduplicate by image content.

 Homepage

 Benchmarks

 Edit
 Add a new result Link an existing benchmark

 Trend Task Dataset Variant Best Model Paper Code
 Video Generation LAION-400M Imagen original 
 

 Papers
 Paper Code Results Date Stars
 

 Dataset Loaders

 Edit
 Add Remove

 No data loaders found. You can submit your data loader here.

 Tasks

 Edit

 * Video Generation
 * Image-to-Text Retrieval
 * Semantic Image-Text Similarity

 Similar Datasets


 ELEVATER

 ELEVATER


 EntitySeg


 LAION-5B


 VALSE


 Usage

 License

 Edit
 * Creative Common CC-BY 4.0

 Modalities

 Edit

 * Images
 * Texts

 Languages

 Edit

 Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA .
 Terms Data policy Cookies policy from
LAION-400M is a dataset with CLIP-filtered 400 million image-text pairs, their CLIP embeddings and kNN indices that allow efficient similarity search.
⚠️ Disclaimer &amp; Content Warning (from the authors)
 Our filtering protocol only removed NSFW images detected as illegal, but the dataset still has NSFW content accordingly marked in the metadata. When freely navigating through the dataset, keep in mind that it is a large-scale, non-curated set crawled from the internet for research purposes, such that collected links may lead to discomforting and disturbing content. Therefore, please use the demo links with caution. You can extract a “safe” subset by filtering out samples drawn with NSFW or via stricter CLIP filtering. 
 There is a certain degree of duplication because we used URL+text as deduplication criteria. The same image with the same caption may sit at different URLs, causing duplicates. The same image with other captions is not, however, considered duplicated. 
 Using KNN clustering should make it easy to further deduplicate by image content.
Papers with Code - LAION-400M Dataset
https://paperswithcode.com/dataset/laion-400m
https://www.eff.org/deeplinks/2016/06/fbi-can-search-400-million-face-recognition-photos
New Report: FBI Can Access Hundreds of Millions of Face Recognition Photos
2016-06-15
Jennifer Lynch
<div><div>
 <article>
 
 
 <div><p>Today the federal <a href="http://www.gao.gov/about/index.html">Government Accountability Office</a> (GAO) finally published its exhaustive report on the FBI’s face recognition capabilities. The takeaway: <a href="http://www.gao.gov/assets/680/677098.pdf">FBI has access to <i>hundreds of millions</i> more photos</a> than we ever thought. And the Bureau has been hiding this fact from the public—in flagrant violation of federal law and agency policy—for years.
</p>
<p>According to the <a href="http://www.gao.gov/assets/680/677098.pdf">GAO Report</a>, FBI’s Facial Analysis, Comparison, and Evaluation (<a href="https://www.fbi.gov/foia/privacy-impact-assessments/facial-analysis-comparison-and-evaluation-face-services-unit">FACE</a>) Services unit not only has access to FBI’s <a href="https://www.eff.org/foia/fbis-next-generation-identification-biometrics-database">Next Generation Identification (NGI)</a> face recognition database of nearly 30 million civil and criminal mug shot photos, it also has access to the State Department’s Visa and Passport databases, the Defense Department’s biometric database, and the drivers license databases of at least 16 states. Totaling 411.9 million images, this is an unprecedented number of photographs, most of which are of Americans and foreigners who have committed no crimes.
</p>
<p>The FBI has done little to make sure that its search results (which the Bureau calls “investigative leads”) do not include photos of innocent people, according to the report. The FBI has conducted only very limited testing to ensure the accuracy of NGI's face recognition capabilities. And it has not taken any steps to determine whether the face recognition systems of its external partners—states and other federal agencies—are sufficiently accurate to prevent innocent people from being identified as criminal suspects. As we know from previous research, face recognition is notoriously inaccurate across the board and may also <a href="https://www.theguardian.com/technology/2016/apr/08/facial-recognition-technology-racial-bias-police">misidentify African Americans and ethnic minorities</a>, young people, and women at higher rates than whites, older people, and men, respectively.
</p>
<p></p>
<p>As the Report points out, many of the 411.9 million face images to which FBI has access—like driver’s license and passport and visa photos—were never collected for criminal or national security purposes. And yet, under agreements we’ve never seen between the FBI and its state and federal partners, the FBI may search these civil photos whenever it’s trying to find a suspect in a crime. As the map above shows, 18 more states are in negotiations with the FBI to provide similar access to their driver’s license databases.
</p>
<p></p>
<p>The states have been very involved in the development of the FBI’s own NGI database, which includes nearly 30 million of the 411.9 million face images accessible to the Bureau (we’ve <a href="https://www.eff.org/deeplinks/2014/04/fbi-plans-have-52-million-photos-its-ngi-face-recognition-database-next-year">written </a><a href="https://www.eff.org/deeplinks/2012/07/fbis_facial_recognition_coming_state_near_you">extensively </a>about <a href="https://www.eff.org/deeplinks/2012/07/eff-urges-congress-protect-privacy-face-recognition">NGI </a>in the <a href="https://www.eff.org/deeplinks/2014/06/eff-and-31-other-organizations-call-privacy-assessment-fbis-biometric-database">past</a>). NGI includes more than 20 million civil and criminal images received directly from at least six states, including California, Louisiana, Michigan, New York, Texas, and Virginia. And it appears five additional states—Florida, Maryland, Maine, New Mexico, and Arkansas—can send search requests directly to the NGI database. As of December 2015, FBI is working with eight more states to grant them access to NGI, and an additional 24 states are also interested.
</p>
<p></p>
<p>The GAO Report spends a significant number of pages criticizing FBI for rolling out these massive face recognition capabilities without ever explaining the privacy implications of its actions to the public. Federal law and Department of Justice policies require the FBI to complete a Privacy Impact Assessment (PIA) of all programs that collect data on Americans, both at the beginning of development and any time there’s significant change to the program. While the FBI produced a PIA in 2008, when it first started planning out the face recognition component of NGI, it didn’t update that PIA until late 2015—seven years later and well after it began making significant changes to the program.  It also failed to produce a PIA for the FACE Services unit until May 2015—three years after FACE began supporting FBI with face recognition searches. As GAO notes, the whole point of PIAs is to give the public notice of the privacy implications of data collection programs and to ensure that privacy protections are built into the system from the start. The FBI failed at this.
</p>
<p></p>
<p>The single bright spot in the report reiterates that FBI decided not to allow searches of civil photos enrolled in NGI to “better protect individuals’ privacy.” This is a hollow victory, however, because if you’ve ever been arrested for any crime at all—including blocking a street as part of a public protest—your civil photos will be linked to your booking photo and subject to face recognition searches along with all the other 29.7 million images in NGI.
</p>
<p>The GAO’s findings are especially shocking, given the timing. Just over a month ago the <a href="https://www.eff.org/deeplinks/2016/05/fbi-ngi-privacyact">FBI demanded</a> its face recognition capabilities be exempt from several key provisions of the federal Privacy Act—and provided the public with only 30 days to respond. Over and over, the FBI’s secret data collection practices confirm why we need <i>more</i> transparency, not less. In the coming weeks, we’ll be asking you to sign on to our comments to the FBI’s proposal. Help us send a message to the FBI that its practices are unacceptable and must change.
</p>
<p></p>
<p></p>
</div>
 </article>
 </div><div>
 <h2>Join EFF Lists</h2>
 
 </div>||||I||||Skip to main content
 * About
 + Contact
 + Press
 + People
 + Opportunities
 * Issues
 + Free Speech
 + Privacy
 + Creativity and Innovation
 + Transparency
 + International
 + Security
 * Our Work
 + Deeplinks Blog
 + Press Releases
 + Events
 + Legal Cases
 + Whitepapers
 + Podcast
 * Take Action
 + Action Center
 + Electronic Frontier Alliance
 + Volunteer
 * Tools
 + Privacy Badger
 + HTTPS Everywhere
 + Surveillance Self-Defense
 + Certbot
 + Atlas of Surveillance
 + Cover Your Tracks
 + Crocodile Hunter
 * Donate
 + Donate to EFF
 + Shop
 + Other Ways to Give
 + Membership FAQ
 * Donate
 + Donate to EFF
 + Shop
 + Other Ways to Give

 * Search form

 Search
 Email updates on news, actions,
 and events in your area.
 Join EFF Lists
 * Copyright (CC BY)
 * Trademark
 * Privacy Policy
 * Thanks
 Electronic Frontier Foundation
 Donate
 How to Fix the Internet: Don't Be Afraid to Poke the Tigers
 Electronic Frontier Foundation
 * About
 + Contact
 + Press
 + People
 + Opportunities
 * Issues
 + Free Speech
 + Privacy
 + Creativity and Innovation
 + Transparency
 + International
 + Security
 * Our Work
 + Deeplinks Blog
 + Press Releases
 + Events
 + Legal Cases
 + Whitepapers
 + Podcast
 * Take Action
 + Action Center
 + Electronic Frontier Alliance
 + Volunteer
 * Tools
 + Privacy Badger
 + HTTPS Everywhere
 + Surveillance Self-Defense
 + Certbot
 + Atlas of Surveillance
 + Cover Your Tracks
 + Crocodile Hunter
 * Donate
 + Donate to EFF
 + Shop
 + Other Ways to Give
 + Membership FAQ
 * Donate
 + Donate to EFF
 + Shop
 + Other Ways to Give

 * Search form

 Search

 New Report: FBI Can Access Hundreds of Millions of Face Recognition Photos

 DEEPLINKS BLOG
 Commentary by Jennifer Lynch
 June 15, 2016

 New Report: FBI Can Access Hundreds of Millions of Face Recognition Photos

 Share It Share on Twitter Share on Facebook Copy link

 Today the federal Government Accountability Office (GAO) finally published its exhaustive report on the FBI’s face recognition capabilities. The takeaway: FBI has access to hundreds of millions more photos than we ever thought. And the Bureau has been hiding this fact from the public—in flagrant violation of federal law and agency policy—for years.

 According to the GAO Report, FBI’s Facial Analysis, Comparison, and Evaluation (FACE) Services unit not only has access to FBI’s Next Generation Identification (NGI) face recognition database of nearly 30 million civil and criminal mug shot photos, it also has access to the State Department’s Visa and Passport databases, the Defense Department’s biometric database, and the drivers license databases of at least 16 states. Totaling 411.9 million images, this is an unprecedented number of photographs, most of which are of Americans and foreigners who have committed no crimes.

 The FBI has done little to make sure that its search results (which the Bureau calls “investigative leads”) do not include photos of innocent people, according to the report. The FBI has conducted only very limited testing to ensure the accuracy of NGI's face recognition capabilities. And it has not taken any steps to determine whether the face recognition systems of its external partners—states and other federal agencies—are sufficiently accurate to prevent innocent people from being identified as criminal suspects. As we know from previous research, face recognition is notoriously inaccurate across the board and may also misidentify African Americans and ethnic minorities, young people, and women at higher rates than whites, older people, and men, respectively.

 As the Report points out, many of the 411.9 million face images to which FBI has access—like driver’s license and passport and visa photos—were never collected for criminal or national security purposes. And yet, under agreements we’ve never seen between the FBI and its state and federal partners, the FBI may search these civil photos whenever it’s trying to find a suspect in a crime. As the map above shows, 18 more states are in negotiations with the FBI to provide similar access to their driver’s license databases.

 The states have been very involved in the development of the FBI’s own NGI database, which includes nearly 30 million of the 411.9 million face images accessible to the Bureau (we’ve written extensively about NGI in the past). NGI includes more than 20 million civil and criminal images received directly from at least six states, including California, Louisiana, Michigan, New York, Texas, and Virginia. And it appears five additional states—Florida, Maryland, Maine, New Mexico, and Arkansas—can send search requests directly to the NGI database. As of December 2015, FBI is working with eight more states to grant them access to NGI, and an additional 24 states are also interested.

 The GAO Report spends a significant number of pages criticizing FBI for rolling out these massive face recognition capabilities without ever explaining the privacy implications of its actions to the public. Federal law and Department of Justice policies require the FBI to complete a Privacy Impact Assessment (PIA) of all programs that collect data on Americans, both at the beginning of development and any time there’s significant change to the program. While the FBI produced a PIA in 2008, when it first started planning out the face recognition component of NGI, it didn’t update that PIA until late 2015—seven years later and well after it began making significant changes to the program. It also failed to produce a PIA for the FACE Services unit until May 2015—three years after FACE began supporting FBI with face recognition searches. As GAO notes, the whole point of PIAs is to give the public notice of the privacy implications of data collection programs and to ensure that privacy protections are built into the system from the start. The FBI failed at this.

 The single bright spot in the report reiterates that FBI decided not to allow searches of civil photos enrolled in NGI to “better protect individuals’ privacy.” This is a hollow victory, however, because if you’ve ever been arrested for any crime at all—including blocking a street as part of a public protest—your civil photos will be linked to your booking photo and subject to face recognition searches along with all the other 29.7 million images in NGI.

 The GAO’s findings are especially shocking, given the timing. Just over a month ago the FBI demanded its face recognition capabilities be exempt from several key provisions of the federal Privacy Act—and provided the public with only 30 days to respond. Over and over, the FBI’s secret data collection practices confirm why we need more transparency, not less. In the coming weeks, we’ll be asking you to sign on to our comments to the FBI’s proposal. Help us send a message to the FBI that its practices are unacceptable and must change.

 Related Issues

 Mandatory National IDs and Biometric Databases
 Privacy
 Transparency

 Related Cases

 FBI's Next Generation Identification Biometrics Database
 FBI Facial Recognition Documents

 Tags

 Street Level Surveillance
 Face Recognition
 Biometrics
 Share It Share on Twitter Share on Facebook Copy link

 Join EFF Lists

 Discover more.

 Email updates on news, actions, events in your area, and more.
 Anti-spam question: Enter the three-letter abbreviation for Electronic Frontier Foundation:
 Don't fill out this field (required)
 Thanks, you're awesome! Please check your email for a confirmation link.
 Oops something is broken right now, please try again later.

 Related Updates

 Deeplinks Blog by Jennifer Lynch , Saira Hussain , Andrew Crocker | November 10, 2022

 EFF Files Amicus Brief Challenging Orange County, CA’s Controversial DNA Collection Program

 Should the government be allowed to collect your DNA—and retain it indefinitely—if you’re arrested for a low-level offense like shoplifting a tube of lipstick, driving without a valid license, or walking your dog off leash? We don’t think so. As we argue in an amicus brief filed in...

 Deeplinks Blog by Alexis Hancock , Adam Schwartz , Jon Callas | August 31, 2021

 Vaccine Passport Missteps We Should Not Repeat

 Vaccine mandates are becoming increasingly urgent from public health officials and various governments. As they roll out, we must protect users of vaccine passports and those who do not want to use—or cannot use—a digitally scannable means to prove vaccination. We cannot let the tools used to fight for public...

 Deeplinks Blog by Saira Hussain , Jennifer Lynch , Nathaniel Sobel | October 22, 2020

 EFF Files Comment Opposing the Department of Homeland Security's Massive Expansion of Biometric Surveillance

 EFF, joined by several leading civil liberties and immigrant rights organizations, recently filed a comment calling on the Department of Homeland Security (DHS) to withdraw a proposed rule that would exponentially expand biometrics collection from both U.S. citizens and noncitizens who apply for immigration benefits and would allow...

 Deeplinks Blog by Alexis Hancock | August 31, 2020

 Digital Identification Must Be Designed for Privacy and Equity

 With growing frequency, the digital world calls for verification of our digital identities. Designed poorly, digital identification can invade our privacy and aggravate existing social inequalities. So privacy and equity must be foremost in discussions about how to design digital identification.Many factors contribute to one's identity; degrees, morals, hobbies, schools,...

 Deeplinks Blog by Jason Kelley | March 19, 2020

 Announcing Who Has Your Face

 The government and law enforcement should not be scanning your photos with face recognition technology. But at least half of Americans are likely in government face recognition databases. Who Has Your Face includes a short quiz that you can use to learn which U.S. government agencies may have access to...
 Deeplinks Blog by Jason Kelley | December 27, 2019

 Fighting Back Against Face Surveillance in the Skies: 2019 Year in Review

 While cities and municipalities made clear strides to limit the use of face surveillance technology throughout 2019, airlines and government agencies tasked with identifying travelers have spent much of the year trying to expand its use. But while the Department of Homeland Security (DHS) and Customs and Border Patrol (CBP),...

 Deeplinks Blog by Jason Kelley | May 8, 2019

 Shareholders Demand To Know How Northrop Grumman Will Protect Human Rights While Building Massive DHS Database

 Over the next few years, the Department of Homeland Security (DHS) plans to implement an enormous biometric collection program which will endanger the rights of citizens and foreigners alike. The agency intends to collect at least seven types of biometric identifiers, including face and voice data, DNA, scars, and tattoos,...

 Deeplinks Blog by Karen Gullo | March 29, 2019

 Don’t Buy California’s Callous Attempt to Ignore People’s DNA Privacy Rights, EFF Tells Court

 Analyzing and indefinitely keeping the DNA profiles of thousands of Californians arrested for felonies, but never charged with a crime, is not just an ominously overbroad practice by law enforcement—it’s an invasion of privacy that violates the state’s constitution. Last year EFF and our co-counsel Michael Risher filed a ...

 Deeplinks Blog by Shahid Buttar | October 9, 2018

 Chicago Should Reject a Proposal for Private-Sector Face Surveillance

 A proposed amendment to the Chicago municipal code would allow businesses to use face surveillance systems that could invade biometric and location privacy, and violate a pioneering state privacy law adopted by Illinois a decade ago. EFF joined a letter with several allied privacy organizations explaining our concerns, which...

 Deeplinks Blog by Jennifer Lynch | June 7, 2018

 HART: Homeland Security’s Massive New Database Will Include Face Recognition, DNA, and Peoples’ “Non-Obvious Relationships”

 So why do we know so little about it?The U.S. Department of Homeland Security (DHS) is quietly building what will likely become the largest database of biometric and biographic data on citizens and foreigners in the United States. The agency’s new Homeland Advanced Recognition Technology (HART) database will include...

 Discover more.

 Email updates on news, actions, events in your area, and more.
 Anti-spam question: Enter the three-letter abbreviation for Electronic Frontier Foundation:
 Don't fill out this field (required)
 Thanks, you're awesome! Please check your email for a confirmation link.
 Oops something is broken right now, please try again later.
 Share It Share on Twitter Share on Facebook Copy link

 Related Issues

 Mandatory National IDs and Biometric Databases
 Privacy
 Transparency

 Related cases

 FBI's Next Generation Identification Biometrics Database
 FBI Facial Recognition Documents

 Related Tags

 Street Level Surveillance
 Face Recognition
 Biometrics

Back to top

 Follow EFF:

 * twitter
 * facebook
 * instagram
 * youtube
 * flicker
 * rss
 * linkedin

 Contact

 * General
 * Legal
 * Security
 * Membership
 * Press

 About

 * Calendar
 * Volunteer
 * Victories
 * History
 * Internships
 * Jobs
 * Staff
 * Diversity & Inclusion

 Issues

 * Free Speech
 * Privacy
 * Creativity & Innovation
 * Transparency
 * International
 * Security

 Updates

 * Blog
 * Press Releases
 * Events
 * Legal Cases
 * Whitepapers
 * EFFector Newsletter

 Press

 * Press Contact

 Donate

 * Join or Renew Membership Online
 * One-Time Donation Online
 * Shop
 * Other Ways to Give
 * Copyright (CC BY)
 * Trademark
 * Privacy Policy
 * Thanks
Today the federal Government Accountability Office (GAO) finally published its exhaustive report on the FBI’s face recognition capabilities. The takeaway: FBI has access to hundreds of millions more photos than we ever thought. And the Bureau has been hiding this fact from the public—in flagrant violation of federal law and agency policy—for years.
According to the GAO Report, FBI’s Facial Analysis, Comparison, and Evaluation (FACE) Services unit not only has access to FBI’s Next Generation Identification (NGI) face recognition database of nearly 30 million civil and criminal mug shot photos, it also has access to the State Department’s Visa and Passport databases, the Defense Department’s biometric database, and the drivers license databases of at least 16 states. Totaling 411.9 million images, this is an unprecedented number of photographs, most of which are of Americans and foreigners who have committed no crimes.
The FBI has done little to make sure that its search results (which the Bureau calls “investigative leads”) do not include photos of innocent people, according to the report. The FBI has conducted only very limited testing to ensure the accuracy of NGI's face recognition capabilities. And it has not taken any steps to determine whether the face recognition systems of its external partners—states and other federal agencies—are sufficiently accurate to prevent innocent people from being identified as criminal suspects. As we know from previous research, face recognition is notoriously inaccurate across the board and may also misidentify African Americans and ethnic minorities, young people, and women at higher rates than whites, older people, and men, respectively.
 
As the Report points out, many of the 411.9 million face images to which FBI has access—like driver’s license and passport and visa photos—were never collected for criminal or national security purposes. And yet, under agreements we’ve never seen between the FBI and its state and federal partners, the FBI may search these civil photos whenever it’s trying to find a suspect in a crime. As the map above shows, 18 more states are in negotiations with the FBI to provide similar access to their driver’s license databases.
 
The states have been very involved in the development of the FBI’s own NGI database, which includes nearly 30 million of the 411.9 million face images accessible to the Bureau (we’ve written  extensively about NGI in the past). NGI includes more than 20 million civil and criminal images received directly from at least six states, including California, Louisiana, Michigan, New York, Texas, and Virginia. And it appears five additional states—Florida, Maryland, Maine, New Mexico, and Arkansas—can send search requests directly to the NGI database. As of December 2015, FBI is working with eight more states to grant them access to NGI, and an additional 24 states are also interested.
 
The GAO Report spends a significant number of pages criticizing FBI for rolling out these massive face recognition capabilities without ever explaining the privacy implications of its actions to the public. Federal law and Department of Justice policies require the FBI to complete a Privacy Impact Assessment (PIA) of all programs that collect data on Americans, both at the beginning of development and any time there’s significant change to the program. While the FBI produced a PIA in 2008, when it first started planning out the face recognition component of NGI, it didn’t update that PIA until late 2015—seven years later and well after it began making significant changes to the program.  It also failed to produce a PIA for the FACE Services unit until May 2015—three years after FACE began supporting FBI with face recognition searches. As GAO notes, the whole point of PIAs is to give the public notice of the privacy implications of data collection programs and to ensure that privacy protections are built into the system from the start. The FBI failed at this.
 
The single bright spot in the report reiterates that FBI decided not to allow searches of civil photos enrolled in NGI to “better protect individuals’ privacy.” This is a hollow victory, however, because if you’ve ever been arrested for any crime at all—including blocking a street as part of a public protest—your civil photos will be linked to your booking photo and subject to face recognition searches along with all the other 29.7 million images in NGI.
The GAO’s findings are especially shocking, given the timing. Just over a month ago the FBI demanded its face recognition capabilities be exempt from several key provisions of the federal Privacy Act—and provided the public with only 30 days to respond. Over and over, the FBI’s secret data collection practices confirm why we need more transparency, not less. In the coming weeks, we’ll be asking you to sign on to our comments to the FBI’s proposal. Help us send a message to the FBI that its practices are unacceptable and must change.
 
 
 
  
 Join EFF Lists
New Report: FBI Can Access Hundreds of Millions of Face Recognition Photos
https://www.eff.org/deeplinks/2016/06/fbi-can-search-400-million-face-recognition-photos
https://towardsdatascience.com/face-recognition-on-330-million-images-at-400-images-per-second-b85e594eab66
Face recognition on 330 million faces at 400 images per second | Towards Data Science
2020-08-14T06:17:05Z
Vladimir Iglovikov
<html><body><div><div><p>In 2018 every student or even every high school kid did a pet project with face recognition. The task becomes much harder if your dataset is not one million people but:</p><ul><li><strong>330</strong> million of the user accounts</li><li><strong>20</strong> million photos are uploaded every day</li><li><strong>0.2</strong> seconds is the maximum time that you can spend per image</li><li>your hardware is limited</li></ul><p>In this article, we share our experience with the development and deployment of the face recognition system in the social network <a href="https://ok.ru/">OK.Ru</a>.</p><p>In this article, we share:</p><ul><li>the math behind the algorithm</li><li>details of the implementation</li><li>results of the deploy</li><li>PR event “StarFace” that we used to promote our solution.</li></ul><h2>Problem</h2><p>There are 330 million user accounts at ok.ru. They contain 30 billion images in total.</p><p>The users of the ok.ru upload 20 million photos per day. 9 million of these images have at least one face. In total, we have 23 million faces. On average there are 2.5 faces per image.</p><p>Users could tag people at the pictures, but they do not do it. Our team decided to build a solution that will automatically tag people in the images. <br/>It is a better user experience if the user will know when someone else uploads images with her.</p><figure></figure><p>We want the user to confirm face tags in the same instant as the photo is uploaded. As a result, in the worst-case scenario, the processing should take no more than 200 milliseconds.</p><h2>The face recognition pipeline</h2><h2>Face recognition at the image</h2><p>The user uploads an image from a client: browser, mobile phone with iOS or Android. After this, the image goes through the face detector.</p><figure><figcaption>Face recognition</figcaption></figure><p>To run automatic face recognition, we need to build a facial profile of every user and fill the database.</p><h2>The creation of the facial profiles</h2><p>In general, to build a facial profile, it is enough to have only one picture. The profile picture is not always enough. Users like to put group photos or the photo of the celebrity there.</p><figure><figcaption>The hard profile</figcaption></figure><p>Let’s consider a user that has the only group pictures. We can deduce the owner of the profile if we know his gender and age and already built friend profiles.</p><figure><figcaption>The process of building user profiles</figcaption></figure><h2>To build the profile we:</h2><p><strong>Step 1: Use only high-quality pictures.</strong></p><p>If the user has a lot of high-quality photos we used the best 100. To estimate the quality of an image we checked</p><ul><li>Did the user tag any of the people?</li><li>Does the photo have meta-information about the type of the camera?</li><li>At some point, the user had this picture as a profile picture.</li></ul><p><strong>Step 2: Look for faces at the image.</strong></p><p>If there are other people in the image, it is not a big deal.</p><p><strong>Step 3: Calculate the embedding vector for every face.</strong></p><p><strong>Step 4: Perform the clustering of the embedding vectors.</strong></p><p>The goal of the clustering is to figure out which set of the embedding corresponds to the user. There are friends and relatives in the photos. We use DBScan for clustering.</p><p><strong>Step 5: Find the “main” cluster</strong></p><p>For every cluster we calculated its weight using:</p><ol><li>The size of the cluster.</li><li>Quality of the images forming the cluster.</li><li>If there were face tags on the photos from the cluster.</li><li>Match between age and gender in the user profile and at the images.</li><li>The distance between the centroid of the cluster to the clusters of the friends.</li></ol><p>We used linear regression to calculate the coefficients of the parameters.<br/>The correct calculation of age and gender is a separate task that we will cover later.</p><p>To call the cluster a “leader,” we need his weight to be larger than the closest cluster’s weight by a margin. We compute this margin on the train set. If we did not find the “leader,” we repeat the procedure but use a broader collection of images. There are users for which we had two clusters. Some families have a joined profile.</p><p><strong>Step 6: Calculate the “main” embedding of the user.</strong></p><p>The main embedding of the user is the centroid of the leading cluster. There are many ways that could be used to build a centroid. After extensive experimentation, we decided to use the simplest: an average of all embedding in the cluster.</p><p>If a user has several clusters, we compute one “main embedding” for each cluster.</p><p>We processed 8 billion images during our experiments, worked with 330 user-profiles, and built embedding for 300 million of them. On average, we used 26 pictures per user to create a profile. In general, we could use one image per person. But more images per profile used — the better was our confidence that the profile that we build corresponds to the actual user.<br/>We repeated the process of rebuilding all 300 million profiles a few times. The reason is that the information about friends’ facial profiles improves the quality of the user profile.</p><p>We processed 8 billion images during our experiments, worked with 330 user-profiles, and built embedding for 300 million of them. On average, we used 26 pictures per user to create a profile. In general, we could use one image per person. But more images per profile used — the better was our confidence that the profile that we build corresponds to the actual user.</p><p>We repeated the process of rebuilding all 300 million profiles a few times. The reason is that the information about friends’ facial profiles improves the quality of the user profile.</p><h2><strong>Face detector</strong></h2><p>The first version of the face detector OK launched in 2013. It was the solution based on the third-party provider. The algorithm was similar to the <a href="https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework">Viola-Jones</a>. In the following years, the solution became outdated — modern approaches based on <a href="https://arxiv.org/abs/1604.02878">MTCNN</a> show twice better performance. We decided to follow the trend and use the cascade of the convolutional neural networks (MTCNN).</p><p>For the old detector, we used 100 “old” CPU servers. Nearly all modern face detection algorithms use convolutional neural networks (CNN) that perform the best on GPUs. We did not have an option to buy enough GPUs: cryptocurrency miners purchased all that was available.</p><blockquote><p>The comment from the translator: The summer of 2018 was the cryptocurrency boom. GPU prices went up a few times, and even with such high rates, it was hard to buy a GPU.</p></blockquote><p>We decided to run our new detector on the CPUs that we used for the old one.</p><p>To detect faces on the images that users upload, we use the cluster of 30 machines. Face detection and the generation of the embedding (iterations on the user accounts) we perform on 1000 virtual machines with the low priority in our cloud.</p><p>(The details of our cloud solution are described in the article <a href="https://habrahabr.ru/company/odnoklassniki/blog/346868/">One-cloud — ОС уровня дата-центра в Одноклассниках</a> by <a href="https://habrahabr.ru/users/m0nstermind/">Оleg Anastasiev</a>. The material is in Russian!)</p><p>When we analyzed our face detector’s performance, we found the corner case: the first stage of the cascade allows too many candidates to the second stage. The detector started working slower.</p><p>For example, at these photos, it was taking up to 1.5 seconds per image.</p><figure><figcaption>An example of having a lot of candidates after the first stage of the cascade.</figcaption></figure><p>To fix the issue, we used the assumption that typically photo does not have too many people. We modified the pipeline to keep no more than 200 most confident candidates after the first stage. This optimization improved the worst prediction four times. From 1.5 seconds to 350 milliseconds.</p><p>There were a few more optimizations. For example, we replaced <a href="https://www.pyimagesearch.com/2014/11/17/non-maximum-suppression-object-detection-python/">Non Maximum Suppression</a> (NMS) after the first stage with <a href="https://www.learnopencv.com/blob-detection-using-opencv-python-c/">Blob Detection</a>. The modification decreased the inference time 1.4 times without the loss in quality.</p><p>We need to admit that there is a lot of progress in the field, and there are works like <a href="https://arxiv.org/abs/1708.05234">FaceBoxes</a>. It can happen that we will update our face detection solution in the future.</p><h2>Face identification (embedding extraction)</h2><p>We experimented with the <a href="https://arxiv.org/abs/1605.07146">Wide ResNet</a>, <a href="https://arxiv.org/abs/1602.07261">Inception-ResNet</a>, <a href="https://arxiv.org/abs/1511.02683">Light CNN</a> network architectures. Inception-Resnet was slightly better, and at the moment, it was our choice.</p><p>To perform face identification, we need a neural network. One option is to find a pre-trained model on the internet. The other is to train it ourselves.</p><p>To train the network you need train and validation datasets. Face identification is a well-studied task. There are datasets: MSCeleb, VGGFace/VGGFace2, MegaFace. The problem is that the generalization ability of the networks trained on them is not good.</p><p>Faces at our portal are quite different from faces that exist at the research datasets:</p><ol><li>Different age distribution. We have kids in our images.</li><li>Different ethnical distribution.</li><li>We have pictures of low quality and resolution. We have group photos.</li></ol><p>To overcome the third point, we can use image augmentation. Downsample and JpegCompression transforms will do the job.</p><blockquote><p>The comment from the translator: <a href="https://albumentations.ai/docs/api_reference/augmentations/transforms/">Albumentations</a> library has the number of transforms that can mimic the image’s low quality.</p></blockquote><p>After a few iterations, we converged on the following procedure:</p><p><strong>Step 1: Download photos from ~100k open profiles. </strong><br/>We chose the profiles randomly, minimizing the number of friend connections between them. As a result, we assume that each person will appear only in one profile.</p><p><strong>Step 2: Build face embeddings.</strong></p><p>We use <a href="https://github.com/davidsandberg/facenet">Facenet</a> pre-trained network.</p><p><strong>Step 3: Cluster faces for every account.</strong></p><ol><li>We do not know how many people will appear on the images corresponding to one account. As a result, we should not use “number_of_clusters” as an input to our clustering algorithm.</li><li>In the ideal situation, we expect to get very similar embeddings for the user forming a dense sphere. In practice, it is not the case. For example, we can have a clot for a person in glasses and a person without them. As a result, centroid based methods will not work for us. We need to use density-based.</li></ol><p>For these two reasons, we used <a href="http://scikit-learn.org/stable/modules/clustering.html#dbscan">DBScan</a>. We manually choose hyperparameters and performed visual validation, which is pretty standard. For the main hyperparameter “eps,” we introduced a heuristic based on the number of people in the account.</p><p><strong>Step 4: Cluster filtering</strong></p><p>The primary sources of the noise in the dataset and how we dealt with them:</p><ol><li>Clusters corresponding to different people merge into one cluster. It may happen due to the limitations of the Face-Id network or the density-based nature of the DBScan. The solution: if two or more faces came to the cluster from the same image, we removed this cluster. As a result, people who like selfie collages did not become parts of our dataset. The fix decreased a few times the number of false merges, so it was worth it.</li><li>A few clusters may correspond to the same person. A person in and without glasses. A person using makeup or not. We came up with the following solution: We measure the distance between clusters. If it below a threshold, we merge clusters. If not, we remove one of the clusters.</li><li>A face detector makes a mistake and detects something that is not a face. Luckily, this is easy to fix.</li></ol><p><strong>Step 5: Fine-tune the network on the data that we got and go back to step 2.</strong></p><p>Repeat 3–4 times till convergence. Gradually the network becomes better. At the last iterations, networks became so good that our heuristics stop adding value.</p><p>We decided that more diversity better and extended our newly obtained OKFace dataset (3.7 million faces, 77k people) with something else.</p><p>Our experiments showed that the most addition is <a href="http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/">VGGFace2</a>. It is large and diverse (angles and lightning). As usual, it consists of the celebrity photos found on Google. It has a large amount of noise in labels. Luckily, cleaning it with the network, trained on OKFace, is a relatively straightforward task.</p><h2>Loss function</h2><p>What is the best loss function for Embedding learning is an open question. For our work, we choose to use the loss function that is as similar to the use in production as possible.</p><p>We use cosine similarity to compare the embedding <em>“x” </em>with the centroids from the candidate embeddings (user and his friends) <em>c1, …, cn. </em>If cos(x, ci) ≥ mp, we claim that it is indeed the user at the photo.</p><p>As a result, we want:</p><ul><li>For the “correct” candidate <em>t, cos(x, ct) </em>was larger than margin value <em>mp.</em></li><li>For others, it should be smaller than <em>mp.</em></li></ul><p>We penalize the deviation from the desired result with the following loss function. This one worked the best in our experiments.</p><figure></figure><p>Centroids <em>c1, …, cn</em> are the network’s parameters and are learned during training with the gradient descent.</p><p>This loss function has its problems. First, it does not converge well when you train it from scratch. Second, tuning two parameters “<em>mp</em>” and “<em>mn</em>” is cumbersome. Nevertheless, fine-tuning with this loss function allowed us to get better performance than with other functions: <a href="https://www.researchgate.net/publication/308190438_A_Discriminative_Feature_Learning_Approach_for_Deep_Face_Recognition">Center Loss</a>, <a href="https://arxiv.org/abs/1707.07391">Contrastive-Center Loss</a>, <a href="https://www.semanticscholar.org/paper/SphereFace%3A-Deep-Hypersphere-Embedding-for-Face-Re-Liu-Wen/366adbacaeea8c35435bf41134b78bb7aba73315">A-Softmax (SphereFace)</a>, <a href="https://arxiv.org/abs/1801.09414">LMCL (CosFace)</a>.</p><h2>Was it worth it?</h2><figure><figcaption>Numbers are an average of 10 measurements and standard deviation.</figcaption></figure><p>An important metric for us is TP@FP: The percent of people that we identify correctly when the number of false positives is fixed. (0.1% in the table)</p><p>With the limit of one mistake per 1000 and without fine-tuning we were able to identify only half of the people correctly.</p><h2>The minimization of false face detections</h2><p>Sometimes face detector finds human faces in the places where there are no faces exist. (4% of the detections).</p><p>It is undesirable to get such a “noise” to the train set. It is a bad user experience when we ask users “to put the name” on the bouquet of roses or in the middle of the carpet.</p><p>The most straightforward fix is to add to the train set for a face detector, a collection of images that do not contain human faces.</p><figure></figure><p>We decided to solve the problem differently.</p><ol><li>Collect a few images that do not contain faces from the internet.</li><li>Create random crops, build embeddings, and cluster them. We got 14 clusters.</li><li>If the face’s embedding is close to the centroid of the “nonface” cluster, we conclude that the proposed face s not face.</li><li>Celebrate our method.</li></ol><p>The described schema could be implemented as a two-layer network with 14 hidden units on top of the embeddings.</p><p>Curiously, the face id network sends all variety of the nonface images only to a few regions of the embedding space.</p><h2>Everyone lies or how to find the real age and gender of the person in a social network</h2><p>Pretty often, users do not show their age or write an incorrect number. To deduce the correct value of the user, we use the graph of friends. In general, the age of the user lies in the largest cluster of his friends&#39; ages. The gender we deduced from the first and last names.</p><p>(<a href="https://habrahabr.ru/users/khud/">Vitaly Hudobashov</a> talks about the details in his lecture <a href="https://ok.ru/video/313332866731">Как узнать возраст человека в социальной сети, даже если он не указан</a>. The lecture is in Russian.)</p><p>All the internal infrastructure at OK.Ru is in Java. Hence all the components of our pipeline are wrapped in Java. Face detector and Face recognizer perform inference in TensorFlow with Java API. Face Detector works at CPU. It is fast enough and works well on the existing hardware. For the recognizer, we installed 72 GPUs. It is more efficient to run Inception Resnet on GPU rather than CPU.</p><p>To store the embedding, we use Cassandra. The total volume of the embeddings for all our users ~300Gb, hence for the fast access, we add cache. We implement the cache as an off-heap.</p><p>(The details are described in the article by <a href="https://habrahabr.ru/users/apangin/">Andrey Pangin</a>: <a href="https://habrahabr.ru/company/odnoklassniki/blog/148139/">Использование разделяемой памяти в Java и off-heap кеширование</a>. In Russian)</p><p>The pipeline that we built can work with up to 1 billion photos per day for iteration. In parallel, we process images that users upload. ~20 million per day.</p><figure></figure><h2>Results</h2><p>We implemented a system that is trained on the real data from the social network. It was trained with limited resources and showed good results.</p><p>The quality of the recognition on the dataset built on the real OK.ru data was TP=97.5%, with the FP=0.1%. The average time to process an image is 120 ms, and the 99th percentile is below 200ms. The system is self-improving. More images the use tags, better the face embedding profile of the user.</p><p>Now, after the upload, users that we found at the image get a notification. They can confirm the proposed automatic label or can delete it if they do not like the picture.</p><figure></figure><p>Automatic recognition twofold increased the number of views in the feed with updates, and the number of clicks increased threefold. The interest of the users’ to the new feature is apparent. We plan to improve it further with a better UI and new applications like Starface.</p><p><a href="https://ok.ru/video/712314849816">Video about UX. (In Russian)</a></p><h2>Flashmob StarFace</h2><p>To introduce the new functionality to users, OK.ru started a challenge. Users upload their images with the stars of Russian sport, business, media, famous bloggers, journalists who have active accounts in OK.ru. Users get a badge to the profile picture or a subscription to the paid-only service. (<a href="https://insideok.ru/blog/odnoklassniki-zapustili-raspoznavanie-lic-na-foto-na-osnove-neyrosetey">Details</a>. In Russian.)</p><p>Users uploaded more than 10,000 pictures with celebrities in the first days of the flashmob. Users uploaded a selfie with stars, photos near the posters, and photoshopped images.</p><figure></figure><h2>Plans</h2><p>The most time-consuming part of the pipeline is the detector. The next step is to replace it or move to the GPU.</p><p>We can try to combine different face recognition model. If it boosts the accuracy of the model, it is worth exploring.</p><p>The next step from the user perspective would be the recognition of the people in the video. We are also planning to inform users about their “clone profiles.”</p><p>Feel free to add ideas for the applications of our face recognition system in the comments at “<a href="https://habr.com/ru/company/odnoklassniki/blog/350566/">Щи, или Распознавание 330 млн лиц на скорости 400 фото / сек</a>”.</p></div></div></body></html>
In 2018 every student or even every high school kid did a pet project with face recognition. The task becomes much harder if your dataset is not one million people but:   330 million of the user accounts  20 million photos are uploaded every day  0.2 seconds is the maximum time that you can spend per image your hardware is limited  In this article, we share our experience with the development and deployment of the face recognition system in the social network OK.Ru. In this article, we share:  the math behind the algorithm details of the implementation results of the deploy PR event “StarFace” that we used to promote our solution.  Problem There are 330 million user accounts at ok.ru. They contain 30 billion images in total. The users of the ok.ru upload 20 million photos per day. 9 million of these images have at least one face. In total, we have 23 million faces. On average there are 2.5 faces per image. Users could tag people at the pictures, but they do not do it. Our team decided to build a solution that will automatically tag people in the images. It is a better user experience if the user will know when someone else uploads images with her.   We want the user to confirm face tags in the same instant as the photo is uploaded. As a result, in the worst-case scenario, the processing should take no more than 200 milliseconds. The face recognition pipeline Face recognition at the image The user uploads an image from a client: browser, mobile phone with iOS or Android. After this, the image goes through the face detector.  Face recognition  To run automatic face recognition, we need to build a facial profile of every user and fill the database. The creation of the facial profiles In general, to build a facial profile, it is enough to have only one picture. The profile picture is not always enough. Users like to put group photos or the photo of the celebrity there.  The hard profile  Let’s consider a user that has the only group pictures. We can deduce the owner of the profile if we know his gender and age and already built friend profiles.  The process of building user profiles  To build the profile we:  Step 1: Use only high-quality pictures.  If the user has a lot of high-quality photos we used the best 100. To estimate the quality of an image we checked  Did the user tag any of the people? Does the photo have meta-information about the type of the camera? At some point, the user had this picture as a profile picture.   Step 2: Look for faces at the image.  If there are other people in the image, it is not a big deal.  Step 3: Calculate the embedding vector for every face.   Step 4: Perform the clustering of the embedding vectors.  The goal of the clustering is to figure out which set of the embedding corresponds to the user. There are friends and relatives in the photos. We use DBScan for clustering.  Step 5: Find the “main” cluster  For every cluster we calculated its weight using:  The size of the cluster. Quality of the images forming the cluster. If there were face tags on the photos from the cluster. Match between age and gender in the user profile and at the images. The distance between the centroid of the cluster to the clusters of the friends.  We used linear regression to calculate the coefficients of the parameters.The correct calculation of age and gender is a separate task that we will cover later. To call the cluster a “leader,” we need his weight to be larger than the closest cluster’s weight by a margin. We compute this margin on the train set. If we did not find the “leader,” we repeat the procedure but use a broader collection of images. There are users for which we had two clusters. Some families have a joined profile.  Step 6: Calculate the “main” embedding of the user.  The main embedding of the user is the centroid of the leading cluster. There are many ways that could be used to build a centroid. After extensive experimentation, we decided to use the simplest: an average of all embedding in the cluster. If a user has several clusters, we compute one “main embedding” for each cluster. We processed 8 billion images during our experiments, worked with 330 user-profiles, and built embedding for 300 million of them. On average, we used 26 pictures per user to create a profile. In general, we could use one image per person. But more images per profile used — the better was our confidence that the profile that we build corresponds to the actual user.We repeated the process of rebuilding all 300 million profiles a few times. The reason is that the information about friends’ facial profiles improves the quality of the user profile. We processed 8 billion images during our experiments, worked with 330 user-profiles, and built embedding for 300 million of them. On average, we used 26 pictures per user to create a profile. In general, we could use one image per person. But more images per profile used — the better was our confidence that the profile that we build corresponds to the actual user. We repeated the process of rebuilding all 300 million profiles a few times. The reason is that the information about friends’ facial profiles improves the quality of the user profile.  Face detector  The first version of the face detector OK launched in 2013. It was the solution based on the third-party provider. The algorithm was similar to the Viola-Jones. In the following years, the solution became outdated — modern approaches based on MTCNN show twice better performance. We decided to follow the trend and use the cascade of the convolutional neural networks (MTCNN). For the old detector, we used 100 “old” CPU servers. Nearly all modern face detection algorithms use convolutional neural networks (CNN) that perform the best on GPUs. We did not have an option to buy enough GPUs: cryptocurrency miners purchased all that was available.  The comment from the translator: The summer of 2018 was the cryptocurrency boom. GPU prices went up a few times, and even with such high rates, it was hard to buy a GPU.  We decided to run our new detector on the CPUs that we used for the old one. To detect faces on the images that users upload, we use the cluster of 30 machines. Face detection and the generation of the embedding (iterations on the user accounts) we perform on 1000 virtual machines with the low priority in our cloud. (The details of our cloud solution are described in the article One-cloud — ОС уровня дата-центра в Одноклассниках by Оleg Anastasiev. The material is in Russian!) When we analyzed our face detector’s performance, we found the corner case: the first stage of the cascade allows too many candidates to the second stage. The detector started working slower. For example, at these photos, it was taking up to 1.5 seconds per image.  An example of having a lot of candidates after the first stage of the cascade.  To fix the issue, we used the assumption that typically photo does not have too many people. We modified the pipeline to keep no more than 200 most confident candidates after the first stage. This optimization improved the worst prediction four times. From 1.5 seconds to 350 milliseconds. There were a few more optimizations. For example, we replaced Non Maximum Suppression (NMS) after the first stage with Blob Detection. The modification decreased the inference time 1.4 times without the loss in quality. We need to admit that there is a lot of progress in the field, and there are works like FaceBoxes. It can happen that we will update our face detection solution in the future. Face identification (embedding extraction) We experimented with the Wide ResNet, Inception-ResNet, Light CNN network architectures. Inception-Resnet was slightly better, and at the moment, it was our choice. To perform face identification, we need a neural network. One option is to find a pre-trained model on the internet. The other is to train it ourselves. To train the network you need train and validation datasets. Face identification is a well-studied task. There are datasets: MSCeleb, VGGFace/VGGFace2, MegaFace. The problem is that the generalization ability of the networks trained on them is not good. Faces at our portal are quite different from faces that exist at the research datasets:  Different age distribution. We have kids in our images. Different ethnical distribution. We have pictures of low quality and resolution. We have group photos.  To overcome the third point, we can use image augmentation. Downsample and JpegCompression transforms will do the job.  The comment from the translator: Albumentations library has the number of transforms that can mimic the image’s low quality.  After a few iterations, we converged on the following procedure:  Step 1: Download photos from ~100k open profiles.  We chose the profiles randomly, minimizing the number of friend connections between them. As a result, we assume that each person will appear only in one profile.  Step 2: Build face embeddings.  We use Facenet pre-trained network.  Step 3: Cluster faces for every account.   We do not know how many people will appear on the images corresponding to one account. As a result, we should not use “number_of_clusters” as an input to our clustering algorithm. In the ideal situation, we expect to get very similar embeddings for the user forming a dense sphere. In practice, it is not the case. For example, we can have a clot for a person in glasses and a person without them. As a result, centroid based methods will not work for us. We need to use density-based.  For these two reasons, we used DBScan. We manually choose hyperparameters and performed visual validation, which is pretty standard. For the main hyperparameter “eps,” we introduced a heuristic based on the number of people in the account.  Step 4: Cluster filtering  The primary sources of the noise in the dataset and how we dealt with them:  Clusters corresponding to different people merge into one cluster. It may happen due to the limitations of the Face-Id network or the density-based nature of the DBScan. The solution: if two or more faces came to the cluster from the same image, we removed this cluster. As a result, people who like selfie collages did not become parts of our dataset. The fix decreased a few times the number of false merges, so it was worth it. A few clusters may correspond to the same person. A person in and without glasses. A person using makeup or not. We came up with the following solution: We measure the distance between clusters. If it below a threshold, we merge clusters. If not, we remove one of the clusters. A face detector makes a mistake and detects something that is not a face. Luckily, this is easy to fix.   Step 5: Fine-tune the network on the data that we got and go back to step 2.  Repeat 3–4 times till convergence. Gradually the network becomes better. At the last iterations, networks became so good that our heuristics stop adding value. We decided that more diversity better and extended our newly obtained OKFace dataset (3.7 million faces, 77k people) with something else. Our experiments showed that the most addition is VGGFace2. It is large and diverse (angles and lightning). As usual, it consists of the celebrity photos found on Google. It has a large amount of noise in labels. Luckily, cleaning it with the network, trained on OKFace, is a relatively straightforward task. Loss function What is the best loss function for Embedding learning is an open question. For our work, we choose to use the loss function that is as similar to the use in production as possible. We use cosine similarity to compare the embedding “x” with the centroids from the candidate embeddings (user and his friends) c1, …, cn. If cos(x, ci) ≥ mp, we claim that it is indeed the user at the photo. As a result, we want:  For the “correct” candidate t, cos(x, ct) was larger than margin value mp.  For others, it should be smaller than mp.   We penalize the deviation from the desired result with the following loss function. This one worked the best in our experiments.   Centroids c1, …, cn are the network’s parameters and are learned during training with the gradient descent. This loss function has its problems. First, it does not converge well when you train it from scratch. Second, tuning two parameters “mp” and “mn” is cumbersome. Nevertheless, fine-tuning with this loss function allowed us to get better performance than with other functions: Center Loss, Contrastive-Center Loss, A-Softmax (SphereFace), LMCL (CosFace). Was it worth it?  Numbers are an average of 10 measurements and standard deviation.  An important metric for us is TP@FP: The percent of people that we identify correctly when the number of false positives is fixed. (0.1% in the table) With the limit of one mistake per 1000 and without fine-tuning we were able to identify only half of the people correctly. The minimization of false face detections Sometimes face detector finds human faces in the places where there are no faces exist. (4% of the detections). It is undesirable to get such a “noise” to the train set. It is a bad user experience when we ask users “to put the name” on the bouquet of roses or in the middle of the carpet. The most straightforward fix is to add to the train set for a face detector, a collection of images that do not contain human faces.   We decided to solve the problem differently.  Collect a few images that do not contain faces from the internet. Create random crops, build embeddings, and cluster them. We got 14 clusters. If the face’s embedding is close to the centroid of the “nonface” cluster, we conclude that the proposed face s not face. Celebrate our method.  The described schema could be implemented as a two-layer network with 14 hidden units on top of the embeddings. Curiously, the face id network sends all variety of the nonface images only to a few regions of the embedding space. Everyone lies or how to find the real age and gender of the person in a social network Pretty often, users do not show their age or write an incorrect number. To deduce the correct value of the user, we use the graph of friends. In general, the age of the user lies in the largest cluster of his friends&#39; ages. The gender we deduced from the first and last names. (Vitaly Hudobashov talks about the details in his lecture Как узнать возраст человека в социальной сети, даже если он не указан. The lecture is in Russian.) All the internal infrastructure at OK.Ru is in Java. Hence all the components of our pipeline are wrapped in Java. Face detector and Face recognizer perform inference in TensorFlow with Java API. Face Detector works at CPU. It is fast enough and works well on the existing hardware. For the recognizer, we installed 72 GPUs. It is more efficient to run Inception Resnet on GPU rather than CPU. To store the embedding, we use Cassandra. The total volume of the embeddings for all our users ~300Gb, hence for the fast access, we add cache. We implement the cache as an off-heap. (The details are described in the article by Andrey Pangin: Использование разделяемой памяти в Java и off-heap кеширование. In Russian) The pipeline that we built can work with up to 1 billion photos per day for iteration. In parallel, we process images that users upload. ~20 million per day.   Results We implemented a system that is trained on the real data from the social network. It was trained with limited resources and showed good results. The quality of the recognition on the dataset built on the real OK.ru data was TP=97.5%, with the FP=0.1%. The average time to process an image is 120 ms, and the 99th percentile is below 200ms. The system is self-improving. More images the use tags, better the face embedding profile of the user. Now, after the upload, users that we found at the image get a notification. They can confirm the proposed automatic label or can delete it if they do not like the picture.   Automatic recognition twofold increased the number of views in the feed with updates, and the number of clicks increased threefold. The interest of the users’ to the new feature is apparent. We plan to improve it further with a better UI and new applications like Starface.  Video about UX. (In Russian)  Flashmob StarFace To introduce the new functionality to users, OK.ru started a challenge. Users upload their images with the stars of Russian sport, business, media, famous bloggers, journalists who have active accounts in OK.ru. Users get a badge to the profile picture or a subscription to the paid-only service. (Details. In Russian.) Users uploaded more than 10,000 pictures with celebrities in the first days of the flashmob. Users uploaded a selfie with stars, photos near the posters, and photoshopped images.   Plans The most time-consuming part of the pipeline is the detector. The next step is to replace it or move to the GPU. We can try to combine different face recognition model. If it boosts the accuracy of the model, it is worth exploring. The next step from the user perspective would be the recognition of the people in the video. We are also planning to inform users about their “clone profiles.” Feel free to add ideas for the applications of our face recognition system in the comments at “Щи, или Распознавание 330 млн лиц на скорости 400 фото / сек”.
Face recognition on 330 million faces at 400 images per second | Towards Data Science
https://towardsdatascience.com/face-recognition-on-330-million-images-at-400-images-per-second-b85e594eab66
https://www.cs.toronto.edu/~kriz/cifar.html
The CIFAR-10 dataset
2009-01-01
None
<div>
<p><a href="index.html">&lt; Back to Alex Krizhevsky's home page</a></p>
The CIFAR-10 and CIFAR-100 are labeled subsets of the <a href="http://people.csail.mit.edu/torralba/tinyimages/">80 million tiny images</a> dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
<h2>The CIFAR-10 dataset</h2>
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
<p>
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
</p><p>
Here are the classes in the dataset, as well as 10 random images from each:
</p><table>
 <tbody><tr>
 <td>airplane</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>automobile</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>bird</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>cat</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>deer</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>dog</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>frog</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>horse</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>ship</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td>truck</td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 <td></td>
 </tr>
</tbody></table>
<br />
The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. "Automobile" includes sedans, SUVs, things of that sort. "Truck" includes only big trucks. Neither includes pickup trucks.
<h3>Download</h3>
If you're going to use this dataset, please cite the tech report at the bottom of this page. <br />
<table>
 <tbody><tr>
 <td>Version</td>
 <td>Size</td>
 <td>md5sum</td>
 </tr>
 <tr>
 <td><a href="cifar-10-python.tar.gz">CIFAR-10 python version</a></td>
 <td>163 MB</td>
 <td>c58f30108f718f92721af3b95e74349a</td>
 </tr>
 <tr>
 <td><a href="cifar-10-matlab.tar.gz">CIFAR-10 Matlab version</a></td>
 <td>175 MB</td>
 <td>70270af85842c9e89bb428ec9976c926</td>
 </tr>
 <tr>
 <td><a href="cifar-10-binary.tar.gz">CIFAR-10 binary version (suitable for C programs)</a></td>
 <td>162 MB</td>
 <td>c32a1d4ab5d03f1284b67883e8d87530</td>
 </tr>
</tbody></table> 
<h3>Baseline results</h3>
You can find some baseline replicable results on this dataset <a href="http://code.google.com/p/cuda-convnet/">on the project page for cuda-convnet</a>. These results were obtained with a convolutional neural network. Briefly, they are 18% test error without data augmentation and 11% with. Additionally, <a href="http://www.cs.toronto.edu/~jasper/">Jasper Snoek</a> has a <a href="http://hips.seas.harvard.edu/content/practical-bayesian-optimization-machine-learning-algorithms">new paper</a> in which he used Bayesian hyperparameter optimization to find nice settings of the weight decay and other hyperparameters, which allowed him to obtain a test error rate of 15% (without data augmentation) using the architecture of the net that got 18%.
<h3>Other results</h3>
<a href="http://rodrigob.github.com/">Rodrigo Benenson</a> has been kind enough to collect results on CIFAR-10/100 and other datasets on his website; <a href="http://rodrigob.github.com/are_we_there_yet/build/classification_datasets_results.html">click here</a> to view.
<h3>Dataset layout</h3>
<h4>Python / Matlab versions</h4>
I will describe the layout of the Python version of the dataset. The layout of the Matlab version is identical.
<p>
The archive contains the files data_batch_1, data_batch_2, ..., data_batch_5, as well as test_batch. Each of these files is a Python "pickled" object produced with <a href="http://www.python.org/doc/2.5/lib/module-cPickle.html">cPickle</a>. Here is a python2 routine which will open such a file and return a dictionary:
</p><pre>def unpickle(file):
 import cPickle
 with open(file, 'rb') as fo:
 dict = cPickle.load(fo)
 return dict
</pre>
And a python3 version:
<pre>def unpickle(file):
 import pickle
 with open(file, 'rb') as fo:
 dict = pickle.load(fo, encoding='bytes')
 return dict
</pre>
Loaded in this way, each of the batch files contains a dictionary with the following elements:
<ul>
 <li><b>data</b> -- a 10000x3072 <a href="http://numpy.scipy.org/">numpy</a> array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
 </li><li><b>labels</b> -- a list of 10000 numbers in the range 0-9. The number at index <i>i</i> indicates the label of the <i>i</i>th image in the array <b>data</b>.</li>
</ul>
<br />
The dataset contains another file, called batches.meta. It too contains a Python dictionary object. It has the following entries:
<ul>
 <li><b>label_names</b> -- a 10-element list which gives meaningful names to the numeric labels in the <b>labels</b> array described above. For example, label_names[0] == "airplane", label_names[1] == "automobile", etc.</li>
</ul> 
<h4>Binary version</h4>
The binary version contains the files data_batch_1.bin, data_batch_2.bin, ..., data_batch_5.bin, as well as test_batch.bin. Each of these files is formatted as follows:
<pre>&lt;1 x label&gt;&lt;3072 x pixel&gt;
...
&lt;1 x label&gt;&lt;3072 x pixel&gt;</pre>
In other words, the first byte is the label of the first image, which is a number in the range 0-9. The next 3072 bytes are the values of the pixels of the image. The first 1024 bytes are the red channel values, the next 1024 the green, and the final 1024 the blue. The values are stored in row-major order, so the first 32 bytes are the red channel values of the first row of the image.
<p>
Each file contains 10000 such 3073-byte "rows" of images, although there is <b>nothing delimiting the rows</b>. Therefore each file should be exactly 30730000 bytes long.
</p><p>
There is another file, called batches.meta.txt. This is an ASCII file that maps numeric labels in the range 0-9 to meaningful class names. It is merely a list of the 10 class names, one per row. The class name on row <i>i</i> corresponds to numeric label <i>i</i>.
</p><h2>The CIFAR-100 dataset</h2>
This dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.
The 100 classes in the CIFAR-100 are grouped into 20 superclasses. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs).<br />
Here is the list of classes in the CIFAR-100:<table>
 <tbody><tr>
 <td>Superclass</td>
 <td>Classes</td>
 </tr>
 <tr>
 <td>aquatic mammals</td>
 <td>beaver, dolphin, otter, seal, whale</td>
 </tr>
 <tr>
 <td>fish</td>
 <td>aquarium fish, flatfish, ray, shark, trout</td>
 </tr>
 <tr>
 <td>flowers</td>
 <td>orchids, poppies, roses, sunflowers, tulips</td>
 </tr>
 <tr>
 <td>food containers</td>
 <td>bottles, bowls, cans, cups, plates</td>
 </tr>
 <tr>
 <td>fruit and vegetables</td>
 <td>apples, mushrooms, oranges, pears, sweet peppers</td>
 </tr>
 <tr>
 <td>household electrical devices</td>
 <td>clock, computer keyboard, lamp, telephone, television</td>
 </tr>
 <tr>
 <td>household furniture</td>
 <td>bed, chair, couch, table, wardrobe</td>
 </tr>
 <tr>
 <td>insects</td>
 <td>bee, beetle, butterfly, caterpillar, cockroach</td>
 </tr>
 <tr>
 <td>large carnivores</td>
 <td>bear, leopard, lion, tiger, wolf</td>
 </tr>
 <tr>
 <td>large man-made outdoor things</td>
 <td>bridge, castle, house, road, skyscraper</td>
 </tr>
 <tr>
 <td>large natural outdoor scenes</td>
 <td>cloud, forest, mountain, plain, sea</td>
 </tr>
 <tr>
 <td>large omnivores and herbivores</td>
 <td>camel, cattle, chimpanzee, elephant, kangaroo</td>
 </tr>
 <tr>
 <td>medium-sized mammals</td>
 <td>fox, porcupine, possum, raccoon, skunk</td>
 </tr>
 <tr>
 <td>non-insect invertebrates</td>
 <td>crab, lobster, snail, spider, worm</td>
 </tr>
 <tr>
 <td>people</td>
 <td>baby, boy, girl, man, woman</td>
 </tr>
 <tr>
 <td>reptiles</td>
 <td>crocodile, dinosaur, lizard, snake, turtle</td>
 </tr>
 <tr>
 <td>small mammals</td>
 <td>hamster, mouse, rabbit, shrew, squirrel</td>
 </tr>
 <tr>
 <td>trees</td>
 <td>maple, oak, palm, pine, willow</td>
 </tr>
 <tr>
 <td>vehicles 1</td>
 <td>bicycle, bus, motorcycle, pickup truck, train</td>
 </tr>
 <tr>
 <td>vehicles 2</td>
 <td>lawn-mower, rocket, streetcar, tank, tractor</td>
 </tr>
</tbody></table>
<br />
Yes, I know mushrooms aren't really fruit or vegetables and bears aren't really carnivores.
<br />
<h3>Download</h3>
<table>
 <tbody><tr>
 <td>Version</td>
 <td>Size</td>
 <td>md5sum</td>
 </tr>
 <tr>
 <td><a href="cifar-100-python.tar.gz">CIFAR-100 python version</a></td>
 <td>161 MB</td>
 <td>eb9058c3a382ffc7106e4002c42a8d85</td>
 </tr>
 <tr>
 <td><a href="cifar-100-matlab.tar.gz">CIFAR-100 Matlab version</a></td>
 <td>175 MB</td>
 <td>6a4bfa1dcd5c9453dda6bb54194911f4</td>
 </tr>
 <tr>
 <td><a href="cifar-100-binary.tar.gz">CIFAR-100 binary version (suitable for C programs)</a></td>
 <td>161 MB</td>
 <td>03b5dce01913d631647c71ecec9e9cb8</td>
 </tr>
</tbody></table> 
<h3>Dataset layout</h3>
<h4>Python / Matlab versions</h4>
The python and Matlab versions are identical in layout to the CIFAR-10, so I won't waste space describing them here.
<h4>Binary version</h4>
The binary version of the CIFAR-100 is just like the binary version of the CIFAR-10, except that each image has two label bytes (coarse and fine) and 3072 pixel bytes, so the binary files look like this: 
<pre>&lt;1 x coarse label&gt;&lt;1 x fine label&gt;&lt;3072 x pixel&gt;
...
&lt;1 x coarse label&gt;&lt;1 x fine label&gt;&lt;3072 x pixel&gt;</pre>
<h3>Indices into the original 80 million tiny images dataset</h3>
Sivan Sabato was kind enough to provide <a href="cifar_indexes">this file</a>, which maps CIFAR-100 images to images in the 80 million tiny images dataset. Sivan Writes:
<pre>The file has 60000 rows, each row contains a single index into the tiny db,
where the first image in the tiny db is indexed "1". "0" stands for an image that is not from the tiny db.
The first 50000 lines correspond to the training set, and the last 10000 lines correspond
to the test set.
</pre>
<h3>Reference</h3>
This tech report (Chapter 3) describes the dataset and the methodology followed when collecting it in much greater detail. Please cite it if you intend to use this dataset.
<br />
<ul>
 <li><a href="learning-features-2009-TR.pdf">Learning Multiple Layers of Features from Tiny Images</a>, Alex Krizhevsky, 2009.</li>
</ul>
||||I||||< Back to Alex Krizhevsky's home page

The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.

The CIFAR-10 dataset

The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

Here are the classes in the dataset, as well as 10 random images from each: airplane 
automobile 
bird 
cat 
deer 
dog 
frog 
horse 
ship 
truck 


The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. "Automobile" includes sedans, SUVs, things of that sort. "Truck" includes only big trucks. Neither includes pickup trucks.

Download

If you're going to use this dataset, please cite the tech report at the bottom of this page.
Version Size md5sum 
CIFAR-10 python version 163 MB c58f30108f718f92721af3b95e74349a
CIFAR-10 Matlab version 175 MB 70270af85842c9e89bb428ec9976c926
CIFAR-10 binary version (suitable for C programs) 162 MB c32a1d4ab5d03f1284b67883e8d87530


Baseline results

You can find some baseline replicable results on this dataset on the project page for cuda-convnet. These results were obtained with a convolutional neural network. Briefly, they are 18% test error without data augmentation and 11% with. Additionally, Jasper Snoek has a new paper in which he used Bayesian hyperparameter optimization to find nice settings of the weight decay and other hyperparameters, which allowed him to obtain a test error rate of 15% (without data augmentation) using the architecture of the net that got 18%.

Other results

Rodrigo Benenson has been kind enough to collect results on CIFAR-10/100 and other datasets on his website; click here to view.

Dataset layout

Python / Matlab versions

I will describe the layout of the Python version of the dataset. The layout of the Matlab version is identical.

The archive contains the files data_batch_1 , data_batch_2 , ..., data_batch_5 , as well as test_batch . Each of these files is a Python "pickled" object produced with cPickle. Here is a python2 routine which will open such a file and return a dictionary:

def unpickle(file):
 import cPickle
 with open(file, 'rb') as fo:
 dict = cPickle.load(fo)
 return dict

And a python3 version:

def unpickle(file):
 import pickle
 with open(file, 'rb') as fo:
 dict = pickle.load(fo, encoding='bytes')
 return dict

Loaded in this way, each of the batch files contains a dictionary with the following elements:
 * data -- a 10000x3072 numpy array of uint8 s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
 * labels -- a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data.

The dataset contains another file, called batches.meta . It too contains a Python dictionary object. It has the following entries:
 * label_names -- a 10-element list which gives meaningful names to the numeric labels in the labels array described above. For example, label_names[0] == "airplane" , label_names[1] == "automobile" , etc.

Binary version

The binary version contains the files data_batch_1.bin , data_batch_2.bin , ..., data_batch_5.bin , as well as test_batch.bin . Each of these files is formatted as follows:

<1 x label><3072 x pixel>
...
<1 x label><3072 x pixel>
In other words, the first byte is the label of the first image, which is a number in the range 0-9. The next 3072 bytes are the values of the pixels of the image. The first 1024 bytes are the red channel values, the next 1024 the green, and the final 1024 the blue. The values are stored in row-major order, so the first 32 bytes are the red channel values of the first row of the image.

Each file contains 10000 such 3073-byte "rows" of images, although there is nothing delimiting the rows. Therefore each file should be exactly 30730000 bytes long.

There is another file, called batches.meta.txt . This is an ASCII file that maps numeric labels in the range 0-9 to meaningful class names. It is merely a list of the 10 class names, one per row. The class name on row i corresponds to numeric label i.

The CIFAR-100 dataset

This dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. The 100 classes in the CIFAR-100 are grouped into 20 superclasses. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs).
Here is the list of classes in the CIFAR-100:

Superclass Classes 
aquatic mammals beaver, dolphin, otter, seal, whale 
fish aquarium fish, flatfish, ray, shark, trout 
flowers orchids, poppies, roses, sunflowers, tulips 
food containers bottles, bowls, cans, cups, plates 
fruit and vegetables apples, mushrooms, oranges, pears, sweet peppers 
household electrical devices clock, computer keyboard, lamp, telephone, television
household furniture bed, chair, couch, table, wardrobe 
insects bee, beetle, butterfly, caterpillar, cockroach 
large carnivores bear, leopard, lion, tiger, wolf 
large man-made outdoor things bridge, castle, house, road, skyscraper 
large natural outdoor scenes cloud, forest, mountain, plain, sea 
large omnivores and herbivores camel, cattle, chimpanzee, elephant, kangaroo 
medium-sized mammals fox, porcupine, possum, raccoon, skunk 
non-insect invertebrates crab, lobster, snail, spider, worm 
people baby, boy, girl, man, woman 
reptiles crocodile, dinosaur, lizard, snake, turtle 
small mammals hamster, mouse, rabbit, shrew, squirrel 
trees maple, oak, palm, pine, willow 
vehicles 1 bicycle, bus, motorcycle, pickup truck, train 
vehicles 2 lawn-mower, rocket, streetcar, tank, tractor 


Yes, I know mushrooms aren't really fruit or vegetables and bears aren't really carnivores.

Download

Version Size md5sum 
CIFAR-100 python version 161 MB eb9058c3a382ffc7106e4002c42a8d85
CIFAR-100 Matlab version 175 MB 6a4bfa1dcd5c9453dda6bb54194911f4
CIFAR-100 binary version (suitable for C programs) 161 MB 03b5dce01913d631647c71ecec9e9cb8


Dataset layout

Python / Matlab versions

The python and Matlab versions are identical in layout to the CIFAR-10, so I won't waste space describing them here.

Binary version

The binary version of the CIFAR-100 is just like the binary version of the CIFAR-10, except that each image has two label bytes (coarse and fine) and 3072 pixel bytes, so the binary files look like this:

<1 x coarse label><1 x fine label><3072 x pixel>
...
<1 x coarse label><1 x fine label><3072 x pixel>

Indices into the original 80 million tiny images dataset

Sivan Sabato was kind enough to provide this file, which maps CIFAR-100 images to images in the 80 million tiny images dataset. Sivan Writes:

The file has 60000 rows, each row contains a single index into the tiny db,
where the first image in the tiny db is indexed "1". "0" stands for an image that is not from the tiny db.
The first 50000 lines correspond to the training set, and the last 10000 lines correspond
to the test set.


Reference

This tech report (Chapter 3) describes the dataset and the methodology followed when collecting it in much greater detail. Please cite it if you intend to use this dataset.
 * Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009.
&lt; Back to Alex Krizhevsky's home page 
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
The CIFAR-10 dataset
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
 
Here are the classes in the dataset, as well as 10 random images from each:
 
  
 airplane
  
  
  
  
  
  
  
  
  
  
 
 
 automobile
  
  
  
  
  
  
  
  
  
  
 
 
 bird
  
  
  
  
  
  
  
  
  
  
 
 
 cat
  
  
  
  
  
  
  
  
  
  
 
 
 deer
  
  
  
  
  
  
  
  
  
  
 
 
 dog
  
  
  
  
  
  
  
  
  
  
 
 
 frog
  
  
  
  
  
  
  
  
  
  
 
 
 horse
  
  
  
  
  
  
  
  
  
  
 
 
 ship
  
  
  
  
  
  
  
  
  
  
 
 
 truck
  
  
  
  
  
  
  
  
  
  
 
 
The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. "Automobile" includes sedans, SUVs, things of that sort. "Truck" includes only big trucks. Neither includes pickup trucks.
Download
If you're going to use this dataset, please cite the tech report at the bottom of this page. 
  
 Version
 Size
 md5sum
 
 
  CIFAR-10 python version 
 163 MB
 c58f30108f718f92721af3b95e74349a
 
 
  CIFAR-10 Matlab version 
 175 MB
 70270af85842c9e89bb428ec9976c926
 
 
  CIFAR-10 binary version (suitable for C programs) 
 162 MB
 c32a1d4ab5d03f1284b67883e8d87530
 
  
Baseline results
You can find some baseline replicable results on this dataset on the project page for cuda-convnet. These results were obtained with a convolutional neural network. Briefly, they are 18% test error without data augmentation and 11% with. Additionally, Jasper Snoek has a new paper in which he used Bayesian hyperparameter optimization to find nice settings of the weight decay and other hyperparameters, which allowed him to obtain a test error rate of 15% (without data augmentation) using the architecture of the net that got 18%.
Other results
Rodrigo Benenson has been kind enough to collect results on CIFAR-10/100 and other datasets on his website; click here to view.
Dataset layout
Python / Matlab versions
I will describe the layout of the Python version of the dataset. The layout of the Matlab version is identical.
The archive contains the files data_batch_1, data_batch_2, ..., data_batch_5, as well as test_batch. Each of these files is a Python "pickled" object produced with cPickle. Here is a python2 routine which will open such a file and return a dictionary:
 def unpickle(file):
 import cPickle
 with open(file, 'rb') as fo:
 dict = cPickle.load(fo)
 return dict
And a python3 version:
def unpickle(file):
 import pickle
 with open(file, 'rb') as fo:
 dict = pickle.load(fo, encoding='bytes')
 return dict
Loaded in this way, each of the batch files contains a dictionary with the following elements:
  data -- a 10000x3072 numpy array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
   labels -- a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data.
The dataset contains another file, called batches.meta. It too contains a Python dictionary object. It has the following entries:
  label_names -- a 10-element list which gives meaningful names to the numeric labels in the labels array described above. For example, label_names[0] == "airplane", label_names[1] == "automobile", etc.
 
Binary version
The binary version contains the files data_batch_1.bin, data_batch_2.bin, ..., data_batch_5.bin, as well as test_batch.bin. Each of these files is formatted as follows:
&lt;1 x label&gt;&lt;3072 x pixel&gt;
...
&lt;1 x label&gt;&lt;3072 x pixel&gt;
In other words, the first byte is the label of the first image, which is a number in the range 0-9. The next 3072 bytes are the values of the pixels of the image. The first 1024 bytes are the red channel values, the next 1024 the green, and the final 1024 the blue. The values are stored in row-major order, so the first 32 bytes are the red channel values of the first row of the image.
Each file contains 10000 such 3073-byte "rows" of images, although there is nothing delimiting the rows. Therefore each file should be exactly 30730000 bytes long.
 
There is another file, called batches.meta.txt. This is an ASCII file that maps numeric labels in the range 0-9 to meaningful class names. It is merely a list of the 10 class names, one per row. The class name on row i corresponds to numeric label i.
 The CIFAR-100 dataset
This dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.
The 100 classes in the CIFAR-100 are grouped into 20 superclasses. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs).
Here is the list of classes in the CIFAR-100:
  
 Superclass
 Classes
 
 
 aquatic mammals
 beaver, dolphin, otter, seal, whale
 
 
 fish
 aquarium fish, flatfish, ray, shark, trout
 
 
 flowers
 orchids, poppies, roses, sunflowers, tulips
 
 
 food containers
 bottles, bowls, cans, cups, plates
 
 
 fruit and vegetables
 apples, mushrooms, oranges, pears, sweet peppers
 
 
 household electrical devices
 clock, computer keyboard, lamp, telephone, television
 
 
 household furniture
 bed, chair, couch, table, wardrobe
 
 
 insects
 bee, beetle, butterfly, caterpillar, cockroach
 
 
 large carnivores
 bear, leopard, lion, tiger, wolf
 
 
 large man-made outdoor things
 bridge, castle, house, road, skyscraper
 
 
 large natural outdoor scenes
 cloud, forest, mountain, plain, sea
 
 
 large omnivores and herbivores
 camel, cattle, chimpanzee, elephant, kangaroo
 
 
 medium-sized mammals
 fox, porcupine, possum, raccoon, skunk
 
 
 non-insect invertebrates
 crab, lobster, snail, spider, worm
 
 
 people
 baby, boy, girl, man, woman
 
 
 reptiles
 crocodile, dinosaur, lizard, snake, turtle
 
 
 small mammals
 hamster, mouse, rabbit, shrew, squirrel
 
 
 trees
 maple, oak, palm, pine, willow
 
 
 vehicles 1
 bicycle, bus, motorcycle, pickup truck, train
 
 
 vehicles 2
 lawn-mower, rocket, streetcar, tank, tractor
 
 
Yes, I know mushrooms aren't really fruit or vegetables and bears aren't really carnivores.
Download
  
 Version
 Size
 md5sum
 
 
  CIFAR-100 python version 
 161 MB
 eb9058c3a382ffc7106e4002c42a8d85
 
 
  CIFAR-100 Matlab version 
 175 MB
 6a4bfa1dcd5c9453dda6bb54194911f4
 
 
  CIFAR-100 binary version (suitable for C programs) 
 161 MB
 03b5dce01913d631647c71ecec9e9cb8
 
  
Dataset layout
Python / Matlab versions
The python and Matlab versions are identical in layout to the CIFAR-10, so I won't waste space describing them here.
Binary version
The binary version of the CIFAR-100 is just like the binary version of the CIFAR-10, except that each image has two label bytes (coarse and fine) and 3072 pixel bytes, so the binary files look like this: 
&lt;1 x coarse label&gt;&lt;1 x fine label&gt;&lt;3072 x pixel&gt;
...
&lt;1 x coarse label&gt;&lt;1 x fine label&gt;&lt;3072 x pixel&gt;
Indices into the original 80 million tiny images dataset
Sivan Sabato was kind enough to provide this file, which maps CIFAR-100 images to images in the 80 million tiny images dataset. Sivan Writes:
The file has 60000 rows, each row contains a single index into the tiny db,
where the first image in the tiny db is indexed "1". "0" stands for an image that is not from the tiny db.
The first 50000 lines correspond to the training set, and the last 10000 lines correspond
to the test set.
Reference
This tech report (Chapter 3) describes the dataset and the methodology followed when collecting it in much greater detail. Please cite it if you intend to use this dataset.
  Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009.
The CIFAR-10 dataset
https://www.cs.toronto.edu/~kriz/cifar.html
https://www.shutterstock.com/search/400-million-years
246 400 Million Years Images, Stock Photos, 3D objects, & Vectors | Shutterstock
2016-04-04
None
<div><div><div><p>Access millions of high-quality images, video clips, music tracks and sound effects!</p><div><p><a href="https://www.shutterstock.com/pricing">See pricing</a></p></div></div><div><header><div><div><div><p><a href="https://www.shutterstock.com/images"></a></p></div><div><p><a href="https://www.shutterstock.com/video"></a></p></div><div><p><a href="https://www.shutterstock.com/images"></a></p></div><div><p><a href="https://www.shutterstock.com/creative-flow"></a></p></div><div><hr/><div><p><a href="https://www.shutterstock.com/business"></a></p></div></div></div><div><div><div><p><a href="https://www.shutterstock.com/pricing"></a></p></div><hr/></div></div><div><p><a href="">Log in</a></p></div></div></header><div><div><div><div><p></p></div></div><div><p></p><div><a href="https://www.shutterstock.com/search/stone%2c-gemstones%2c-and-concrete"><div><p></p></div></a><a href="https://www.shutterstock.com/search/megaannum"><div><p></p></div></a><a href="https://www.shutterstock.com/search/fossil"><div><p></p></div></a><a href="https://www.shutterstock.com/search/sedimentary-rock"><div><p></p></div></a><a href="https://www.shutterstock.com/search/siltstone"><div><p></p></div></a><a href="https://www.shutterstock.com/search/geology"><div><p></p></div></a><a href="https://www.shutterstock.com/search/mudstone"><div><p></p></div></a><a href="https://www.shutterstock.com/search/cornwall"><div><p></p></div></a><a href="https://www.shutterstock.com/search/class"><div><p></p></div></a><a href="https://www.shutterstock.com/search/paleozoic"><div><p></p></div></a></div></div><div><nav><ol><li><h6><a href="https://www.shutterstock.com/images">Images</a></h6></li><li><h6><a href="https://www.shutterstock.com/category/backgrounds-textures">Design, Templates, and Materials</a></h6></li><li><h6><a href="https://www.shutterstock.com/search/stone%2c-gemstones%2c-and-concrete">Stone, Gemstones, and Concrete</a></h6></li></ol></nav></div><div><h2>400 Million Years royalty-free images</h2><div><h2>246 400 million years stock photos, vectors, and illustrations are available royalty-free for download.</h2> <p><a href="https://www.shutterstock.com/video/search/400-million-years">See 400 million years stock video clips</a></p></div></div><div><div><p><a href="https://www.shutterstock.com/search/400-million-years"></a><a href="https://www.shutterstock.com/search/400-million-years?image_type=photo"></a><a href="https://www.shutterstock.com/search/400-million-years?image_type=vector"></a><a href="https://www.shutterstock.com/search/400-million-years?image_type=illustration"></a><a href="https://www.shutterstock.com/search/400-million-years?image_type=3d-object"></a></p></div><div><p></p><div><p>Any time</p></div></div></div><div><div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div><div></div></div><div><div><a href="https://www.shutterstock.com/search/stone%2c-gemstones%2c-and-concrete"><div><p></p></div></a><a href="https://www.shutterstock.com/search/megaannum"><div><p></p></div></a><a href="https://www.shutterstock.com/search/fossil"><div><p></p></div></a><a href="https://www.shutterstock.com/search/sedimentary-rock"><div><p></p></div></a><a href="https://www.shutterstock.com/search/siltstone"><div><p></p></div></a><a href="https://www.shutterstock.com/search/geology"><div><p></p></div></a><a href="https://www.shutterstock.com/search/mudstone"><div><p></p></div></a><a href="https://www.shutterstock.com/search/cornwall"><div><p></p></div></a><a href="https://www.shutterstock.com/search/class"><div><p></p></div></a><a href="https://www.shutterstock.com/search/paleozoic"><div><p></p></div></a></div><div><div><p><a href="https://www.shutterstock.com/search/400-million-years?page=2">Next</a></p></div><div><a href="https://www.shutterstock.com/search/400-million-years?page=2"></a><p></p></div></div></div></div></div></div></div></div></div>||||I|||| Access millions of high-quality images, video clips, music tracks and sound effects!

 See pricing
 Images
 Video
 Browse Content
 Design
 Enterprise
 0 Credits Available
 Pricing
 Menu
 Log inSign up
 All images
 ​
 ​
 Search by image
 Related searches:
 Stone, Gemstones, and Concrete
 Megaannum
 Fossil
 Sedimentary rock
 Siltstone
 Geology
 Mudstone
 Cornwall
 Class
 Paleozoic

 1. Images

 2. 

 3. Design, Templates, and Materials

 4. 

 5. Stone, Gemstones, and Concrete

 400 Million Years royalty-free images

 246 400 million years stock photos, vectors, and illustrations are available royalty-free for download.

 See 400 million years stock video clips
 Filters
 All images Photos Vectors Illustrations 3D Objects
 Upload date
 Any time
 Stone, Gemstones, and Concrete
 Megaannum
 Fossil
 Sedimentary rock
 Siltstone
 Geology
 Mudstone
 Cornwall
 Class
 Paleozoic
 Next
 ​
 of 3
Access millions of high-quality images, video clips, music tracks and sound effects!   See pricing                                                       Log in                                                                                    Images     Design, Templates, and Materials     Stone, Gemstones, and Concrete       400 Million Years royalty-free images  246 400 million years stock photos, vectors, and illustrations are available royalty-free for download.  See 400 million years stock video clips                       Any time                                                                                                                 Next
246 400 Million Years Images, Stock Photos, 3D objects, & Vectors | Shutterstock
https://www.shutterstock.com/search/400-million-years