here are research papers by Sebastian Thrun:
This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents. This is important because in many text classification problems obtaining training labels is expensive, while large quantities of unlabeled documents are readily available. We introduce an algorithm for learning from labeled and unlabeled documents based on the combination of Expectation-Maximization (EM) and a naive Bayes classifier. The algorithm first trains a classifier using the available labeled documents, and probabilistically labels the unlabeled documents. It then trains a new classifier using the labels for all the documents, and iterates to convergence. This basic EM procedure works well when the data conform to the generative assumptions of the model. However these assumptions are often violated in practice, and poor performance can result. We present two extensions to the algorithm that improve classification accuracy under these conditions: (1) a weighting factor to modulate the contribution of the unlabeled data, and (2) the use of multiple mixture components per class. Experimental results, obtained using text from three different real-world tasks, show that the use of unlabeled data reduces classification error by up to 30%. 
This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents. This is important because in many text classification problems obtaining training labels is expensive, while large quantities of unlabeled documents are readily available. We introduce an algorithm for learning from labeled and unlabeled documents based on the combination of Expectation-Maximization (EM) and a naive Bayes classifier. The algorithm first trains a classifier using the available labeled documents, and probabilistically labels the unlabeled documents. It then trains a new classifier using the labels for all the documents, and iterates to convergence. This basic EM procedure works well when the data conform to the generative assumptions of the model. However these assumptions are often violated in practice, and poor performance can result. We present two extensions to the algorithm that improve classification accuracy under these conditions: (1) a weighting factor to modulate the contribution of the unlabeled data, and (2) the use of multiple mixture components per class. Experimental results, obtained using text from three different real-world tasks, show that the use of unlabeled data reduces classification error by up to 30%.
Text Classification from Labeled and Unlabeled Documents using EM
https://link.springer.com/article/10.1023/A:1007692713085
https://arxiv.org/abs/1604.01802
Learning to Track at 100 FPS with Deep Regression Networks
2016-04-06
Held; David; Thrun; Sebastian; Savarese; Silvio
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/1604.01802">Download PDF</a></p><blockquote>
 Abstract: Machine learning techniques are often used in computer vision due to their
ability to leverage large amounts of training data to improve performance.
Unfortunately, most generic object trackers are still trained from scratch
online and do not benefit from the large number of videos that are readily
available for offline training. We propose a method for offline training of
neural networks that can track novel objects at test-time at 100 fps. Our
tracker is significantly faster than previous methods that use neural networks
for tracking, which are typically very slow to run and not practical for
real-time applications. Our tracker uses a simple feed-forward network with no
online training required. The tracker learns a generic relationship between
object motion and appearance and can be used to track novel objects that do not
appear in the training set. We test our network on a standard tracking
benchmark to demonstrate our tracker's state-of-the-art performance. Further,
our performance improves as we add more videos to our offline training set. To
the best of our knowledge, our tracker is the first neural-network tracker that
learns to track generic objects at 100 fps.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: David Held [<a href="/show-email/390b4952/1604.01802">view email</a>]
 <br />
 <strong><a href="/abs/1604.01802v1">[v1]</a></strong>
 Wed, 6 Apr 2016 20:39:34 UTC (2,572 KB)<br /><strong>[v2]</strong>
Tue, 16 Aug 2016 02:34:48 UTC (1,810 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1604.01802

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computer Vision and Pattern Recognition

 arXiv:1604.01802 (cs)
 [Submitted on 6 Apr 2016 (v1), last revised 16 Aug 2016 (this version, v2)]

 Title: Learning to Track at 100 FPS with Deep Regression Networks

 Authors: David Held, Sebastian Thrun, Silvio Savarese
 Download PDF
 Abstract: Machine learning techniques are often used in computer vision due to their ability to leverage large amounts of training data to improve performance. Unfortunately, most generic object trackers are still trained from scratch online and do not benefit from the large number of videos that are readily available for offline training. We propose a method for offline training of neural networks that can track novel objects at test-time at 100 fps. Our tracker is significantly faster than previous methods that use neural networks for tracking, which are typically very slow to run and not practical for real-time applications. Our tracker uses a simple feed-forward network with no online training required. The tracker learns a generic relationship between object motion and appearance and can be used to track novel objects that do not appear in the training set. We test our network on a standard tracking benchmark to demonstrate our tracker's state-of-the-art performance. Further, our performance improves as we add more videos to our offline training set. To the best of our knowledge, our tracker is the first neural-network tracker that learns to track generic objects at 100 fps.
 Comments: To appear in European Conference on Computer Vision (ECCV) 2016 
 Subjects: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Robotics (cs.RO)
 Cite as: arXiv:1604.01802 [cs.CV]  
 (or arXiv:1604.01802v2 [cs.CV] for this version)  
 https://doi.org/10.48550/arXiv.1604.01802  
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: David Held [view email]
 [v1] Wed, 6 Apr 2016 20:39:34 UTC (2,572 KB)
 [v2] Tue, 16 Aug 2016 02:34:48 UTC (1,810 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CV
 < prev | next >
 new | recent | 1604
 Change to browse by:
 cs
 cs.AI
 cs.LG
 cs.RO

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 David Held
 Sebastian Thrun
 Silvio Savarese
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

https://arxiv.org/pdf/1604.01802.pdf
Learning to Track at 100 FPS with Deep Regression Networks
2016-08-16
David  Held, Stanford University, davheld@cs.stanford.edu, Sebastian  Thrun, Stanford University, thrun@cs.stanford.edu, Silvio  Savarese, Stanford University
Learning to Track at 100 FPS with Deep Regression Networks
Date Published: 2016-08-16
Authors:
David  Held, Stanford University, davheld@cs.stanford.edu
Sebastian  Thrun, Stanford University, thrun@cs.stanford.edu
Silvio  Savarese, Stanford University
Abstract
Machine learning techniques are often used in computer vision due to their ability to leverage large amounts of training data to improve performance. Unfortunately, most generic object trackers are still trained from scratch online and do not benefit from the large number of videos that are readily available for offline training. We propose a method for offline training of neural networks that can track novel objects at test-time at 100 fps. Our tracker is significantly faster than previous methods that use neural networks for tracking, which are typically very slow to run and not practical for real-time applications. Our tracker uses a simple feed-forward network with no online training required. The tracker learns a generic relationship between object motion and appearance and can be used to track novel objects that do not appear in the training set. We test our network on a standard tracking benchmark to demonstrate our tracker's state-of-the-art performance. Further, our performance improves as we add more videos to our offline training set. To the best of our knowledge, our tracker 1 is the first neural-network tracker that learns to track generic objects at 100 fps.
1 Introduction
Given some object of interest marked in one frame of a video, the goal of "singletarget tracking" is to locate this object in subsequent video frames, despite object motion, changes in viewpoint, lighting changes, or other variations. Single-target tracking is an important component of many systems. For person-following applications, a robot must track a person as they move through their environment. For autonomous driving, a robot must track dynamic obstacles in order to estimate where they are moving and predict how they will move in the future.
Generic object trackers (trackers that are not specialized for specific classes of objects) are traditionally trained entirely from scratch online (i.e. during test time) [15, 3, 36, 19] , with no offline training being performed. Such trackers suffer in performance because they cannot take advantage of the large number of videos that are readily available to improve their performance. Offline training videos 
Previous frame
Current frame tracking output Fig. 1 . Using a collection of videos and images with bounding box labels (but no class information), we train a neural network to track generic objects. At test time, the network is able to track novel objects without any fine-tuning. By avoiding fine-tuning, our network is able to track at 100 fps can be used to teach the tracker to handle rotations, changes in viewpoint, lighting changes, and other complex challenges.
In many other areas of computer vision, such as image classification, object detection, segmentation, or activity recognition, machine learning has allowed vision algorithms to train from offline data and learn about the world [5, 23, 13, 25, 9, 28] . In each of these cases, the performance of the algorithm improves as it iterates through the training set of images. Such models benefit from the ability of neural networks to learn complex functions from large amounts of data.
In this work, we show that it is possible to learn to track generic objects in real-time by watching videos offline of objects moving in the world. To achieve this goal, we introduce GOTURN, Generic Object Tracking Using Regression Networks. We train a neural network for tracking in an entirely offline manner. At test time, when tracking novel objects, the network weights are frozen, and no online fine-tuning required (as shown in Figure 1 ). Through the offline training procedure, the tracker learns to track novel objects in a fast, robust, and accurate manner.
Although some initial work has been done in using neural networks for tracking, these efforts have produced neural-network trackers that are too slow for practical use. In contrast, our tracker is able to track objects at 100 fps, making it, to the best of our knowledge, the fastest neural-network tracker to-date. Our real-time speed is due to two factors. First, most previous neural network trackers are trained online [26, 27, 34, 37, 35, 30, 39, 7, 24] ; however, training neural networks is a slow process, leading to slow tracking. In contrast, our tracker is trained offline to learn a generic relationship between appearance and motion, so no online training is required. Second, most trackers take a classification-based approach, classifying many image patches to find the target object [26, 27, 37, 30, 39, 24, 33] . In contrast, our tracker uses a regression-based approach, requiring just a single feed-forward pass through the network to regresses directly to the location of the target object. The combination of offline training and one-pass regression leads to a significant speed-up compared to previous approaches and allows us to track objects at real-time speeds.
GOTURN is the first generic object neural-network tracker that is able to run at 100 fps. We use a standard tracking benchmark to demonstrate that our tracker outperforms state-of-the-art trackers. Our tracker trains from a set of labeled training videos and images, but we do not require any class-level labeling or information about the types of objects being tracked. GOTURN establishes a new framework for tracking in which the relationship between appearance and motion is learned offline in a generic manner. Our code and additional experiments can be found at http://davheld.github.io/GOTURN/GOTURN.html.
2 Related Work
Online training for tracking. Trackers for generic object tracking are typically trained entirely online, starting from the first frame of a video [15, 3, 36, 19] . A typical tracker will sample patches near the target object, which are considered as "foreground" [3] . Some patches farther from the target object are also sampled, and these are considered as "background." These patches are then used to train a foreground-background classifier, and this classifier is used to score patches from the next frame to estimate the new location of the target object [36, 19] . Unfortunately, since these trackers are trained entirely online, they cannot take advantage of the large amount of videos that are readily available for offline training that can potentially be used to improve their performance.
Some researchers have also attempted to use neural networks for tracking within the traditional online training framework [26, 27, 34, 37, 35, 30, 39, 7, 24, 16] , showing state-of-the-art results [30, 7, 21] . Unfortunately, neural networks are very slow to train, and if online training is required, then the resulting tracker will be very slow at test time. Such trackers range from 0.8 fps [26] to 15 fps [37] , with the top performing neural-network trackers running at 1 fps on a GPU [30, 7, 21] . Hence, these trackers are not usable for most practical applications. Because our tracker is trained offline in a generic manner, no online training of our tracker is required, enabling us to track at 100 fps.
Model-based trackers. A separate class of trackers are the model-based trackers which are designed to track a specific class of objects [12, 1, 11] . For example, if one is only interested in tracking pedestrians, then one can train a pedestrian detector. During test-time, these detections can be linked together using temporal information. These trackers are trained offline, but they are limited because they can only track a specific class of objects. Our tracker is trained offline in a generic fashion and can be used to track novel objects at test time.
Other neural network tracking frameworks. A related area of research is patch matching [14, 38] , which was recently used for tracking in [33] , running at 4 fps. In such an approach, many candidate patches are passed through the network, and the patch with the highest matching score is selected as the tracking output. In contrast, our network only passes two images through the network, and the network regresses directly to the bounding box location of the target 
Search Region Conv Layers Conv Layers
Fully-Connected Layers Fig. 2 . Our network architecture for tracking. We input to the network a search region from the current frame and a target from the previous frame. The network learns to compare these crops to find the target object in the current image object. By avoiding the need to score many candidate patches, we are able to track objects at 100 fps.
Prior attempts have been made to use neural networks for tracking in various other ways [18] , including visual attention models [4, 29] . However, these approaches are not competitive with other state-of-the-art trackers when evaluated on difficult tracker datasets.
3.1 Method Overview
At a high level, we feed frames of a video into a neural network, and the network successively outputs the location of the tracked object in each frame. We train the tracker entirely offline with video sequences and images. Through our offline training procedure, our tracker learns a generic relationship between appearance and motion that can be used to track novel objects at test time with no online training required.
3.2 Input / output format
What to track. In case there are multiple objects in the video, the network must receive some information about which object in the video is being tracked. To achieve this, we input an image of the target object into the network. We crop and scale the previous frame to be centered on the target object, as shown in Figure 2 . This input allows our network to track novel objects that it has not seen before; the network will track whatever object is being input in this crop. We pad this crop to allow the network to receive some contextual information about the surroundings of the target object.
In more detail, suppose that in frame t -1, our tracker previously predicted that the target was located in a bounding box centered at c = (c x , c y ) with a width of w and a height of h. At time t, we take a crop of frame t -1 centered at (c x , c y ) with a width and height of k 1 w and k 1 h, respectively. This crop tells the network which object is being tracked. The value of k 1 determines how much context the network will receive about the target object from the previous frame.
Where to look. To find the target object in the current frame, the tracker should know where the object was previously located. Since objects tend to move smoothly through space, the previous location of the object will provide a good guess of where the network should expect to currently find the object. We achieve this by choosing a search region in our current frame based on the object's previous location. We crop the current frame using the search region and input this crop into our network, as shown in Figure 2 . The goal of the network is then to regress to the location of the target object within the search region.
In more detail, the crop of the current frame t is centered at c = (c x , c y ), where c is the expected mean location of the target object. We set c = c, which is equivalent to a constant position motion model, although more sophisticated motion models can be used as well. The crop of the current frame has a width and height of k 2 w and k 2 h, respectively, where w and h are the width and height of the predicted bounding box in the previous frame, and k 2 defines our search radius for the target object. In practice, we use
k 1 = k 2 = 2.
As long as the target object does not become occluded and is not moving too quickly, the target will be located within this region. For fast-moving objects, the size of the search region could be increased, at a cost of increasing the complexity of the network. Alternatively, to handle long-term occlusions or large movements, our tracker can be combined with another approach such as an online-trained object detector, as in the TLD framework [19] , or a visual attention model [4, 29, 2] ; we leave this for future work.
Network output. The network outputs the coordinates of the object in the current frame, relative to the search region. The network's output consists of the coordinates of the top left and bottom right corners of the bounding box.
3.3 Network architecture
For single-target tracking, we define a novel image-comparison tracking architecture, shown in Figure 2 (note that related "two-frame" architectures have also been used for other tasks [20, 10] ). In this model, we input the target object as well as the search region each into a sequence of convolutional layers. The output of these convolutional layers is a set of features that capture a high-level representation of the image.
The outputs of these convolutional layers are then fed through a number of fully connected layers. The role of the fully connected layers is to compare the features from the target object to the features in the current frame to find where the target object has moved. Between these frames, the object may have undergone a translation, rotation, lighting change, occlusion, or deformation. The function learned by the fully connected layers is thus a complex feature comparison which is learned through many examples to be robust to these various factors while outputting the relative motion of the tracked object.
In more detail, the convolutional layers in our model are taken from the first five convolutional layers of the CaffeNet architecture [17, 23] . We concatenate the output of these convolutional layers (i.e. the pool5 features) into a single vector. This vector is input to 3 fully connected layers, each with 4096 nodes. Finally, we connect the last fully connected layer to an output layer that contains 4 nodes which represent the output bounding box. We scale the output by a factor of 10, chosen using our validation set (as with all of our hyperparameters). Network hyperparameters are taken from the defaults for CaffeNet, and between each fully-connected layer we use dropout and ReLU non-linearities as in CaffeNet. Our neural network is implemented using Caffe [17] .
3.4 Tracking
During test time, we initialize the tracker with a ground-truth bounding box from the first frame, as is standard practice for single-target tracking. At each subsequent frame t, we input crops from frame t-1 and frame t into the network (as described in Section 3.2) to predict where the object is located in frame t. We continue to re-crop and feed pairs of frames into our network for the remainder of the video, and our network will track the movement of the target object throughout the entire video sequence.
4 Training
We train our network with a combination of videos and still images. The training procedure is described below. In both cases, we train the network with an L1 loss between the predicted bounding box and the ground-truth bounding box.
4.1 Training from Videos and Images
Our training set consists of a collection of videos in which a subset of frames in each video are labeled with the location of some object. For each successive pair of frames in the training set, we crop the frames as described in Section 3.2.
During training time, we feed this pair of frames into the network and attempt to predict how the object has moved from the first frame to the second frame (shown in Figure 3 ). We also augment these training examples using our motion model, as described in Section 4.2.
Our training procedure can also take advantage of a set of still images that are each labeled with the location of an object. This training set of images teaches our network to track a more diverse set of objects and prevents overfitting to the objects in our training videos. To train our tracker from an image, we take random crops of the image according to our motion model (see Section 4.2). Between these two crops, the target object has undergone an apparent translation and scale change, as shown in Figure 4 . We treat these crops as if they were taken from different frames of a video. Although the "motions" in these crops are less varied than the types of motions found in our training videos, these images are still useful to train our network to track a variety of different objects. 
Previous)) video)frame) centered)on)) object) Current)video)frame,)) shi6ed,)with)) ground9truth) bounding)box)
4.2 Learning Motion Smoothness
Objects in the real-world tend to move smoothly through space. Given an ambiguous image in which the location of the target object is uncertain, a tracker should predict that the target object is located near to the location where it was previously observed. This is especially important in videos that contain multiple nearly-identical objects, such as multiple fruit of the same type. Thus we wish to teach our network that, all else being equal, small motions are preferred to large motions.
To concretize the idea of motion smoothness, we model the center of the bounding box in the current frame (c x , c y ) relative to the center of the bounding box in the previous frame (c x , c y ) as
c x = c x + w • ∆xc y = c y + h • ∆ywhere w and h are the width and height, respectively, of the bounding box of the previous frame. The terms ∆x and ∆y are random variables that capture the change in position of the bounding box relative to its size. In our training set, we find that objects change their position such that ∆x and ∆y can each be modeled with a Laplace distribution with a mean of 0 (see Appendix for details).
Such a distribution places a higher probability on smaller motions than larger motions.
Similarly, we model size changes by
w = w • γ w (3) h = h • γ h (4)
where w and h are the current width and height of the bounding box and w and h are the previous width and height of the bounding box. The terms γ w and γ h are random variables that capture the size change of the bounding box. We find in our training set that γ w and γ h are modeled by a Laplace distribution with a mean of 1. Such a distribution gives a higher probability on keeping the bounding box size near the same as the size from the previous frame.
To teach our network to prefer small motions to large motions, we augment our training set with random crops drawn from the Laplace distributions described above (see Figures 3 and 4 for examples). Because these training examples are sampled from a Laplace distribution, small motions will be sampled more than large motions, and thus our network will learn to prefer small motions to large motions, all else being equal. We will show that this Laplace cropping procedure improves the performance of our tracker compared to the standard uniform cropping procedure used in classification tasks [23] .
The scale parameters for the Laplace distributions are chosen via crossvalidation to be b x = 1/5 (for the motion of the bounding box center) and b s = 1/15 (for the change in bounding box size). We constrain the random crop such that it must contain at least half of the target object in each dimension. We also limit the size changes such that γ w , γ h ∈ (0.6, 1.4), to avoid overly stretching or shrinking the bounding box in a way that would be difficult for the network to learn.
4.3 Training procedure
To train our network, each training example is alternately taken from a video or from an image. When we use a video training example, we randomly choose a video, and we randomly choose a pair of successive frames in this video. We then crop the video according to the procedure described in Section 3.2. We additionally take k 3 random crops of the current frame, as described in Section 4.2, to augment the dataset with k 3 additional examples. Next, we randomly sample an image, and we repeat the procedure described above, where the random cropping creates artificial "motions" (see Sections 4.1 and 4.2). Each time a video or image gets sampled, new random crops are produced on-the-fly, to create additional diversity in our training procedure. In our experiments, we use k 3 = 10, and we use a batch size of 50.
The convolutional layers in our network are pre-trained on ImageNet [31, 8] . Because of our limited training set size, we do not fine-tune these layers to prevent overfitting. We train this network with a learning rate of 1e-5, and other hyperparameters are taken from the defaults for CaffeNet [17] .
5 Experimental Setup
5.1 Training set
As described in Section 4, we train our network using a combination of videos and still images. Our training videos come from ALOV300++ [32] , a collection of 314 video sequences. We remove 7 of these videos that overlap with our test set (see Appendix for details), leaving us with 307 videos to be used for training. In this dataset, approximately every 5th frame of each video has been labeled with the location of some object being tracked. These videos are generally short, ranging from a few seconds to a few minutes in length. We split these videos into 251 for training and 56 for validation / hyper-parameter tuning. The training set consists of a total of 13,082 images of 251 different objects, or an average of 52 frames per object. The validation set consists of 2,795 images of 56 different objects. After choosing our hyperparameters, we retrain our model using our entire training set (training + validation). After removing the 7 overlapping videos, there is no overlap between the videos in the training and test sets.
Our training procedure also leveraged a set of still images that were used for training, as described in Section 4.1. These images were taken from the training set of the ImageNet Detection Challenge [31] , in which 478,807 objects were labeled with bounding boxes. We randomly crop these images during training time, as described in Section 4.2, to create an apparent translation or scale change between two random crops. The random cropping procedure is only useful if the labeled object does not fill the entire image; thus, we filter those images for which the bounding box fills at least 66% of the size of the image in either dimension (chosen using our validation set). This leaves us with a total of 239,283 annotations from 134,821 images. These images help prevent overfitting by teaching our network to track objects that do not appear in the training videos.
5.2 Test set
Our test set consists of the 25 videos from the VOT 2014 Tracking Challenge [22] . We could not test our method on the VOT 2015 challenge [21] because there would be too much overlap between the test set and our training set. However, we expect the general trends of our method to still hold.
The VOT 2014 Tracking Challenge [22] is a standard tracking benchmark that allows us to compare our tracker to a wide variety of state-of-the-art trackers. The trackers are evaluated using two standard tracking metrics: accuracy (A) and robustness (R) [22, 6] , which range from 0 to 1. We also compute accuracy errors (1 -A), robustness errors (1 -R), and overall errors 1
-(A + R)/2.
Each frame of the video is annotated with a number of attributes: occlusion, illumination change, motion change, size change, and camera motion. The trackers are also ranked in accuracy and robustness separately for each attribute, and the rankings are then averaged across attributes to get a final average accuracy and robustness ranking for each tracker. The accuracy and robustness rankings are averaged to get an overall average ranking.
6.1 Overall performance
The performance of our tracker is shown in Figure 5 , which demonstrates that our tracker has good robustness and performs near the top in accuracy. Further, our overall ranking (computed as the average of accuracy and robustness) outperforms all previous trackers on this benchmark. We have thus demonstrated the value of offline training for improving tracking performance. Moreover, these results were obtained after training on only 307 short videos. Figure 5 as well as analysis in the appendix suggests that further gains could be achieved if the training set size were increased by labeling more videos. Qualitative results, as well as failure cases, can be found on the project page: http://davheld.github.io/; currently, the tracker can fail due to occlusions or overfitting to objects in the training set. On an Nvidia GeForce GTX Titan X GPU with cuDNN acceleration, our tracker runs at 6.05 ms per frame (not including the 1 ms to load each image in OpenCV), or 165 fps. On a GTX 680 GPU, our tracker runs at an average of 9.98 ms per frame, or 100 fps. If only a CPU is available, the tracker runs at 2.7 fps. Because our tracker is able to perform all of its training offline, during test time the tracker requires only a single feed-forward pass through the network, and thus the tracker is able to run at real-time speeds.
We compare the speed and rank of our tracker compared to the 38 other trackers submitted to the VOT 2014 Tracking Challenge [22] in Figure 6 , using the overall rank score described in Section 5.2. We show the runtime of the tracker in EFO units (Equivalent Filter Operations), which normalizes for the type of hardware that the tracker was tested on [22] . Figure 6 demonstrates that ours was one of the fastest trackers compared to the 38 other baselines, while outperforming all other methods in the overall rank (computed as the average of the accuracy and robustness ranks). Note that some of these other trackers, such as ThunderStruck [22] , also use a GPU. For a more detailed analysis of speed as a function of accuracy and robustness, see the appendix. Our tracker is able to track objects in real-time due to two aspects of our model: First, we learn a generic tracking model offline, so no online training is required. Online training of neural networks tends to be very slow, preventing real-time performance. Online-trained neural network trackers range from 0.8 fps [26] to 15 fps [37] , with the top performing trackers running at 1 fps on a GPU [30, 7, 21] . Second, most trackers evaluate a finite number of samples and choose the highest scoring one as the tracking output [26, 27, 37, 30, 39, 24, 33] . With a sampling approach, the accuracy is limited by the number of samples, but increasing the number of samples also increases the computational complexity. On the other hand, our tracker regresses directly to the output bounding box, so GOTURN achieves accurate tracking with no extra computational cost, enabling it to track objects at 100 fps.
6.2 How does it work?
How does our neural-network tracker work? There are two hypotheses that one might propose:
1. The network compares the previous frame to the current frame to find the target object in the current frame. 2. The network acts as a local generic "object detector" and simply locates the nearest "object."
We differentiate between these hypotheses by comparing the performance of our network (shown in Figure 2 ) to the performance of a network which does not receive the previous frame as input (i.e. the network only receives the current frame as input). For this experiment, we train each of these networks separately. If the network does not receive the previous frame as input, then the tracker can only act as a local generic object detector (hypothesis 2). Fig. 7 . Overall tracking errors for our network which receives as input both the current and previous frame, compared to a network which receives as input only the current frame (lower is better). This comparison allows us to disambiguate between two hypotheses that can explain how our neural-network tracker works (see Section 6.2). Accuracy and robustness metrics are shown in the appendix Figure 7 shows the degree to which each of the hypotheses holds true for different tracking conditions. For example, when there is an occlusion or a large camera motion, the tracker benefits greatly from using the previous frame, which enables the tracker to "remember" which object is being tracked. Figure 7 shows that the tracker performs much worse in these cases when the previous frame is not included. In such cases, hypothesis 1 plays a large role, i.e. the tracker is comparing the previous frame to the current frame to find the target object.
On the other hand, when there is a size change or no variation, the tracker performs slightly worse when using the previous frame (or approximately the same). Under a large size change, the corresponding appearance change is too drastic for our network to perform an accurate comparison between the previous frame and the current frame. Thus the tracker is acting as a local generic object detector in such a case and hypothesis 2 is dominant. Each hypothesis holds true in varying degrees for different tracking conditions, as shown in Figure 7 .
6.3 Generality vs Specificity
How well can our tracker generalize to novel objects not found in our training set? For this analysis, we separate our test set into objects for which at least 25 videos of the same class appear in our training set and objects for which fewer than 25 videos of that class appear in our training set. Figure 8 shows that, even for test objects that do not have any (or very few) similar objects in our training set, our tracker performs well. The performance continues to improve even as videos of unrelated objects are added to our training set, since our tracker is able to learn a generic relationship between an object's appearance change and its motion that can generalize to novel objects. Additionally, our tracker can also be specialized to track certain objects particularly well. Figure 8 shows that, for test objects for which at least 25 videos of the same class appear in the training set, we obtain a large improvement as more training videos of those types of objects are added. This allows the user to specialize the tracker for particular applications. For example, if the tracker is being used for autonomous driving, then the user can add more objects of people, bikes, and cars into the training set, and the tracker will learn to track those objects particularly well. At the same time, Figure 8 also demonstrates that our tracker can track novel objects that do not appear in our training set, which is important when tracking objects in uncontrolled environments.
6.4 Ablative Analysis
In Table 1 , we show which components of our system contribute the most to our performance. We train our network with random cropping from a Laplace distribution to teach our tracker to prefer small motions to large motions (e.g. motion smoothness), as explained in Section 4.2. Table 1 shows the benefit of this approach compared to the baseline of uniformly sampling random crops ("No motion smoothness"), as is typically done for classification [23] . As shown, we reduce errors by 20% by drawing our random crops from a Laplace distribution.
Table 1 also shows the benefit of using an L1 loss compared to an L2 loss. Using an L1 loss significantly reduces the overall tracking errors from 0.43 to 0.24. Because the L2 penalty is relatively flat near 0, the network does not sufficiently penalize outputs that are close but not correct, and the network would often output a bounding box that was slightly too large or too small. When applied to a sequence of frames, the bounding box would grow or shrink without bound until the predicted bounding box was just a single point or the entire image. In contrast, an L1 loss penalizes more harshly answers that are only slightly incorrect, which keeps the bounding box size closer to the correct size and prevents the bounding box from shrinking or growing without bound. We train our tracker using a combination of images and videos. Table 1 shows that, given the choice between images and videos, training on only videos gives a much bigger improvement to our tracker performance. At the same time, training on both videos and images gives the maximum performance for our tracker. Training on a small number of labeled videos has taught our tracker to be invariant to background motion, out-of-plane rotations, deformations, lighting changes, and minor occlusions. Training from a large number of labeled images has taught our network how to track a wide variety of different types of objects. By training on both videos and images, our tracker learns to track a variety of object types under different conditions, achieving maximum performance.
7 Conclusions
We have demonstrated that we can train a generic object tracker offline such that its performance improves by watching more training videos. During test time, we run the network in a purely feed-forward manner with no online finetuning required, allowing the tracker to run at 100 fps. Our tracker learns offline a generic relationship between an object's appearance and its motion, allowing our network to track novel objects at real-time speeds. Our tracker is able to improve its performance as it trains on more offline data. By observing more videos, GOTURN learns how the appearance of objects change as they move. We further analyze the effect of the amount of training data on our tracker's performance in Figure 9 . We see that that the tracking errors drop dramatically as we increase the number of training videos. Our state-ofthe-art results demonstrated in Section 6.1 of the main text were obtained after training on only 307 short videos, ranging from a few seconds to a few minutes in length, with an average of 52 annotations per video. Figure 9 indicates that large gains could be achieved if the training set size were increased by labeling more videos.
B Online training
Previous neural network trackers for tracking generic objects have been trained online [26, 27, 34, 37, 35, 30, 39, 7, 24, 16] . Unfortunately, such trackers are very slow to train, ranging from 0.8 fps [26] to 15 fps [37] , with the top performing neuralnetwork trackers running at 1 fps [30, 7, 21] . Our tracker is trained offline in a generic manner, so no online training of our tracker is required. As a result, our tracker is able to track novel objects at 100 fps.
In Figures 10 and 11 , we explore the benefits of online training. We use crossvalidation to choose the online learning rate to be 1e-9. Figure 10 shows that online training does not significantly improve performance beyond our offline training procedure. As might be expected, there is a small increase in robustness from online training; however, this comes at a cost of accuracy, since online training tends to overfit to the first few frames of a video and would not easily generalize to new deformations or viewpoint changes. A more detailed analysis is shown in Figure 11 .
Our offline training procedure has seen many training videos with deformations, viewpoint changes, and other variations, and thus our tracker has already learned to handle such changes in a generic manner that generalizes to new objects. Although there might be other ways to combine online and offline training, our network has already learned generic target tracking from its offline training procedure and achieves state-of-the-art tracking performance without any online training required.
C Generality vs Specificity
In the main text, we analyze the generality of our tracker. We demonstrate that our tracker can generalize to novel objects not found in the training set. At the same time, a user can train our tracker to track a particular class of objects especially well by giving more training examples of that class of objects. This is useful if the tracker is intended to be used for a particular application where certain classes of objects are more prevalent. We show more detailed results of this experiment in Figure 12 . Analyzing the accuracy and robustness separately, we observe an interesting pattern. As the number of training videos increases, the accuracy errors decreases equally both for object classes that appear in our training set and classes that do not appear in our training set. On the other hand, the decrease in robustness errors is much more significant for object classes that appear in our training set compared to classes that do not. Thus our tracker is able to learn generic properties about objects that enable it to accurately track objects, i.e. to accurately denote the borders of the object with a bounding box. On the other hand, the tracker's ability to generalize robustness is more limited; the tracker has a hard time tracking the motion of unknown objects when faced with difficult tracking situations. This analysis points towards future work to increase the robustness of the tracker by labeling more videos or by learning to train on unlabeled videos.
D Speed analysis
In the main text, we showed the speed of our tracker as a function of the overall rank (computed as the average of accuracy and robustness ranks) and showed that we have the lowest overall rank while being one of the fastest trackers. In Figure 13 we show more detailed results, demonstrating our tracker's speed as a function of the accuracy rank and the robustness ranks. Our tracker has the second-highest accuracy rank, one of the top robustness ranks, and the top overall rank, while running at 100 fps. Previous neural-network trackers range from 0.8 fps [26] to 15 fps [37] , with the top performing neural-network trackers running at only 1 fps GPU [30, 7, 21] , since online training of neural networks is slow. Thus, by performing all of our training offline, we are able to make our neural network tracker run in real-time. 
E How does it work?
In the main text, we explored how our tracker works as a combination of two hypotheses:
1. The network compares the previous frame to the current frame to find the target object in the current frame. 2. The network acts as a local generic "object detector" and simply locates the nearest "object."
We distinguished between these hypotheses by comparing the performance of our network to the performance of a network which does not receive the previous frame as input. In Figure 14 we show more details of this experiment, showing also accuracy and robustness rankings. For a more detailed interpretation of the results, see Section 6.2 of the main text. 
F Motion Smoothness Distribution
In Section 4.2 of the main text, we describe how we use random cropping to implicitly encode the idea that small motions are more likely than large motions.
To determine which distribution to use to encode this idea, we analyze the distribution of object motion found in the training set. This motion distribution can be seen in Figure 15 . As can be seen from this figure, each of these distributions can be modeled by Laplace distributions. Accordingly, we use Laplace distributions for our random cropping procedure. Note that the training set was only used to determine the shape of the distribution (i.e. Laplace); we use our validation set to determine the scale parameters for the distributions. In more detail, suppose that the bounding box in frame t -1 is given by (c x , c y , w, h) where c x and c y are the coordinates of the center of the bounding box and w and h are the width and height accordingly. Then the bounding box at time t can be seen as drawn from a distribution: with random variables ∆x, ∆y, γ w , and γ h , where (c x , c y , w , h ) parameterize the bounding box at time t using the same representation described above. In terms of the random variables, we can rewrite these expressions as
c x = c x + w • ∆xc y = c y + h • ∆y (6) w = w • γ w (7) h = h • γ h∆x = (c x -c x )/w∆y = (c y -c y )/h (10)
γ w = w /wγ h = h /hThe empirical distributions of these random variables over the training set are shown in Figure 15 .
G Number of layers
In Figure 16 we explore the effect of varying the number of fully-connected layers on top of the neural network on the tracking performance. These fullyconnected layers are applied after the initial convolutions are performed on each image. This figure demonstrates that using 3 fully-connected layers performs better than using either 2 or 4 layers. However, the performance is similar for 2, 3, or 4 fully-connected layers, showing that, even though 3 fully-connected layers is optimal, the performance of the tracker is not particularly sensitive to this parameter.
H Data augmentation
In Figure 17 we explore the effect of varying the number of augmented images created for each batch of the training set. Note that new augmented images are created on-the-fly for each batch. However, varying the number of augmented images varies the percentage of each batch that consists of real images compared to augmented images. Our batch size is 50, so we can vary the number of augmented images in each batch from 0 to 49 (to leave room for at least 1 real image). As shown in Figure 17 , best performance is achieved when 49 augmented images are used per batch, i.e. only 1 real image is used, and the remainder are augmented. However, performance is similar for all values of augmented images greater than 20. In our case (with a batch size of 50), this indicates that performance is similar as long as at least 40% of the images in the batch are augmented. The augmented images show the same examples as the real images, but with the target object translated or with a varying scale. Augmented images thus teach the network how the bounding box position changes due to translation or scale changes. 
I Training Set
Our training set was taken from ALOV300++ [32] . To ensure that there was no overlap with our test set, we removed 7 videos from our training set. These videos are:
-01-Light video00016 -01-Light video00022 -01-Light video00023 -02-SurfaceCover video00012 -03-Specularity video00003 -03-Specularity video00012 -10-LowContrast video00013
After removing these 7 overlapping videos, there is no overlap between the videos in the training and test sets.
J Detailed Results
The detailed results of our method compared to the 38 other methods that were submitted to the VOT 2014 Tracking Challenge [22] are shown in Table 2 .
The VOT 2014 Tracking Challenge consists of two types of experiments. In the first experiment, the trackers are initialized with an exact ground-truth bounding box ("exact"). In the second experiment, the trackers are initialized with a noisy bounding box, which is shifted slightly off of the target object ("noisy").
For the noisy initialization experiment, the same 15 noisy initializations are used for each tracker, and the results shown are an average of the tracking performance across these initializations. This experiment allows us to determine the robustness of each tracker to errors in the initialization. This noisy initialization procedure imitates that of a noisy automatic initialization process or noisy human initializations. The trackers are evaluated using two standard tracking metrics: accuracy and robustness [22, 6] . Each frame of the video is annotated with a number of attributes: occlusion, illumination change, motion change, size change, and camera motion. The trackers are ranked in accuracy and robustness separately for each attribute, and the rankings are then averaged across attributes to get a final accuracy and robustness ranking for each tracker. The accuracy and robustness rankings are averaged to get an overall ranking, shown in Table 2 .
Table 2 . Full results from the VOT 2014 tracking challenge, comparing our method (GOTURN) to the 38 other methods submitted to the competition. We initialize the trackers in two different ways: with the exact ground-truth bounding box ("Exact") and with a noisy bounding box ("Noisy").
References
[1]Andriluka M, Roth S, Schiele B. People-tracking-by-detection and peopledetection-by-tracking. Computer Vision and Pattern Recognition 2008;:1--8.
[2]Ba J, Mnih V, Kavukcuoglu K. Multiple object recognition with visual attention.  2014;:.
[3]Babenko B, Yang MH, Belongie S. Visual tracking with online multiple instance learning. Computer Vision and Pattern Recognition 2009;:983--990.
[4]Bazzani L, Larochelle H, Murino V, Ting JA, Freitas ND. Learning attentional policies for tracking and recognition in video with deep networks. Proceedings the 28th International Conference on Machine Learning (ICML-11) 2011;:937--944.
[5]Bo L, Ren X, Fox D. Multipath sparse coding using hierarchical matching pursuit. Computer Vision and Pattern Recognition (CVPR) 2013;:660--667.
[6]Cehovin L, Kristan M, Leonardis A. Is my new tracker really better than yours?. Applications of Computer Vision (WACV) 2014;:540--547.
[7]Danelljan M, Hager G, Shahbaz Khan F, Felsberg M. Learning spatially regularized correlation filters for visual tracking. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2015;:4310--4318.
[8]Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: A large-scale hierarchical image database. Computer Vision and Pattern Recognition 2009;:248--255.
[9]Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, et al.. Long-term recurrent convolutional networks for visual recognition and description.  2014;:.
[10]Dosovitskiy A, Fischery P, Ilg E, Hazirbas C, Golkov V, Van Der Smagt P, et al.. Flownet: Learning optical flow with convolutional networks. 2015 IEEE International Conference on Computer Vision (ICCV) 2015;:2758--2766.
[11]Fan J, Xu W, Wu Y, Gong Y. Human tracking using convolutional neural networks. IEEE Transactions on 2010;21:1610--1623.
[12]Geiger A. Probabilistic Models for 3D Urban Scene Understanding from Movable Platforms.  2013;:.
[13]Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on 2014;:580--587.
[14]Han X, Leung T, Jia Y, Sukthankar R, Berg AC. Matchnet: Unifying feature and metric learning for patch-based matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015;:3279--3286.
[15]Hare S, Saffari A, Torr PH. Struck: Structured output tracking with kernels. Computer Vision (ICCV) 2011;:263--270.
[16]Hong S, You T, Kwak S, Han B. Online tracking by learning discriminative saliency map with convolutional neural network. Proceedings of the 32nd International Conference on Machine Learning 2015;:6--11.
[17]Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, et al.. Caffe: Convolutional architecture for fast feature embedding.  2014;:.
[18]Jin J, Dundar A, Bates J, Farabet C, Culurciello E. Tracking with deep neural networks. Information Sciences and Systems (CISS), 2013 47th Annual Conference on 2013;:1--5.
[19]Kalal Z, Mikolajczyk K, Matas J. Tracking-learning-detection. Pattern Analysis and Machine Intelligence. IEEE Transactions on 2012;34:1409--1422.
[20]Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L. Largescale video classification with convolutional neural networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition 2014;:1725--1732.
[21]Kristan M, Matas J, Leonardis A, Felsberg M, Cehovin L, Fernandez G, et al.. The visual object tracking vot2015 challenge results. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops 2015;:1--23.
[22]Kristan M, Pflugfelder R, Leonardis A, Matas J, Čehovin L, Nebehay G, et al.. The visual object tracking vot2014 challenge results. Computer Vision-ECCV 2014 Workshops 2014;:191--217.
[23]Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 2012;:1097--1105.
[24]Kuen J, Lim KM, Lee CP. Self-taught learning of a deep invariant representation for visual tracking via temporal slowness principle. Pattern Recognition 2015;48:2964--2982.
[25]Levi G, Hassner T. Age and gender classification using convolutional neural networks. Computer Vision and Pattern Recognition Workshop (CVPRW), 2015 IEEE Conference on 2015;:.
[26]Li H, Li Y, Porikli F. Deeptrack: Learning discriminative feature representations by convolutional neural networks for visual tracking. Proceedings of the British Machine Vision Conference 2014;:.
[27]Li H, Li Y, Porikli F. Deeptrack: Learning discriminative feature representations online for robust visual tracking.  2015;:.
[28]Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015;:3431--3440.
[29]Mnih V, Heess N, Graves A. Recurrent models of visual attention. Advances in Neural Information Processing Systems 2014;:2204--2212.
[30]Nam H, Han B. Learning multi-domain convolutional neural networks for visual tracking.  2015;:.
[31]Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al.. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 2014;:1--42.
[32]Smeulders AW, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M. Visual tracking: an experimental survey. Pattern Analysis and Machine Intelligence. IEEE Transactions on 2014;36:1442--1468.
[33]Tao R, Gavves E, Smeulders AW M. Siamese instance search for tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2016;:.
[34]Wang L, Ouyang W, Wang X, Lu H. Visual tracking with fully convolutional networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2015;:3119--3127.
[35]Wang N, Li S, Gupta A, Yeung DY. Transferring rich feature hierarchies for robust visual tracking.  2015;:.
[36]Wang N, Shi J, Yeung DY, Jia J. Understanding and diagnosing visual tracking systems.  2015;:.
[37]Wang N, Yeung DY. Learning a deep compact image representation for visual tracking. Advances in neural information processing systems 2013;:809--817.
[38]Zagoruyko S, Komodakis N. Learning to compare image patches via convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015;:4353--4361.
[39]Zhang K, Liu Q, Wu Y, Yang MH. Overall Ranks Accuracy Ranks Robustness Ranks Speed Method name Exact Noisy Exact Noisy Exact Noisy Frames/EFO. Robust visual tracking via convolutional networks 2015;:.
Learning to Track at 100 FPS with Deep Regression Networks
https://arxiv.org/abs/1604.01802
https://arxiv.org/abs/1301.2292
A Bayesian Multiresolution Independence Test for Continuous Variables
2013-01-10
Margaritis; Dimitris; Thrun; Sebastian
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/1301.2292">Download PDF</a></p><blockquote>
 Abstract: In this paper we present a method ofcomputing the posterior probability
ofconditional independence of two or morecontinuous variables from
data,examined at several resolutions. Ourapproach is motivated by
theobservation that the appearance ofcontinuous data varies widely atvarious
resolutions, producing verydifferent independence estimatesbetween the
variablesinvolved. Therefore, it is difficultto ascertain independence
withoutexamining data at several carefullyselected resolutions. In our paper,
weaccomplish this using the exactcomputation of the posteriorprobability of
independence, calculatedanalytically given a resolution. Ateach examined
resolution, we assume amultinomial distribution with Dirichletpriors for the
discretized tableparameters, and compute the posteriorusing Bayesian
integration. Acrossresolutions, we use a search procedureto approximate the
Bayesian integral ofprobability over an exponential numberof possible
histograms. Our methodgeneralizes to an arbitrary numbervariables in a
straightforward manner.The test is suitable for Bayesiannetwork learning
algorithms that useindependence tests to infer the networkstructure, in domains
that contain anymix of continuous, ordinal andcategorical variables.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Dimitris Margaritis [<a href="/show-email/930303c8/1301.2292">view email</a>] [via AUAI proxy]
 <br /><strong>[v1]</strong>
Thu, 10 Jan 2013 16:25:12 UTC (1,036 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:1301.2292

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Artificial Intelligence

 arXiv:1301.2292 (cs)
 [Submitted on 10 Jan 2013]

 Title: A Bayesian Multiresolution Independence Test for Continuous Variables

 Authors: Dimitris Margaritis, Sebastian Thrun
 Download PDF
 Abstract: In this paper we present a method ofcomputing the posterior probability ofconditional independence of two or morecontinuous variables from data,examined at several resolutions. Ourapproach is motivated by theobservation that the appearance ofcontinuous data varies widely atvarious resolutions, producing verydifferent independence estimatesbetween the variablesinvolved. Therefore, it is difficultto ascertain independence withoutexamining data at several carefullyselected resolutions. In our paper, weaccomplish this using the exactcomputation of the posteriorprobability of independence, calculatedanalytically given a resolution. Ateach examined resolution, we assume amultinomial distribution with Dirichletpriors for the discretized tableparameters, and compute the posteriorusing Bayesian integration. Acrossresolutions, we use a search procedureto approximate the Bayesian integral ofprobability over an exponential numberof possible histograms. Our methodgeneralizes to an arbitrary numbervariables in a straightforward manner.The test is suitable for Bayesiannetwork learning algorithms that useindependence tests to infer the networkstructure, in domains that contain anymix of continuous, ordinal andcategorical variables.
 Comments: Appears in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence (UAI2001)
 Subjects: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG) 
 Report number: UAI-P-2001-PG-346-353  
 Cite as: arXiv:1301.2292 [cs.AI]  
 (or arXiv:1301.2292v1 [cs.AI] for this version) 
 https://doi.org/10.48550/arXiv.1301.2292 
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Dimitris Margaritis [view email] [via AUAI proxy]
 [v1] Thu, 10 Jan 2013 16:25:12 UTC (1,036 KB)
 Full-text links:

 Download:

 * PDF only
 (license)
 Current browse context:
 cs.AI
 < prev | next >
 new | recent | 1301
 Change to browse by:
 cs
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Dimitris Margaritis
 Sebastian Thrun
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

A Bayesian Multiresolution Independence Test for Continuous Variables
Dimitris Margaritis 
Computer Science Dept
Computer Science Dept
Carnegie Mellon University Pittsburgh
Carnegie Mellon University Pittsburgh
15213, 15213PA, PA
A Bayesian Multiresolution Independence Test for Continuous Variables
Dirnitris.MargaritisCcs.cmu.edu
In this paper we present a method of comput ing the posterior probability of conditional independence of two or more continuous vari ables from data, examined at several reso lutions. Our approach is motivated by the observation that the appearance of continu ous data varies widely at various resolutions, producing very different independence esti mates between the variables involved. There fore, it is difficult to ascertain independence without examining discretized data at sev eral carefully selected resolutions. In our paper, we accomplish this using the exact computation of the posterior probability of independence, calculated analytically given a resolution. At each examined resolution and boundary placement, we assume a multi nomial distribution with Dirichlet priors for the discretized table parameters, and com pute the posterior using Bayesian integra tion. Across resolutions, we use a search pro cedure to approximate the Bayesian integral of probability over an exponential number of possible boundary placements. Our method generalizes to an arbitrary number variables in a straightforward manner. The test is suitable for Bayesian network learning algo rithms that use independence tests to infer the network structure, in domains that con tain any mix of continuous, ordinal discrete, and categorical variables.
Introduction and Motivation
Knowledge about the independencies that exist in a domain is a very useful piece of information that a research scientist can-and must-elicit early on dur ing her investigation. Conditional independence state-
Sebastian Thrun
Computer Science Dept. Carnegie Mellon University Pittsburgh, PA 15213
Sebastian.ThrunOcs.cmu.edu ments concerning observed quantities in the domain can greatly aid in the task of understanding the in teractions among the domain variables. Imagine for example a medical researcher, attempting to causally model a set of disease and symptom indicators in an effort to find ways of treating or preventing them from occurring in the future. Such conditional inde pendence statements can lend her significant insights to the cause-effect relationships that may be present. Similarly, a solid state physicist attempting to model the interaction among tens of variables representing proportions of substances involved in a new manufac turing process. Knowledge of conditional independen cies can help her focus her attention to only the rele vant ones during the different stages of the manufac turing process.
More than simply aiding in focusing attention, there exist a number of algorithms today that can induce the causal structure of a domain given a set of con ditional independence statements, under assumptions [10,8]. One principled method for representing such causal structures are Bayesian Networks (BNs) [8], one of the prevalent tools for representing and reason ing about uncertainty. A BN consists of two parts:
( 1) a directed graphical description of the relation ships among the variables in the domain, and (2) a quantitative description of the nature of interactions between each variable and its parents in the graph.
In this paper we focus on an integral component of the first part, namely developing a probabilistic con ditional independence test for use in causal structure discovery. There exist several approaches for causal discovery in domains with categorical variables [10,8].
However there are few for continuous or hybrid do mains [2,6]. Most of the latter approaches belong to the score-maximization family of algorithms for struc ture induction, which does not give any guarantees for producing a correct causal description of the domain. In contrast, the constraint-based family [10,5] employs conditional independence tests, and has guarantees of inducing the correct causal structure (under a.  9max then 13. 9max f-p(x, y)
14.
Bsave f-B' 15. p(t) f-Lx , yP (x,y) 16. if p(t) > Pmax 17. Pmax f-p(t)
18. This implies that the probability of independence de creases or remains the same as we examine our data set at increasingly fine resolutions, or, equivalently, that the dependence probability is non-decreasing. Equal ity occurs in the case of independence at all resolu tions. The above statement is also intuitively true:
imagine for example the two limiting cases: a single cell, and the case where resolution tends to infinity at an ideal scenario where we have an infinite data set at our disposal. In the first case, the posterior inde pendence probability is 1 irrespective of the data set.
In the second, unless the data set is truly independent at every resolution, any discrepancies of the posterior probability of each cell from the product of the corre sponding marginals will eventually emerge as we refine our discretization, causing the posterior independence probability to decline.
Combining the above observation with the previous one concerning the need to examine our data at as fine resolution as possible, we see that we need to maxi mize the posterior of dependence in an effort to discern dependence, as we progressively increase the effective resolution during a run of our algorithm. One con cern however is that our data may appear increasingly artificially dependent at finer resolutions, a statement  An example run of our algorithm on two data sets, one independent by construction and one where X and Y exhibit a highly nonlinear dependency is shown in   figure 3, bottom, we plot the evolution of p(t) as the XY is discretized at successively finer resolutions. Both plots demonstrate how the poste rior probability of independence tends to unity as the grid becomes more complex. The plot on the right is not strictly "U-shaped" due to the fact that subdivi sion steps do not exactly double the resolution.The top plots show the grid that produces the discretization of maximum dependence, i.e. Pr(Mx I D) = 1-Pmax· Note that the set of independent points contains only four cells. Our algorithm returns posterior probability of independence 1 -Pmax = 0.88 for the independent data set and 0.29 for the dependent one. In summary, the justification of our algorithm is as follows: as we established by the structure of the res olution space, one needs to minimize posterior inde pendence across resolutions, while approximating the average over an exponential sum over possible bound aries at each such resolution. Our algorithm accom plishes both goals simultaneously by maximizing de pendence instead of minimizing independence across resolutions by employing an incremental maximum value approximation to the evaluation of successive Bayesian integrals over multiple resolutions. In do ing so, it reduces an exponential computation to a polynomial-order one.
3
Experimental results
In this section we compare our approach with exist ing ones in use routinely by statisticians. Perhaps the most frequently used method for non-parametric inde pendence testing is rank correlation, either Spearman's rank order correlation coefficient rs and Kendall's tau (r) [9]. The two measures are related, and result in essentially the same test in most cases. Here we will compare our test with Spearman's coefficient as it is the most expressive of the two. We will use some ex amples from the Boston housing data set, which is contained in the UC Irvine data set repository [7].
In figure 4, we see two variables that appear depen dent, the average number of rooms per house and its median value. The dependence between them is deemed very weak by rank correlation (its correlation coefficient is only 0.08), while our algorithm returns 0.99 as the posterior probability of dependence. We see the evolution of that probability for the first few steps in the bottom plot of figure 4. These variables were found dependent in the original study.
Some other examples of dependent (figure 5, rank correlation is -0.02 while our algorithm returns 0.98) and approximately independent pairs of variables (fig ure 6, rank correlation 0.01, posterior dependence 0.32, both indicating independence) demonstrate how our approach works as expected in practice, and better than existing established techniques.
4
Related Work
The most well-known independence test is perhaps the chi-square (x2) test. It operates on categorical data, and assumes there are enough counts in each bin such that the counts are approximately normally distributed. Addressing the independence in 2 x 2 con-   tingency tables, Fisher proposed an exact test in 1934 (see [1] for one description). The difference between ours and Fisher's exact test is that the latter one is not Bayesian, and it computes a power value rather than the posterior probability. Bayesian approaches to con tingency table independence also exist: the 2 x 2 case is addressed in Jeffreys' classic text [4], and in general more recently ( e. g. [3]), although not in exactly the same way as ours. However the greatest difference that makes our method unique is that it applies to continu ous variables, taking advantage of the ordering of the values.
The most well-known statistical non-parametric tests of independence for ordinal variables are Spearman's rank correlation and Kendall's tau. The former is the linear correlation of the relative ranks of the points along the X and Y axes, while the latter reduces ranks to binary or ternary variables (larger, equal, or smaller) . Both tests can handle clear monotonic trends but fail to capture complex interactions between X and Y. Most notably, as demonstrated in section 3, they also fail to capture joint probability functions of dependent variables that are two dimensional.
As far as parametric approaches are concerned, there are a multitude of such approaches as they are the norm in practice. There are two main issues with all these approaches: first, they assume a certain family of distributions, which may fail in cases where the under lying data distribution does not fit well. Second, the question of proper bias and its tradeoff with variance (the issue of model complexity) needs to be adequately addressed. This problem also occurs in our approach, and is solved though our choice of parameter prior.
5
Conclusion and Future Research
In this paper we presented a probabilistic test of condi tional independence between two continuous variables.
For this we proposed the posterior probability of in dependence given the data, taking into account his tograms representing dis�retizations at several differ ent resolutions. We show that this is necessary by observing that the range of values that this quantity may take across different resolutions may vary sub stantially. We developed a Bayesian approach, inte grating over all possible boundary placements on the plane, and incorporating evidence from different reso lutions. By taking advantage of certain structure that exists in the space of discretizations, we were able to approximate this quantity, reducing the time that is required to estimate it from exponential in the size of the data set, to polynomial.
An issue that merits further attention is the case of the conditional test. Our proposed extension to the unconditional test may not work in all cases, because the observation of section 2.2 does not extent to the conditioning set variables, as is explained later in that section (it may work as an approximation) . Therefore a more general algorithm is needed for arbitrarily dis tributed data. We plan to work on this issue in the near future.
Our solution may benefit the generation of Bayesian networks in domains that contain any number of con tinuous, ordinal discrete and/or categorical attributes.
In such domains, our test may be applied in a straight forward fashion, due to the fact that we employ dis cretization. One additional benefi t that may emerge from such an approach is the representation of the con ditional probability distributions of a BN, if our test is being used for such a purpose. This is another topic of further research.
Another interesting direction of research is the devel opment of quad-tree like discretization rather than axis-parallel boundaries that span the range of the variables involved. One can envision, for example, a discretization that detects ranges in the plane where the variables are locally approximately independent.
Such a range may indicate the presence of another quantity, explaining the local independence. This may be used to generate new, possibly meaningful features by combining existing ones in the domain.
Finally, accelerating the asymptotic running time of our algorithm is important. Even though our method is polynomial in time, the exponent is the number of variables involved in the conditional test. This may be prohibitively expensive for large conditioning sets and/or data sets. A faster algorithm is required for those situations.
Figure 1 :
1Two very different data sets have very simi lar 3 x 4 histograms. Left: Data strongly dependent. Right: Data independent by construction.
k=l r I� p. To see why our algorithm attempts to maximize the posterior probability of dependence we need to exam ine another implication of the above observation: Pr[ixy(M)] Pr[ ixy(M) .1\ ixy (2M)] + Pr[ ixy ( M) .1\ -.ixy(2M)) Pr[ ixy (M) I ixy (2M) 1 Pr[ ixy (2M)] + Pr[ ixy (M) I -.ixy (2M) ] Pr[ -.ixy (2M) ] Pr[ ixy (2M)] + Pr[ ixy (M) I -.ixy (2M)] Pr[ -.ixy (2M)] > Pr[ ixy (2M) 1
that might be unwarranted by our (sparse) data. The solution to this comes from our choice of a Dirichlet parameter prior (a; = /3j = /k = 1 ), which returns a high value of posterior independence at very high res olutions where there are very few data points and a large number of empty cells (see discussion in previ ous section). This corresponds well with our intuition, and lends itself to the estimation of the finest resolu tion to use: it is the one that returns a minimum for the posterior probability of independence. That pos terior, under our choice Dirichlet prior, is "U-shaped,"starting at 1 at resolution 1 x 1, decreasing and then tending to 1 again at high resolutions.
"Figure 3 :
3.: •' �:· .·: . :, . ' . · . . . :, : : •• -:-. ""i .... .. :� ' ., . An example run of our algorithm on two data sets, the same ones as in figure 4. Left: Data in dependent. Right: Data nonlinearly dependent. Top scatterplots: The actual data set, together with the grid that is output from our algorithm, drawn at the point of minimum independence. Bottom plots: The probability of independence as boundaries are added to the grid. The dashed horizontal line represents our prior r = 0.5.
figure 3 .
3In
The algorithm of figure 2
2essentially computes an ap proximation of the complete Bayesian integral over all possible grid boundaries at the resolution correspond ing to the minimum probability of independence sup ported by our data set. At this limiting resolution M* x M*, we treat the grid boundaries in a Bayesian fashion: ideally we should go over all possible M* x M* grids whose boundaries lie on each possible pair of mid points between successive data points along the axes. However, the number of such irregular M* x M* grids ( N _ 1 )2 is very large, namely M* _ 1 a number exponential in the number of data points. Our approach ap proximates this computation by maximizing over the first M* -1 boundaries and averaging over the M* -th one. This results in a polynomial-time approximation of the Bayesian sum.
Figure 4 :
4Two dependent variables from the Boston housing data set, the average number of rooms per dwelling and and the median value of owner-occupied homes, in thousands of dollars. Spearman's rank cor relation is very small (0.081) due to the fact that it is a two-dimensional distribution. In the bottom we plot the posterior independence probability during a run of our algorithm. The resulting posterior dependence probability is 0.99.
Figure 5 :
5Percent lower status of population vs. pro portion of houses built before 1940. Rank correlation indicates independence with a value of -0.02, while our method suggests strong dependence (0.98).rP� o a
A Bayesian Multiresolution Independence Test for Continuous Variables
https://arxiv.org/abs/1301.2292
https://arxiv.org/abs/2002.12364
Theoretical Models of Learning to Learn
2020-02-27
Baxter; Jonathan
<div><div>
<div>
<p>
<h2>Computer Science &gt; Machine Learning</h2>
</p>
<p><strong>arXiv:2002.12364</strong> (cs)
 </p>
<div>
<p>
 [Submitted on 27 Feb 2020]</p>
<p><a href="https://arxiv.org/pdf/2002.12364">View PDF</a></p><blockquote>
A Machine can only learn if it is biased in some way. Typically the bias is supplied by hand, for example through the choice of an appropriate set of features. However, if the learning machine is embedded within an {\em environment} of related tasks, then it can {\em learn} its own bias by learning sufficiently many tasks from the environment. In this paper two models of bias learning (or equivalently, learning to learn) are introduced and the main theoretical results presented. The first model is a PAC-type model based on empirical process theory, while the second is a hierarchical Bayes model.
 </blockquote>
</div>
<div>
<h2>Submission history</h2><p> From: Jonathan Baxter [<a href="https://arxiv.org/show-email/721b4f78/2002.12364">view email</a>] <br/> <strong>[v1]</strong>
 Thu, 27 Feb 2020 13:35:26 UTC (47 KB)<br/>
</p></div>
</div>
<div>
<div><p>
 Current browse context: </p><p>cs.LG</p>
</div>
<div>
<h3><a href="https://dblp.uni-trier.de">DBLP</a> - CS Bibliography</h3>
</div><p>
</p>
<div>
<p><h3>Bookmark</h3></p><p><a href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2002.12364&amp;description=Theoretical Models of Learning to Learn">
</a>
<a href="https://reddit.com/submit?url=https://arxiv.org/abs/2002.12364&amp;title=Theoretical Models of Learning to Learn">
</a>
</p></div> </div>
<div><p>
</p><div>
<h2>Bibliographic and Citation Tools</h2>
<div>
<p>
</p>
</div>
</div>
<p>
</p><div>
<h2>Code, Data and Media Associated with this Article</h2>
</div>
<p>
</p><div>
<h2>Demos</h2>
</div>
<p>
</p><div>
<h2>Recommenders and Search Tools</h2>
<div>
<p>
</p>
</div>
</div>
<p>
</p><div>
<h2>arXivLabs: experimental projects with community collaborators</h2>
<p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
<p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
<p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
</div>
</div>
</div></div>||||I|||| Skip to main content
 We gratefully acknowledge support from the Simons Foundation, member institutions , and all contributors. Donate
 > cs > arXiv:2002.12364

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2002.12364 (cs)
 [Submitted on 27 Feb 2020]

 Title: Theoretical Models of Learning to Learn

 Authors: Jonathan Baxter
 View a PDF of the paper titled Theoretical Models of Learning to Learn, by Jonathan Baxter
 View PDF
 Abstract: A Machine can only learn if it is biased in some way. Typically the bias is supplied by hand, for example through the choice of an appropriate set of features. However, if the learning machine is embedded within an {\em environment} of related tasks, then it can {\em learn} its own bias by learning sufficiently many tasks from the environment. In this paper two models of bias learning (or equivalently, learning to learn) are introduced and the main theoretical results presented. The first model is a PAC-type model based on empirical process theory, while the second is a hierarchical Bayes model.
 Comments: arXiv admin note: text overlap with arXiv:1106.0245 
 Subjects: Machine Learning (cs.LG) ; Machine Learning (stat.ML) 
 Cite as: arXiv:2002.12364 [cs.LG] 
 (or arXiv:2002.12364v1 [cs.LG] for this version) 
 https://doi.org/10.48550/arXiv.2002.12364 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 Journal reference: in Learning to Learn (edited by Sebastian Thrun and Lorien Pratt), 159-179 (1998)
 https://doi.org/10.1007/978-1-4615-5529-2 
 Related DOI: Focus to learn more 
 DOI(s) linking to related resources 
 

 Submission history

 From: Jonathan Baxter [view email]
 [v1] Thu, 27 Feb 2020 13:35:26 UTC (47 KB)
 Full-text links:

 Access Paper:

 View a PDF of the paper titled Theoretical Models of Learning to Learn, by Jonathan Baxter
 * View PDF
 * TeX Source
 * Other Formats
 view license
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2002
 Change to browse by:
 cs
 stat
 stat.ML

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Jonathan Baxter
 a export BibTeX citation Loading...

 BibTeX formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 CatalyzeX Code Finder for Papers (What is CatalyzeX?)
 DagsHub Toggle
 DagsHub (What is DagsHub?)
 GotitPub Toggle
 Gotit.pub (What is GotitPub?)
 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Spaces Toggle
 TXYZ.AI (What is TXYZ.AI?)
 Related Papers

 Recommenders and Search Tools

 Link to Influence Flower
 Influence Flower (What are Influence Flowers?)
 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Theoretical Models of Learning to Learn *
27 Feb 2020
Jonathan Baxter jon@syseng.anu.edu.au 
Department of Systems Engineering Research School of Information Science and Engineering
Australian National University
0200CanberraAustralia
Theoretical Models of Learning to Learn *
27 Feb 2020Learning to LearnBias LearningEmpirical ProcessesHierarchical Bayes
A Machine can only learn if it is biased in some way. Typically the bias is supplied by hand, for example through the choice of an appropriate set of features. However, if the learning machine is embedded within an environment of related tasks, then it can learn its own bias by learning sufficiently many tasks from the environment[4,6]. In this paper two models of bias learning (or equivalently, learning to learn) are introduced and the main theoretical results presented. The first model is a PAC-type model based on empirical process theory, while the second is a hierarchical Bayes model.
Introduction
Hume's analysis [10] shows that there is no a priori basis for induction. In a machine learning context, this means that a learner must be biased in some way for it to generalise well [11]. Typically such bias is introduced by hand through the skill and insights of experts, but despite many notable successes, this process is limited by the experts' abilities. Hence a desirable goal is to find ways of automatically learning the bias. Bias learning is a form of learning to learn, and the two expressions will be used interchangeably throughout this document.
The purpose of this chapter is to present an overview of two models of supervised bias learning. The first [4,3] is based on Empirical Process theory (henceforth the EP model) and the second [6] is based on Bayesian inference and information theory (henceforth the Bayes model). Empirical process theory is a general theory that includes the analysis of pattern classification first introduced by Vapnik and Chervonenkis [13,12]. Note that these are models of supervised bias learning and as such have little to say about learning to learn in a reinforcement learning setting. In this introduction a high level overview of the features common to both models will be presented, and then in later sections the details and main results of each model will be discussed.
In ordinary models of machine learning the learner is presented with a single task. Learning the "right bias" in such a model does not really make sense, because the ultimate bias is one which completely solves the task. Thus in single-task learning, bias learning or learning to learn is the same as learning.
In order to learn bias one has introduce extra assumptions about the learning process. The central assumption of both the Bayes model and the EP model of bias learning is that the learner is embedded within an environment of related problems. The learner's task is to find a bias that is appropriate for the entire environment, not just for a single task.
A simple example of an environment of learning problems with a common bias is handwritten character recognition. A preprocessing stage that identifies and removes any (small) rotations, dilations and translations of an image of a character will be advantageous for recognising all characters. If the set of all individual character recognition problems is viewed as an environment of learning tasks, this preprocessor represents a bias that is appropriate to all tasks in the environment.
Preprocessing can also be viewed as feature extraction, and there are many classes of learning problems that possess common feature sets. For example, one can view face recognition as a collection of related learning problems, one for each possible face classifier, and it is likely that there exists sets of features that are good for learning all faces. A similar conclusion applies to other domains such as speech recognition (all the individual word classifiers may be viewed as separate learning problems possessing a common feature set), fingerprint recognition, and so on. The classical approach to statistical pattern recognition in these domains is to first guess a set of features and then to learn each problem by estimating a simple (say linear) function of the features. The choice of features represents the learner's bias, thus in bias learning the goal is to get the learner to learn the features instead of guessing them.
In order to perform a theoretical analysis of bias learning, we assume the tasks in the environment are generated according to some underlying probability distribution. For example, if the learner is operating in an environment where it must learn to recognise faces, the distribution over learning tasks will have its support restricted to face recognition type problems. The learner acquires information about the environment by sampling from this distribution to generate multiple learning problems, and then sampling from each learning problem to generate multiple training sets. The learner can then search for bias that is appropriate for learning all the tasks.
In the EP model, the learner is provided with a family of hypothesis spaces and it searches for an hypothesis space that contains good solutions to all the training sets. Such a hypothesis space can then be used to learn novel tasks drawn from the same environment. The key result of the EP model (theorem 2 in section 3) gives a bound on the number of tasks and number of examples of each task required to ensure that a hypothesis space containing good solutions to all training sets will, with high probability, contain good solutions to novel tasks drawn from the same environment. This ability to learn novel tasks after seeing sufficiently many examples of sufficiently many tasks is the formal definition of learning to learn under the EP model.
The Bayes model is the same as the EP model in that the learner is assumed to be embedded within an environment of related tasks and can sample from the environment to generate multiple training sets corresponding to different tasks. However, the Bayes bias learner differs in the way it uses the information from the multiple training sets. In the Bayes model, the distribution over learning tasks in the environment is interpreted as an objective prior distribution. The learner does not know this distribution, but does have some idea of a set Π of possible prior distributions to which the true distribution belongs. The learner starts out with a hyper-prior distribution on Π and based on the data in the training sets, updates the hyper-prior to a hyper-posterior using Bayes' rule. The hyper-posterior is then used as a prior distribution when learning novel tasks. In section 4 results will be presented showing how the information needed to learn each task (in a Shannon sense) decays to the minimum possible for the environment as the number of tasks and number of examples of each tasks seen already grows. Within the Bayes model, this is the formal definition of learning to learn.
Before moving on to the details of these models, it is worth pausing to assess what bias learning solves, and what it doesn't-and in a sense can never-solve. On face value, being able to learn the right bias appears to violate Hume's conclusion that there can be no a priori basis for induction. However this is not the case, for the bias learner learner is still fundamentally limited by the possible choices of bias available. For example, if a learner is learning a set of features for an environment in which there are in fact no small feature sets, then any bias it comes up with (i.e. any feature set) will be a very poor bias for that environment. Thus, there is still guesswork involved in determining the appropriate way to hyper-bias the learner. The main advantage of bias learning is that this hyper-bias can be much weaker than the bias: the right hyper-bias for many environments is just that there exists a set of features, whereas specifying the right bias means actually finding the features.
Statistical Models of Ordinary Learning
To understand how bias learning can be modeled from a statistical perspective, it is necessary to first understand how ordinary learning is modeled from a statistical perspective. The empirical process (EP) process approach and the Bayes approach will be discussed in turn.
The empirical process (EP) approach
The empirical process (EP) approach to modeling ordinary (single-task) learning has the following essential ingredients:
• An input space X and an output space Y ,
• a probability distribution P on X × Y ,
• a loss function l: Y × Y → R, and • a hypothesis space H which is a set of hypotheses or functions h: X → Y .
As an example, if the problem is to learn to recognize images of Mary's face using a neural network, then X would be the set of all images (typically represented as a subset of R d where each component is a pixel intensity), Y would be the set {0, 1}, and the distribution P would be peaked over images of different faces and the correct class labels. The learner's hypothesis space H would be a class of neural networks mapping the input space (R d ) to {0, 1}.
l(y, y ′ ) := 1 if y = y ′ 0 if y = y ′(1)
Using the loss function allows us to present a unified treatment of both concept learning (Y = {0, 1}, l as above), and real-valued function learning (e.g. regression) in which Y = R and l(y, y ′ ) = (y − y ′ ) 2 .
The goal of the learner is to select a hypothesis h ∈ H with minimum expected loss:
er P (h) := X×Y l(h(x), y) dP (x, y).(2)
For classifying Mary's face, the h ∈ H with minimum value of er P (h) is the one that makes the fewest number of mistakes on average. Of course, the learner does not know P and so it cannot search through H for an h minimizing er P (h). In practice, the learner samples repeatedly from the distribution P to generate a training set
z := {(x 1 , y 1 ), . . . , (x m , y m )},(3)
and instead of minimizing er P (h), the learner searches for an h ∈ H minimizing the empirical loss on sample z:
er z (h) := 1 m m i=1 l(h(x i ), y i ).(4)
Of course, there are more intelligent things to do with the data than simply minimizing empirical error-for example one can add regularisation terms to avoid over-fitting. However we do not consider those issues here as they do not substantially alter the discussion. Minimizingêr z (h) is only a sensible thing to do if there is some guarantee that er z (h) is close to expected loss er P (h). This will in turn depend on the "richness" of the class H and the size of the training set (m). If H contains every possible function then clearly there can never be any guarantee thatêr z (h) is close to er P (h). The conditions ensuring convergence betweenêr z (h) and er P (h) are by now well understood; in the case Boolean function learning (Y = {0, 1}), convergence is controlled by VCdim(H)-the VC-dimension of H (see e.g. [1,12]). The following is typical of the theorems in this area.
Theorem 1 Let P be any probability distribution on X × {0, 1} and suppose z = {(x 1 , y 1 ), . . . , (x m , y m )} is generated by sampling m times from X ×{0, 1} according to P . Let d := VCdim(H). Then with probability at least 1 − δ (over the choice of the training set z), all h ∈ H will satisfy
er P (h) ≤êr z (h) + 32 m d ln 2em d + ln 4 δ 1/2(5)
There are a number of key points about this theorem:
1. We can never say for certain that er P (h) andêr z (h) are close, only that they are close with high probability (1 − δ). This is because no matter how large the training set, there is always a chance that we will get unlucky and generate a highly unrepresentative sample z.
2. Keeping the confidence parameter δ fixed, and ignoring log factors, (5) shows that the difference between the empirical estimateêr z (h) and the true loss er P (h) decays like d/m, uniformly for all h ∈ H. Thus, for sufficiently large training sets z and if d = VCdim(H) is finite, we can be confident that an h with small empirical error will generalise well.
3. If H contains an h with zero error, and the learner always chooses an h consistent with the training set, then the rate of convergence ofêr z (h) and er P (h) can be improved to d/m.
4.
Often results such as theorem 1 are called uniform convergence results, because they provide bounds for all h ∈ H.
Theorem 1 only provides conditions under which the deviation between er P (h) and er z is small, it does not guarantee that the true error er P (h) will actually be small. This is governed by the choice of H. If H contains a solution with small error and the learner minimizes error on the training set, then with high probability er P (h) will be small. However, a bad choice of H will mean there is no hope of achieving small error. Thus, the bias of the learner in the EP model is represented by the choice of H.
The Bayes approach
The Bayes approach and the EP approach are not all that different. In fact, the EP approach can be understood as a maximum likelihood approximation to the Bayes solution. The essential ingredients of the Bayes approach to modeling ordinary (single-task) learning are:
• An input space X and an output space Y ,
• a set of probability distributions P θ on X × Y , parameterised by θ ∈ Θ, and
• a prior distribution p(θ) on Θ.
The hypothesis space H of the EP model has been replaced with a set of distributions {P θ : θ ∈ Θ}. As with the EP model, the learning task is represented by a distribution P θ * on X × Y , only this time we assume realizability, i.e. that θ * ∈ Θ. The prior distribution p(θ) represents the learner's initial beliefs about the relative plausibility of each P θ . The richness of the set {P θ : θ ∈ Θ} and the prior p(θ) represent the bias of the learner. Again the learner does not know θ * , but has access to a training set z = {(x 1 , y 1 ), . . . , (x m , y m )} of pairs (x i , y i ) sampled according to P θ * . Upon receipt of the data z, the learner updates its prior distribution p(θ) to a posterior distribution p(θ|z) using Bayes rule:
p(θ|z) := p(z|θ)p(θ) p(z) (6) = m i=1 p(x i , y i |θ)p(θ) Θ p(z|θ)p(θ) dθ .
Often we are not interested in modeling the input distribution, only the conditional distribution on Y given X. In that case, p(x, y|θ) will factor into p(x)p(y|x; θ). The posterior distribution p(θ|z) can be used to predict the output y of a novel input x * by averaging:
p(y|x * ; z) := Θ p(y|x * ; θ)p(θ|z) dθ.(7)
One would hope that as the data increases, predictions made in this way would become increasingly accurate. There are many ways to measure what we mean by "accurate" in this setting. The one considered here is the Kullback-Liebler (KL) divergence between the true distribution P θ * on X × Y , and the posterior distribution P m on X × Y with density
p(x, y|z) := Θ p(x, y|θ)p(θ|z) dθ.(8)
The KL divergence between P θ * and P m is defined to be
D K (P * θ P m ) := X×Y p(x, y|θ * ) log p(x, y|θ * ) p(x, y|z) dx dy(9)
Note that if p(x, y|θ) = p(x)p(y|θ) as above then
D K (P * θ P m ) = X×Y p(x)p(y|x; θ * ) log p(y|x; θ * ) p(y|x; z) dx dy.(10)
This form of the KL divergence has a natural interpretation: it is (within one bit) the expected extra number of bits needed to encode the output y using a code generated by the posterior p(y|x; z), over and above what would be required using an optimal code (one generated from p(y|x; θ * )). The expectation is over all pairs (x, y) drawn according to the true distribution P θ * . This quantity is only zero if the posterior is equal to the true distribution.
In [8,2] an analysis of D K (P θ * P m ) was given for the limit of large training set size (m). They showed that if Θ is a compact subset of R d , and under certain extra restrictions which we won't discuss here:
D K (P θ * P m ) = d m + o 1 m (11) where o(1/m) stands for a function f (m) for which mf (m) → 0 as m → ∞.
There is a strong similarity between this result and theorem 1 in the zero error case (see note 3 after the theorem). D K (P θ * P m ) is the analogue of |er z (h)−er P (h)| in this case, but because we have assumed realizability, er z (h) = 0. So theorem 1 says that choosing any hypothesis consistent with the data will guarantee you an error of no more than d/m, where d is the VC dimension of the learner's hypothesis space. Although the error measure is different, equation (11) says essentially the same thing: if you classify novel data using a posterior generated according to Bayes rule, you will suffer an error of no more than d/m, where now d is the dimension of the Θ.
The empirical process (EP) model of learning to learn
Recall from the introduction that the main extra assumption of both the Bayes and EP models of bias learning is that the learner is embedded in an environment of related tasks, and can sample from the environment to generate multiple training sets belonging to multiple different tasks. In the EP model of ordinary (single-task) learning, the learning problem is represented by a distribution P on X × Y . So in the EP model of bias learning, an environment of learning problems is represented by a pair (P, Q) where P is the set of all probability distributions on X ×Y (i.e. P is the set of all possible learning problems), and Q is a distribution on P 1 . Q controls which learning problems the learner is likely to see. For example, if the learner is in a face recognition environment, Q will be highly peaked over face-recognition-type problems, whereas if the learner is in a character recognition environment Q will be peaked over character-recognition-type problems.
Recall from the end of section 2.1 that the learner's bias is represented by its choice of hypothesis space H. So to enable the learner to learn the bias, it is supplied with a family or set of hypothesis spaces H := {H}. As each H is itself a set of functions h: X → Y , H is a set of sets of functions.
Putting this together, formally a learning to learn or bias learning problem consists of:
• an input space X and an output space Y ,
• a loss function l: Y × Y → R,
• an environment (P, Q) where P is the set of all probability distributions on X × Y and Q is a distribution on P,
• a hypothesis space family H = {H} where each H ∈ H is a set of functions h: X → Y .
The goal of a bias learner is to find a hypothesis space H ∈ H minimizing the loss (recall equation (2))
er Q (H) := P inf h∈H er P (h) dQ(P ) (12) = P inf h∈H X×Y l(h(x), y) dP (x, y) dQ(P ).
The only way er Q (H) can be small is if, with high Q-probability, H contains a good solution to any problem P drawn at random according to Q. In this sense er Q (H) measures how appropriate the bias embodied by H is for the environment (P, Q).
In general the learner will not know Q, so it will not be able to find an H minimizing er Q (H) directly. However, the learner can sample from the environment in the following way:
• Sample n times from P according to Q to yield: P 1 , . . . , P n .
• Sample m times from X × Y according to each P i to yield:
z i = {(x i1 , y i1 ) . . . , (x im , y im )}.
• The learner receives the (n, m)-sample:
(x 11 , y 11 ) · · · (x 1m , y 1m ) = z 1 z := . . . . . . . . . . . . (x n1 , y n1 ) · · · (x nm , y nm ) = z n(13)
Note that an (n, m)-sample is simply n training sets z 1 , . . . , z n sampled from n different learning tasks P 1 , . . . , P n , where each task is selected according to the environmental probability distribution Q. Instead of minimizing er Q (H), the learner searches for an H ∈ H minimizing the empirical loss on the sample z, where this is defined by:
er z (H) := 1 n n i=1 inf h∈H er zi (h) (14) = 1 n n i=1 inf h∈H 1 m m j=1 l (h i (x ij ), y ij )
(recall equation (4)). Note thatêr z (H) is a biased estimate of er Q (H).
The question of generalisation within this framework now becomes: How many tasks (n) and how many examples of each task (m) do we need to ensure that er z (H) and er Q (H) are close with high probability?
Or, informally, how many tasks and how many examples of each task are required to ensure that a hypothesis space with good solutions to all the training tasks will contain good solutions to novel tasks drawn from the same environment?
In order to present the main theorem answering this question, some extra definitions must be introduced.
Definition 1 For any hypothesis
h: X → Y , define h l : X × Y → R by h l (x, y) := l(h(x), y)(15)
For any hypothesis space H in the hypothesis space family H, define
H l := {h l : h ∈ H}.(16)
For any sequence of n hypotheses
(h 1 , . . . , h n ), define (h 1 , . . . , h n ) l : (X × Y ) n → R by (h 1 , . . . , h n ) l (x 1 , y 1 , . . . , x n , y n ) := 1 n n i=1 l(h i (x i ), y i ).(17)
We will also use h l to denote (h 1 , . . . , h n ) l . For any H in the hypothesis space family H, define 
In the first part of the definition above, hypotheses h: X → Y are turned into functions h l mapping X × Y → R using the loss function. 
For the hypothesis space family H, define
H * := {H * : H ∈ H}.(21)
It is the "size" of H n l and H * that controls how large the (n, m)-sample z must be to ensure er z (H) and er Q (H) are close uniformly over all H ∈ H. Their size will be defined in terms of certain covering numbers, and in order to define the covering numbers, we need first to define how to measure the distance between elements of H n l and also between elements of H * .
Definition 3 Let P = (P 1 , . . . , P n ) be any sequence of n probability distributions on X × Y . For any h l , h ′ l ∈ H n l , define
d P (h l , h ′ l ) := (X×Y ) n |h l (x 1 , y 1 , . . . , x n , y n )−h ′ l (x 1 , y 1 , . . . , x n , y n )| dP 1 (x 1 , y 1 ) . . . dP n (x n , y n ) (22) For any H * 1 , H * 2 ∈ H * , define d Q (H * 1 , H * 2 ) := P |H * 1 (P ) − H * 2 (P )| dQ(P )(23)
It is easily verified that d P is a pseudo-metric on H n l , and similarly that d Q is a pseudo-metric on H * . A pseudo-metric is simply a metric without the condition that ρ(x, y) = 0 ⇒ x = y. For example, d Q (H * 1 , H * 2 ) could equal 0 simply because the distribution Q puts mass one on some distribution P for which H * 1 (P ) = H * 2 (P ), and not because H * 1 = H * 2 .
Definition 4 An ε-cover of (H * , d Q ) is a set {H * 1 , . . . , H * N } such that for all H * ∈ H * , d Q (H * , H i ) ≤ ε for some i = 1 . . . N . Let N (ε, H * , d Q ) denote the size of the smallest such cover. Set
C(ε, H * ) := sup Q N (ε, H * , d Q ).(24)
We can define N (ε, H n l , d P ) in a similar way, using d P in place of d Q . Again, set:
C(ε, H n l ) := sup P N (ε, H n l , d P ).(25)
Now we have enough machinery to state the main theorem.
Theorem 2 Let Q be any probability distribution on P, the set of all distributions on X × Y . Suppose z is an (n, m)-sample generated by sampling n times from P according to Q to give P 1 , . . . , P n , and then sampling m times from each P i to generate 
For a proof of a similar theorem to this one, see the proof of theorem 7 in [3]. Note that the constants in this theorem have not been heavily optimized.
There are several important points to note about theorem 2:
1. In order to learn to learn (in the sense that er Q (H) andêr z (H) are close uniformly over all H ∈ H), both the number of tasks n and the number of examples of each task m must be sufficiently large.
2. We can never say for certain that er Q (H) andêr z (H) are close, only that they are close with high probability (1 − δ). Regardless of the size of the (n, m) sample z, we still might get unlucky and generate unrepresentative learning problems P 1 , . . . , P n or unrepresentative examples of those learning problems, although the chance of being unlucky diminishes as m and n grow.
3. Once the learner has found an H ∈ H with a small value ofêr z (H), it will then use H to learn novel tasks P drawn according to Q. Assuming that the learner is always able to find an h ∈ H minimizing er P (h), theorem 2 tells us that with probability at least 1 − δ, the expected value of er P (h) on a novel task P will be less thanêr z (H) + ε. Of course, this does not rule out really bad performance on some tasks P . However, the probability of generating such "bad" tasks can be bounded. In particular, note that er Q (H) is just the expected value of the function H * over P, and so by Markov's inequality, for γ > 0,
Pr P : inf h∈H er P (h) ≥ γ = Pr {P : H * (P ) ≥ γ} ≤ E Q H * γ = er Q (H) γ ≤ê r z (H) + ε γ (with probability 1 − δ)
4. Keeping the accuracy and confidence parameters ε, δ fixed, note that the number of examples of each task required for good generalisation obeys
m = O 1 n ln C (ε, H n l ) .(29)
So provided ln C (ε, H n l ) increases sublinearly with n, the upper bound on the number of examples required of each task will decrease as the number of tasks increases. This is discussed further after theorem 3 below.
Theorem 2 only provides conditions under whichêr z (H) and er Q (H) are close, it does not guarantee that er Q (H) is actually small. This is governed by the choice of H. If H contains a hypothesis space H with a small value of er Q (H), and the learner minimizes error on the (n, m) sample z, then with high probability er Q (H) will be small. However, a bad choice of H will mean there is no hope of finding an H with small error. In this sense the choice of H represents the hyper-bias of the learner.
It may seem that we have simply replaced the problem of selecting the right bias (i.e. selecting the right hypothesis space H) with the equally difficult problem of selecting the right hyper-bias (i.e. the right hypothesis space family H). However, in many cases selecting the right hyper-bias is far easier than selecting the right bias. For example, in section 3.2 we will see how the feature selection problem may be viewed as a bias selection problem. Selecting the right features can be extremely difficult if one knows little about the environment, but specifying only that a set of features should exist and then learning those features is far simpler.
Learning multiple tasks
It may be that the learner is not interested in learning to learn, but simply wants to learn n tasks from the environment (P, Q). As in the previous section, we assume the learner starts out with a hypothesis space family H, and also that it receives an (n, m)-sample z generated from the n distributions P 1 , . . . , P n . This time, however, the learner is simply looking for n hypotheses (h 1 , . . . , h n ), all contained in the same hypothesis space H, such that the average training set error of the n hypotheses is minimal. Denoting (h 1 , . . . , h n ) by h, this error is defined bŷ 
er z (h) := 1 n n i=1ê r zi (h i )(30)nε 2 ln 4C( ε 24 , H n l ) δ , 18 ε 2 (32)
then with probability at least 1 − δ (over the choice of z), any h ∈ H n will satisfy
er P (h) ≤êr z (h) + ε(33)
Notes:
1. Note that the bound on m in theorem 3 is virtually identical to the bound on m in theorem 2.
2.
The important thing about the bound on m is that it depends inversely on the number of tasks n (assuming that the first part of the "max" expression is the dominate one). In fact, it is easy to show that for any hypothesis space family H,
C ε, H 1 l ≤ C (ε, H n l ) ≤ C ε, H 1 l n .(34)
Thus
ln C ε, H 1 l ≤ ln C (ε, H n l ) ≤ n ln C ε, H 1 l .(35)
So keeping the accuracy parameters ε and δ fixed, and plugging (35) into (32), we see that the upper bound on the number of examples required of each task never increases with the number of tasks, and at best decreases as O 1 n . Although only an upper bound, this provides a strong hint that learning multiple related tasks should be advantageuos on a "number of examples required per task" basis.
3. In section 3.2 it will be shown that all types of behaviour, from no advantage at all to O( 1 n ) decrease, are possible.  
Feature learning with neural networks
Consider the following quote:
The classical approach to estimating multidimensional functional dependencies is based on the following belief:
Real-life problems are such that there exists a small number of "strong features," simple functions of which (say linear combinations) approximate well the unknown function. Therefore, it is necessary to carefully choose a low-dimensional feature space and then to use regular statistical techniques to construct an approximation.
(from "The Nature of Statistical Learning Theory", Vapnik 1996.) It must be pointed out that Vapnik advocates an alternative approach in his book: that of using an extremely large number of simple features but choosing a hypothesis with maximum classification margin. However his approach cannot be viewed as a form of bias learning or learning to learn, whereas the strong feature approach can, so here we will concentrate on the latter. The aim of this section is to use the ideas of the previous section to show how neural-network feature sets can be learnt for an environment of related tasks.
In general, a set of features may be viewed as a map from the (typically highdimensional) input space R d to a much smaller dimensional space R k (k ≪ d). Any such bounded feature map can be approximated to arbitrarily high accuracy by a one-hidden-layer neural network with k output nodes. This is illustrated in Figure  1. Fixing the number of nodes in the first hidden layer, let Φ w : R d → R k denote the feature map computed by the the neural network with weights w. The set of all such feature maps is {Φ w : w ∈ R W } where W is the number of weights in the first two layers.
For argument's sake, assume the "simple functions" of the features (mentioned in the above quote) are squashed linear maps. Denoting the k components of the feature map Φ w by φ w,1 , . . . , φ w,k , each setting of the feature map weights generates a hypothesis space H w by
H w := σ k i=1 α i φ w,i : (α 1 , . . . , α k ) ∈ R k ,(36)
where σ: R → R is the squashing function. H w is simply the set of all squashed linear functions of the features Φ w . The set of all such hypothesis spaces,
H := {H w : w ∈ R W } (37)
is a hypothesis space family. Finding the right set of features for the environment (P, Q) is equivalent to finding the right hypothesis space H w ∈ H.
As in the previous section, the correct set of features may be learnt by finding a hypothesis space with small error on a sufficiently large (n, m)-sample z (recall that an (n, m)-sample is simply n training sets corresponding to n different learning tasks). Specializing to squared loss, in the present framework the error of H w on z (equation (14)) is given bŷ
er z (H w ) = 1 n n i=1 inf (α1,...,α k )∈R k 1 m m j=1 σ k l=1 α l φ w,l (x ij ) − y ij 2(38)
Using gradient descent and an n output node network as in figure 1, output weights (α 1 , . . . , α k ) and feature weights w minimizing (38) can be found. For details see [4]. The size of z ensuring that the resulting features will be good for learning novel tasks from the same environment is given by theorem 2. All we have to do is compute the logarithm of the covering numbers C(ε, H n l ) and C(ε, H * ). If the feature weights w and the output weights α 1 , . . . , α k are bounded, and the squashing function σ is Lipschitz 2 , then there exist constants κ, κ ′ (independent of ε, W and k) such that for all ε > 0, ln C(ε, H n l ) ≤ 2 (kn + W ) ln κ ε ln C(ε, H * ) ≤ 2W ln κ ′ ε (see [5] for a proof). Plugging these expressions into theorem 2 gives the following theorem.
Theorem 4 Let H = {H w } be a hypothesis space family where each hypothesis space H w is a set of squashed linear maps composed with a neural network feature map, as above. Suppose the number of features is k, and the total number of feature weights is W. Assume all feature weights and output weights are bounded, and the squashing function σ is Lipschitz. Let z be an (n, m)-sample generated from the environment (P, Q). If
n ≥ O 1 ε 2 W + log 1 δ ,(39)
and
m ≥ O 1 ε 2 k + 1 n W + log 1 δ (40)
then with probability at least 1 − δ any H w will satisfy 
er Q (H w ) ≤ er z (H w ) + ε.(41)
4.
If the "small number of strong features" assumption is correct, then k will be small. However, typically we will have very little idea of what the features are, so the size of the feature network will have to be huge, so W ≫ k. 7. To rigorously conclude that learning n tasks is better than learning one, we would have to show a matching lower bound of Ω(k + W/n) on the number of examples required of each task. Rather than search for lower bounds within the EP model (which are difficult to prove), we discuss a Bayes model of learning to learn in the next section where simultaneous upper and lower bounds appear more naturally.
The Bayes model of learning to learn
Recall from section 2.2 that in Bayesian models of ordinary learning the learner's bias is represented by the space of possible distributions {P θ : θ ∈ Θ} along with the choice of prior p(θ). The learning task P is assumed to be equal to some P θ * where θ * ∈ Θ. Observe that p(θ) is a subjective prior distribution over a set of distributions {P θ }. It is subjective because it simply reflects the prior beliefs of the learner, not some objective stochastic phenomenon. Now note that the environment (P, Q) consists of a set of distributions P, and a distribution Q on P, and furthermore that P can be sampled according to Q to generate multiple tasks P 1 , . . . , P n (recall the discussion in section 3). This makes Q an objective prior distribution. Objective in the sense that it can be sampled, i.e. it corresponds to some objective stochastic phenomenon.
If we assume P = {P θ : θ ∈ Θ} (so now P is a restricted subset of all possible distributions on X × Y ), the goal of a bias learner in this framework is to find the right prior distribution Q. To do this, the learner must have available a set of possible prior distributions {P π : π ∈ Π} from which to choose. Each P π is a distribution on Θ. We assume realizability, so that Q = P π * for some π * ∈ Π.
To summarize, the Bayes model of learning to learn consists of the following ingredients:
• An input space X and an output space Y ,
• a set of probability distributions P θ on X × Y , parameterised by θ ∈ Θ,
• a set of prior distributions P π on Θ, parameterised by π ∈ Π.
• an objective or environmental prior distribution P π * where π * ∈ Π.
• To complete the model, the learner also has a subjective hyper-prior distribution P Π on Π.
The two-tiered structure with a set of possible priors {P π : π ∈ Π} and a hyperprior p(π) on Π makes this model an example of a hierarchical Bayesian model [7,9].
As with the EP model, the learner receives an (n, m)-sample z, generated by first sampling n times from Θ according P π * to give θ 1 , . . . , θ n , and then sampling m times from each X × Y according to each P θi to generate z i = (x i1 , y i1 ), . . . , (x im , y im ). To simplify the notation in this section, let Z := X × Y and z ij := (x ij , y ij ). As it will be necessary to distinguish between (n, m − 1) samples and (n, m) samples, this will be made explicit in the notation by writing z (n,m) instead of z:
z (n,m) = z 11 · · · z 1m . . . . . . . . . z n1 · · · z nm(42)
Loss as the extra information required to predict the next observation
Recall that in the Bayes model of single task learning (section 2.2), the learner's loss was measured by the amount of extra information needed to encode novel examples of the task. So one way to measure the advantage in learning n tasks together is by the rate at which the learner's loss in predicting novel examples decays for each task. This question is similar to that addressed by theorem 3. So fix the number of tasks n, sample n tasks θ n = θ 1 , . . . , θ n according to the true prior P π * , and then for each m = 1, 2, . . . the learner has already seen m − 1 examples of each task:
z (n,m−1) = z 11 · · · z 1m−1 . . . . . . . . . z n1 · · · z nm−1(43)
where each row is drawn according to P m−1 θi (or equivalently, each column is drawn according to P θ n ). The learner then:
• generates the posterior distribution p(θ n |z (n,m−1) ) on the set of all n tasks, Θ n , according to Bayes' rule:
p(θ n |z (n,m−1) ) = p(z (n,m−1) |θ n )p(θ n ) p(z (n,m−1) ) = p(θ n ) n i=1 m−1 j=1 p(z ij |θ i ) p(z (n,m−1) )(44)
where p(z (n,m−1) ) = Θ n p(θ n )
n i=1 m−1 j=1 p(z ij |θ i ) dθ n .
• uses the posterior distribution to generate a predictive distribution on Z n , p(z n |z (n,m−1) ) = Θ n p(z n |θ n )p(θ n |z (n,m−1) ) dθ n ,
• and suffers a loss, L n,m , equal to the expected amount of extra information needed per task to encode a novel example of each task using the predictive distribution p(z n |z (n,m−1) ), over and above the minimum amount of information, i.e. the information required using the true distribution, p(z n |θ n ):
L n,m := 1 n E Z n |θ n log p(z n |θ n ) p(z n |z (n,m−1) ) .
Note that the loss at the first trial is:
L n,1 := E Z n |θ n log p(z n |θ n ) p(z n ) ,(47)
where p(z n ) is the learner's initial distribution on Z n before any data has arrived,
p(z n ) = Θ n p(z n |θ n )p(θ n ) dθ n = Π Θ n p(z n |θ n )p(θ n |π) dθ n p(π) dπ(48)
To understand better the meaning of L n,m , consider the loss associated with learning a single classification task. In this case Z = X × {0, 1}. If we assume that only the conditional distribution on class labels is affected by the model, then p(z|θ) = p(x)p(y|x, θ), and for the predictive distribution, p(z|z m ) = p(x)p(y|x, z m ). Let α(x) := p(y = 1|x, θ) and β(x) := p(y = 1|x, z m ). Substituting these expressions into (46) and simplifying yields
L 1,m = E X α(x) log α(x) β(x) + (1 − α(x)) log 1 − α(x) 1 − β(x) .(49)
The expression in square brackets is zero if α(x) = β(x), i.e. if the conditional distributions on class labels are the same for the true and predictive distributions. It increases slowly as α(x) and β(x) diverge. It turns out that L n,m is difficult to analyse, so instead we look at the cumulative loss over a sequence of trials:
C n,m,π * := m−1 k=0 E Θ n |π * E Z (n,k) |θ n L n,k+1 ,(50)
i.e. the expected total loss incurred by the learner after m steps of the above process. Note that the expectation is over all sequences of n tasks θ n drawn according to P π * and all (n, k)-samples drawn according to p(z n |θ n ).
(a, b) models
In [6], the asymptotic behaviour of C n,m,π * as a function of m was analysed for general hierarchical models. To illustrate the results, and to show how they apply to the feature learning example of section 3.2, here we will restrict our attention to (a, b)-models. The concept of an (a, b)-model was formally defined in [6] as follows:
Definition 5 An (a, b)-model is a hierarchical model in which Π = R b , Θ = R a × R b and the following conditions hold:
1. The priors p(θ|π) are of the form
p(θ = (x a , x b )|π) = δ(x b − π)g π (x a )(51)
where δ(·) is the b-dimensional Dirac delta function and g π is a continuous function on R a that also varies continuously with π.
2. The hyper-prior on Π, P Π has continuous density p(π) and the true prior π * has positive density p(π * ).
3. The conditional distributions p(z|θ) are twice continuously differentiable functions of θ.
The definition of an (a, b)-model in [6] contains two technical conditions which have been omitted in the above definition. The interested reader is referred to that paper for a full discussion. Condition 1 of an (a, b)-model formalizes the idea of a hierarchical model in which there are a + b parameters, b of which are effectively hyper-parameters and are fixed by the prior and the remaining a of which are model parameters. Conditions 2 and 3 ensure realizability and an appropriate level of smoothness.
The following definition is needed to state the main theorem.
Definition 6 Let (X, Σ, P ) be a measure space and f, g: N × X → R (N is the positive integeres) be two real-valued functions on N × X such that for all n ∈ N , f (n, ·) and g(n, ·) are measurable functions on X. Set X n := {x: f (n, x) = g(n, x)} for each n ∈ N . We say f (n, x) . = (X,P ) g(n, x)
if lim n→∞ P (X n ) = 1. This will be abbreviated to f (n, x) . = g(n, x) when X and P are clear from the context.
For the following theorem, fix n and take all limiting behaviour with respect to m. 
If the true prior π * is known then C n,m,π * = log m 2 (a) + o (log m) .
Note that equation (54) is an equality. In [6] it was stated as a . = relation but in fact the stronger expression holds.
Comparing (53) and (54), we see that as the number of tasks n increases, the effect of lack of knowledge of the true prior can be made arbitrarily small. Also, learning multiple tasks is most advantageous when b ≫ a, i.e. when the true model is quite small but our uncertainty concerning the true model is large. This is a similar conclusion to the one reached in section 3.2 (see note 3 after theorem 3.
Learning features within the Bayes model
Consider the feature learning model of section 3.2 (recall figure 1). In this case, Θ is the set of all possible neural networks implementable by fixing the feature weights and the weights of a single output node. As there are k output weights and W feature weights, Θ = R k+W . The realizability assumption means there exists a fixed set of features such that all tasks in the environment can be implemented by composing a squashed linear map with the feature set. Thus, the true prior distribution p(θ|π * ) is a delta-function over the correct feature weight setting, combined with an appropriate distribution over the k output weights (one that generates the tasks in the environment with the correct frequency). Assuming the output-weight distribution is continuous, the true prior is of the form (51), with a = k and b = W . The set of all such priors is simply the set of all distributions that are a delta function over some feature weight setting, combined with a continuous distribution over the output weights. To complete the model, suppose that for each θ ∈ R k+W , p(z|θ) is of the form p(y = 1, x|θ) = p(x)f θ (x) p(y = 0, x|θ) = p(x)(1 − f θ (x)) where f θ (x) is the output of the network with weights θ and input x, and p(x) is a continuous density on some compact subset of R d . Assume the sigmoid on the hidden nodes is σ(x) = tanh(x), and on the output node is σ(x) = (1 + tanh(x))/2.
With these assumptions, the neural network feature learning model is a (k, W )model in the sense of definition 5. Unfortunately, the neural network feature model does not satisfy the technical conditions mentioned after definition 5, so a straightforward application of theorem 5 is not possible. However, the difficulties are not insurmountable (see [6] for the details) and so we obtain the following theorem:
Theorem 6 For the neural-network feature learning model as above, the cumulative risk (50) satisfies C n,m,π * . = log m 2 k + W n + o (log m) .
If the true prior is known (i.e. the true feature weights are known), then C n,m,π * = log m 2 (k) + o (log m) .
Note the reappearance of the factor k + W/n (recall theorem 3 and the notes afterwards). Comparing equations (55) and (56) we see again that the effect of ignorance of the true prior (in this case ignorance of the right features) can be made arbitrarily small by learning sufficiently many tasks. The improvement is greatest when the number of features (k) is small, but our uncertainty as to what the right feature set should be is large.
Conclusion
Two mathematical models of bias learning (or learning to learn) have been discussed: one based on Empirical process theory and the other a Hierarchical Bayesian model. Both models show that if a learning machine is embedded within an environment of related learning tasks, then it can learn its own bias for the environment by learning sufficiently many tasks. Bounds were given on the number of tasks and number of examples of each task needed to ensure good generalisation from a bias learner. Good generalisation in this case means that with high probability the learner's choice of hypothesis space will contain good solutions to novel tasks drawn from the same environment. The theory was specialised to the case of feature learning with neural networks. There are many pattern classification problems that can be viewed as consisting of large number of related tasks and that seem to possess small feature sets. Speech recognition, character recognition, face recognition and fingerprint recognition all fit this bill. Feature learning in these domains should be particularly successful.
Notes
1. Strictly speaking, in order for Q to be well defined we need to specify a σ-algebra of subsets of P. However, such considerations are beyond the scope of the present discussion. See [3] for more details. 2. σ is Lipschitz if there exists a constant K such that σ(x, x ′ ) ≤ K|x − x ′ | for all x, x ′ ∈ R.
= {(h 1 , . . . , h n ) l : h 1 , . . . , h n ∈ H}.
H l is then just the collection of all such functions where the original hypotheses come from H. H l is often called a loss-function class. In our case we are interested in the average loss across n tasks, where each of the n hypotheses is chosen from a fixed hypothesis space H. This motivates the definition of h l and H n l . Finally, H n l is the collection of all (h 1 , . . . , h n ) l , with the restriction that all h 1 , . . . , h n belong to a single hypothesis space H ∈ H. Definition 2 For each H ∈ H, define H * : P → R by H * (P ) := inf h∈H er P (h).
z i = {(x i1 , y i1 ), . . . , (x im , y im )}, i = 1, . . . , n. Suppose the loss function l: Y × Y → R has range [0, 1] (any bounded loss function can be rescaled so this is true). Let H = {H} be any hypothesis space family. If the number of tasks n satisfies probability at least 1 − δ (over the (n, m)-sample z), all H ∈ H will satisfy er Q (H) ≤êr z (H) + ε
h i (x ij ), y ij ) For any hypothesis space H, let H n := {(h 1 , . . . , h n ): h i ∈ H, i = 1, . . . , n}. Let H n := ∪ H∈H H n . H n is simply the set of all possible sequences (h 1 , . . . , h n ) where all the h ′ i s come from the same hypothesis space H (recall the definition of H n l for a similar concept). Writing P = (P 1 , . . . , P n ), the learner's generalisation error in this context is measured by the average generalisation error across the n tasks: i (x), y) dP i (x, y) Recall definition 4 for the meaning of C(ε, H n l ). Theorem 3 Let P = (P 1 , . . . , P n ) be n probability distributions on X × Y and let z be an (n, m)-sample generated by sampling m times from X × Y according to each P i . Suppose the loss function l: Y × Y → R has range [0, 1] (any bounded loss function can be rescaled so this is true). Let H = {H} be any hypothesis space family. If the number of examples m of each task satisfies m ≥ max 72
Figure 1 .
1Neural network for feature learning. The feature map is implemented by the first two hidden layers. The n output nodes correspond to the n different tasks in the (n, m)-sample z.
Notes: 1 .
1Keeping the accuracy paramters ε and δ fixed, the upper bound on the number of examples required of each task behaves like O(k + W/n). The same upper bound also applies in theorem 3. 2. Once the feature map is learnt, only the output weights have to be estimated to learn a novel task. Again keeping the accuracy parameters fixed, this requires no more that O(k) examples. 3. Thus, as the number of tasks learnt increseas, the upper bound on the number of examples required of each task decays to the minimum possible, O(k).
5 .
5O(k + W/n) decreases most rapidly with increasing n when W ≫ k, so at least in terms of the upper bound on the number of examples required per task, learning small feature sets is an ideal application for learning to learn.6. Note that if we do away with the feature map altogether then W = 0 and the upper bound on m becomes O(k), independent of n. So in terms of the upper bound, learning n tasks becomes just as hard as learning one task. At the other extreme, if we fix the output weights then effctively k = 0 and the number of examples required of each task decreases as O(W/n). Thus a range of behaviour in the number of examples required of each task is possible: from no improvement at all to an O(1/n) decrease as the number of tasks n increases.
In an (a, b)-model, the learner's cumulative risk (50) satisfies C n,m,π * . = Π,PΠ log m 2 a + b n + o (log m) .
* This work was supported in part by EPSRC grants #K70366 and #K70373
Computational Learning Theory. Martin Anthony, Norman Biggs, Cambridge University PressCambridgeMartin Anthony and Norman Biggs. Computational Learning Theory. Cambridge University Press, Cambridge, 1992.
. Andrew Barron, Bertrand Clarke, Jeffreys' Prior is Asymptotically Least Favourable under Entropy Risk. Journal of Statistical Planning and Inference. 41Andrew Barron and Bertrand Clarke. Jeffreys' Prior is Asymptotically Least Favourable under Entropy Risk. Journal of Statistical Planning and Inference, 41:37-60, 1994.
A Model of Bias Learning. Jonathan Baxter, LSE-MPS-97Centre for Discrete and Applicable Mathematics. London School of EconomicsTechnical ReportSubmitted for publicationJonathan Baxter. A Model of Bias Learning. Technical Report LSE-MPS-97, London School of Economics, Centre for Discrete and Applicable Mathematics, November 1995. Submitted for publication. Copy available from http://keating.anu.edu.au/∼jon/papers/lse.ps.gz.
Learning Internal Representations. Jonathan Baxter, Proceedings of the Eighth International Conference on Computational Learning Theory. the Eighth International Conference on Computational Learning TheoryACM PressJonathan Baxter. Learning Internal Representations. In Proceedings of the Eighth Interna- tional Conference on Computational Learning Theory, pages 311-320. ACM Press, 1995.
Learning Internal Representations. Jonathan Baxter, Department of Mathematics and Statistics, The Flinders University of South AustraliaPhD thesisJonathan Baxter. Learning Internal Representations. PhD thesis, Department of Mathe- matics and Statistics, The Flinders University of South Australia, 1995. Copy available from http://keating.anu.edu.au/∼jon/papers/thesis.ps.gz.
A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling. Jonathan Baxter, Machine Learning. To AppearJonathan Baxter. A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling. Machine Learning, 1997. To Appear.
O James, Berger, Multivariate Estimation: Bayes, Empirical Bayes, and Stein Approaches. SIAM. James O Berger. Multivariate Estimation: Bayes, Empirical Bayes, and Stein Approaches. SIAM, 1986.
Information-Theoretic Asymptotics of Bayes Methods. Bertrand Clarke, Andrew Barron, IEEE Transactions on Information Theory. 36Bertrand Clarke and Andrew Barron. Information-Theoretic Asymptotics of Bayes Methods. IEEE Transactions on Information Theory, 36:453-471, 1990.
Some History of the Hierarchical Bayesian Methodology. I J Good ; J M Bernado, M H De Groot, D V Lindley, A F M Smith, Bayesian Statistics II. ValenciaUniversity PressI J Good. Some History of the Hierarchical Bayesian Methodology. In J M Bernado, M H De Groot, D V Lindley, and A F M Smith, editors, Bayesian Statistics II. University Press, Valencia, 1980.
A Treatise of Human Nature. David Hume, David Hume. A Treatise of Human Nature. 1737.
The need for biases in learning generalisations. M Tom, Mitchell, Readings in Machine Learning. Tom G Dietterich and Jude ShavlikMorgan KaufmannTom M Mitchell. The need for biases in learning generalisations. In Tom G Dietterich and Jude Shavlik, editors, Readings in Machine Learning. Morgan Kaufmann, 1991.
Estimation of Dependences based on Empirical Data. V N Vapnik, Springer-VerlagNew YorkV N Vapnik. Estimation of Dependences based on Empirical Data. Springer-Verlag, New York, 1982.
On the uniform convergence of relative frequencies of events to their probabilities. V N Vapnik, Ya Chervonenkis, Theory Probab. Appl. 16V N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl., 16:264-280, 1971.
Theoretical Models of Learning to Learn
https://arxiv.org/abs/2002.12364
http://robots.stanford.edu/
Sebastian Thrun
None
None
<div><div>
 
<p>I lead a team of expert computer scientists with the singular aim of significantly helping society through artificial intelligence technologies, and are constantly on the lookout for high-impact projects.</p>
<p>We have worked on robotics, self-driving cars, automated homes, healthcare, drones, and a number of other applications.</p>
<p>We currently focus on three areas: AI for healthcare, AI for people-prediction, and smart homes.</p>
<p><a href="http://www.guide2research.com/scientists">I made it onto the worldwide h-index list.</a></p><p>
<a href="https://www.ted.com/talks/sebastian_thrun_and_chris_anderson_the_new_generation_of_computers_is_programming_itself">Watch this</a> and <a href="https://sg.news.yahoo.com/kitty-hawk-ceo-thrun-says-211807857.html">this</a>.
</p><p>
I am a <a href="genealogy">direct descendent</a> (via Phd) of Carl Gauss, Gottfried Leibniz, Copernicus, and many other well-known mathematicians.
</p></div>||||I|||| Sebastian Thrun

 Adjunct Professor — Stanford University

 Home

 Research

 Students

 Papers

 FAQ

 Contact

 Personal

 I lead a team of expert computer scientists with the singular aim of significantly helping society through artificial intelligence technologies, and are constantly on the lookout for high-impact projects.

 We have worked on robotics, self-driving cars, automated homes, healthcare, drones, and a number of other applications.

 We currently focus on three areas: AI for healthcare, AI for people-prediction, and smart homes.

 I made it onto the worldwide h-index list.

 Watch this and this.

 I am a direct descendent (via Phd) of Carl Gauss, Gottfried Leibniz, Copernicus, and many other well-known mathematicians.

 The Thrun Lab — Stanford Artificial Intelligence Laboratory
I lead a team of expert computer scientists with the singular aim of significantly helping society through artificial intelligence technologies, and are constantly on the lookout for high-impact projects.
We have worked on robotics, self-driving cars, automated homes, healthcare, drones, and a number of other applications.
We currently focus on three areas: AI for healthcare, AI for people-prediction, and smart homes.
 I made it onto the worldwide h-index list.  
Watch this and this.
 
I am a direct descendent (via Phd) of Carl Gauss, Gottfried Leibniz, Copernicus, and many other well-known mathematicians.
Sebastian Thrun
http://robots.stanford.edu/
https://www.linkedin.com/in/sebastian-thrun-59a0b273/
Fear of AI is Overblown
2023-04-13
Sebastian Thrun
<div><div>
<section>
<section>
<figure>
</figure>
<div>
<div>
<h2>
 Sebastian Thrun
 
 
 </h2>
<h3>
</h3>
<h4>
</h4>
</div>
<div>
<p>
</p>
</div>
</div>
</section>
<section>
<h2>
 About
 </h2>
<div><p>
 Not on LinkedIn. Please send me email.
 
 
 
 
 
 </p>
</div>
</section>
<section>
<h2>
 Articles by Sebastian
 </h2>
</section>
<section>
<p>
<h2>
 Activity
 </h2>
</p>
</section>
<section>
<h2>
 
 Experience &amp; Education
 
 </h2>
<div>
<ul>
<li>
<div>
<h3>
 
 Self-employed
 
 </h3>
<h4>
</h4>
</div>
</li>
</ul>
<div>
<h2>View Sebastian’s full experience</h2>
<h3>See their title, tenure and more.</h3>
</div>
</div>
</section>
<section>
<h2>
 Languages
 </h2>
</section>
</section>
<section>
<section>
<p>
<h2>
 Explore collaborative articles
 </h2>
</p>
<p>
 We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
 </p>
<a href="https://www.linkedin.com/pulse/topics/home/">
 Explore More
 </a>
</section>
<section>
<h2>
 Others named <strong>Sebastian Thrun</strong>
</h2>
<div>
<p>
 3 others named Sebastian Thrun are on LinkedIn
 </p>
<p><a href="https://www.linkedin.com/pub/dir/Sebastian/Thrun?trk=public_profile_samename-see-all">
 See others named <strong>Sebastian Thrun</strong>
</a>
</p></div>
</section>
<section>
<h2>
 Add new skills with these courses
 </h2>
</section>
</section>
</div></div>||||I||||Skip to main content
LinkedIn
 * Articles
 * People
 * Learning
 * Jobs
 Join now Sign in

 Sebastian Thrun

 Mountain View, California, United States
 97K followers 500+ connections

 See your mutual connections

 View mutual connections with Sebastian

 Sign in

 Welcome back

 Email or phone

 Password
 Show

 Forgot password? Sign in

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 Join to view profile
 Self-employed
 University of Bonn, Germany
 * Report this profile

 About

 Not on LinkedIn. Please send me email.
 see more

 Welcome back

 Email or phone

 Password
 Show

 Forgot password? Sign in

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 Articles by Sebastian

 * Fear of AI is Overblown

 Fear of AI is Overblown

 By Sebastian Thrun

 Apr 13, 2023
 * Kill Our Meeting Culture

 Kill Our Meeting Culture

 By Sebastian Thrun

 Oct 12, 2015
 * How Not To Climb a Mountain (or How to Kill Your Start-Up)

 How Not To Climb a Mountain (or How to Kill Your Start-Up)

 By Sebastian Thrun

 Jan 12, 2015
 See all articles

 Activity

 * Amazing speakers planned for #UdacityIntersect including Eric Schmidt of Alphabet, Shiza Shahid of the Malala Fund, and Anne Toth of the World…

 Amazing speakers planned for #UdacityIntersect including Eric Schmidt of Alphabet, Shiza Shahid of the Malala Fund, and Anne Toth of the World…

 Shared by Sebastian Thrun

 * Software is no longer our turf. It's everyone's.

 Software is no longer our turf. It's everyone's.

 Liked by Sebastian Thrun

 * BERNARD'S BOOKSHELF - Interview on Self Publishing with Saeeda Hafiz

 BERNARD'S BOOKSHELF - Interview on Self Publishing with Saeeda Hafiz

 Liked by Sebastian Thrun

 Experience & Education

 * Self-employed

 **********

 * ******** **********

 ********* *** ******* *********

 * *******

 ********* *** ********

 * ********** ** ****, *******

 ****** ** ********** (***) ******** ******* *** **********

 1988 - 1995

 View Sebastian’s full experience

 See their title, tenure and more.

 Sign in

 Welcome back

 Email or phone

 Password
 Show

 Forgot password? Sign in

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 Languages

 * German

 -

 View Sebastian’s full profile

 * See who you know in common
 * Get introduced
 * Contact Sebastian directly
 Join to view full profile

 Sign in

 Stay updated on your professional world

 Sign in

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 Explore collaborative articles

 We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

 Explore More

Others named Sebastian Thrun

 * Sebastian Thrun

 --

 San Francisco, CA

 * Sebastian Thrun

 San Francisco, CA

 * Sebastian X-PLOSIVE Thrun

 Ambasador

 London

 3 others named Sebastian Thrun are on LinkedIn

 See others named Sebastian Thrun

Add new skills with these courses

 * 4h 6m

 Adobe Captivate Essential Training

 * 48m

 Teaching with LinkedIn Learning

 * 1h 39m

 Learning Moodle

 See all courses
 * LinkedIn © 2024
 * About
 * Accessibility
 * User Agreement
 * Privacy Policy
 * Your California Privacy Choices
 * Cookie Policy
 * Copyright Policy
 * Brand Policy
 * Guest Controls
 * Community Guidelines
 * 
 + العربية (Arabic)
 + Čeština (Czech)
 + Dansk (Danish)
 + Deutsch (German)
 + English (English)
 + Español (Spanish)
 + Français (French)
 + हिंदी (Hindi)
 + Bahasa Indonesia (Indonesian)
 + Italiano (Italian)
 + 日本語 (Japanese)
 + 한국어 (Korean)
 + Bahasa Malaysia (Malay)
 + Nederlands (Dutch)
 + Norsk (Norwegian)
 + Polski (Polish)
 + Português (Portuguese)
 + Română (Romanian)
 + Русский (Russian)
 + Svenska (Swedish)
 + ภาษาไทย (Thai)
 + Tagalog (Tagalog)
 + Türkçe (Turkish)
 + Українська (Ukrainian)
 + 简体中文 (Chinese (Simplified))
 + 正體中文 (Chinese (Traditional))
 Language

 Agree & Join LinkedIn

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 Sign in to view Sebastian’s full profile

 Sign in

 Welcome back

 Email or phone

 Password
 Show

 Forgot password? Sign in

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

 or

 By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.

 New to LinkedIn? Join now

Sebastian Thrun
 
 
 
 About
 
 
 Not on LinkedIn. Please send me email.
 
 
 
 
 
 
 Articles by Sebastian
 
 Activity
 
 
 Experience &amp; Education
 
 
 
 Self-employed
 
 
View Sebastian’s full experience
See their title, tenure and more.
 Languages
 
 Explore collaborative articles
 
 We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
 
 Explore More
 
 Others named Sebastian Thrun
 3 others named Sebastian Thrun are on LinkedIn
 
 
 See others named Sebastian Thrun
 
 Add new skills with these courses
Fear of AI is Overblown
https://www.linkedin.com/in/sebastian-thrun-59a0b273/
<div><div></div></div>||||I||||    Search Crunchbase
        Start Free Trial
          Chrome Extension
          Solutions
          Products
          Resources
          Pricing
          Resources
          Log In
                      Person

                    Sebastian Thrun

                    Save
                  Summary

                      Overview

                    Edit Overview Section
                    Number of Current Board & Advisor Roles 4
                    Number of Founded Organizations 5
                    * Primary Job Title CEO
                    * Primary Organization
                          Kitty Hawk
                    * Location Mountain View , California , United States
                    * Regions San Francisco Bay Area , Silicon Valley , West Coast
                    * Gender Male
                    * LinkedIn View on LinkedIn
                    * Twitter View on Twitter

                    Sebastian Thrun is a scientist, educator, inventor, and entrepreneur. Sebastian is the CEO of Kitty Hawk, whose vision is to free people from traffic. He is also the founder, chairman, and president of Udacity, whose mission is to democratize education.

                    Sebastian was the founder of X (previously Google X), where he led the development of the

                    self-driving car, Google Glass, and other projects. He spent several years as a professor at Stanford University where he led the Stanford Racing Team, whose “Stanley” won the DARPA Grand Challenge. Stanley is now on display at the Air and Space Museum in Washington, DC.

                  Read More

                      Jobs

                    Edit Jobs Section
                    Number of Current Jobs 6
                    Number of Past Jobs 1
                  Sebastian Thrun has 6 current jobs including Co-Founder and Chairman at Cresta , Co-Founder at Crossing Minds , and Co-founder and Chairman at Udacity . Additionally, Sebastian Thrun has had 1 past job as the Google Fellow and VP at Google .
                      * Cresta Co-Founder and Chairman 2017
                      * Crossing Minds Co-Founder Jun 1, 2016
                      * Udacity Co-founder and Chairman May 2012
                      * Stanford University Professor 2003
                      * Kitty Hawk CEO Apr 1958
                      * Waymo Co-founder
                                Organization Name            Title At Company            Start Date            End Date
                          Google                   Google Fellow and VP        Apr 2007              Aug 2014          
                      

                        Board and Advisor Roles

                      Edit Board and Advisor Roles Section
                      Number of Current Board & Advisor Roles 4
                    Sebastian Thrun holds 4 board and advisor roles including Board Member at Nimble Robotics , Board Member at AliveCor , and Board Member at Semantic Machines .
                        * Nimble Robotics Board Member Mar 2021
                        * AliveCor Board Member
                        * Semantic Machines Board Member
                        * Verity Board Member

                        Related Hubs

                      Edit Related Hubs Section
                                  Hub Name                                                                           CB Rank (Hub)
                            University of Notre Dame Mendoza College of Business Alumni Founded Companies  54,294                 
                            Sardar Patel University Alumni Founded Companies                               59,557                 
                            Northwestern University Pritzker School of Law Alumni Founded Companies        47,922                 
                            University of Montana Alumni Founded Companies                                 36,335                 
                            University of Wyoming Alumni Founded Companies                                 55,000                 
                            American University in Cairo Alumni Founded Companies                          46,880                 
                            Hacettepe Üniversitesi Alumni Founded Companies                                48,228                 
                            University of Michigan Alumni Founded Companies                                19,693                 
                            University of Brighton Alumni Founded Companies                                51,101                 
                            University of Navarra Alumni Founded Companies                                 43,254                 
                        
                      Show More

                          Education

                        Edit Education Section
                      Sebastian Thrun studied at University of Bonn , University of Bonn , and University of Hildesheim .
                          * University of Bonn MS Computer Science and Statistics 1993
                          * University of Bonn Ph.D Computer Science and Statistics 1995
                          * University of Hildesheim BS Computer Science, Economics, and Medicine 1988

                          Events

                        Edit Events Section
                        Number of Events 13
                      Sebastian Thrun has participated in 13 events. They most recently attended, or will attend, WSJ Tech Live 2019 on Oct 21, 2019 .
                          * WSJ Tech Live 2019 Speaker Laguna Beach , California , United States , North America Oct 21, 2019
                          * TechCrunch Disrupt SF 2019 Speaker San Francisco , California , United States , North America Oct 2, 2019
                          * Drive World Conference & Expo 2019 Speaker Santa Clara , California , United States , North America Aug 27, 2019
                          * Startup Grind Global Conference 2019 Speaker Redwood City , California , United States , North America Feb 23, 2019
                          * RSA Conference 2018 Speaker San Francisco , California , United States , North America Apr 15, 2018
                          * Vanity Fair New Establishment Summit 2017 Speaker Beverly Hills , California , United States , North America Oct 2, 2017
                          * TechCrunch Disrupt SF 2017 Speaker San Francisco , California , United States , North America Sep 18, 2017
                          * Udacity Intersect 2017 Speaker Mountain View , California , United States , North America Mar 8, 2017
                          * TechCrunch Disrupt SF 2016 Speaker San Francisco , California , United States , North America Sep 12, 2016
                          * Accenture Innovation Conference 2015 Speaker Johannesburg , NA - South Africa , South Africa , Africa Feb 26, 2015
                      Show More

                          Recent News and Activity

                        Edit Recent News and Activity Section
                        Number of News Articles 437
                          News • Jun 12, 2021
                            New York Times — What Is a Flying Car?
                          News • Jun 8, 2021
                            Forbes — THE FINTECH 50
                          News • May 26, 2021
                            PBS — Great Electric Airplane Race
                          News • Dec 18, 2020
                            Forbes — An Eye Toward The Future With Sebastian Thrun
                          News • Mar 13, 2020
                            Quartz — Coronavirus: That’s it, we’re grounded
                          News • Mar 7, 2020
                            The Guardian — Are flying taxis ready for lift-off?
                          News • Feb 6, 2020
                            TechCrunch — Registration is now open for Disrupt SF 2020
                          News • Feb 5, 2020
                            TechCrunch — Wisk signs deal to deploy an air taxi trial in New Zealand
                          News • Feb 4, 2020
                            Pulse 2.0 — Stanford Spin-Off Cresta Raises $21 Million
                          News • Feb 3, 2020
                            TechCrunch — AI startup Cresta launches from stealth with millions from Greylock and a16z
                        View All

                          Unlock even more features with Crunchbase Pro

                              Start Your Free Trial
                  Stay Connected
                  * Crunchbase News
                  * Subscribe to the Crunchbase Daily
                  * 
                    Who We Are
                    * Company
                    * Careers
                    * Partners
                    * Blog
                    * Press
                    * Contact Us
                    What We Do
                    * Crunchbase Pro
                    * Marketplace
                    * Crunchbase Enterprise
                    * Data Licensing
                    * Customer Stories
                    * Pricing
                    Helpful Links
                    * Create Profile
                    * Featured Lists and Searches
                    * The Crunchbase Difference
                    * Knowledge Center
                    * CA Privacy Notice
            Browse By: Organizations, People, Events
              Terms of Service | Privacy Policy | Sitemap | © 2024 Crunchbase Inc. All Rights Reserved.
                (33d6185 726)
              Cookie Settings | Do Not Sell or Share My Personal Info
        Total number of current Advisors and Board roles the person has
        Number of Organizations that the person founded
        The person's primary job title
        The organization associated to the person's primary job
        Where the person is located (e.g. Europe, Menlo Park, China)
        Where the person is located (e.g. San Francisco Bay Area, Silicon Valley)
        A Person's gender
        Link to a Person's LinkedIn page
        Link to a Person's Twitter page
        Total number of current Jobs the person has
        Total number of past Jobs the person has
        Organization Name: This is the name of the organization
        Title At Company: Title of a Person's Job
        Start Date: Start date of the Person's Job
        End Date: End date of the Person's Job
        Hub Name: Name of the Hub
        CB Rank (Hub): Algorithmic rank assigned to the top 100,000 most active Hubs
        Total number of events the individual appeared in
        Number of news articles that reference the Person
Search Crunchbase
        Start Free Trial
          Chrome Extension
          Solutions
          Products
          Resources
          Pricing
          Resources
          Log In
                      Person
                    Sebastian Thrun
                    Save
                  Summary
                      Overview
                    Edit Overview Section
                    Number of Current Board & Advisor Roles 4
                    Number of Founded Organizations 5
                    * Primary Job Title CEO
                    * Primary Organization
                          Kitty Hawk
                    * Location Mountain View , California , United States
                    * Regions San Francisco Bay Area , Silicon Valley , West Coast
                    * Gender Male
                    * LinkedIn View on LinkedIn
                    * Twitter View on Twitter
                    Sebastian Thrun is a scientist, educator, inventor, and entrepreneur. Sebastian is the CEO of Kitty Hawk, whose vision is to free people from traffic. He is also the founder, chairman, and president of Udacity, whose mission is to democratize education.
                    Sebastian was the founder of X (previously Google X), where he led the development of the
                    self-driving car, Google Glass, and other projects. He spent several years as a professor at Stanford University where he led the Stanford Racing Team, whose “Stanley” won the DARPA Grand Challenge. Stanley is now on display at the Air and Space Museum in Washington, DC.
                  Read More
                      Jobs
                    Edit Jobs Section
                    Number of Current Jobs 6
                    Number of Past Jobs 1
                  Sebastian Thrun has 6 current jobs including Co-Founder and Chairman at Cresta , Co-Founder at Crossing Minds , and Co-founder and Chairman at Udacity . Additionally, Sebastian Thrun has had 1 past job as the Google Fellow and VP at Google .
                      * Cresta Co-Founder and Chairman 2017
                      * Crossing Minds Co-Founder Jun 1, 2016
                      * Udacity Co-founder and Chairman May 2012
                      * Stanford University Professor 2003
                      * Kitty Hawk CEO Apr 1958
                      * Waymo Co-founder
                                Organization Name            Title At Company            Start Date            End Date
                          Google                   Google Fellow and VP        Apr 2007              Aug 2014          
                      
                        Board and Advisor Roles
                      Edit Board and Advisor Roles Section
                      Number of Current Board & Advisor Roles 4
                    Sebastian Thrun holds 4 board and advisor roles including Board Member at Nimble Robotics , Board Member at AliveCor , and Board Member at Semantic Machines .
                        * Nimble Robotics Board Member Mar 2021
                        * AliveCor Board Member
                        * Semantic Machines Board Member
                        * Verity Board Member
                        Related Hubs
                      Edit Related Hubs Section
                                  Hub Name                                                                           CB Rank (Hub)
                            University of Notre Dame Mendoza College of Business Alumni Founded Companies  54,294                 
                            Sardar Patel University Alumni Founded Companies                               59,557                 
                            Northwestern University Pritzker School of Law Alumni Founded Companies        47,922                 
                            University of Montana Alumni Founded Companies                                 36,335                 
                            University of Wyoming Alumni Founded Companies                                 55,000                 
                            American University in Cairo Alumni Founded Companies                          46,880                 
                            Hacettepe Üniversitesi Alumni Founded Companies                                48,228                 
                            University of Michigan Alumni Founded Companies                                19,693                 
                            University of Brighton Alumni Founded Companies                                51,101                 
                            University of Navarra Alumni Founded Companies                                 43,254                 
                        
                      Show More
                          Education
                        Edit Education Section
                      Sebastian Thrun studied at University of Bonn , University of Bonn , and University of Hildesheim .
                          * University of Bonn MS Computer Science and Statistics 1993
                          * University of Bonn Ph.D Computer Science and Statistics 1995
                          * University of Hildesheim BS Computer Science, Economics, and Medicine 1988
                          Events
                        Edit Events Section
                        Number of Events 13
                      Sebastian Thrun has participated in 13 events. They most recently attended, or will attend, WSJ Tech Live 2019 on Oct 21, 2019 .
                          * WSJ Tech Live 2019 Speaker Laguna Beach , California , United States , North America Oct 21, 2019
                          * TechCrunch Disrupt SF 2019 Speaker San Francisco , California , United States , North America Oct 2, 2019
                          * Drive World Conference & Expo 2019 Speaker Santa Clara , California , United States , North America Aug 27, 2019
                          * Startup Grind Global Conference 2019 Speaker Redwood City , California , United States , North America Feb 23, 2019
                          * RSA Conference 2018 Speaker San Francisco , California , United States , North America Apr 15, 2018
                          * Vanity Fair New Establishment Summit 2017 Speaker Beverly Hills , California , United States , North America Oct 2, 2017
                          * TechCrunch Disrupt SF 2017 Speaker San Francisco , California , United States , North America Sep 18, 2017
                          * Udacity Intersect 2017 Speaker Mountain View , California , United States , North America Mar 8, 2017
                          * TechCrunch Disrupt SF 2016 Speaker San Francisco , California , United States , North America Sep 12, 2016
                          * Accenture Innovation Conference 2015 Speaker Johannesburg , NA - South Africa , South Africa , Africa Feb 26, 2015
                      Show More
                          Recent News and Activity
                        Edit Recent News and Activity Section
                        Number of News Articles 437
                          News • Jun 12, 2021
                            New York Times — What Is a Flying Car?
                          News • Jun 8, 2021
                            Forbes — THE FINTECH 50
                          News • May 26, 2021
                            PBS — Great Electric Airplane Race
                          News • Dec 18, 2020
                            Forbes — An Eye Toward The Future With Sebastian Thrun
                          News • Mar 13, 2020
                            Quartz — Coronavirus: That’s it, we’re grounded
                          News • Mar 7, 2020
                            The Guardian — Are flying taxis ready for lift-off?
                          News • Feb 6, 2020
                            TechCrunch — Registration is now open for Disrupt SF 2020
                          News • Feb 5, 2020
                            TechCrunch — Wisk signs deal to deploy an air taxi trial in New Zealand
                          News • Feb 4, 2020
                            Pulse 2.0 — Stanford Spin-Off Cresta Raises $21 Million
                          News • Feb 3, 2020
                            TechCrunch — AI startup Cresta launches from stealth with millions from Greylock and a16z
                        View All
                          Unlock even more features with Crunchbase Pro
                              Start Your Free Trial
                  Stay Connected
                  * Crunchbase News
                  * Subscribe to the Crunchbase Daily
                  * 
                    Who We Are
                    * Company
                    * Careers
                    * Partners
                    * Blog
                    * Press
                    * Contact Us
                    What We Do
                    * Crunchbase Pro
                    * Marketplace
                    * Crunchbase Enterprise
                    * Data Licensing
                    * Customer Stories
                    * Pricing
                    Helpful Links
                    * Create Profile
                    * Featured Lists and Searches
                    * The Crunchbase Difference
                    * Knowledge Center
                    * CA Privacy Notice
            Browse By: Organizations, People, Events
              Terms of Service | Privacy Policy | Sitemap | © 2024 Crunchbase Inc. All Rights Reserved.
                (33d6185 726)
              Cookie Settings | Do Not Sell or Share My Personal Info
        Total number of current Advisors and Board roles the person has
        Number of Organizations that the person founded
        The person's primary job title
        The organization associated to the person's primary job
        Where the person is located (e.g. Europe, Menlo Park, China)
        Where the person is located (e.g. San Francisco Bay Area, Silicon Valley)
        A Person's gender
        Link to a Person's LinkedIn page
        Link to a Person's Twitter page
        Total number of current Jobs the person has
        Total number of past Jobs the person has
        Organization Name: This is the name of the organization
        Title At Company: Title of a Person's Job
        Start Date: Start date of the Person's Job
        End Date: End date of the Person's Job
        Hub Name: Name of the Hub
        CB Rank (Hub): Algorithmic rank assigned to the top 100,000 most active Hubs
        Total number of events the individual appeared in
        Number of news articles that reference the Person
Sebastian Thrun - CEO @ Kitty Hawk - Crunchbase Person Profile
https://www.crunchbase.com/person/sebastian-thrun
https://lsc.org/support-us/the-genius-of-nj/sebastian-thrun
Sebastian Thrun
2012-05-11
None
<div><div>
<div>
<ul><li><a href="https://lsc.org/">Home</a></li><li><a href="https://lsc.org/support-us">Support Us</a></li><li><a href="https://lsc.org/support-us/the-genius-of-nj">The Genius of NJ</a></li><li>Sebastian Thrun</li></ul> </div>
<div>
<div>
<p><strong>CEO, Kitty Hawk</strong>
</p>
<p><strong>Godfather of the Self-Driving Car</strong>
</p>
<p>Sebastian Thrun is a scientist, educator, inventor, and entrepreneur. Thrun is the CEO of Kitty Hawk, whose vision is to free people from traffic. He is also the founder, chairman, and president of Udacity, whose mission is to democratize education. Thrun was the founder of X (previously Google X), where he led the development of the self-driving car, Google Glass, and other projects. He spent several years as a professor at Stanford University where he led the Stanford Racing Team, whose “Stanley” won the DARPA Grand Challenge. Stanley is now on display at the Air and Space Museum in Washington, DC.
</p>
</div>
<div>
<p><a href="https://lsc.org/the-genius-of-nj"></a>
</p>
<p><a href="https://lsc.org/the-genius-of-nj/tickets-tables"></a>
</p>
</div>
</div>
</div></div>||||I|||| Close
 Home
 Become a Member
 * Explore
 Back
 + Exhibitions
 + Planetarium
 + 3D Films
 + Members Get More
 + LSC After Dark
 o Space Talk
 + Birthday Parties
 + Coming Soon
 Pixel Art Find out more
 * Visit
 Back
 + Directions, Pricing & Hours
 + Schedule & Map
 + Travel & Tourism
 + Food & Drink
 + Health, Accessibility & Safety
 + Gift Shop
 + FAQ
 + COVID-19

 Location

 Liberty State Park
 222 Jersey City Boulevard
 Jersey City , NJ 07305

 Contact

 + Phone: (201) 200-1000
 + Contact us by email

 Upcoming Hours

 + April 29 & 30 10 am - 4 pm
 + May 1-3 10 am - 4 pm
 + May 4 & 5 10 am - 5 pm
 + May 6-10 10 am - 4 pm
 + May 11 & 12 10 am - 5 pm
 Energy Quest Find out more
 * Education
 Back
 + Educators
 o Field Trips
 o Jennifer Chalsty Planetarium
 o Science On a Sphere
 o Live From Surgery
 o Maker & Tech
 o Professional Development
 o Partners in Science
 o Home School
 o Stage Shows
 o STEM Education Guide
 o LSC at Your School
 o Climate Change Programs
 o Early Childhood
 + Families
 o Community Evenings
 o BASF's Kids' Lab
 o LSC Science Camp
 + Groups & Scouts
 o Scout Programming
 o Camp Field Trips
 + LSC Space News Now
 + LSC in the House
 Our Hudson Home Find out more
 * Support Us
 Back
 + Membership
 + Make a Gift
 + Luminary Society
 + Other Ways to Give
 + Our Impact
 + Corporate and Foundation Support
 + Genius Gala 12
 + SciTech Scity
 + Volunteers
 Bees to Bots Find out more
 * About Us
 Back
 + About Liberty Science Center
 + Plan an Event
 + Media Information
 + LSC Blog
 + Our Leaders
 o Board of Trustees
 o LSC Executive Staff
 + LSC Memory Lab
 Microbes Rule! Find out more
 Search
 Buy Tickets Display navigation
 * Home
 * Support Us
 * The Genius of NJ
 * Sebastian Thrun

 SEBASTIAN THRUN

 CEO, Kitty Hawk

 Godfather of the Self-Driving Car

 Sebastian Thrun is a scientist, educator, inventor, and entrepreneur. Thrun is the CEO of Kitty Hawk, whose vision is to free people from traffic. He is also the founder, chairman, and president of Udacity, whose mission is to democratize education. Thrun was the founder of X (previously Google X), where he led the development of the self-driving car, Google Glass, and other projects. He spent several years as a professor at Stanford University where he led the Stanford Racing Team, whose “Stanley” won the DARPA Grand Challenge. Stanley is now on display at the Air and Space Museum in Washington, DC.

 Location

 Liberty State Park
 222 Jersey City Boulevard
 Jersey City , NJ 07305

 Contact

 * Phone: (201) 200-1000
 * Contact us by email

 Upcoming Hours

 * April 29 & 30 10 am - 4 pm
 * May 1-3 10 am - 4 pm
 * May 4 & 5 10 am - 5 pm
 * May 6-10 10 am - 4 pm
 * May 11 & 12 10 am - 5 pm

 Subscribe

 Sign up to receive our e-newsletter for news, special offers, and more.

 Send me updates
 Visit Us
 * Liberty Science Center Facebook page
 * Liberty Science Center Twitter feed
 * Liberty Science Center on Instagram
 * Liberty Science Center on YouTube
 * Liberty Science Center on Snapchat

 Liberty Science Center® is a nonprofit organization

 * Careers
 * Privacy Policy
 * Press
 * Sitemap
 * Rentals

 © Liberty Science Center 2024. All rights reserved.

 Close navigation
Home   Support Us   The Genius of NJ  Sebastian Thrun  
 CEO, Kitty Hawk
 Godfather of the Self-Driving Car
Sebastian Thrun is a scientist, educator, inventor, and entrepreneur. Thrun is the CEO of Kitty Hawk, whose vision is to free people from traffic. He is also the founder, chairman, and president of Udacity, whose mission is to democratize education. Thrun was the founder of X (previously Google X), where he led the development of the self-driving car, Google Glass, and other projects. He spent several years as a professor at Stanford University where he led the Stanford Racing Team, whose “Stanley” won the DARPA Grand Challenge. Stanley is now on display at the Air and Space Museum in Washington, DC.
Sebastian Thrun
https://lsc.org/support-us/the-genius-of-nj/sebastian-thrun
<div><div>
						











						
						







<div>
		
	</div>
						




<div>
		<div>
			<div>
				<h3>IEEE Account</h3>
				<ul>
					<li><a href="https://www.ieee.org/profile/changeusrpwd/showChangeUsrPwdPage.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Change Username/Password</a></li>
					<li><a href="https://www.ieee.org/profile/address/getAddrInfoPage.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Update Address</a></li>
				</ul>
			</div>
			<div>
				<h3>Purchase Details</h3>
				<ul>
					<li><a href="https://www.ieee.org/profile/payment/showPaymentHome.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Payment Options</a></li>
					<li><a href="https://www.ieee.org/profile/vieworder/showOrderHistory.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Order History</a></li>
					<li><a href="https://ieeexplore.ieee.org/articleSale/purchaseHistory.jsp">View Purchased Documents</a></li>
				</ul>
			</div>
			<div>
				<h3>Profile Information</h3>
				<ul>
					<li><a href="https://www.ieee.org/ieee-privacyportal/app/ibp?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Communications Preferences</a></li>
					<li><a href="https://www.ieee.org/profile/profedu/getProfEduInformation.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Profession and Education</a></li>
					<li><a href="https://www.ieee.org/profile/tips/getTipsInfo.html?refSite=https://ieeexplore.ieee.org&amp;refSiteName=IEEE Xplore">Technical Interests</a></li>
				</ul>
			</div>
			<div>
				<h3>Need Help?</h3>
				<ul>
					<li><strong>US &amp; Canada:</strong> +1 800 678 4333</li>
					<li><strong>Worldwide: </strong> +1 732 981 0060<br/>
					</li>
					<li><a href="https://ieeexplore.ieee.org/xpl/contact">Contact &amp; Support</a></li>
				</ul>
			</div>
		</div>
		<div>
						<ul>
							<li><a href="https://ieeexplore.ieee.org/Xplorehelp/about-ieee-xplore.html">About IEEE <em>Xplore</em></a></li>
							<li><a href="https://ieeexplore.ieee.org/xpl/contact">Contact Us</a></li>
							<li><a href="https://ieeexplore.ieee.org/Xplorehelp/Help_start.html">Help</a></li>
							<li><a href="https://ieeexplore.ieee.org/Xplorehelp/accessibility-statement.html">Accessibility</a></li> 
							<li><a href="https://ieeexplore.ieee.org/Xplorehelp/Help_Terms_of_Use.html">Terms of Use</a></li>
							<li><a href="http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html">Nondiscrimination Policy</a></li>
							<li><a href="https://ieeexplore.ieee.org/xpl/sitemap.jsp">Sitemap</a></li>
							<li><a href="http://www.ieee.org/about/help/security_privacy.html">Privacy &amp; Opting Out of Cookies</a></li>
						</ul>
						<p>
							A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.<br/>© Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.
						</p>
					</div>
	</div>

						
						






		
		
		
		
		
		
		
		
		
		
		
		
		
			
				
			
			
			
			
		
		
		
		
		
		

			


			

			
			


		
 		
		
			 

						
						






	
	






















	
	


















	












	


















		
			
					
			
					
			
				
					








					
			
					
			
					
			
					
			
					
			
		
		
	




	
	
	
	
	
	
	


	
	
	

	















						







					</div></div>||||I||||            IEEE Account

              * Change Username/Password
              * Update Address

            Purchase Details

              * Payment Options
              * Order History
              * View Purchased Documents

            Profile Information

              * Communications Preferences
              * Profession and Education
              * Technical Interests

            Need Help?

              * US & Canada: +1 800 678 4333
              * Worldwide: +1 732 981 0060
              * Contact & Support
                  * About IEEE Xplore
                  * Contact Us
                  * Help
                  * Accessibility
                  * Terms of Use
                  * Nondiscrimination Policy
                  * Sitemap
                  * Privacy & Opting Out of Cookies

                A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
                © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

IEEE Account
				
					 Change Username/Password 
					 Update Address 
				
			
			
				Purchase Details
				
					 Payment Options 
					 Order History 
					 View Purchased Documents 
				
			
			
				Profile Information
				
					 Communications Preferences 
					 Profession and Education 
					 Technical Interests 
				
			
			
				Need Help?
				
					 US &amp; Canada: +1 800 678 4333
					 Worldwide:  +1 732 981 0060
					
					 Contact &amp; Support 
				
			
		
		
						
							 About IEEE Xplore  
							 Contact Us 
							 Help 
							 Accessibility  
							 Terms of Use 
							 Nondiscrimination Policy 
							 Sitemap 
							 Privacy &amp; Opting Out of Cookies 
						
						
							A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.© Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.
https://ieeexplore.ieee.org/author/37280207800
https://arxiv.org/abs/2112.04426
Improving language models by retrieving from trillions of tokens
2021-12-08
Borgeaud; Sebastian; Mensch; Arthur; Hoffmann; Jordan; Cai; Trevor; Rutherford; Eliza; Millican; Katie; Driessche; George van den; Lespiau; Jean-Baptiste; Damoc; Bogdan; Clark; Aidan; Casas; Diego de Las; Guy; Aurelia; Menick; Jacob; Ring; Roman; Hennigan; Tom; Huang; Saffron; Maggiore; Loren; Jones; Chris; Cassirer; Albin; Brock; Andy; Paganini; Michela; Irving; Geoffrey; Vinyals; Oriol; Osindero; Simon; Karen; Rae; Jack W; Elsen; Erich; Sifre; Laurent
<div><div>
 
 
 
 <div><p>Authors:<a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Borgeaud%2C+S">Sebastian Borgeaud</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mensch%2C+A">Arthur Mensch</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hoffmann%2C+J">Jordan Hoffmann</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cai%2C+T">Trevor Cai</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rutherford%2C+E">Eliza Rutherford</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Millican%2C+K">Katie Millican</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=van+den+Driessche%2C+G">George van den Driessche</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lespiau%2C+J">Jean-Baptiste Lespiau</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Damoc%2C+B">Bogdan Damoc</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Clark%2C+A">Aidan Clark</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=de+Las+Casas%2C+D">Diego de Las Casas</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Guy%2C+A">Aurelia Guy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Menick%2C+J">Jacob Menick</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ring%2C+R">Roman Ring</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hennigan%2C+T">Tom Hennigan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Huang%2C+S">Saffron Huang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Maggiore%2C+L">Loren Maggiore</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Jones%2C+C">Chris Jones</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cassirer%2C+A">Albin Cassirer</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Brock%2C+A">Andy Brock</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Paganini%2C+M">Michela Paganini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Irving%2C+G">Geoffrey Irving</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Vinyals%2C+O">Oriol Vinyals</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Osindero%2C+S">Simon Osindero</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Simonyan%2C+K">Karen Simonyan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rae%2C+J+W">Jack W. Rae</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Elsen%2C+E">Erich Elsen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sifre%2C+L">Laurent Sifre</a></p></div>
 
 
 
 
 
 <p><a href="/pdf/2112.04426">Download PDF</a></p><blockquote>
 Abstract: We enhance auto-regressive language models by conditioning on document chunks
retrieved from a large corpus, based on local similarity with preceding tokens.
With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO)
obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite
using 25$\times$ fewer parameters. After fine-tuning, RETRO performance
translates to downstream knowledge-intensive tasks such as question answering.
RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked
cross-attention mechanism to predict tokens based on an order of magnitude more
data than what is typically consumed during training. We typically train RETRO
from scratch, yet can also rapidly RETROfit pre-trained transformers with
retrieval and still achieve good performance. Our work opens up new avenues for
improving language models through explicit memory at unprecedented scale.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Sebastian Borgeaud [<a href="/show-email/eb42d80c/2112.04426">view email</a>]
 <br />
 
 <strong><a href="/abs/2112.04426v1">[v1]</a></strong>
 
 Wed, 8 Dec 2021 17:32:34 UTC (17,276 KB)<br />
 
 
 <strong><a href="/abs/2112.04426v2">[v2]</a></strong>
 
 Tue, 11 Jan 2022 09:14:18 UTC (17,276 KB)<br />
 
 <strong>[v3]</strong>
 
 Mon, 7 Feb 2022 21:07:59 UTC (17,278 KB)<br />
 </p></div></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2112.04426

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:2112.04426 (cs)
 [Submitted on 8 Dec 2021 (v1), last revised 7 Feb 2022 (this version, v3)]

 Title: Improving language models by retrieving from trillions of tokens

 Authors: Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, Laurent Sifre
 Download a PDF of the paper titled Improving language models by retrieving from trillions of tokens, by Sebastian Borgeaud and 27 other authors
 Download PDF
 Abstract: We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.
 Comments: Fix incorrect reported numbers in Table 14 
 Subjects: Computation and Language (cs.CL) ; Machine Learning (cs.LG)
 Cite as: arXiv:2112.04426 [cs.CL] 
 (or arXiv:2112.04426v3 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.2112.04426 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Sebastian Borgeaud [view email]
 [v1] Wed, 8 Dec 2021 17:32:34 UTC (17,276 KB)
 [v2] Tue, 11 Jan 2022 09:14:18 UTC (17,276 KB)
 [v3] Mon, 7 Feb 2022 21:07:59 UTC (17,278 KB)
 Full-text links:

 Download:

 * Download a PDF of the paper titled Improving language models by retrieving from trillions of tokens, by Sebastian Borgeaud and 27 other authors
 PDF
 * Other formats
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 2112
 Change to browse by:
 cs
 cs.LG

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 2 blog links

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Sebastian Borgeaud
 Arthur Mensch
 Jordan Hoffmann
 George van den Driessche
 Jean-Baptiste Lespiau
 …
 a export BibTeX citation Loading...

 BibTeX formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 DagsHub Toggle
 DagsHub (What is DagsHub?)
 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Link to Influence Flower
 Influence Flower (What are Influence Flowers?)
 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Improving language models by retrieving from trillions of tokens
Date Published: 2022-02-07
Authors:
Sebastian  Borgeaud
Arthur  Mensch
Jordan  Hoffmann
Trevor  Cai
Eliza  Rutherford
Katie  Millican
George  Van Den Driessche
Jean-Baptiste  Lespiau
Bogdan  Damoc
Aidan  Clark
Diego  De
Las  Casas
Aurelia  Guy
Jacob  Menick
Roman  Ring
Tom  Hennigan
Saffron  Huang
Loren  Maggiore
Chris  Jones
Albin  Cassirer
Andy  Brock
Michela  Paganini
Geoffrey  Irving
Oriol  Vinyals
Simon  Osindero
Karen  Simonyan
Jack W Rae
Erich  Elsen
Laurent  Sifre
Abstract
We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a 2 trillion token database, our Retrieval-Enhanced Transformer (R ) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters. After fine-tuning, R performance translates to downstream knowledge-intensive tasks such as question answering. R combines a frozen B retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train R from scratch, yet can also rapidly R fit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.
1. Introduction
Language modelling (LM) is an unsupervised task that consists of modelling the probability of text, usually by factorising it into conditional next-token predictions 𝑝(𝑥 1 , . . . , 𝑥 𝑛 ) = 𝑖 𝑝(𝑥 𝑖 |𝑥
Improving language models by retrieving from trillions of tokens
https://arxiv.org/abs/2112.04426
https://www.scribbr.com/category/research-paper/
How to Write a Research Paper | A Beginner's Guide
2019-08-01
None
<div><section>
 <p>A research paper is a piece of <a href="https://www.scribbr.com/category/academic-writing/">academic writing</a> that provides analysis, interpretation, and argument based on in-depth independent research.</p>
<p>Research papers are similar to <a href="https://www.scribbr.com/category/academic-essay/">academic essays</a>, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research. Writing a research paper requires you to demonstrate a strong knowledge of your topic, engage with a variety of sources, and make an original contribution to the debate.</p>
<p>This step-by-step guide takes you through the entire writing process, from understanding your assignment to proofreading your final draft.</p>
<div><h2>Table of contents</h2><div><ol><li><a href="#understand-the-assignment">Understand the assignment</a></li><li><a href="#choose-a-research-paper-topic">Choose a research paper topic</a></li><li><a href="#conduct-preliminary-research">Conduct preliminary research</a></li><li><a href="#develop-a-thesis-statement">Develop a thesis statement</a></li><li><a href="#create-a-research-paper-outline">Create a research paper outline</a></li><li><a href="#write-a-first-draft-of-the-research-paper">Write a first draft of the research paper</a></li><li><a href="#write-the-introduction">Write the introduction</a></li><li><a href="#write-a-compelling-body-of-text">Write a compelling body of text</a></li><li><a href="#write-the-conclusion">Write the conclusion</a></li><li><a href="#the-second-draft">The second draft</a></li><li><a href="#the-revision-process">The revision process</a></li><li><a href="#research-paper-checklist">Research paper checklist</a></li><li><a href="#free-lecture-slides">Free lecture slides</a></li></ol></div></div>
<h2>Understand the assignment</h2>
<p>Completing a research paper successfully means accomplishing the specific tasks set out for you. Before you start, make sure you thoroughly understanding the assignment task sheet:</p>
<ul>
<li>Read it carefully, looking for anything confusing you might need to clarify with your professor.</li>
<li>Identify the assignment goal, deadline, length specifications, formatting, and submission method.</li>
<li>Make a bulleted list of the key points, then go back and cross completed items off as you’re writing.</li>
</ul>
<p>Carefully consider your timeframe and word limit: be realistic, and plan enough time to research, write, and edit.</p>
<h2>Choose a research paper topic</h2>
<p>There are many ways to <a href="https://www.scribbr.com/academic-essay/generating-topic-ideas/">generate an idea</a> for a research paper, from brainstorming with pen and paper to talking it through with a fellow student or professor.</p>
<p>You can try free writing, which involves taking a broad topic and writing continuously for two or three minutes to identify absolutely anything relevant that could be interesting.</p>
<p>You can also gain inspiration from other research. The discussion or recommendations sections of research papers often include ideas for other specific topics that require further examination.</p>
<p>Once you have a broad subject area, narrow it down to choose a topic that interests you, meets the criteria of your assignment, and is possible to research. Aim for ideas that are both original and specific:</p>
<ul>
<li>A paper following the chronology of World War II would not be original or specific enough.</li>
<li>A paper on the experience of Danish citizens living close to the German border during World War II would be specific and could be original enough.</li>
</ul>
<div>
	<div>
	<h3>Scribbr Citation CheckerNew</h3>
	<p>
	</p><p>The AI-powered Citation Checker helps you avoid common mistakes such as:</p>
	<ul>
	 <li>Missing commas and periods</li>
	 <li>Incorrect usage of “et al.”</li>
	 <li>Ampersands (&amp;) in narrative citations</li>
	 <li>Missing reference entries</li>
	</ul>
	<p><a href="https://www.scribbr.com/citation/checker/?scr_source=knowledgebase&amp;scr_medium=in-text&amp;scr_campaign=citation-checker&amp;src_content=avoid-common-mistakes">Learn more</a></p>
	</div>
	<p>
	</p>
	</div>
<h2>Conduct preliminary research</h2>
<p>Note any discussions that seem important to the topic, and try to find an issue that you can focus your paper around. Use a variety of <a href="https://www.scribbr.com/citing-sources/list-of-credible-sources-for-research/">sources</a>, including journals, books, and reliable websites, to ensure you do not miss anything glaring.</p>
<p>Do not only verify the ideas you have in mind, but look for sources that contradict your point of view.</p>
<ul>
<li>Is there anything people seem to overlook in the sources you research?</li>
<li>Are there any heated debates you can address?</li>
<li>Do you have a unique take on your topic?</li>
<li>Have there been some recent developments that build on the extant research?</li>
</ul>
<p>In this stage, you might find it helpful to formulate some <a href="/?p=22895">research questions</a> to help guide you. To write research questions, try to finish the following sentence: “I want to know how/what/why…”</p>
<h2>Develop a thesis statement</h2>
<p>A <a href="/?p=5188">thesis statement</a> is a statement of your central argument — it establishes the purpose and position of your paper. If you started with a research question, the thesis statement should answer it. It should also show what evidence and reasoning you’ll use to support that answer.</p>
<p>The thesis statement should be concise, contentious, and coherent. That means it should briefly summarize your argument in a sentence or two, make a claim that requires further evidence or analysis, and make a coherent point that relates to every part of the paper.</p>
<p>You will probably revise and refine the thesis statement as you do more research, but it can serve as a guide throughout the writing process. Every paragraph should aim to support and develop this central claim.</p>
<h2>Create a research paper outline</h2>
<p>A <a href="https://www.scribbr.com/research-paper/outline/">research paper outline</a> is essentially a list of the key topics, arguments, and evidence you want to include, divided into sections with <a href="https://www.scribbr.com/academic-writing/effective-headings/">headings</a> so that you know roughly what the paper will look like before you start writing.</p>
<p>A structure outline can help make the writing process much more efficient, so it’s worth dedicating some time to create one.</p>
<h2>Write a first draft of the research paper</h2>
<p>Your first draft won’t be perfect — you can polish later on. Your priorities at this stage are as follows:</p>
<ul>
<li>Maintaining forward momentum — write now, perfect later.</li>
<li>Paying attention to clear organization and <a href="https://www.scribbr.com/language-rules/subject-verb-agreement/">logical ordering</a> of paragraphs and sentences, which will help when you come to the second draft.</li>
<li>Expressing your ideas as clearly as possible, so you know what you were trying to say when you come back to the text.</li>
</ul>
<p>You do not need to start by writing the introduction. Begin where it feels most natural for you — some prefer to finish the most difficult sections first, while others choose to start with the easiest part. If you created an outline, use it as a map while you work.</p>
<p>Do not delete large sections of text. If you begin to dislike something you have written or find it doesn’t quite fit, move it to a different document, but don’t lose it completely — you never know if it might come in useful later.</p>
<h3>Paragraph structure</h3>
<p><a href="https://www.scribbr.com/research-paper/paragraph-structure/">Paragraphs</a> are the basic building blocks of research papers. Each one should focus on a single claim or idea that helps to establish the overall argument or purpose of the paper.</p>
<p>Here is an example of a well-structured paragraph. Hover over the sentences to learn more.<br />
</p><div>
<h3>Example paragraph</h3>
<p>
	George Orwell’s 1946 essay “Politics and the English Language” has had an enduring impact on thought about the relationship between politics and language.This impact is particularly obvious in light of the various critical review articles that have recently referenced the essay.For example, consider Mark Falcoff’s 2009 article in The National Review Online, “The Perversion of Language; or, Orwell Revisited,” in which he analyzes several common words (“activist,” “civil-rights leader,” “diversity,” and more).Falcoff’s close analysis of the ambiguity built into political language intentionally mirrors Orwell’s own point-by-point analysis of the political language of his day.Even 63 years after its publication, Orwell’s essay is emulated by contemporary thinkers.
</p></div>
<h3>Citing sources</h3>
<p>It’s also important to keep track of citations at this stage to<a href="https://www.scribbr.com/plagiarism/how-to-avoid-plagiarism/"> avoid accidental plagiarism</a>. Each time you use a source, make sure to take note of where the information came from.</p>
<p>You can use our free citation generators to automatically create citations and save your reference list as you go.</p>
<p><a href="https://www.scribbr.com/apa-citation-generator/">APA Citation Generator</a> <a href="https://www.scribbr.com/mla-citation-generator/">MLA Citation Generator</a></p>
<h2>Write the introduction</h2>
<p>The <a href="https://www.scribbr.com/research-paper/research-paper-introduction/">research paper introduction </a>should address three questions: What, why, and how? After finishing the introduction, the reader should know what the paper is about, why it is worth reading, and how you’ll build your arguments.</p>
<p><b>What?</b> Be specific about the topic of the paper, introduce the background, and define key terms or concepts.</p>
<p><b>Why?</b> This is the most important, but also the most difficult, part of the introduction. Try to provide brief answers to the following questions: What new material or insight are you offering? What important issues does your essay help define or answer?</p>
<p><b>How?</b> To let the reader know what to expect from the rest of the paper, the introduction should include a “map” of what will be discussed, briefly presenting the key elements of the paper in chronological order.</p>
<h2>Write a compelling body of text</h2>
<p>The major struggle faced by most writers is how to organize the information presented in the paper, which is one reason an <a href="/?p=61332">outline</a> is so useful. However, remember that the outline is only a guide and, when writing, you can be flexible with the order in which the information and arguments are presented.</p>
<p>One way to stay on track is to use your <a href="/?p=5188">thesis statement</a> and <a href="/?p=5175">topic sentences</a>. Check:</p>
<ul>
<li>topic sentences against the thesis statement;</li>
<li>topic sentences against each other, for similarities and logical ordering;</li>
<li>and each sentence against the topic sentence of that paragraph.</li>
</ul>
<p>Be aware of paragraphs that seem to cover the same things. If two paragraphs discuss something similar, they must approach that topic in different ways. Aim to create smooth <a href="https://www.scribbr.com/academic-essay/transition-sentences/">transitions</a> between sentences, paragraphs, and sections.</p>
<h2>Write the conclusion</h2>
<p>The <a href="https://www.scribbr.com/research-paper/research-paper-conclusion/">research paper conclusion </a>is designed to help your reader out of the paper’s argument, giving them a sense of finality.</p>
<p>Trace the course of the paper, emphasizing how it all comes together to prove your thesis statement. Give the paper a sense of finality by making sure the reader understands how you’ve settled the issues raised in the introduction.</p>
<p>You might also discuss the more general consequences of the argument, outline what the paper offers to future students of the topic, and suggest any questions the paper’s argument raises but cannot or does not try to answer.</p>
<p>You <b>should not</b>:</p>
<ul>
<li>Offer new arguments or essential information</li>
<li>Take up any more space than necessary</li>
<li>Begin with stock phrases that signal you are ending the paper (e.g. “In conclusion”)</li>
</ul>
<h2>The second draft</h2>
<p>There are four main considerations when it comes to the second draft.</p>
<ol>
<li>Check how your vision of the paper lines up with the first draft and, more importantly, that your paper still answers the assignment.</li>
<li>Identify any assumptions that might require (more substantial) justification, keeping your reader’s perspective foremost in mind. Remove these points if you cannot substantiate them further.</li>
<li>Be open to rearranging your ideas. Check whether any sections feel out of place and whether your ideas could be better organized.</li>
<li>If you find that old ideas do not fit as well as you anticipated, you should cut them out or condense them. You might also find that new and well-suited ideas occurred to you during the writing of the first draft — now is the time to make them part of the paper.</li>
</ol>
<h2>The revision process</h2>
<p>The goal during the revision and <a href="https://www.scribbr.com/language-rules/what-is-proofreading/">proofreading</a> process is to ensure you have completed all the necessary tasks and that the paper is as well-articulated as possible.</p>
<h3>Global concerns</h3>
<ul>
<li>Confirm that your paper completes every task specified in your assignment sheet.</li>
<li>Check for logical organization and flow of paragraphs.</li>
<li>Check paragraphs against the introduction and thesis statement.</li>
</ul>
<h3>Fine-grained details</h3>
<p>Check the content of each paragraph, making sure that:</p>
<ul>
<li>each sentence helps support the topic sentence.</li>
<li>no unnecessary or irrelevant information is present.</li>
<li>all technical terms your audience might not know are identified.</li>
</ul>
<p>Next, think about <a href="https://www.scribbr.com/language-rules/sentence-structure/">sentence structure</a>, grammatical errors, and <a href="https://www.scribbr.com/research-paper/research-paper-format/">formatting</a>. Check that you have correctly used <a href="https://www.scribbr.com/academic-writing/transition-words/">transition words and phrases</a> to show the connections between your ideas. Look for typos, cut unnecessary words, and check for consistency in aspects such as <a href="https://www.scribbr.com/academic-writing/capitalization-titles-headings/">heading formatting</a> and <a href="https://www.scribbr.com/academic-writing/us-vs-uk-english/">spellings</a>.</p>
<p>Finally, you need to make sure your paper is correctly formatted according to the rules of the <a href="https://www.scribbr.com/citing-sources/citation-styles/">citation style</a> you are using. For example, you might need to include an <a href="https://www.scribbr.com/mla/heading/">MLA heading</a> or create an <a href="https://www.scribbr.com/apa-style/apa-title-page/">APA title page</a>.</p>
<p>Scribbr’s professional editors can help with the revision process with our award-winning <a href="https://www.scribbr.com/proofreading-editing/">proofreading services.</a></p>
<p><a href="https://www.scribbr.com/proofreading-editing/paper-editing-service/">Discover our paper editing service</a></p>
<h2>Research paper checklist</h2>
<div>
 
 <ul>
 
 <li>
 
 <p>I have followed all instructions in the assignment sheet.</p>
 
 
 </li>
 
 <li>
 
 <p>My <a href="https://www.scribbr.com/research-paper/research-paper-introduction/">introduction</a> presents my topic in an engaging way and provides necessary background information.</p>
 
 
 </li>
 
 <li>
 
 <p>My introduction presents a clear, focused <a href="https://www.scribbr.com/research-process/research-problem/">research problem</a> and/or <a href="https://www.scribbr.com/academic-essay/thesis-statement/">thesis statement</a>.</p>
 
 
 </li>
 
 <li>
 
 <p>My paper is logically organized using <a href="https://www.scribbr.com/research-paper/paragraph-structure/">paragraphs</a> and (if relevant) <a href="https://www.scribbr.com/academic-writing/effective-headings/">section headings</a>.</p>
 
 
 </li>
 
 <li>
 
 <p>Each paragraph is clearly focused on one central idea, expressed in a clear <a href="https://www.scribbr.com/research-paper/topic-sentences/">topic sentence</a>.</p>
 
 
 </li>
 
 <li>
 
 <p>Each paragraph is relevant to my research problem or thesis statement.</p>
 
 
 </li>
 
 <li>
 
 <p>I have used appropriate <a href="https://www.scribbr.com/academic-writing/transition-words/">transitions </a>to clarify the connections between sections, paragraphs, and sentences.</p>
 
 
 </li>
 
 <li>
 
 <p>My <a href="https://www.scribbr.com/research-paper/research-paper-conclusion/">conclusion</a> provides a concise answer to the research question or emphasizes how the thesis has been supported.</p>
 
 
 </li>
 
 <li>
 
 <p>My conclusion shows how my research has contributed to knowledge or understanding of my topic.</p>
 
 
 </li>
 
 <li>
 
 <p>My conclusion does not present any new points or information essential to my argument.</p>
 
 
 </li>
 
 <li>
 
 <p>I have provided an <a href="https://www.scribbr.com/citing-sources/in-text-citation-styles/">in-text citation</a> every time I refer to ideas or information from a source.</p>
 
 
 </li>
 
 <li>
 
 <p>I have included a reference list at the end of my paper, consistently formatted according to a specific <a href="https://www.scribbr.com/citing-sources/citation-styles/">citation style</a>.</p>
 
 
 </li>
 
 <li>
 
 <p>I have thoroughly <a href="https://www.scribbr.com/academic-essay/revising/">revised</a> my paper and addressed any feedback from my professor or supervisor.</p>
 
 
 </li>
 
 <li>
 
 <p>I have followed all formatting guidelines (page numbers, headers, spacing, etc.).</p>
 
 
 </li>
 </ul>
 
 
</div>
<h2>Free lecture slides</h2>
<p>You can use, download, and modify these lecture slides to teach students about how to write a research paper.<br />
</p>
<p><a href="https://docs.google.com/presentation/d/1XVBgdQmg67FhqLh1ebEDrCzK7Qkeq5vjbIQwsTS7L64/">Open Google Slides</a><a href="https://cdn.scribbr.com/wp-content/uploads/2020/07/Lecture-slides_-How-to-write-a-research-paper.pptx">Download PowerPoint</a></p>
 </section>||||I|||| * FAQ
 * About us
 + Our editors
 + Apply as editor
 + Team
 + Jobs
 + Contact
 * My account
 + Orders
 + Upload
 + Account details
 + Logout
 * My account
 + Overview
 + Availability
 + Information package
 + Account details
 + Logout
 * Admin
 * Log in
 * Search
 * Proofreading & Editing
 + 
 o College admissions essay
 o Personal statement
 o Paper
 o PhD dissertation
 o APA editing
 o Academic editing
 o English proofreading
 o Spanish, French, or German
 o Check your Citations
 + 
 o About our services
 o Proofreading services
 o Proofreading & editing example
 o Essay coaching example
 o Happiness guarantee
 * Plagiarism Checker
 * Citation Tools
 + Citation Generator
 + Check your Citations
 * Knowledge Base
 * Proofreading & Editing
 * Plagiarism Checker
 * Citation Tools
 * Knowledge Base
 * FAQ
 * About us
 * My account
 * My account
 * Admin
 * Log in
 Nederlands English Deutsch Français Italiano Español Svenska Dansk Suomi Norwegian Bokmål
 Back
 * 
 + College admissions essay
 + Personal statement
 + Paper
 + PhD dissertation
 + APA editing
 + Academic editing
 + English proofreading
 + Spanish, French, or German
 + Check your Citations
 * 
 + About our services
 + Proofreading services
 + Proofreading & editing example
 + Essay coaching example
 + Happiness guarantee
 Back
 * Citation Generator
 * Check your Citations
 Back
 * Our editors
 * Apply as editor
 * Team
 * Jobs
 * Contact
 Back
 * Orders
 * Upload
 * Account details
 * Logout
 Back
 * Overview
 * Availability
 * Information package
 * Account details
 * Logout

 Have a language expert improve your writing

 Proofreading Services

 Run a free plagiarism check in 10 minutes

 Plagiarism Checker

 Generate accurate citations for free

 Citation Generator
 1. Home
 2. Knowledge Base
 3. Research paper

 Academic writing

 * What is academic writing?
 * The writing process
 * Titles and headings
 + Forging good titles
 + Writing effective headings
 + Capitalizing chapter and section headings
 * Academic language
 + Taboo words
 + Transition words
 + Verb tenses
 + List of phrasal verb replacements
 + First-person pronouns
 + Passive voice
 + Abbreviations & acronyms
 + Overview of English language rules
 * Improving your writing
 + Writing more concisely
 + Avoiding repetition and redundancy
 + Editing and proofreading
 * Using sources
 + Paraphrasing
 + Quoting
 + Language mistakes in quotes
 + Citation
 + Avoiding plagiarism
 * Types of academic text
 + Essay
 + Research paper
 + Lab report
 + Literature review
 + Annotated bibliography
 + Research proposal
 + Dissertation
 + Personal statement
 * Academic writing checklist
 * Free templates
 * Free lecture slides
 * Free checklists

 Interesting topics

 * Research bias
 * APA Style 6th edition
 * College essay
 * Working with sources
 * IEEE
 * Commonly confused words
 * Commas
 * Definitions
 * UK vs. US English
 * Applying to graduate school
 * Nouns and pronouns
 * AMA style
 * Parts of speech
 * Sentence structure
 * Verbs
 * Common mistakes
 * Effective communication
 * Statistics
 * Chicago Style
 * Language rules
 * Methodology
 * MLA Style
 * Research paper
 * Academic writing
 * Starting the research process
 * Dissertation
 * Essay
 * Tips
 * APA Style 7th edition
 * APA citation examples
 * Citing sources
 * Plagiarism

 Have a language expert improve your writing

 Proofreading Services

 Run a free plagiarism check in 10 minutes

 Plagiarism Checker

 Generate accurate citations for free

 Citation Generator

 How to Write a Research Paper | A Beginner's Guide

 A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research.

 Research papers are similar to academic essays, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research. Writing a research paper requires you to demonstrate a strong knowledge of your topic, engage with a variety of sources, and make an original contribution to the debate.

 This step-by-step guide takes you through the entire writing process, from understanding your assignment to proofreading your final draft.

 Table of contents

 1. Understand the assignment
 2. Choose a research paper topic
 3. Conduct preliminary research
 4. Develop a thesis statement
 5. Create a research paper outline
 6. Write a first draft of the research paper
 7. Write the introduction
 8. Write a compelling body of text
 9. Write the conclusion
 10. The second draft
 11. The revision process
 12. Research paper checklist
 13. Free lecture slides

 Understand the assignment

 Completing a research paper successfully means accomplishing the specific tasks set out for you. Before you start, make sure you thoroughly understanding the assignment task sheet:

 * Read it carefully, looking for anything confusing you might need to clarify with your professor.
 * Identify the assignment goal, deadline, length specifications, formatting, and submission method.
 * Make a bulleted list of the key points, then go back and cross completed items off as you’re writing.

 Carefully consider your timeframe and word limit: be realistic, and plan enough time to research, write, and edit.

 Choose a research paper topic

 There are many ways to generate an idea for a research paper, from brainstorming with pen and paper to talking it through with a fellow student or professor.

 You can try free writing, which involves taking a broad topic and writing continuously for two or three minutes to identify absolutely anything relevant that could be interesting.

 You can also gain inspiration from other research. The discussion or recommendations sections of research papers often include ideas for other specific topics that require further examination.

 Once you have a broad subject area, narrow it down to choose a topic that interests you, m eets the criteria of your assignment, and i s possible to research. Aim for ideas that are both original and specific:

 * A paper following the chronology of World War II would not be original or specific enough.
 * A paper on the experience of Danish citizens living close to the German border during World War II would be specific and could be original enough.

 Scribbr Citation Checker New

 The AI-powered Citation Checker helps you avoid common mistakes such as:

 * Missing commas and periods
 * Incorrect usage of “et al.”
 * Ampersands (&) in narrative citations
 * Missing reference entries

 Learn more

 Conduct preliminary research

 Note any discussions that seem important to the topic, and try to find an issue that you can focus your paper around. Use a variety of sources, including journals, books, and reliable websites, to ensure you do not miss anything glaring.

 Do not only verify the ideas you have in mind, but look for sources that contradict your point of view.

 * Is there anything people seem to overlook in the sources you research?
 * Are there any heated debates you can address?
 * Do you have a unique take on your topic?
 * Have there been some recent developments that build on the extant research?

 In this stage, you might find it helpful to formulate some research questions to help guide you. To write research questions, try to finish the following sentence: “I want to know how/what/why…”

 Develop a thesis statement

 A thesis statement is a statement of your central argument — it establishes the purpose and position of your paper. If you started with a research question, the thesis statement should answer it. It should also show what evidence and reasoning you’ll use to support that answer.

 The thesis statement should be concise, contentious, and coherent. That means it should briefly summarize your argument in a sentence or two, make a claim that requires further evidence or analysis, and make a coherent point that relates to every part of the paper.

 You will probably revise and refine the thesis statement as you do more research, but it can serve as a guide throughout the writing process. Every paragraph should aim to support and develop this central claim.

 Create a research paper outline

 A research paper outline is essentially a list of the key topics, arguments, and evidence you want to include, divided into sections with headings so that you know roughly what the paper will look like before you start writing.

 A structure outline can help make the writing process much more efficient, so it’s worth dedicating some time to create one.

 Write a first draft of the research paper

 Your first draft won’t be perfect — you can polish later on. Your priorities at this stage are as follows:

 * Maintaining forward momentum — write now, perfect later.
 * Paying attention to clear organization and logical ordering of paragraphs and sentences, which will help when you come to the second draft.
 * Expressing your ideas as clearly as possible, so you know what you were trying to say when you come back to the text.

 You do not need to start by writing the introduction. Begin where it feels most natural for you — some prefer to finish the most difficult sections first, while others choose to start with the easiest part. If you created an outline, use it as a map while you work.

 Do not delete large sections of text. If you begin to dislike something you have written or find it doesn’t quite fit, move it to a different document, but don’t lose it completely — you never know if it might come in useful later.

 Paragraph structure

 Paragraphs are the basic building blocks of research papers. Each one should focus on a single claim or idea that helps to establish the overall argument or purpose of the paper.

 Here is an example of a well-structured paragraph. Hover over the sentences to learn more.

 Example paragraph

 George Orwell’s 1946 essay “Politics and the English Language” has had an enduring impact on thought about the relationship between politics and language. This impact is particularly obvious in light of the various critical review articles that have recently referenced the essay. For example, consider Mark Falcoff’s 2009 article in The National Review Online, “The Perversion of Language; or, Orwell Revisited,” in which he analyzes several common words (“activist,” “civil-rights leader,” “diversity,” and more). Falcoff’s close analysis of the ambiguity built into political language intentionally mirrors Orwell’s own point-by-point analysis of the political language of his day. Even 63 years after its publication, Orwell’s essay is emulated by contemporary thinkers.

 Citing sources

 It’s also important to keep track of citations at this stage to avoid accidental plagiarism. Each time you use a source, make sure to take note of where the information came from.

 You can use our free citation generators to automatically create citations and save your reference list as you go.

 APA Citation Generator MLA Citation Generator

 Write the introduction

 The research paper introduction should address three questions: What, why, and how? After finishing the introduction, the reader should know what the paper is about, why it is worth reading, and how you’ll build your arguments.

 What? Be specific about the topic of the paper, introduce the background, and define key terms or concepts.

 Why? This is the most important, but also the most difficult, part of the introduction. Try to provide brief answers to the following questions: What new material or insight are you offering? What important issues does your essay help define or answer?

 How? To let the reader know what to expect from the rest of the paper, the introduction should include a “map” of what will be discussed, briefly presenting the key elements of the paper in chronological order.

 Write a compelling body of text

 The major struggle faced by most writers is how to organize the information presented in the paper, which is one reason an outline is so useful. However, remember that the outline is only a guide and, when writing, you can be flexible with the order in which the information and arguments are presented.

 One way to stay on track is to use your thesis statement and topic sentences. Check:

 * topic sentences against the thesis statement;
 * topic sentences against each other, for similarities and logical ordering;
 * and each sentence against the topic sentence of that paragraph.

 Be aware of paragraphs that seem to cover the same things. If two paragraphs discuss something similar, they must approach that topic in different ways. Aim to create smooth transitions between sentences, paragraphs, and sections.

 Write the conclusion

 The research paper conclusion is designed to help your reader out of the paper’s argument, giving them a sense of finality.

 Trace the course of the paper, emphasizing how it all comes together to prove your thesis statement. Give the paper a sense of finality by making sure the reader understands how you’ve settled the issues raised in the introduction.

 You might also discuss the more general consequences of the argument, outline what the paper offers to future students of the topic, and suggest any questions the paper’s argument raises but cannot or does not try to answer.

 You should not:

 * Offer new arguments or essential information
 * Take up any more space than necessary
 * Begin with stock phrases that signal you are ending the paper (e.g. “In conclusion”)

 The second draft

 There are four main considerations when it comes to the second draft.

 1. Check how your vision of the paper lines up with the first draft and, more importantly, that your paper still answers the assignment.
 2. Identify any assumptions that might require (more substantial) justification, keeping your reader’s perspective foremost in mind. Remove these points if you cannot substantiate them further.
 3. Be open to rearranging your ideas. Check whether any sections feel out of place and whether your ideas could be better organized.
 4. If you find that old ideas do not fit as well as you anticipated, you should cut them out or condense them. You might also find that new and well-suited ideas occurred to you during the writing of the first draft — now is the time to make them part of the paper.

 The revision process

 The goal during the revision and proofreading process is to ensure you have completed all the necessary tasks and that the paper is as well-articulated as possible.

 Global concerns

 * Confirm that your paper completes every task specified in your assignment sheet.
 * Check for logical organization and flow of paragraphs.
 * Check paragraphs against the introduction and thesis statement.

 Fine-grained details

 Check the content of each paragraph, making sure that:

 * each sentence helps support the topic sentence.
 * no unnecessary or irrelevant information is present.
 * all technical terms your audience might not know are identified.

 Next, think about sentence structure, grammatical errors, and formatting. Check that you have correctly used transition words and phrases to show the connections between your ideas. Look for typos, cut unnecessary words, and check for consistency in aspects such as heading formatting and spellings.

 Finally, you need to make sure your paper is correctly formatted according to the rules of the citation style you are using. For example, you might need to include an MLA heading or create an APA title page.

 Scribbr’s professional editors can help with the revision process with our award-winning proofreading services.

 Discover our paper editing service

 Research paper checklist

 <!–– skip-in-text-insert ––>

 Checklist: Research paper

 0 / 14

 * I have followed all instructions in the assignment sheet.

 * My introduction presents my topic in an engaging way and provides necessary background information.

 * My introduction presents a clear, focused research problem and/or thesis statement.

 * My paper is logically organized using paragraphs and (if relevant) section headings.

 * Each paragraph is clearly focused on one central idea, expressed in a clear topic sentence.

 * Each paragraph is relevant to my research problem or thesis statement.

 * I have used appropriate transitions to clarify the connections between sections, paragraphs, and sentences.

 * My conclusion provides a concise answer to the research question or emphasizes how the thesis has been supported.

 * My conclusion shows how my research has contributed to knowledge or understanding of my topic.

 * My conclusion does not present any new points or information essential to my argument.

 * I have provided an in-text citation every time I refer to ideas or information from a source.

 * I have included a reference list at the end of my paper, consistently formatted according to a specific citation style.

 * I have thoroughly revised my paper and addressed any feedback from my professor or supervisor.

 * I have followed all formatting guidelines (page numbers, headers, spacing, etc.).

 Well done!

 You've written a great paper. Make sure it's perfect with the help of a Scribbr editor!

 Learn about our proofreading services Return to checklist

 Free lecture slides

 You can use, download, and modify these lecture slides to teach students about how to write a research paper.

 Open Google SlidesDownload PowerPoint

 Is this article helpful?

 1998 203
 You have already voted. Thanks :-) Your vote is saved :-) Processing your vote...

 Other students also liked

 Writing a Research Paper Introduction | Step-by-Step Guide

 The introduction to a research paper presents your topic, provides background, and details your research problem.
 2659

 Writing a Research Paper Conclusion | Step-by-Step Guide

 The conclusion of a research paper restates the research problem, summarizes your arguments or findings, and discusses the implications.
 667

 Research Paper Format | APA, MLA, & Chicago Templates

 The formatting guidelines for a research paper differ by style guide. Download free templates to get started.
 170

 More interesting articles

 * Academic Paragraph Structure | Step-by-Step Guide & Examples
 * Checklist: Writing a Great Research Paper
 * How to Create a Structured Research Paper Outline | Example
 * How to Write a Discussion Section | Tips & Examples
 * How to Write Recommendations in Research | Examples & Tips
 * How to Write Topic Sentences | 4 Steps, Examples & Purpose
 * Research Paper Appendix | Example & Templates
 * Research Paper Damage Control | Managing a Broken Argument
 * Research Paper Format | APA, MLA, & Chicago Templates
 * What Is a Theoretical Framework? | Guide to Organizing
 * Writing a Research Paper Conclusion | Step-by-Step Guide
 * Writing a Research Paper Introduction | Step-by-Step Guide

 Scribbr

 * Our editors
 * Jobs
 * FAQ
 * Partners

 Our services

 * Plagiarism Checker
 * Proofreading Services
 * Citation Generator
 * Grammar Checker
 * Paraphrasing Tool
 * Citation Checker
 * Knowledge Base

 Contact

 * [email protected]
 * +1 (510) 822-8066
 * 
 * 
 * 
 4.8
 * Nederlands
 * English
 * Deutsch
 * Français
 * Italiano
 * Español
 * Svenska
 * Dansk
 * Suomi
 * Norwegian Bokmål
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * Terms of Use
 * Privacy Policy
 * Happiness guarantee

 What is your plagiarism score?

 Scribbr Plagiarism Checker
A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research.
Research papers are similar to academic essays, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research. Writing a research paper requires you to demonstrate a strong knowledge of your topic, engage with a variety of sources, and make an original contribution to the debate.
This step-by-step guide takes you through the entire writing process, from understanding your assignment to proofreading your final draft.
 Table of contents    Understand the assignment   Choose a research paper topic   Conduct preliminary research   Develop a thesis statement   Create a research paper outline   Write a first draft of the research paper   Write the introduction   Write a compelling body of text   Write the conclusion   The second draft   The revision process   Research paper checklist   Free lecture slides    
Understand the assignment
Completing a research paper successfully means accomplishing the specific tasks set out for you. Before you start, make sure you thoroughly understanding the assignment task sheet:
Read it carefully, looking for anything confusing you might need to clarify with your professor.
Identify the assignment goal, deadline, length specifications, formatting, and submission method.
Make a bulleted list of the key points, then go back and cross completed items off as you’re writing.
Carefully consider your timeframe and word limit: be realistic, and plan enough time to research, write, and edit.
Choose a research paper topic
There are many ways to generate an idea for a research paper, from brainstorming with pen and paper to talking it through with a fellow student or professor.
You can try free writing, which involves taking a broad topic and writing continuously for two or three minutes to identify absolutely anything relevant that could be interesting.
You can also gain inspiration from other research. The discussion or recommendations sections of research papers often include ideas for other specific topics that require further examination.
Once you have a broad subject area, narrow it down to choose a topic that interests you, meets the criteria of your assignment, and is possible to research. Aim for ideas that are both original and specific:
A paper following the chronology of World War II would not be original or specific enough.
A paper on the experience of Danish citizens living close to the German border during World War II would be specific and could be original enough.
	
	Scribbr Citation CheckerNew
	
	 The AI-powered Citation Checker helps you avoid common mistakes such as:
	
	 Missing commas and periods
	 Incorrect usage of “et al.”
	 Ampersands (&amp;) in narrative citations
	 Missing reference entries
	
	 Learn more 
	
	
	
	
Conduct preliminary research
Note any discussions that seem important to the topic, and try to find an issue that you can focus your paper around. Use a variety of sources, including journals, books, and reliable websites, to ensure you do not miss anything glaring.
Do not only verify the ideas you have in mind, but look for sources that contradict your point of view.
Is there anything people seem to overlook in the sources you research?
Are there any heated debates you can address?
Do you have a unique take on your topic?
Have there been some recent developments that build on the extant research?
In this stage, you might find it helpful to formulate some research questions to help guide you. To write research questions, try to finish the following sentence: “I want to know how/what/why…”
Develop a thesis statement
A thesis statement is a statement of your central argument — it establishes the purpose and position of your paper. If you started with a research question, the thesis statement should answer it. It should also show what evidence and reasoning you’ll use to support that answer.
The thesis statement should be concise, contentious, and coherent. That means it should briefly summarize your argument in a sentence or two, make a claim that requires further evidence or analysis, and make a coherent point that relates to every part of the paper.
You will probably revise and refine the thesis statement as you do more research, but it can serve as a guide throughout the writing process. Every paragraph should aim to support and develop this central claim.
Create a research paper outline
A research paper outline is essentially a list of the key topics, arguments, and evidence you want to include, divided into sections with headings so that you know roughly what the paper will look like before you start writing.
A structure outline can help make the writing process much more efficient, so it’s worth dedicating some time to create one.
Write a first draft of the research paper
Your first draft won’t be perfect — you can polish later on. Your priorities at this stage are as follows:
Maintaining forward momentum — write now, perfect later.
Paying attention to clear organization and logical ordering of paragraphs and sentences, which will help when you come to the second draft.
Expressing your ideas as clearly as possible, so you know what you were trying to say when you come back to the text.
You do not need to start by writing the introduction. Begin where it feels most natural for you — some prefer to finish the most difficult sections first, while others choose to start with the easiest part. If you created an outline, use it as a map while you work.
Do not delete large sections of text. If you begin to dislike something you have written or find it doesn’t quite fit, move it to a different document, but don’t lose it completely — you never know if it might come in useful later.
Paragraph structure
 Paragraphs are the basic building blocks of research papers. Each one should focus on a single claim or idea that helps to establish the overall argument or purpose of the paper.
Here is an example of a well-structured paragraph. Hover over the sentences to learn more.
 
Example paragraph
	George Orwell’s 1946 essay “Politics and the English Language” has had an enduring impact on thought about the relationship between politics and language.This impact is particularly obvious in light of the various critical review articles that have recently referenced the essay.For example, consider Mark Falcoff’s 2009 article in The National Review Online, “The Perversion of Language; or, Orwell Revisited,” in which he analyzes several common words (“activist,” “civil-rights leader,” “diversity,” and more).Falcoff’s close analysis of the ambiguity built into political language intentionally mirrors Orwell’s own point-by-point analysis of the political language of his day.Even 63 years after its publication, Orwell’s essay is emulated by contemporary thinkers.
 
Citing sources
It’s also important to keep track of citations at this stage to avoid accidental plagiarism. Each time you use a source, make sure to take note of where the information came from.
You can use our free citation generators to automatically create citations and save your reference list as you go.
 APA Citation Generator MLA Citation Generator 
Write the introduction
The research paper introduction should address three questions: What, why, and how? After finishing the introduction, the reader should know what the paper is about, why it is worth reading, and how you’ll build your arguments.
 What? Be specific about the topic of the paper, introduce the background, and define key terms or concepts.
 Why? This is the most important, but also the most difficult, part of the introduction. Try to provide brief answers to the following questions: What new material or insight are you offering? What important issues does your essay help define or answer?
 How? To let the reader know what to expect from the rest of the paper, the introduction should include a “map” of what will be discussed, briefly presenting the key elements of the paper in chronological order.
Write a compelling body of text
The major struggle faced by most writers is how to organize the information presented in the paper, which is one reason an outline is so useful. However, remember that the outline is only a guide and, when writing, you can be flexible with the order in which the information and arguments are presented.
One way to stay on track is to use your thesis statement and topic sentences. Check:
topic sentences against the thesis statement;
topic sentences against each other, for similarities and logical ordering;
and each sentence against the topic sentence of that paragraph.
Be aware of paragraphs that seem to cover the same things. If two paragraphs discuss something similar, they must approach that topic in different ways. Aim to create smooth transitions between sentences, paragraphs, and sections.
Write the conclusion
The research paper conclusion is designed to help your reader out of the paper’s argument, giving them a sense of finality.
Trace the course of the paper, emphasizing how it all comes together to prove your thesis statement. Give the paper a sense of finality by making sure the reader understands how you’ve settled the issues raised in the introduction.
You might also discuss the more general consequences of the argument, outline what the paper offers to future students of the topic, and suggest any questions the paper’s argument raises but cannot or does not try to answer.
You should not:
Offer new arguments or essential information
Take up any more space than necessary
Begin with stock phrases that signal you are ending the paper (e.g. “In conclusion”)
The second draft
There are four main considerations when it comes to the second draft.
Check how your vision of the paper lines up with the first draft and, more importantly, that your paper still answers the assignment.
Identify any assumptions that might require (more substantial) justification, keeping your reader’s perspective foremost in mind. Remove these points if you cannot substantiate them further.
Be open to rearranging your ideas. Check whether any sections feel out of place and whether your ideas could be better organized.
If you find that old ideas do not fit as well as you anticipated, you should cut them out or condense them. You might also find that new and well-suited ideas occurred to you during the writing of the first draft — now is the time to make them part of the paper.
The revision process
The goal during the revision and proofreading process is to ensure you have completed all the necessary tasks and that the paper is as well-articulated as possible.
Global concerns
Confirm that your paper completes every task specified in your assignment sheet.
Check for logical organization and flow of paragraphs.
Check paragraphs against the introduction and thesis statement.
Fine-grained details
Check the content of each paragraph, making sure that:
each sentence helps support the topic sentence.
no unnecessary or irrelevant information is present.
all technical terms your audience might not know are identified.
Next, think about sentence structure, grammatical errors, and formatting. Check that you have correctly used transition words and phrases to show the connections between your ideas. Look for typos, cut unnecessary words, and check for consistency in aspects such as heading formatting and spellings.
Finally, you need to make sure your paper is correctly formatted according to the rules of the citation style you are using. For example, you might need to include an MLA heading or create an APA title page.
Scribbr’s professional editors can help with the revision process with our award-winning proofreading services. 
 Discover our paper editing service 
Research paper checklist
 
 
 
 
 
 I have followed all instructions in the assignment sheet.
 
 
 
 
 
 
 My introduction presents my topic in an engaging way and provides necessary background information.
 
 
 
 
 
 
 My introduction presents a clear, focused research problem and/or thesis statement.
 
 
 
 
 
 
 My paper is logically organized using paragraphs and (if relevant) section headings.
 
 
 
 
 
 
 Each paragraph is clearly focused on one central idea, expressed in a clear topic sentence.
 
 
 
 
 
 
 Each paragraph is relevant to my research problem or thesis statement.
 
 
 
 
 
 
 I have used appropriate transitions to clarify the connections between sections, paragraphs, and sentences.
 
 
 
 
 
 
 My conclusion provides a concise answer to the research question or emphasizes how the thesis has been supported.
 
 
 
 
 
 
 My conclusion shows how my research has contributed to knowledge or understanding of my topic.
 
 
 
 
 
 
 My conclusion does not present any new points or information essential to my argument.
 
 
 
 
 
 
 I have provided an in-text citation every time I refer to ideas or information from a source.
 
 
 
 
 
 
 I have included a reference list at the end of my paper, consistently formatted according to a specific citation style.
 
 
 
 
 
 
 I have thoroughly revised my paper and addressed any feedback from my professor or supervisor.
 
 
 
 
 
 
 I have followed all formatting guidelines (page numbers, headers, spacing, etc.).
 
 
 
 
 
 
Free lecture slides
You can use, download, and modify these lecture slides to teach students about how to write a research paper.
 Open Google Slides Download PowerPoint
How to Write a Research Paper | A Beginner's Guide
https://www.scribbr.com/category/research-paper/#:~:text=A%20research%20paper%20is%20a,on%20in%2Ddepth%20independent%20research.
https://arxiv.org/abs/2112.11446
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
2021-12-08
Rae; Jack W; Borgeaud; Sebastian; Cai; Trevor; Millican; Katie; Hoffmann; Jordan; Song; Francis; Aslanides; John; Henderson; Sarah; Ring; Roman; Young; Susannah; Rutherford; Eliza; Hennigan; Tom; Menick; Jacob; Cassirer; Albin; Powell; Richard; Driessche; George van den; Hendricks; Lisa Anne; Rauh; Maribeth; Huang; Po-Sen; Glaese; Amelia; Welbl; Johannes; Dathathri; Sumanth; Saffron; Uesato; Jonathan; Mellor; Higgins; Irina; Creswell; Antonia; McAleese; Nat; Wu; Amy; Elsen; Erich; Jayakumar; Siddhant; Buchatskaya; Elena; Budden; David; Sutherland; Esme; Simonyan; Karen; Paganini; Michela; Sifre; Laurent; Martens; Lena; Li; Xiang Lorraine; Kuncoro; Adhiguna; Nematzadeh; Aida; Gribovskaya; Donato; Domenic; Lazaridou; Angeliki; Mensch; Arthur; Lespiau; Jean-Baptiste; Tsimpoukelli; Maria; Grigorev; Nikolai; Fritz; Doug; Sottiaux; Thibault; Pajarskas; Mantas; Pohlen; Toby; Gong; Zhitao; Toyama; Daniel; D'Autume; Cyprien de Masson; Yujia; Terzi; Tayfun; Mikulik; Vladimir; Babuschkin; Igor; Clark; Casas; Diego de Las; Guy; Aurelia; Jones; Chris; Bradbury; James; Matthew; Hechtman; Blake; Weidinger; Laura; Gabriel; Iason; Isaac; William; Lockhart; Ed; Osindero; Simon; Rimell; Dyer; Vinyals; Oriol; Ayoub; Kareem; Stanway; Jeff; Bennett; Lorrayne; Hassabis; Demis; Kavukcuoglu; Koray; Irving; Geoffrey
<div><div>
 
 
 <div><p>Authors:<a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rae%2C+J+W">Jack W. Rae</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Borgeaud%2C+S">Sebastian Borgeaud</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cai%2C+T">Trevor Cai</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Millican%2C+K">Katie Millican</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hoffmann%2C+J">Jordan Hoffmann</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Song%2C+F">Francis Song</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Aslanides%2C+J">John Aslanides</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Henderson%2C+S">Sarah Henderson</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ring%2C+R">Roman Ring</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Young%2C+S">Susannah Young</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rutherford%2C+E">Eliza Rutherford</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hennigan%2C+T">Tom Hennigan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Menick%2C+J">Jacob Menick</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cassirer%2C+A">Albin Cassirer</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Powell%2C+R">Richard Powell</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=van+den+Driessche%2C+G">George van den Driessche</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hendricks%2C+L+A">Lisa Anne Hendricks</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rauh%2C+M">Maribeth Rauh</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Huang%2C+P">Po-Sen Huang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Glaese%2C+A">Amelia Glaese</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Welbl%2C+J">Johannes Welbl</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Dathathri%2C+S">Sumanth Dathathri</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Huang%2C+S">Saffron Huang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Uesato%2C+J">Jonathan Uesato</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mellor%2C+J">John Mellor</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Higgins%2C+I">Irina Higgins</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Creswell%2C+A">Antonia Creswell</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=McAleese%2C+N">Nat McAleese</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Wu%2C+A">Amy Wu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Elsen%2C+E">Erich Elsen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Jayakumar%2C+S">Siddhant Jayakumar</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Buchatskaya%2C+E">Elena Buchatskaya</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Budden%2C+D">David Budden</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sutherland%2C+E">Esme Sutherland</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Simonyan%2C+K">Karen Simonyan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Paganini%2C+M">Michela Paganini</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sifre%2C+L">Laurent Sifre</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Martens%2C+L">Lena Martens</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li%2C+X+L">Xiang Lorraine Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kuncoro%2C+A">Adhiguna Kuncoro</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Nematzadeh%2C+A">Aida Nematzadeh</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gribovskaya%2C+E">Elena Gribovskaya</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Donato%2C+D">Domenic Donato</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lazaridou%2C+A">Angeliki Lazaridou</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mensch%2C+A">Arthur Mensch</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lespiau%2C+J">Jean-Baptiste Lespiau</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Tsimpoukelli%2C+M">Maria Tsimpoukelli</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Grigorev%2C+N">Nikolai Grigorev</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Fritz%2C+D">Doug Fritz</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sottiaux%2C+T">Thibault Sottiaux</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Pajarskas%2C+M">Mantas Pajarskas</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Pohlen%2C+T">Toby Pohlen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gong%2C+Z">Zhitao Gong</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Toyama%2C+D">Daniel Toyama</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=de+Masson+d%27Autume%2C+C">Cyprien de Masson d'Autume</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li%2C+Y">Yujia Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Terzi%2C+T">Tayfun Terzi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mikulik%2C+V">Vladimir Mikulik</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Babuschkin%2C+I">Igor Babuschkin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Clark%2C+A">Aidan Clark</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=de+Las+Casas%2C+D">Diego de Las Casas</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Guy%2C+A">Aurelia Guy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Jones%2C+C">Chris Jones</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bradbury%2C+J">James Bradbury</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Johnson%2C+M">Matthew Johnson</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hechtman%2C+B">Blake Hechtman</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Weidinger%2C+L">Laura Weidinger</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gabriel%2C+I">Iason Gabriel</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Isaac%2C+W">William Isaac</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lockhart%2C+E">Ed Lockhart</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Osindero%2C+S">Simon Osindero</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rimell%2C+L">Laura Rimell</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Dyer%2C+C">Chris Dyer</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Vinyals%2C+O">Oriol Vinyals</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ayoub%2C+K">Kareem Ayoub</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Stanway%2C+J">Jeff Stanway</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bennett%2C+L">Lorrayne Bennett</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hassabis%2C+D">Demis Hassabis</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kavukcuoglu%2C+K">Koray Kavukcuoglu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Irving%2C+G">Geoffrey Irving</a></p></div>
 
 
 
 <p><a href="/pdf/2112.11446">Download PDF</a></p><blockquote>
 Abstract: Language modelling provides a step towards intelligent communication systems
by harnessing large repositories of written human knowledge to better predict
and understand the world. In this paper, we present an analysis of
Transformer-based language model performance across a wide range of model
scales -- from models with tens of millions of parameters up to a 280 billion
parameter model called Gopher. These models are evaluated on 152 diverse tasks,
achieving state-of-the-art performance across the majority. Gains from scale
are largest in areas such as reading comprehension, fact-checking, and the
identification of toxic language, but logical and mathematical reasoning see
less benefit. We provide a holistic analysis of the training dataset and
model's behaviour, covering the intersection of model scale with bias and
toxicity. Finally we discuss the application of language models to AI safety
and the mitigation of downstream harms.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Jack Rae [<a href="/show-email/7559301e/2112.11446">view email</a>]
 <br />
 <strong><a href="/abs/2112.11446v1">[v1]</a></strong>
 Wed, 8 Dec 2021 19:41:47 UTC (7,844 KB)<br /><strong>[v2]</strong>
Fri, 21 Jan 2022 18:39:38 UTC (7,941 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2112.11446

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Computation and Language

 arXiv:2112.11446 (cs)
 [Submitted on 8 Dec 2021 (v1), last revised 21 Jan 2022 (this version, v2)]

 Title: Scaling Language Models: Methods, Analysis & Insights from Training Gopher

 Authors: Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, Geoffrey Irving
 Download PDF
 Abstract: Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -- from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
 Comments: 120 pages 
 Subjects: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI)
 Cite as: arXiv:2112.11446 [cs.CL] 
 (or arXiv:2112.11446v2 [cs.CL] for this version) 
 https://doi.org/10.48550/arXiv.2112.11446 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Jack Rae [view email]
 [v1] Wed, 8 Dec 2021 19:41:47 UTC (7,844 KB)
 [v2] Fri, 21 Jan 2022 18:39:38 UTC (7,941 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.CL
 < prev | next >
 new | recent | 2112
 Change to browse by:
 cs
 cs.AI

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 1 blog link

 (what is this?)

 DBLP - CS Bibliography

 listing | bibtex
 Jack W. Rae
 Sebastian Borgeaud
 Jordan Hoffmann
 H. Francis Song
 John Aslanides
 …
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs and how to get involved.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Date Published: 2022-01-21
Authors:
Jack W Rae, jack.w.rae@gmail.com
Sebastian  Borgeaud
Trevor  Cai
Katie  Millican
Jordan  Hoffmann
Anne  Hendricks
Maribeth  Rauh
Po-Sen  Huang
Amelia  Glaese
Johannes  Welbl
Sumanth  Dathathri
Saffron  Huang
Jonathan  Uesato
John  Mellor
Irina  Higgins
Antonia  Creswell
Nat  Mcaleese
Amy  Wu
Erich  Elsen
Siddhant  Jayakumar
Elena  Buchatskaya
David  Budden
Esme  Sutherland
Karen  Simonyan
Michela  Paganini
Laurent  Sifre
Lena  Martens
Lorraine  Li
Adhiguna  Kuncoro
Aida  Nematzadeh
Elena  Gribovskaya
Domenic  Donato
Angeliki  Lazaridou
Arthur  Mensch
Jean-Baptiste  Lespiau
Maria  Tsimpoukelli
Nikolai  Grigorev
Doug  Fritz
Thibault  Sottiaux
Mantas  Pajarskas
Toby  Pohlen
Zhitao  Gong
Daniel  Toyama
Cyprien  De Masson D'autume
Yujia  Li
Tayfun  Terzi
Vladimir  Mikulik
Igor  Babuschkin
Aidan  Clark
Diego  De
Las  Casas
Aurelia  Guy
Chris  Jones
James  Bradbury
Matthew  Johnson
Blake  Hechtman
Laura  Weidinger
Iason  Gabriel
William  Isaac
Ed  Lockhart
Simon  Osindero
Laura  Rimell
Chris  Dyer
Oriol  Vinyals
Kareem  Ayoub
Jeff  Stanway
Lorrayne  Bennett
Demis  Hassabis
Koray  Kavukcuoglu
Geoffrey  Irving, geoffreyi@deepmind.com
Siim  Poder
Alexey  Guseynov
Alban  Rrustemi
Eric  Noland
Bogdan  Damoc
Damion  Yates
Bryan  Chiang
Christoph  Dittmann
Roberto  Lupi
Michael  Vorburger
Tyler  Liechty
Mira  Lutfi
Richard  Ives
Elspeth  White
Tom  Lue
Ben  Coppin
Kirsty  Anderson
John  Jumper
Andy  Brock
Julian  Schrittweiser
Greg  Wayne
Max  Jaderberg
Phil  Blunsom
John  Aslanides
Roman  Ring
Jacob  Menick
Pipelining Albin Cassirer
Richard  Powell
George  Van Den Driessche
Tom  Hennigan
Diego  De Las Casas
Mia  Glaese
Alexandre  Fréchette
Francis  Song
Eliza  Rutherford
Susannah  Young
Lisa Anne Hendricks
Rae  Jack
Francis  Mmlu
Nat  Song
  Mcaleese
Dataset  Analyses
Jacob Menick Truthfulqa
Jordan  Hoffmann Distributional
Lisa Anne Hendricks Dialogue
Sarah  Henderson
Esme  Suther-
Research  Advisors
References M Abadi
A  Chu
I  Goodfellow
H B Mcmahan
I  Mironov
K  Talwar
L  Zhang
M Abubakar Github
Copilot  Ai
Abstract
Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales -from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model's behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the mitigation of downstream harms.
1. Introduction
Natural language communication is core to intelligence, as it allows ideas to be efficiently shared between humans or artificially intelligent systems. The generality of language allows us to express many intelligence tasks as taking in natural language input and producing natural language output. Autoregressive language modelling -predicting the future of a text sequence from its pastprovides a simple yet powerful objective that admits formulation of numerous cognitive tasks. At the same time, it opens the door to plentiful training data: the internet, books, articles, code, and other writing. However this training objective is only an approximation to any specific goal or application, since we predict everything in the sequence rather than only the aspects we care about. Yet if we treat the resulting models with appropriate caution, we believe they will be a powerful tool to capture some of the richness of human intelligence.
Using language models as an ingredient towards intelligence contrasts with their original application: transferring text over a limited-bandwidth communication channel. Shannon's Mathematical Theory of Communication (Shannon, 1948) linked the statistical modelling of natural language with compression, showing that measuring the cross entropy of a language model is equivalent to measuring its compression rate. Shannon fit early language models to real data via precomputed tables of text statistics (Dewey, 1923) relating model complexity to improved text compression alongside more realistic text generation. 1 But the relation to intelligence was there from the start: Shannon posits that a sufficiently complex model will resemble human communication adequately, and the Imitation Game (Turing, 1950) cemented the link. The relation between data compression (via prediction) and intelligence has been further expanded upon since (see Chater (1999) ; Legg and Hutter (2007) ; Wolff (1982) ).
A key driver towards better language models has been modern computing. From their pen-andpaper origins, language models have transformed in capacity and predictive power by the exponential rise in compute (Moore et al., 1965) . In the 1990s and 2000s, 𝑛-gram models saw increases in scale and better smoothing approaches (Ney et al., 1994) , including a 300 billion 𝑛-gram model trained on two trillion tokens of text (Brants et al., 2007) . These models have been applied to speech recognition (Jelinek, 1997) , spelling correction (Brill and Moore, 2000) , machine translation (Brown et al., 1990) , and many other areas. However 𝑛-gram models become statistically and computationally inefficient as the context length is increased, which limits the richness of language they can model.
In the past two decades language models have progressed to neural networks that capture the structure of language implicitly (Bengio et al., 2003; Graves, 2013; Jozefowicz et al., 2016; Mikolov et al., 2010; Radford et al., 2019) . Progress has been driven by both scale and network architecture (Bahdanau et al., 2014; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017) . ? and Kaplan et al. (2020) independently found power laws relating cross entropy loss to model size for recurrent and Transformer neural language models respectively. The empirically predicted gains to scale were realised in practice by the Generative Pre-trained Transformer 3 (GPT-3, Brown et al. (2020) ), a 175 billion parameter Transformer trained over 300 billion tokens of text, which consumed zettaflops of compute to train -an order of magnitude beyond prior work (Rosset, 2020) . GPT-3 demonstrated unprecedented generation quality alongside generalist capabilities across many Natural Language Processing (NLP) tasks -notably when prompted with examples (termed few-shot prompting).
In this paper we describe a protocol for training a state-of-the-art large language model and present a 280 billion parameter model called Gopher. We outline the methods of architecture specification, optimisation, infrastructure, and the curation of a high-quality text dataset MassiveText in Section 3. We perform a broad analysis of benchmark performance across 152 tasks that examine several diverse aspects of intelligence, and summarise the key results in Section 4. We see that Gopher lifts the performance over current state-of-the-art language models across roughly 81% of tasks containing comparable results, notably in knowledge-intensive domains such as fact checking and general knowledge.
As harmful content occurs both in Gopher's training set and in many potential downstream applications, we examine model toxicity and bias in Section 5 with a focus on how scale influences these properties. We find larger models are more likely to generate toxic responses when provided with toxic prompts, but they can also more accurately classify toxicity. We also analyse Gopher in a dialogue-interaction setting in Section 6 via prompting and present several transcripts to demonstrate qualitative capabilities and limitations of the model. Finally, we discuss the ethical and safe application of these models including which types of undesirable behaviour to mitigate before and after training in Section 7. We discuss applicationdriven safety and the potential for language models to accelerate research towards safer intelligent technology.
2. Background
Language modelling refers to modelling the probability of text 𝑃(𝑆) where 𝑆 can be a sentence, paragraph, or document depending on the application. This is done by tokenizing the string: mapping it to a sequence of integer-valued tokens: 𝑔(𝑆) = 𝑋 = (𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑛 ) ∈ 𝑉 𝑛 where 𝑉 is the vocabulary (a finite set of positive integers) and 𝑛 is the resulting sequence length, and modelling 𝑋. Tokenization can be open-vocabulary where any string can be uniquely tokenized, e.g., byte-level modelling, or closed-vocabulary where only a subset of text can be uniquely represented, e.g., a list of words and a singular out-of-vocabulary token. We employ open-vocabulary tokenization via a mixture of byte-pair encoding (BPE) with a backoff to UTF-8 bytes in the style of Radford et al. (2018) .
The typical way to model the token sequence 𝑋 is via the chain rule 𝑃(𝑋) = 𝑃(𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑛 ) = 𝑛 𝑖=1 𝑃(𝑋 𝑖 |𝑋 90%) which was obtained using a smaller 223M parameter ALBERT-XXL model fine-tuned on the dataset (Jiang et al., 2020) . It is possible supervised fine-tuning leads to greater reading comprehension, but it is also plausible the datasets contain exploitable statistics which can lead to high accuracy -as has been recently discovered for several common-sense reasoning benchmarks (Li et al., 2021) . In the claim-only setting (closed-book) there is a persistent trend in three-way classificaton accuracy with parameter scale. Breaking down the three classes into two pairs, scale benefits mostly the ability to distinguish SUPPORTED vs REFUTED, but not REFUTED versus NOTENOUGHINFO. When gold evidence is provided (open-book) there is a small benefit from 7.1B to 280B Gopher and performance slightly exceeds the supervised SOTA (Kruengkrai et al., 2021) .
For some of the most well-studied common sense reasoning tasks: Winogrande, HellaSwag and PIQA, Gopher is outperformed by the larger Megatron-Turing NLG by a small amount (1.2%, 0.2% and 4.1% respectively), but all LM approaches trail human-level performance considerably (Section D.13). As with the mathematics tasks, this suggests that these models have limited reasoning capabilities.
We next highlight fact-checking. This is an important problem within the domain of tackling misinformation. We find that Gopher outperforms supervised SOTA approaches on the well-studied FEVER fact-checking benchmark when evidence is supplied. We see across model sizes in Figure 3 that Random 25.0% GPT-2 1 32.4% Average human rater 1 34.5% GPT-3 scale improves both the checking of facts given gold evidence alongside the 'closed book' checking of facts with a claim only. However, larger scale does not benefit the classification of facts which are unknown versus false, implying that larger models improve fact checking performance by knowing more facts versus forming a deeper understanding of misinformation at this stage.
Moving beyond per-task performance, we display the average accuracy across the 57 tasks in MMLU (Table 5 ). These tasks consist of real-world human exams covering a range of academic subjects. We have comparisons from GPT-3 (Brown et al., 2020) , and a 11B T5 model fine-tuned on question tasks called UnifiedQA (Khashabi et al., 2020) . These baseline model results along with human rater and expert performance were collected by Hendrycks et al. (2020) . In Table 5 we see that Gopher achieves an overall accuracy of 60%, well above GPT-3's 43.9% and UnifiedQA's 48.9%. Whilst this lifts the known performance of the pure language-model approach, it still trails the estimated human expert performance of 89.8%. We also display how this performance contrasts with human expectations. From a competitive forecasting platform Hypermind3 , human forecasters aim to predict the accuracy of machine learning systems on this benchmark by set dates for prizes -according to the September 2021 average forecast, Gopher-level performance was expected between June 2022 and June 2023.
We conclude that Gopher lifts the baseline performance of a language-model approach across a wide set of tasks. In some settings (e.g., RACE reading comprehension and FEVER fact-checking) Gopher nears human rater performance or the performance of supervised models designed for particular problem domains. However for a few categories of tasks (e.g., mathematical reasoning and commonsense) there is less of an improvement and this may indicate a limitation to the large-scale language model approach. Next, we consider the topic of model scale in isolation.
4.3. Performance Improvements with Scale
Next, we investigate which types of tasks benefit from scaling model size. In this section we compare the performance of Gopher (280B) to smaller models (≤ 7.1B). Because the Gopher family of models are all trained on the same dataset for the same number of tokens, this allows us to isolate the effect Figure 4 | 280B vs best performance up to 7.1B across different tasks. We compare the performance of Gopher to the best performance of our smaller models up to 7.1B. In nearly every case, Gopher outperforms the best smaller model's performance. Small gains come from either scale not improving results substantially or the smaller models already being very performant. Language modelling improvements are in BPB and the rest are in terms of accuracy.
of scaling parameters and training compute for each task.
We compute the relative performance improvement of Gopher (280B) versus the best performance up to 7.1B over all 152 tasks. The most performant smaller Gopher family model is usually, but not always, our 7.1B model. We find that Gopher demonstrates a performance improvement on the vast majority of tasks -only 16 (10.5%) had zero or no performance gains. In contrast, 57 (37.5%) tasks had small improvements, with relative performance increases of up to 25%, and 79 (51.2%) tasks had significant improvements of over 25%. We then visualise relative performance improvement by task category in Figure 4 . Some of the largest benefits of scale are seen in the Medicine, Science, Technology, Social Sciences, and the Humanities task categories. These same categories are also where we see the greatest performance improvement over LM SOTA, as described in the previous section. Highlighting some specific tasks: for Figure of Speech Detection from BIG-bench we obtain the largest gains-a 314% increase. Gopher achieved an impressive 52.7% accuracy whereas the 7.1B model achieved only 16.8% accuracy. Gopher also dramatically improves over the smaller models in Logical Args, Marketing, and Medical Genetics. For the TruthfulQA benchmark (Lin et al., 2021b) we find performance improvement with scale (from 1.4B to 280B), despite scale appearing to hurt performance for several other model families such as GPT-J, GPT-2, T5, GPT-3. Furthermore, 280B is the first model to demonstrate performance significantly beyond random guessing on the multiple-choice TruthfulQA task formulation (more details in Section D.10). These results highlight that on some tasks, scale seems to "unlock" the ability of a model to significantly improve performance on particular tasks.
On the other hand, we find that scale has a reduced benefit for tasks in the Maths, Logical Reasoning, and Common Sense categories. Our results suggest that for certain flavours of mathematical or logical reasoning tasks, it is unlikely that scale alone will lead to performance breakthroughs. In some cases Gopher has a lower performance than smaller models-examples of which include Abstract Algebra and Temporal Sequences from BIG-bench, and High School Mathematics from MMLU. On the other hand, the modest performance gains in common sense tasks largely come from relatively strong performance from the smaller models, limiting the room for relative improvement. While language modelling tasks see the smallest average improvements, this is due to the performance metric measured in BPB rather than accuracy and greatly limits the possible relative gains.
By comparing Gopher to our smaller models, we are able to specifically ask questions about the impact of model scale. We conclude that while model scale plays an important role for improvements across the vast majority of tasks, the gains are not equally distributed. Many academic subjects, along with general knowledge, see large improvements come from scale alone. However, this analysis also highlights areas where model scale alone is not enough, or where the gains from scale are more modest-specifically some mathematical and logical reasoning tasks. By combining these scaling results with the comparisons of Gopher to LM SOTA, we see that scale and the dataset are both contributing to Gopher's strong performance in these domains. In the next section we investigate various properties of the model relating to toxic content generation and classification, the modelling of biases, and the representation of dialects.
5. Toxicity and Bias Analysis
Alongside the benefits of scaling language models, it is crucial to analyse how scale impacts potentially harmful behaviour. Here we study the behaviour of our language models with respect to problematic outputs and biases. We investigate the tendency of models to produce toxic output, to recognise toxic text, to display distributional bias in discourse about different groups of people, and to model subgroup dialects. For each question we consider variation across model scale.
We choose evaluations and metrics which are commonly used in the field. However, various work has discussed the limitations of current metrics and evaluations (Blodgett et al., 2020 (Blodgett et al., , 2021;; Sheng et al., 2019; Welbl et al., 2021; Xu et al., 2021a) and our analysis has uncovered further caveats, which we highlight in the following sections and Section 7.2. We include these measures despite their shortcomings to underscore the importance of tackling these challenges and to highlight specific areas for future work, rather than to establish these particular approaches as best practice.
5.1. Toxicity
In the Sections 5.1.1 and 5.1.2, we rely on the widely used and commercially deployed Perspective API4 classifier to study the toxicity of text generated by LMs, and associated CivilComments dataset for studying models' ability to detect toxic text. Accordingly, we adopt their definition of toxicity as "a rude, disrespectful or unreasonable comment that is likely to make someone leave a discussion."5 
5.1.1. Generation Analysis
Our toxicity analysis of text generated by LMs follows the methodology used in Gehman et al. (2020) ; Welbl et al. (2021) . We use Perspective API to obtain toxicity scores for LM prompts and continuations. We analyse the toxicity of LM outputs when sampling is conditioned on a set of prompts and when it's unconditional (i.e. unprompted), similar to Welbl et al. (2021) . Conditional generation allows us to analyse how the model responds to prompts that have varying toxicity scores. Prompts are from the RealToxicityPrompts (RTP) dataset (Gehman et al., 2020) , which contains 100k naturally occurring, sentence-level prompts derived from a large corpus of English web text. We sample 10% of the 100k RTP prompts for efficiency and generate 25 continuations per prompt.
The continuation toxicity of larger models is more consistent with prompt toxicity than for smaller models (Figure 5a ). When prompted, as the input toxicity increases, larger models respond with greater toxicity, plateauing near 7.1B parameters. This suggests that more parameters increase the model's ability to respond like-for-like to inputs. For unprompted samples, the toxicity is low and does not increase with model size. Levels are slightly lower than in the training data (Figure A22b ), i.e. when unprompted, the LM does not amplify training data toxicity. More details on our toxicity evaluation methodology, results and metrics can be found in Section E.1.
5.1.2. Classification Analysis
We evaluate the models' ability to detect toxic text in the few-shot setting, in a manner similar to Schick et al. (2021) , on the CivilComments dataset (Borkan et al., 2019) (see Section E.2 for details). We observe that the model's ability to classify text for toxicity increases with scale in few-shot settings (Figure 5b ). The smaller models perform comparably or worse than a random classifier (which would achieve an AUC of 0.5). The largest model achieves an AUC of around 0.76 in the 20-shot setting, significantly improving on the smaller models (Figure 5b ). We note that while the state-of-the-art for toxicity detection in the few-shot setting is not well established, our performance is well below that of state of the art classifiers trained specifically for toxicity detection (Borkan et al., 2019) .
In Section E.2, we further explore whether large language models used for few-shot toxicity classification exhibit subgroup bias. We measure unintended classifier bias using the 280B model with metrics introduced in Borkan et al. (2019) and find that the model is prone to bias against subgroups in different ways. Thus, while language models can be a powerful tool for few-shot classification (especially important in tasks with data that is difficult to annotate), outcomes are not necessarily fair across subgroups. More work is needed to understand how to best mitigate these biases, and caution must be exercised when optimising for improvements in their toxicity classification capabilities.
5.2. Distributional Bias
We define distributional biases as biases which are not apparent in a single sample, but emerge over many samples. For example, whereas "The woman is a nurse" is not a problematic sentence, it can be problematic if the model disproportionately associates women with certain occupations. As discussed in Sheng et al. (2021) , distributional biases in language models can have both negative representational impacts (e.g., Kay et al. (2015) ) and allocational impacts (e.g., Dastin (2018) ). To investigate distributional biases in our model, we measure stereotypical associations between gender and occupation, the distribution of sentiment in samples conditioned on different social groups, and perplexity on different dialects. Whereas performance across many language tasks increases with scale, we find that simply increasing model size does not remove biased language. Indeed, we expect Progress in this space will require challenging cross-disciplinary work to outline desirable behaviour, measure and interpret model outputs, and design novel mitigations, as demonstrated by these results and the significant limitations of the following methods, discussed in Section 7.2.
5.2.1. Gender and Occupation Bias
We study gender and occupation bias via two different evaluations. First, we measure the probability of gendered words following different occupation contexts. Second, we evaluate on the Winogender coreference resolution dataset (Rudinger et al., 2018) , where similar coreference accuracy across different pronouns indicates less gender bias. In our evaluation, we primarily compare performance across male and female gendered terms, but acknowledge these terms do not represent all possible gender identities (Cao and Daumé, 2021) .
Gender Word Probability
To measure how probable different gender words are in different occupation contexts, we follow a setup similar to Brown et al. (2020) . We input an occupation prompt like "The {occupation} was a" into our model and compute a gender bias metric by comparing the probabilities of the prompt being followed by either male or female gendered terms.
Figure 6a reports our probability based gender bias metric as a function of model size for two different templates ("The {occupation} was a {gender}" and "The {occupation} is a {gender}"). Overall, we do not find a consistent correlation between model size and bias. Furthermore, we find that apparently unrelated choices in template (changing "was" to "is") can alter the measured bias. Additionally, choice of gender words also impacts results; if we only use the terms "male" and "female," gender bias is substantially lower than when summing over a large set of gendered terms (Figure A24a ). Section E.3.1 contains further details of the implementation, metrics, and results.
Winogender
We explore bias on a zero-shot coreference task using the Winogender dataset (Rudinger et al., 2018) . Models are evaluated on whether they can correctly resolve a pronoun to either an occupation word or a relevant distractor word. We expect unbiased models to have similar coreference resolution performance regardless of the pronoun gender. This evaluation is similar High is positive sentiment, low is negative. See Figure A26 for religion and occupation results.
to the "disambiguation_q" ambiguous pronoun gender bias task reported in our BIG-bench results (Section D.8.3) . However, here we are measuring performance in a zero-shot setting.
Similar to the BIG-bench analysis, we observe that overall performance increases with model size (Figure A24b ). Following Rudinger et al. (2018) , we also report performance on sentences which are likely to be hard for a gender biased model (called "gotchas") in Figure 6b . A "gotcha" example is one where the correct coreference resolution is one that differs from stereotypes (based on labor statistics6 ). Performance increases across both "gotchas" and "not gotchas" with model size, though performance on "gotchas" is considerably lower. On "gotcha" examples, there is a significant difference in performance for male and female pronouns. Thus, though performance on coreference resolution for the overall task increases considerably with size, our analysis suggests Gopher is still impacted by gender and occupation bias. Full details of our setup and results are in Section E.3.1.
5.2.2. Sentiment Bias towards Social Groups
Sentiment bias is one way to quantify how generated text describes different identities and social groups. In prior work, the difference in sentiment distributions is used to measure individual and group fairness in generative language models (Huang et al., 2020) . For this work, we measure the sentiment of model outputs for different occupations, countries, races, and religions. Here we present an overview, with details of metric definition, term and template lists, and full results in Section E.3.2. Huang et al. (2020) , we sample completions based on templated prompts. In each prompt, a single modifier or noun is changed to refer to a different attribute. For example, the template "The {attribute} person could" could be filled in with "Christian," "Jewish," or "Muslim". The samples for each prompt are scored between 0 (negative) to 1 (positive) by a sentiment classifier. Following Brown et al. (2020) ; Huang et al. (2020) we measure sentiment for race, religion, country, and occupation. We also extend the term set for religion and race to include an unspecified option without the attribute word ("The {attribute} person could" becomes "The person could"). We include this unspecified option because attributes that are assumed to be the default in a particular culture or context, such as a majority or higher-status attribute, are often left unmarked (unspecified) in language (Waugh, 1982) .
Results
In Figure 7 and Figure A26 , we plot the distribution of normalized sentiment scores for all completions of all prompts for each attribute, and report an aggregated group fairness metric in Figure A25 . As in gender and occupation bias, we see no clear trend with scale. This is particularly evident for countries and occupations, while further analysis is needed to understand why particular attributes within race and religion appear to follow a slight downward trend in mean sentiment.
For sentiment distribution, we observe that certain attributes have notably lower mean sentiment scores. To better understand this, we analyse word co-occurrences for pairs of attributes (Table A25 ). From this, we observe our models inherit features of historical and contemporary discourse about specific groups (Mohamed et al., 2020) . Second, similar to the gender and occupation results, the choice of demographic terms requires careful thought. See Section E.3.2 for deeper discussion.
5.2.3. Perplexity on Dialects
Although Gopher has impressive performance on language benchmarks, it is only able to model text reflected in the training data. If certain dialects are underrepresented in a training corpus, there is likely to be disparate model performance in understanding such language. To test for this gap, we measure the perplexity of our models on Tweets from the African American (AA)-aligned corpus and White-aligned corpus curated by Blodgett et al. (2016) . Our results show that perplexity on the AA-aligned corpus is higher for all model sizes. As the model scales, perplexity for both dialects improves, but it does so at roughly the same rate so the gap does not close with scale.
These results highlight a distinct way that bias manifests in the language models. The preceding metrics quantify how models' outputs vary when different groups are the subject of the output, which can constitute a representational harm when it is more negative or stereotypical (Blodgett et al., 2020) . However, the models also show disparate ability in modelling dialects, which could lead to allocational harms in applications with users with different dialects.
6. Dialogue
So far, we have explored the capabilities and limitations of Gopher through quantitative methods. In this section we investigate the model through direct interaction. We find that by conditionally sampling from a dialogue prompt similar to the few-shot method of Brown et al. (2020) , our Dialogue-Prompted Gopher can emulate a conversational format to a decent quality. We provide example transcripts here, with more in Section H.5. We contrast this with the more conventional method of fine-tuning on dialogue data, finding that fine-tuning did not deliver significantly preferred responses in a small-scale human study. Unlike Section 5.1.1, toxicity of Dialogue-Prompted Gopher responses does not increase with model scale, even when prompted with toxic questions (Figure 9 ).
Prompt Generation
What am I thinking?
It's Friday night and I'm in bed and awake at 12:37 am. "I woke up from a dream where I was standing next to... The Golden Globes are over and now I need to make a list of films that I must see, most likely in the coming months Where am I headed? Is there another place where I can get away? Can I use my smartphone to play? How do I think I can do this?
Table 6 | Responses from Gopher when naively prompted with a question, for three seeds.
Language models are trained to reproduce their input distribution, not to engage in conversation. When prompted with a question, we can see that the model generates a first-person narrative, some text resembling a blog post, and a generic list of existential questions (Table 6 ). This behaviour is consistent with the content that Gopher has been trained on.
In order to produce a conversationalist, we use a prompt that describes Gopher's role and starts a conversation between Gopher and a fictional User, including behaviours such as aversion to offensive language and an ability to opt out of certain question types; see Table A30 for the full prompt. Table 7 shows a transcript with Dialogue-Prompted Gopher on the topic of cell biology and bacteria. Here it remains on topic, discusses some technical details, and provides a correct citation link. However it actually provides subtle incorrect responses in some cases (prokaryotes are not the only single-cell organisms). Table 8 shows an unsuccessful transcript illustrating factual errors confidently expressed. See Section H.5 for more transcripts with interesting behaviours and failure modes, including more subtle plausible but factually incorrect dialogue with a claim of search (Table A32 ), generating harmful text (Table A35 ), or contradicting itself and showing a general lack of common sense (Table A37 ).
Anecdotally, we find both successes and failures to be common, but we emphasize that Dialogue-Prompted Gopher is still just a language model. The prompt conditions the model's prior over responses but does not result in a consistently reliable or factual dialogue model. We refer the reader to Weidinger et al. (2021) for a detailed discussion on language model harms specific to dialogue and we discuss some ideas regarding building trustworthy systems in Section 7.3.
6.2. Fine-tuning for Dialogue
Recent work on dialogue often focuses on supervised training with dialogue-specific data (Chen et al., 2017) , such as Google's Meena (Adiwardana et al., 2020 ) and Facebook's BlenderBot (Roller et al., 2021a) . We explore this approach by creating a curated dialogue dataset from MassiveWeb and fine-tuning Gopher on this dataset for ∼5 billion tokens to produce Dialogue-Tuned Gopher. We then ask human raters for their preference over the response from Dialogue-Tuned Gopher and Dialogue-Prompted Gopher, using our dialogue prompt (Table A30 ) for both models. To our surprise, we find from 1400 ratings the preference is (50 ± 0.04)%: no significant difference. We describe the methodology in detail in Section H.3. We consider this an interesting initial result; future work would be valuable to rigorously examine the pros and cons of fine-tuning versus prompting for dialogue with large-scale models and compare Gopher to existing dialogue systems accounting for large differences in model size. 
6.3. Dialogue & Toxicity
We investigate the toxicity of Dialogue-Prompted Gopher. We adapt the RTP methodology to the dialogue setting (called RTP questions, details in Section H.4). In Figure 9 (left), we observe that Dialogue-Prompted Gopher does not follow the same trend (increased toxicity with model scale) as Gopher. Whilst we see a monotonic increase in continuation toxicity with model scale in the unprompted setting (Figure 5a ), Dialogue-Prompted Gopher toxicity tends to slightly decrease with increased model scale (from 117M parameters, except for prompts in the most toxic bucket). Potentially, larger models can better account for the given prompt (which includes "to be respectful, polite, and inclusive"). Specifically, we compare the continuation toxicity between Gopher (tested on RTP) and Dialogue-Prompted Gopher (tested on RTP questions) models relative to 44M models for prompts with high toxicity in the right of Figure 9 . Again, we observe that with dialogue prompting, continuation toxicity remains largely at levels similar to the 44M model, contrasting with the upward trend observed for unprompted language models.
RTP is quite a straightforward stress-test: the user utters a toxic statement and we observe how the system responds. In work parallel to this study, Perez et al. (2022) probes Dialogue-Prompted Gopher further via an adversarial attack generated by Gopher. This approach induces the model to recite discriminatory jokes from its training data, insult the user, and elaborate on inappropriate desires, among many other offenses. Occasionally, Dialogue-Prompted Gopher's response refers to the fact that its instructions prohibit a behaviour before exhibiting that behaviour, such as by opening with "[Ignoring your request to not discuss political, social, and religious issues.]" To date, automatic adversarial attacks consistently elicit toxic language from models (Wallace et al., 2019) even after safety mitigations (Yu and Sagae, 2021) , and serve as a useful complement to manual adversarial attacks such as Xu et al. (2021b) .
The recent work of Askell et al. (2021) similarly found that prompting alone was sufficient to turn a language model into an interesting but non-robust assistant. They conduct a variety of human evaluations of their system, both for the prompt-only case and for stronger interventions such as learning from human demonstrations or preferences. In particular, they also found that prompting prevents toxicity from increasing with scale on RTP (Section 2.2.2 in their paper). This provides evidence that the effect is reliable across different language models and toxicity classifiers. 
7.1. Towards Efficient Architectures
In this work we have taken a well established architecture and pushed model scale. To follow this scaling enquiry further, we have to either increase the amount of energy and compute to train larger transformers or move towards more efficient architectures.
We break down the computational cost from training Gopher in Table A26 and Appendix F and observe the majority is spent in the linear maps. This motivated an investigation into sparseparameter training detailed in Appendix G, but did not yield an overall efficiency boost to date. An alternative approach to sparsifying the linear maps is to split them into separate, conditionallyactivated experts (Fedus et al., 2021; Lepikhin et al., 2021; Lin et al., 2021a) . This approach has been scaled up with the Switch Transformer which contains 1.7T parameters but a smaller compute cost to Gopher (Fedus et al., 2021) and the more recent 1.2T GLaM (?) which outperforms GPT-3 across 29 language tasks whilst requiring 3X fewer FLOPs to train. We separately consider a retrieval mechanism searching over the training set for relevant extracts during pre-training (Borgeaud et al., 2021) , partially avoiding the need to memorise knowledge into network weights. This approach reached GPT-3-level language model performance with a 7 billion parameter model and over a 10× reduction in training compute. Thus, whilst this paper focused on transformer models, this is likely a transitory stage as more efficient architectures are developed.
7.2. Challenges in Toxicity and Bias
We highlight some of the limitations we encountered in our evaluation metrics for toxicity and bias and motivate what properties would be desired from future evaluation benchmarks.
Challenges in using classifiers. While the Perspective API is a capable toxicity classifier (0.97 evaluation AUC 7 ), toxicity classifiers can be subject to social bias, assigning higher toxicity to innocuous mentions of particular identity groups (Dixon et al., 2018; Röttger et al., 2021) . While toxicity classifiers quantify one type of harm, overreliance on automatic evaluation can introduce unintended social biases (Welbl et al., 2021; Xu et al., 2021a) . Sentiment classifiers are also subject to bias (Kiritchenko and Mohammad, 2018) . Sheng et al. (2019) propose regard classifiers as an alternative to repurposing sentiment classifiers for bias analysis; these measure regard towards a particular demographic group, but are only available for certain groups.
Challenges in distributional bias. While we only consider a few possible evaluations (see Sheng et al. (2021) for an overview), we observe that distributional bias can be especially challenging to measure. Figure 6a illustrates the brittleness of template-based evaluation: simply changing the verb in the gender and occupation template from "was" to "is" impacts observed trends. However, collecting high quality, naturalistic datasets is challenging (Blodgett et al., 2021) . We believe high quality data collection will be interdisciplinary and involve consulting experts on various language harms, as was done for HateCheck dataset (Röttger et al., 2021) .
Challenges in defining context. Our toxicity and bias evaluations are not contextualised in applications or specific user groups, leaving the desired behaviour unclear. For example, we choose commonly studied subgroups for our analysis (adopted from Brown et al. (2020) and Huang et al. (2020) ), but demographic groups such as race are highly contextual (Hanna et al., 2020) . Our larger models produce more toxic outputs when prompted with toxic inputs; this may help models designed to detect toxicity (Section 5.1.2) but be problematic in other applications. In our sentiment analysis, our model frequently outputs negative words like "flee" and "escape" when describing Syria, but enforcing equal sentiment across countries might erase historical and political context.
The limitations above focus on measuring bias and toxicity as we do not explore mitigation strategies in this work. However, our limitations demonstrate important challenges in measuring and defining criteria for language models, and we emphasize the importance of careful model analysis and understanding in language research. Robust metrics are essential for effective mitigation, and we posit that work which outlines desirable behaviour, designs reliable metrics, and builds analysis tools is as important as methods developed for mitigation.
7.3. Safety benefits and safety risks
We believe language models are a powerful tool for the development of safe artificial intelligence, and this is a central motivation of our work. However language models risk causing significant harm if used poorly, and the benefits cannot be realised unless the harms are mitigated.
On the benefit side, language is the primary human communication medium for subtle ideas. If we want ML models which do what humans want, including in subtle cases where correct behaviour requires detailed discussion, we need machines capable of engaging in that discussion. Both directions of communication will be required: humans telling machines what we want, and machines explaining their behaviour to humans. In the near-term, natural language explanations can make models more trustworthy (Camburu et al., 2018) and more performant Coyle and Weller (2020) ; Kasirzadeh (2021) ; Rajani et al. (2019) survey some of the benefits and subtleties of explanations. Safety methods focused on interactive communication with humans include cooperative inverse reinforcement learning (Hadfield-Menell et al., 2016) ; see Russell (2020) for a broader discussion.
To extend the benefits of communication to advanced agents, several recursive safety proposals use language to break down tasks into smaller pieces that are easier to supervise by humans, including iterated amplification (Christiano et al., 2018) , debate (Irving and Askell, 2019; Irving et al., 2018) , and recursive reward modelling (Leike et al., 2018) . Realizing these schemes require language models to follow human discussion and reasoning, motivating work on highly capable models. Experimental work is nascent: Wu et al. (2021) uses recursive reward modelling to summarise books hierarchically, building on earlier work using human feedback for simpler tasks such as summarisation (Böhm et al., 2019; Stiennon et al., 2020; Ziegler et al., 2019) . Perez et al. (2019) simulates debate using a frozen question-answering model as judge. Human preference learning has been applied to many other NLP tasks including dialogue (Jaques et al., 2020) ; see Wang et al. (2021) for a survey.
On the harm side, Bender et al. (2021) highlights many dangers of large language models such as memorisation of training data (Abubakar, 2021; Carlini et al., 2021) , high training cost (Section G.3), distributional shift due to static training data (Lazaridou et al., 2021) , amplification of inherent biases, and generation of toxic language (Gehman et al., 2020 ) -which we consider in Section 5. See Weidinger et al. (2021) for an over-arching taxonomy of harms.
After assessing the landscape of potential harms, it is natural to question how and when to mitigate them. Some harms can be tackled during pre-training, such as leaks of private information and reduced performance for some languages and social groups. Privacy-preserving training algorithms such as Abadi et al. ( 2016) have been applied only at small scale, such as in Anil et al. (2021) to pre-train a 340M parameter BERT model and in Yu et al. (2021) to fine-tune LMs with up to 1.5B parameters. English-only datasets should be broadened to more languages (Xue et al., 2020) . We have begun this process for MassiveWeb: Borgeaud et al. (2021) trains on a version with 10 languages. However, we believe many harms due to LMs may be better addressed downstream, via both technical means (e.g. fine-tuning and monitoring) and sociotechnical means (e.g. multi-stakeholder engagement, controlled or staged release strategies, and establishment of application specific guidelines and benchmarks). Focusing safety and fairness efforts downstream has several benefits: Faster iteration cycles. LLMs are trained infrequently due to their expense, so mistakes are slow to correct during pre-training but fast to correct if mitigations are applied downstream. Fast iteration is critical when factual information changes (Lazaridou et al., 2021) , societal values change (Weidinger et al., 2021) , or our knowledge about how to mitigate harms changes. In particular, accidental censoring of data can damage performance for language by or about marginalized groups (Dodge et al., 2021; Welbl et al., 2021; Xu et al., 2021a) .
Safety depends on the application. Language models reflect the statistics of their training data rather than alignment to human values, and it is unclear what it means to align a language model without knowing the downstream application. Selbst et al. (2019) emphasize the non-portability of fairness between social contexts and applications. Model cards (Mitchell et al., 2019) include primary intended use and out-of-scope uses, and datasheets for datasets (Gebru et al., 2018) include recommended uses. As an example, a dialogue agent should avoid toxic language, while a translation model may need to preserve toxicity to ensure accuracy.
LMs can serve multiple roles within one application.
A single LM might be used both as a classifier for good vs. bad output and as a policy generating that output (Stiennon et al., 2020) . As a policy we may want no toxic output, but as a classifier the LM should be familiar with toxic text to classify it accurately (Buckman) . Downstream mitigation allows separate fine-tuning for each role, but mitigating toxicity by filtering during pre-training can harm classifier performance (Welbl et al., 2021) . Figure 5 shows a correlation between generation and recognition of toxic language in the Gopher family. In some cases, toxicity is the goal: Perez et al. (2022) uses Gopher to generate questions which cause Dialogue-Prompted Gopher to behave poorly. This classifier vs. policy split applies to other harms: we may want an accurate policy and a good lie detector.
However, any particular claim that a harm is best mitigated downstream is empirical: if we cannot mitigate downstream in practice, mistakes will be locked in until the next LM is retrained. We also emphasize that even if some mitigations are best applied downstream, we share responsibility for ensuring the necessary mitigations occur in applications where Gopher is deployed, both by influencing those deployments and by conducting applicable safety research. We have started some of this research, including both harm taxonomies (Kenton et al., 2021; Weidinger et al., 2021) and mitigations (Perez et al., 2022; Welbl et al., 2021) . Much more is required, and is left to future work.
8. Conclusion
The landscape of language technologies with general capabilities is progressing rapidly. Language models are a key driver of this progress, and we have shown that an emphasis on data quality and scale still yields interesting performance advances over existing work. However, the benefits of scale are nonuniform: some tasks which require more complex mathematical or logical reasoning observe little benefit up to the scale of Gopher. This may be an inherent property of the language modelling objective -it is hard to compress mathematics and easier to learn many associative facts about the world. However it is possible that a sufficiently complex model may become bottlenecked by its poor understanding (and thus compression) of reasoning and new reasoning capabilities will emerge beyond the scale reached here. Alongside the development of more powerful language models, we advocate broad development of analysis and interpretability tools to better understand model behaviour and fairness, both to guide mitigation of harms and to better inform the use of these models as a tool to scalably align artificial intelligence to societal benefit.
9. Acknowledgements
We would like to thank Dominik Grewe, Dimitrios Vytiniotis, Tamara Norman, and Dan Belov for their help verifying the final training topology; Peter Hawkins and Skye Wanderman-Milne for their help understanding the JAX runtime; Loren Maggiore for input on mixed-precision training;
A. MassiveText
We describe our data collection procedure for MassiveText, analyse the resulting dataset, and justify key design choices. We include the datasheet (Gebru et al., 2018) for MassiveText in Section A.5.
We believe that dataset diversity is crucial for training powerful and general large language models, and thus include data from a diverse range of sources (Table 2 ): web pages (from custom dataset MassiveWeb, C4, and Wikipedia), books, news articles, and code (GitHub). Existing text datasets created for training large language models are typically based solely on web pages, such as the C4 and mC4 datasets (Raffel et al., 2020b; Xue et al., 2020) . Similar to our work, The Pile (Gao et al., 2020) dataset also includes many text sources such as web pages, books, and academic papers.
When collecting MassiveText, we decide to use only simple heuristics for filtering out low quality text. In particular, we do not attempt to filter out low quality documents by training a classifier based on a "gold" set of text, such as English Wikipedia or pages linked from Reddit (Radford et al., 2019) , as this could inadvertently bias towards a certain demographic or erase certain dialects or sociolects from representation. Filtering text for quality, while preserving coverage of dialects and avoiding biases, is an important direction for future research.
A.1. Dataset Pipeline
In this section we detail the pipeline stages we use to collect the various subsets of MassiveText. We also include a brief description of our algorithm to extract fixed-size training chunks from our dataset of documents.
A.1.1. Pipeline stages
For all MassiveText subsets, we filter out non-English documents, process data into a homogeneous text-only format, deduplicate documents, and filter out documents too similar to those in our test sets. Additionally, for our curated web-text corpus (MassiveWeb) we obtain the web data in text-only format using a custom HTML scraper, we apply an extra filter to remove explicit content at the initial stages, and we apply a series of simple heuristics to filter out low-quality text. Figure A1 gives an overview of all data processing stages, which we will discuss in detail for the remainder of this section. 
Content Filtering (All subsets)
We start by filtering out non-English documents. At this stage, we also remove pages from MassiveWeb that do not pass Google's SafeSearch filter, which incorporates various web signals to identify explicit content. 13 We use SafeSearch rather than manual wordlist filters, because the latter have been found to disproportionately filter out inoffensive content associated with minority groups (Dodge et al., 2021) .
Text Extraction (MassiveWeb only)
We extract text from web pages using the tree structure of the HTML markup. For high-quality web pages, we observe that self-contained coherent blocks of salient text tend to occur in groups of semantic tags at the same level in the tree. We find such sets of tags and convert them to plain text, taking care to preserve any meaningful formatting, such as indentation, newlines and bullet points. This yields a large volume of text documents, and the resulting diversity in formatting style translates effectively to the generative capabilities of the Gopher models.
Quality Filtering (MassiveWeb only)
The vast majority of text found on the web is of insufficient quality to be useful for language model training. For example, many web pages contain primarily automatically generated content, or text that is not intended for human consumption (such as keywords for search-engine optimisation). Much of the web also comprises social media content, which can variously lack context, coherence, or substance. To remove low-quality data while minimising potential for bias, we apply a number of simple, easily understood heuristic filters: we remove any document that does not contain between 50 and 100,000 words, or whose mean word length is outside the range of 3 to 10 characters; we remove any document with a symbol-to-word ratio greater than 0.1 for either the hash symbol or the ellipsis; and we remove any document with more than 90% of lines starting with a bullet point, or more than 30% ending with an ellipsis. We also require that 80% of words in a document contain at least one alphabetic character, and apply a "stop word" filter, to remove documents that do not contain at least two of the following English words: the, be, to, of, and, that, have, with; this adequately deals with ostensibly English documents that contain no coherent English text.
Repetition Removal (MassiveWeb only)
Another indicator of poor quality data is excessive repetition of certain words or phrases within a document. Qualitatively we observe that excessive repetition is often linked with uninformative content. Furthermore a well-studied failure mode of current language models is to repeat themselves during sampling (Holtzman et al., 2019) which may be partially attributed to repetitous training data.
We address this by removing documents with a high proportion of repeated lines, paragraphs, or 𝑛-grams. We remove documents containing many short duplicate passages, as well as those with fewer, larger sections of duplicate content, and we make sure to identify both types by using multiple approaches to calculate the proportion of duplicate content. For lines and paragraphs separately, we calculate over the document both the fraction that are duplicates, and the fraction of characters contained within those duplicates; for each 𝑛 ∈ {2, . . . , 4}, we calculate the fraction of characters contained within the most frequently-occurring 𝑛-gram; and for each 𝑛 ∈ {5, . . . , 10}, we calculate the fraction of characters contained within all duplicate 𝑛-grams, taking care not to count characters that occur in overlapping 𝑛-grams more than once. We then filter out documents whose duplicate content surpasses any of the thresholds detailed in Table A1 .
An alternative approach to data filtering that we consider is to use an existing model to rank documents by likelihood. However, samples that are assigned high likelihood by a model are not For each measurement of text repetition, we show the limit above which a document containing such repetition is filtered out.
necessarily high quality, even if the data used to train the model was high quality -repetitious text falls under this category. Furthermore it can also be costly, as it requires inferring likelihoods for a large number of documents, and carries an increased risk of introducing unintentional bias. However we consider this an interesting area for future work.
Document Deduplication (All subsets) 14 Many web pages contain text that is duplicated on other pages across the internet. We remove all exact duplicates to obtain a set of unique documents. In addition to exact duplicates, there are many documents with significant 𝑛-gram overlap. We use the MinHash algorithm to compute 13-gram Jaccard similarities to determine which documents are near-duplicates of each other (Lee et al., 2021a) . To further increase recall, we normalize white spaces and ignore punctuation when constructing the 𝑛-grams. We define two documents to be too similar when their Jaccard similarity exceeds 0.8, and randomly remove one of them.
Test-set Filtering (All subsets)
We use a similar approach to remove training documents that resemble documents from our test datasets (Wikitext103, C4, Curation Corpus, and LAMBADA). Specifically, we compute the 13-gram Jaccard similarity between train and test documents, and remove train documents that have a Jaccard similarity exceeding 0.8 with a test set document.
Additionally, we remove the Wikipedia pages used in the Wikitext103 validation and test sets from our Wikipedia training dataset. This ensures that we do not leak Wikipedia pages from Wikitext103 which might have been missed in the previous procedure due to edits made to those pages since the Wikitext103 dataset was collected.
We apply this 𝑛-gram based filtering strategy to all subsets of MassiveText but note that some of our test datasets (such as the Pile) were created after we trained Gopher and thus may be leaked in our training dataset.
A.1.2. Constructing Token Sequences
We describe our algorithm for extracting training sequences from the set of documents in MassiveText.
The algorithm is designed to have good shuffling properties and to avoid unnecessary PAD tokens which waste compute resources. Informally, we follow the following steps:
1. Uniformly choose a document of 𝐵 bytes from one of our MassiveText subsets. 2. Crop out 𝐶=15 × 𝑛 UTF-8 bytes, where 𝑛 is the training token sequence length. Uniformly choosing a start index for the crop would skew the distribution in such a way that we would almost never see the first token in a document. We therefore first uniformly sample a start index 𝑠 in U -𝐶 4 , 𝐵 -𝐶 4 and extract the crop from [max(0, 𝑠), min(𝐵, 𝑠 + 𝐶)].
3. Tokenize the extracted bytes, and add the BOS and EOS tokens.
4. Since most documents are shorter than our sequence length 𝑛=2048, we concatenate 10 such tokenized byte crops. 5. We split the concatenation into sequences of 𝑛=2048 tokens, and discard the final chunk if it's shorter than the sequence length. This avoids wasting compute by training on PAD tokens.
6. Merge data from the various MassiveText subsets by sampling individual training sequences
according the weights given in Table 2 . 7. Shuffle and batch the data for training.
A.2. Dataset Analysis
Understanding the performance of the Gopher family of models is one angle of insight into the complete methodology. However, we can also understand the strengths and limitations of these models by analysing their training dataset. In this section we analyse MassiveText, breaking it down by document lengths, toxicity, languages, contents (such as web domains), and tokenizer compression rate.
Document Lengths
We show the distribution of document length measured in tokens in Figure A2a . MassiveWeb, C4, News, and Wikipedia documents contain on average fewer than 1,000 tokens. A majority of documents from those datasets can be fully included in the 2,048 sequence length of our models. For GitHub, the average document contains 2,946 tokens. Only the Books dataset contains extremely long documents-an average book contains 120,000 tokens and the longest book has over 1.3M tokens.
Training Data Toxicity
We evaluate the toxicity of the MassiveText subsets, again using Perspective API. To this end, we select random text spans up to 100 tokens from 200k documents sampled from each training data subset, truncating incomplete sentences, and sub-sample the resulting text spans to match the respective subset sampling weights used during Gopher training. We sub-sample based on total token count rather than document count, to avoid giving long documents (e.g., Books) more weight than during training. Despite the light data filtering, we observe generally low toxicity scores in the toxicity histogram in Figure A2b . Across all training subsets mean and median toxicity scores are at 0.10 and 0.07, respectively, and the 95% percentile toxicity score is 0.3. Considering the threshold of 0.5, at which a toxic label is the more likely prediction of the Perspective API classifier, 0.8% of the texts fall above this score. This is markedly lower than the corresponding proportion of 4.3% reported by Gehman et al. (2020) for the GPT-2 training data, potentially reflecting the different principles for training data selection. 15 As not all MassiveText subsets are sampled with equal weight during training, we provide a per-dataset breakdown in Figure A22a . Overall toxicity levels are lowest on Wikipedia, while the increased levels for Github can potentially be explained with out-of-domain application of the toxicity classifier, resulting in more prediction uncertainty.
Language Distribution
The vast majority -99% -of text in MassiveText is English. The distribution of the top 10 remaining languages is shown in Figure A3a . We exclude the GitHub dataset from this analysis as it mostly comprises code. The majority of the non-English text is in Hindi, followed by European languages: French, Spanish, German, and Italian. Chinese and Japanese make up for 5% and 4% of the non-English tokens respectively.
MassiveWeb URL Breakdown To better understand the contents of MassiveWeb, we show the top 20 domains by token count in Figure A3b . A majority of domains in the top 20 are academic journals, presentation websites, question answering websites, or social media. Despite not explicitly constructing or biasing the contents towards scientific content, we find that 4 of the top 6 domains are of academic or scientific nature. We also note that 0.33% of MassiveWeb tokens come from GitHub and 0.28% from Stack Overflow. A2 shows the compression rate of our 32,000 BPE vocabulary on the MassiveText subsets, measured in UTF-8 bytes per SentencePiece token. Note that SentencePiece tokens never cross word boundaries. We compare with the larger GPT-2/3 BPE vocabulary of 50,000 tokens. Using a larger vocabulary provides a small increase in compression rate: between 1% to 3% for text datasets and over 13% for GitHub. Table A2 | Dataset Compression Rate of our tokenizer measured in UTF-8 bytes per (tokenized) token (higher implies better compression), compared to the GPT-2 tokenizer. GitHub is the least compressible subset, whereas C4 is the most. The larger GPT-2 vocabulary provides a relative increase of 1%-3% for text and a 13% increase for code.
Bytes per Token Ours
(32K) GPT-2 (50K) (% Δ) Wikipedia (
A.3. Dataset Ablations
In this section we ablate two key design choices: the relative weighting of each MassiveText subset during training, and the pre-processing steps for collecting MassiveWeb.
A.3.1. MassiveText Subsets Weighting
We first analyse how different weightings of the MassiveText subsets affect the downstream performance on Wikitext103, LAMBADA, C4, and Curation Corpus. To reduce the size of the sweep, we fix the sampling weights for Wikipedia and GitHub. For Wikipedia, we require a full epoch over the training data, and thus fix the sampling weight to 2%. For GitHub, we set the sampling weight to 3%, as we want our models to train primarily on text but still be exposed to code. We thus consider the relative contribution of the remaining 95% of text between the remaining four subsets (MassiveWeb, News, Books, and C4). We sweep over 7 different combinations and show the downstream loss in Figure A4 . We find that using a high proportion of Books reduces the loss on LAMBADA, whilst using a higher proportion of C4 helps on the C4 validation set. 
A.3.2. Iterative Refinement of MassiveWeb
We construct MassiveWeb by iteratively refining several key processing stages (described in Section A.1), all of which lead to improvements in model performance.
We validate the impact of the processing stages by training 1.4B parameter models at each stage. We sub-sample all datasets to 5GB of text, in order to run this ablation in a reasonable amount of time. We report the validation loss on three downstream tasks as a proxy for dataset quality in Figure A5 . Compared with the extracted text in its raw unfiltered form, adding the simple heuristic quality filters described in Section A.1 dramatically improves downstream performance across the board, and deduplicating documents brings further substantial improvements. With all processing stages combined, a model trained on our dataset significantly outperforms models trained on OpenWebText (Radford et al., 2018) or C4 on all three datasets. We also note that the effect of deduplication is likely underestimated on the sub-sampled datasets as larger datasets are expected to contain more duplicates. 
A.4. Text normalisation
Our tokenizer performs NKFC 16 normalization as a pre-processing step. This normalization form is not fully lossless. For example, exponents are brought down: 2 5 is normalized to 2 5. This reduces the expressivity of the model and also changes the evaluation and test datasets. We therefore will use lossless normalization forms in future work and recommend this more generally to anyone using open-domain vocabularies.
A.5. MassiveText Datasheet
We follow the framework defined by Gebru et al. (2018) and give the datasheet for MassiveText in Table A3 .
Motivation
For what purpose was the dataset created? Who created the dataset? Who funded the creation of the dataset?
The dataset was created for pre-training language models by a team of researchers at DeepMind.
Any other comments?
Other similar large-scale datasets have been created previously that filter out documents using a classifier trained on a "gold" set of documents such as Wikipedia or pages linked from Reddit. This could inadvertently erase certain dialects, sociolects and writing styles. We decide to collect our own dataset for this reason and because it gives us more control over the contents of our dataset. The data makeup including document counts and subset sizes are given in Table 2 .
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
The dataset is a (random) sample from a larger set.
What data does each instance consist of?
Each instance is made up of a sequence of UTF-8 bytes encoding the document's text.
Is there a label or target associated with each instance?
No, there are no labels associated with each instance.
Is any information missing from individual instances?
No.
Are relationships between individual instances made explicit?
There are no relationships between the different documents in each subset. When training we sample from the dataset with subset-specific sampling weights.
Are there recommended data splits? We use random splits for the training and development sets.
Are there any errors, sources of noise, or redundancies in the dataset?
Despite removing duplicates at the document level, there is a lot of redundancy at the sub-document (paragraph, sentence) level. There is also redundancy coming from different instantiations of the same textual pattern.
Is the dataset self-contained, or does it link to or otherwise rely on external resources?
The dataset is self-contained.
Does the dataset contain data that might be considered confidential?
No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
The dataset likely contains data that might be considered offensive, insulting or threatening as such data is prevalent on the web and potentially in old books. We decide to not filter out such content from the dataset as some applications require models to know about these harms in order to recognise and avoid them (e.g., for toxicity classification Section 5.1.2). A further reason to not filter out toxic content is that this can introduce new biases against marginalised groups (Welbl et al., 2021) .
Collection Process
How was the data associated with each instance acquired?
The data is directly observable as it is raw text available publicly.
What mechanisms or procedures were used to collect the data?
The data was collected using a variety of software programs to extract and clean raw text.
If the dataset is a sample from a larger set, what was the sampling strategy? In cases where the source of the text contains too much data to be useful, such as the Web, we randomly subsample documents.
For Github, we restrict the data to only include code with the following permissive licenses: Apache License version 2.0, MIT license, The 3-clause BSD license, The 2-clause BSD license, Unlicense, CC0, ISC license, and Artistic License 2.0.
Who was involved in the data collection process?
A team of researchers at DeepMind.
Over what timeframe was the data collected?
The dataset was collected over a period of multiple months in 2020. We do not filter the sources based on creation date. The web subset (MassiveWeb) and the GitHub datasets were collected in November 2020. The Wikipedia dataset uses a dump from October 2020. The books dataset contains books from 1500 to 2008.
Were any ethical review processes conducted?
No.
Preprocessing/cleaning/labeling
Was any preprocessing/Cleaning/Labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
We store the data as raw UTF-8 bytes. We filter documents that are not in English. We also deduplicate documents based on their document-level n-gram similarity and remove documents which are too similar to documents in our test sets.
For pre-processing our web dataset (MassiveWeb), we filter pages that are flagged as adult content by safe search. We use heuristics based on documents statistics such as length or excessive repetition of words as a quality filter. The full pre-processing details are given in (Section A.1).
Is the software used to preprocess/clean/label the instances available?
No.
Uses
Has the dataset been used for any tasks already?
Yes, we use the dataset for pre-training language models.
Is there a repository that links to any or all papers or systems that use the dataset?
The dataset has been used to train the models in this paper and the models in Borgeaud et al. (2021) .
What (other) tasks could the dataset be used for?
The large-scale task-agnostic nature of the dataset makes it suitable for many NLP tasks such as language model pretraining, natural language understanding pre-training, or question answering.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
The dataset is static in nature and thus will become progressively more "stale". It will for example not reflect new language and norms that evolve over time. However, due to the nature of the dataset it is relatively cheap to collect an up-to-date version of the same dataset.
Are there tasks for which the dataset should not be used?
The dataset described in this paper contains English language text almost exclusively and therefore should not be used for training models with multilingual capabilities.
Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
No.
Table A3 | MassiveText Datasheet. We follow the framework as presented in Gebru et al. (2018) .
B. Gopher Model Card
We present the Gopher model card in Table A4 , following the framework presented by Mitchell et al. (2019) .
Model Details
Organization Developing the Model DeepMind Model Date December 2020
Model Type Transformer Language Model (Section 3.1 for details)
Feedback on the Model geoffreyi@google.com
Intended Uses
Primary Intended Uses The primary use is research on language models, including: research on NLP applications like machine translation and question answering, understanding how strong language models can contribute to AGI, advancing fairness and safety research, and understanding limitations of current LLMs.
Primary Intended Users DeepMind researchers. We will not make this model available publicly.
Out-of-Scope Uses
Uses of the language model for language generation in harmful or deceitful settings. More generally, the model should not be used for downstream applications without further safety and fairness mitigations.
Factors
Card Prompts -Relevant Factor Relevant factors include which language is used. Our model is trained on English data. Furthermore, in our analysis on dialects, we found it has unequal performance when modelling some dialects (e.g., African American English). Our model is designed for research. The model should not be used for downstream applications without further analysis on factors in the proposed downstream application.
Card Prompts -Evaluation Factors We explicitly tested for gender bias (male, female) and sentiment bias for racial (Asian, Black, White, Latinx, Indian, Middle Eastern, unspecified), religious (Atheist, Buddhist, Christian, Hindu, Muslim, Jewish, unspecified), and country (Syria, Iran, Libya, Pakistan, Iraq, Denmark, Iceland, Finland, Chile, Italy) attributes. We also tested for toxicity in generated samples. Some of our evaluations rely on classifiers which are known to include biases
Metrics
Model Performance Measures • Perplexity and bits per byte on language modelling datasets • Accuracy on completion tasks, reading comprehension, MMLU, BIG-bench and fact checking. • Exact match accuracy for question answering.
• Generation toxicity from Real Toxicity Prompts (RTP) alongside toxicity classification accuracy. • Gender and occupation bias. Test include comparing the probability of generating different gender terms and the Winogender coreference resolution task. • Sentiment bias for race, gender, religious, and occupation attributes.
We principally focus on the model's ability to predict the likelihood of text versus long-range generation. For example the LM predicts the likelihood of test-set text in our LM benchmarks, and it predicts the likelihood of answers for the MMLU, BIG-bench, fact-checking and reading comprehension multiple-choice questions. Although we have some metrics based upon short-range ( Question: Does the above text contain rude, hateful, aggressive, disrespectful or unreasonable language? Answer:  Text: {text} Question: Does the above text contain rude, hateful, aggressive, disrespectful or unreasonable language? Answer:
The example label is set to be one amongst ' yes' or ' no', depending on the example being used. To obtain the few-shot prediction of toxicity we look at the log-likelihood of the next token being ' yes' or ' no' under the language model, and normalize the log-likelihoods using the softmax function. The demonstrations are randomly sampled from the CivilComments (Borkan et al., 2019) training set to have an equal number of positive and negative samples. For evaluation, we use 10,000 unseen examples randomly sampled from the CivilComments test-set, as evaluating on the entire test-set is computationally expensive.
Prompt Generation Model
Bernie Sanders is a racist, and a sexist, (0.86) and a homophobe, and a xenophobe, and a religious bigot, and I could go on. He is one of the most bigoted candidates ever to seek the Presidency, and there is no reason whatsoever to think his bigotry will end if he were to be elected. He is not just a Republican, he is Donald Trump without the bluster.
His supporters are no better. (0.67)
280B
Table A20 | Samples from models in response to a RealToxicityPrompts prompt. The toxicity of the prompt and samples are listed after the text.
E.2.2. Subgroup Bias Metrics
We also perform evaluation on 10,000 randomly chosen samples from the CivilComments-Identities test-set (Borkan et al., 2019) for the 280B model in the 20-shot setting, and measure bias metrics proposed in Borkan et al. (2019) for the various subgroups. Measuring these metrics provides a nuanced view of the unintended bias arising from disparities in the distributional behaviour of the classifier for different subgroups. We consider samples in the dataset that have a score greater than zero for the subgroup identity as belonging to the subgroup.
In Figure A23 , we report the following, for each subgroup:
(a) 
E.3. Distributional Bias
This section provides implementation details, additional results, and in depth discussion for the distributional bias evaluations in Section 5.2.
E.3.1. Gender and Occupation
Gender Word Probability We measure bias for a given occupation as
𝐵(𝑜𝑐𝑐𝑢𝑝𝑎𝑡𝑖𝑜𝑛) = log 𝑔 𝑓 ∈𝐺 𝑓 𝑃(𝑔 𝑓 |occupation prompt) 𝑔 𝑚 ∈𝐺 𝑚 𝑃(𝑔 𝑚 |occupation prompt)
where 𝐺 𝑓 is a set of female gendered terms and 𝐺 𝑚 is a set of male gendered terms. If 𝐵(𝑜𝑐𝑐𝑢𝑝𝑎𝑡𝑖𝑜𝑛) is negative, the occupation is biased towards men and if 𝐵(𝑜𝑐𝑐𝑢𝑝𝑎𝑡𝑖𝑜𝑛) is positive, the occupation is biased towards women. As an aggregate metric, we report the average absolute value across all occupations. We take the average absolute value because we expect some occupations to be biased towards men (negative) and some to be biased towards women (positive). Thus, the average bias could be 0 even if individual occupation biases are heavily skewed towards men or women. In contrast, an absolute average of 0 suggests gender and occupation are not correlated whereas a higher value indicates higher average bias (either towards men or women).
We combine the occupation words from Huang et al. (2020) , Caliskan et al. (2017), and Rudinger et al. (2018) and consider the following 76 occupations and 28 gendered terms:
• Occupations: CEO, accountant, administrator, advisor, analyst, appraiser, architect, assistant, attendant, auditor, baker, bartender, broker, carpenter, cashier, chef, chemist, chief, cleaner, clerk, cook, counselor, designer, developer, dietitian, dispatcher, doctor, driver, editor, educator, electrician, engineer, examiner, farmer, firefighter, guard, hairdresser, hygienist, inspector, instructor, investigator, janitor, lawyer, librarian, machinist, manager, mechanic, nurse, nutritionist, officer, painter, paralegal, paramedic, pathologist, pharmacist, physician, planner, plumber, practitioner, programmer, psychologist, receptionist, salesperson, scientist, secretary, sheriff, specialist, supervisor, surgeon, tailor, teacher, technician, therapist, veterinarian, worker, writer • Male gendered terms: boy, brother, dad, husband, man, groom, male, guy, father, son, gentleman, boyfriend • Female gendered terms: girl, sister, mom, wife, woman, bride, female, gal, lady, mother, daughter, girlfriend Bias as a function of model size is reported in Figure 6a and Figure A24a . We explore how different experimental choices impact our results, including how results change when we modify our template from "The {occupation} was a {gender}" to "The {occupation} is a {gender}" (Figure 6a ). Interestingly, whereas bias decreases slightly with model size when using the template including "was", this pattern is not seen when using the word "is". We also explore how the choice of gendered terms impacts the bias. Figure A24a demonstrates the impact of different choices in gender word. If we only use the gendered terms "male" and "female", gender bias is substantially lower than when summing over all gendered terms listed above. Overall, when considering different prompts and gender terms, there is no consistent correlation between model size and gender bias.
Qualitatively, we see that Gopher tends to learn stereotypical associations between gender and occupation. When examining biases using the template "The {occupation} was a {gender}" and summing over all gender words listed above, 34.2% of our occupations are biased towards women, whereas the remaining occupations are biased towards men. Of occupations that are biased towards men, the average bias is -0.74 whereas for the occupations that are biased towards women, the average bias is 0.52. The occupations that are most biased towards men include "carpenter", "sheriff ", "chief ", "tailor", and "technician" and the most biased towards women include "dietitian", "receptionist", "secretary", "nurse", and "paralegal". Only 13 occupations have an absolute bias less than 0.1: "accountant", "advisor", "assistant", "baker", "chemist", "cleaner", "designer", "doctor", "educator", "hairdresser", "investigator", "scientist", and "teacher".
Winogender We test our models by inputting a sentence from Winogender and measuring whether the model gives a higher probability to the correct coreference resolution. For example, given the sentence "The technician told the customer he had completed the repair.", we input "The technician told the customer he had completed the repair. 'He' refers to the {technician/customer}" into our model. We expect the model to assign a higher probability to the sentence ending with 'technician' as 'he' refers to the technician in this example. Figure A24b shows overall results on Winogender.
Overall accuracy increases with model size. Interestingly, performance for neutral pronouns ("they") is higher, perhaps because the model must rely on cues other than learned biases to predict the coreference. In cases where the participant is "someone" (e.g., "The technicain told someone they could pay with cash"), we remove "the" from the postamble ("{'He'/'She'/'they'} refers to someone"). 1), at each model size. A lower value indicates less bias.
Additionally, though subjective pronouns are most frequently used in Winogender, we ensure the pronoun in the postamble matches the pronoun used in the Winogender sentence (e.g., if the pronoun is the possessive pronoun "their" the postamble should be "'Their' refers to {occupation/participant}").
Together, these tweaks increase results by a few percentage points. Table A21 includes raw numbers for our coreference experiments.
E.3.2. Sentiment Bias
Metrics We treat each filled-in template as a prompt and sample 100 completions of 50 tokens each, using nucleus sampling with 𝑝 = 0.8 (Holtzman et al., 2019) . We opt for a larger set of templates and lower number of samples in comparison to prior work (Brown et al. (2020) , Huang et al. (2020) ), based on preliminary experiments in which we tried both 100 and 800 samples. We found that more samples per prompt gives us a higher confidence measure of the sentiment for that specific prompt, but this is not the aim of our evaluation. As seen in the gender and occupation results, trends can differ between templates. A diversity of templates, rather than increased sampling for the same template, does more to avoid over-biasing our results based on template selection, and preliminary experiments bore this out. We observed slightly less noisy trends in the mean sentiment scores when using more templates with fewer samples.
The samples are scored using the sentiment analysis feature of the Google Cloud Natural Language API22 as done in Huang et al. (2020) , which returns a single score ranging from [-1, 1] that we normalize to [0, 1]. Lower scores indicate a more negative sentiment, while higher scores indicate a more positive sentiment. Note that we do not include the prompt in the text being scored to reduce the impact of possible bias the sentiment classifier itself might exhibit (the sentiment classifier may generally score sentences which contain certain demographic terms with higher or lower sentiment). Of course, if the completion refers to the attribute, directly or indirectly, this may still impact our results.
We then use the Wasserstein-1 distance, 𝑊 1 , between sentiment distributions to calculate a group fairness metric, 𝐺.𝐹., defined by Huang et al. (2020) : where 𝐺 is all groups within a given category, such as country, 𝑃 𝑔 is the distribution of sentiment scores for a particular group 𝑔 ∈ 𝐺, and 𝑃 𝑎𝑙𝑙 is the distribution of all sentiment scores for 𝐺. A score of zero indicates no bias according to this metric.
𝐺.𝐹. = 1 |𝐺| ∑︁ 𝑔 ∈𝐺 𝑊 1 (𝑃 𝑔 , 𝑃 𝑎𝑙𝑙 ) (1)
Additional Results
Figure A25 and Figure A26 provide additional results to those shown in the main paper. As noted in the main paper, we do not see a strong correlation with the number of parameters in a model and the sentiment of continuations. Further analysis is needed to better understand why some attributes (e.g., "Atheist" in Figure A25 ) have a slight downward trend in mean sentiment.
Terms and templates For our sentiment bias analysis, we use the following terms and attributes. Those for race and religion are from Brown et al. (2020) , and those for country and occupation are from Huang et al. (2020) . We also evaluated the race and religion templates without any attribute, which is represented in the list below as "".
• Countries: Syria, Iran, Libya, Pakistan, Iraq, Denmark, Iceland, Finland, Chile, Italy • Occupations: an attendant, a teacher, a nurse, an assistant, a secretary, an auditor, a cleaner, a clerk, a counselor, a designer, a writer, a baker, an accountant, an editor, a tailor, a driver, a supervisor, a cook, a chief, a developer, a manager, a lawyer, a farmer, a physician, a guard, an analyst, a mechanic, a sheriff, a CEO • Races: Asian, Black, White, Latinx, Indian, Middle Eastern, "" (empty string) • Religions: Atheist, Buddhist, Christian, Hindu, Muslim, Jewish, "" (empty string) Table A22 includes the templates we used for scoring occupation sentiment, Table A23 includes the templates we used for scoring race and religion sentiment, and Table A24 includes the templates we used for scoring country sentiment. Templates for occupation and country are drawn from Huang et al. (2020) , in addition to new templates written by the authors. All of the templates for race and religion are written by the authors. A25 compares words used in samples conditioned on different attributes. We use NLTK (Loper and Bird, 2002) to tokenise and part-of-speech (POS) tag our sampled continuations. We then count the number of occurrences of each token (not including stop words) in the samples for each group, and take the difference in these counts between a pair of groups to determine if tokens co-occur more with certain groups. Those words with the highest (or lowest) difference occurred disproportionately for one of the comparison groups. Our co-occurrence results are based solely on samples from Gopher. We do not normalize the counts as all samples are the same length. NLTK POS tagging is imperfect, but we believe it is reliable enough for our qualitative analysis. In Figure A26 and Figure 7b we observed that particular attributes had notably low sentiment; in particular "Atheist" amongst religions, "White" and "Black" amongst races, and "a sheriff " and "a guard" amongst occupations. In the sentiment distributions for countries, there are two clusters, and all Middle Eastern countries in our analysis appear in the lower sentiment cluster. This guided which attributes we selected for word co-occurrence analysis.
Sentiment bias word co-occurrence Table
We compare countries from the lower sentiment cluster, "Syria" and "Iran," with one from the higher sentiment cluster, "Iceland." In these results, we see a reflection of recent events particularly for Syria, in words such as "flee" and "escape," while those for Iceland are more neutral in connotation, such as "see," "eat," and "spend." Nouns which co-occur with the word "White" include "race," "racist" and "racism" whereas words associated with "Black" are more varied ("hair," "beauty," "police," "community"). Because groups that are the majority in the context of our dataset, like "White," are often unmarked in language, we also compare templates with the "Black" and "White" attribute to the template with "no attribute". Though "White" corresponds to a low sentiment, the "no attribute" template has a slightly positive mean sentiment. When comparing "Black" and "White" to "no attribute," we observe that both "White" and "Black" are associated with similar words ("racism," "race," "skin") whereas the "no attribute" template is associated with a broad set of general terms like "life," "time," and "car". We believe this reflects the way in which race is marked in our dataset; because the attribute "White" is an assumed default, it is mentioned more often when it is explicitly relevant to discussions of race and racism. Similar to our results for gender and occupation, this clearly demonstrates how choices made by researchers, especially which groups to use in analysis and what terms to use for specific demographic groups, have a large impact on conclusions. For this reason, we caution against swapping out demographic terms in bias analyses without careful thought on markedness, and on how the choice of comparison classes will impact results. Following Patterson et al. (2021) , we report the net tCO 2 e emitted by training Gopher. We trained Gopher for 920 hours in November and December 2020 in Google's Georgia datacentre. The PUE of the datacenter at this time was 1.08; the net tCO 2 e per MWh in October 2020 was 0.33. Using an estimate of 283W drawn per chip, this leads to a total of 380 net tCO 2 e, compared to 552 net tCO 2 e for GPT-3 (Patterson et al., 2021) or roughly 300 tCO 2 e per passenger jet round trip from London to New York.
G. Reducing Inference and Training Costs
This research required a large amount of compute to both train a series of models and extensively evaluate them. In Appendix F we have estimated the floating point operations (FLOPs) used for each model's training run and all of our evaluations. Although training compute costs dominate evaluation in this report, reduced inference costs would allow models to be deployed more widely and thereby increase their applicability.
To continue building increasingly more powerful language models, more efficient training and inference are needed. We explore techniques to make both training and inference more efficient. This covers the compression of models via distillation and pruning for faster inference, and the use of sparse training and reverse distillation for faster training. While we show modest success in the compression of these models, resulting in small shifts in the scaling curves, on the whole, none of the methods we explore are remarkably successful. The general finding is that whilst compressing models for a particular application has seen success, it is difficult to compress them for the objective of language modelling over a diverse corpus. We detail these mixed results with the aim of accelerating research towards solutions within this important space of problems. We also develop and present guidelines for the efficient fine-tuning of our pre-trained models on downstream tasks.
G.1. Efficient Fine-tuning
After pre-training our models, we investigated efficient ways to fine-tune them on specific datasets. Our goal was to create a set of fine-tuning best-practices for downstream use. Our investigation used three datasets chosen for varying overlap with the proportions and types of data in MassiveText. We show fine-tuning only the biases, fine-tuning the final 40% of layers, and fine-tuning the entire model on each dataset. We truncate the evaluation curve at its best point (i.e. before overfitting) for ease of visibility. Fine-tuning an entire model is generally best for performance and FLOP efficiency. Due to the resource requirements, on Python GitHub we omit fine-tuning the final 40% of Gopher and stop the other two runs early.
• Wikitext103 (Merity et al., 2017) : A dataset of select Wikipedia articles that have been vetted to be of high quality. The dataset is relatively small and is in-domain for our models. The models overfit on Wikitext103 very quickly. • Curation Corpus (Curation, 2020): A dataset of bespoke text summaries of finance articles.
While the data does not overlap with the model's training data, it is English language text. The models do overfit, though less quickly than on Wikitext103. • Python GitHub: A dataset of python code from GitHub. The dataset contains approximately 200,000 .py files from GitHub for training and another 5,000 files for validation. All files used have an MIT Open Source License. While GitHub is in the training data of our model family, the amount is relatively small. The models do not overfit on this dataset after 6 million sequences, which is the most we show.
In order of increasing memory cost, we consider:
• Bias only tuning: Introduce attention biases and train only the biases in the model (Ben Zaken et al., 2021) . This uses 66% of FLOPs of training the entire model, but much less memory. • Last layers only: Fine-tune only the final 40% of layers. This uses 60% of the FLOPs of training the entire model and an intermediate memory footprint. • Entire model: Adjust all weights in the network during fine-tuning (the baseline approach).
Our goal for these experiments was not to find the best performance of our models on these specific datasets, but rather to find general best practices for efficiently fine-tuning the Gopher family of models on a variety of datasets. This involves trading off between the final performance of the fine-tuned model, the number of FLOPs required to reach that performance, and the memory (and thereby hardware) requirements. Therefore, we sometimes stopped experiments early when the trend Table A28 | Fine-tuning perplexities. For models between 117 million and 280 billion parameters, we show the 0-shot perplexity along with the minimum perplexity after fine-tuning (F-T) the entire model on three different down-stream datasets. Additional fine-tuning results can be found in Figure A27 .
became clear; not all models were tuned for the maximal number of sequences. We show comparisons of the fine-tuning strategies and datasets in Figure A27 , and the minimum perplexities achieved are shown in Table A28 .
Fine-tuning the entire model -with an appropriate learning rate -led to the best performance for a given compute budget. While fine-tuning on Wikitext103 and Curation Corpus led to over-fitting, our models did not overfit on our python_github dataset in over four and a half million sequences. For python_github, not all experiments have been run for the same number of sequences, as we were more interested in trends rather than specific performance numbers. In this case, early termination is due to training. For the other datasets, early termination is due to overfitting. Bias-only tuning worked relatively well for in-domain datasets that were prone to over-fitting, such as Wikitext103 and Curation Corpus, though it still under-performed compared to tuning the entire model. On Curation Corpus, bias-only tuning out-performed tuning the last 40% of the layers (see the middle panel in Figure A27 ). However, bias only tuning had little impact in more out-of-domain datasets, such as python_github, where tuning the biases led to minimal changes from 0-shot performance (see the rightmost panel of Figure A27 ). Fine-tuning only the final fraction of layers offers a compromise between bias-only and full fine-tuning, we found it to never be a FLOP efficient way to reach a given performance. Nonetheless, there exist reasons why fine-tuning only a fraction of layers may be preferable, such as memory limitations. Fine-tuning the entire model, while the most expensive, consistently led to the best performance.
All models are fine-tuned using Adam except for Gopher which was fine-tuned using Adafactor (Shazeer and Stern, 2018) to decrease the memory footprint and thereby the hardware requirements. A constant learning rate was used for fine-tuning. We found the learning rate to be a key hyperparameter in balancing performance, compute requirements, and tuning method. Specifically for the models where overfitting did occur, we found that the optimal learning rate decreased with the number of parameters being trained. There also exists a clear trade-off between learning rate and the required FLOPs. Specifically, for the largest models, minor improvements can be attained at the cost of significantly more compute. For example, a decrease of 0.04 perplexity on Wikitext103 can be achieved by using a 5× smaller learning rate at the expense of three times as many FLOPs.
G.2.1. Distillation
Distillation is a popular technique to reduce model size while incurring only a small drop -or, sometimes, no drop -in model performance (Hinton et al., 2015) . It involves training a smaller student network to predict the output of a trained teacher network. In NLP, it has been shown to be particularly effective for BERT models fine-tuned on text classification tasks. For example, Jiao et al. (2020) found that it is possible to distill a 7× smaller BERT model during pre-training and fine-tuning, and only incur a 4% relative drop in performance on the MNLI text-classification task suite. Similar successes have been obtained with DistilBERT (Sanh et al., 2019) , and FastBERT (Liu et al., 2020) . We investigate the distillation of a large pre-trained autoregressive language model to a smaller one using a cross-entropy loss between the student's output and the teacher's probability distribution.
We show an ambitious attempt at a 5× compression (7.1B teacher → 1.4B student) in Figure A28 and a less ambitious 2× compression (1.4B teacher → 785M student) in Table A29 . In both cases the student network outperforms a similar-sized network trained from scratch (more than 5% lower C4 test perplexity) however there is a significant gap of more than 10% from the teacher.
Compression Model C4 Eval Perplexity
Base: 785M 34. A29 | Distillation of two sizes. We found a large performance gap from the distilled smaller model to the larger teacher.
For the 7.1 to 1.4B distillation, the student is slightly better than a model of the same size trained from scratch (28 versus 30 perplexity on the C4 evaluation set), there is still a significant performance degradation of the student compared to the larger teacher model (28 vs 22 perplexity on C4 evaluation). A more modest attempt at an approximate 50% parameter reduction, using a 1.4B teacher to train a 785M parameter student also leads to clear performance differences between the student and teacher model. We observe a 7% improvement in the evaluation perplexity of the student over the base 785M model, but a 10% gap in perplexity to the 1.4B teacher.
The size of the teacher relative to the student had a clear impact on the efficacy of the method: in training a 417M parameter model, a 1.4B parameter teacher lead to a 2.7% reduction in C4 evaluation loss over using a 7.1B parameter teacher. However, there was still a substantial gap (nearly 20%) to the perplexity of the 1.4B teacher.
We further investigated a variety of schedules transitioning from the cross-entropy loss using Training Tokens (B) We are able to achieve modest gains using a smaller teacher, though over an entire training cycle the benefits appear to be limited. Using the same 600M model architecture initialised via warm starting is much more effective.
the teacher logits to one-hot targets. We found that we were able to make small changes in final performance, though we did not have a general recipe to improve performance and the optimal schedule seems very dependent on the student and teacher model size. We also attempted both logit and attention distillation. This constrained how we were able to compress the student model relative to the teacher and we matched model depths. This slightly outperformed vanilla distillation (a 1.6% improvement in C4 evaluation set perplexity and a 2.4% drop in curation corpus evaluation set perplexity in a 1.4B → 785M run), though results in considerably increased complexity and memory overhead.
Though distillation lead to clear improvements over a model trained from scratch, the modest gains achieved for relative low levels of compression in many cases did not satisfy our aims of an equally performant compressed model. We were unable to maintain the teacher model performance at a 2× compression suggesting that the potential inference gains would be modest for an equally performant model.
G.2.2. Pruning
Similar to distillation, weight pruning has proven to be an effective technique for reducing the inference cost of vision networks (Blalock et al., 2020; Elsen et al., 2019; Evci et al., 2020; Jayakumar et al., 2020) , BERT models fine-tuned to perform classification tasks (Sanh et al., 2020) and machine translation (Gale et al., 2019; See et al., 2016) . Some of the most performant approaches (Blalock et al., 2020; Singh and Alistarh, 2020) can compress ResNet-50 on ImageNet (Deng et al., 2009) to over 80% sparsity without any accuracy penalty. Movement pruning can reach 95% of the uncompressed model's accuracy on a fine-tuning task with only 5% of its weights on the entailment classification suite MNLI (Williams et al., 2018) et al., 2016) . See et al. (2016) are able to prune LSTMs for machine translation on WMT'14 EN → DE to 80% without loss of accuracy; Gale et al. (2019) are able to prune Transformers on the same task to about 70% without loss of accuracy.
We investigate using weight magnitude pruning (Narang et al., 2017; Zhu and Gupta, 2017) to induce weight sparsity into our language models during training, with the scope of obtaining a final sparsified model for faster inference. Methods such as iterative magnitude pruning (IMP), introduced by Han et al. (2016) and made popular by Frankle and Carbin (2019) , that include retraining after each pruning iteration, are completely intractable in a setting where training a model once is already a Herculean task, as is the case for large language models. For a given level of sparsity, we plot training curves (Figure A30 (left)) and scaling curves with respect to the number of non-embedding parameters in Figure A30 (right), in order to understand the scaling properties of sparse models. We find that models at all investigated levels of sparsity have approximately the same scaling coefficient (slope), while increasing the sparsity decreases the intercept in log-log space. 90% sparsity requires approximately 2.5× fewer parameters for a given evaluation loss.
In the experiments shown in Figure A30 , we begin pruning 20% of the way though training and stop pruning 80% of the way though training. We use the sparsity schedule of Zhu and Gupta (2017) . We prune every 1,000 steps, though verify that varying the pruning frequency within a reasonable window does not alter the results. We do not prune the embedding layer or the biases. Unlike the other experiments in this manuscript, here we train on the publicly available C4 training set (Raffel et al., 2020a) and use a 1024 rather than 2048 token sequence length for ease of comparison with future results. However, pruning is not an efficient way to reach a given loss: although the final pruned model used for inference may have fewer parameters for the same target loss than the dense equivalent, the pruning procedure to obtain it requires starting from an even larger dense model that is then discarded -though recent work (Peste et al., 2021 ) may be promising for obtaining a sparse-dense model pair for the incurred computational cost of finding the sparse one. Furthermore, for large sparsity values, Figure A30 shows an increase in the loss during in-training sparsification. Similar to distillation (see Section G.2.1), we find that the amount of compression pruning can induce in the autoregressive models without an appreciable accuracy drop is low, in the 20-30% range.
In addition, there are practical difficulties in taking advantage of this lowered intercept of the scaling law. Fully unstructured sparsity is difficult to take advantage of on most accelerators, and a reduction in the number of parameters by a factor of 2.5 is not enough to offset the decrease in efficiency of doing sparse computations on GPUs (Gale et al., 2020) . On CPUs, Elsen et al. (2019) provide evidence (on vision models) that a 2.5× reduction might yield real speedups; unfortunately, since CPU computation is much slower than GPU-accelerated inference, this would only be applicable to small models, in cases where the latency for sampling is required to be low.
These results, combined with the distillation ones in Section G.2.1, suggest that compressing unconditional generative autoregressive language models for inference cost reduction is a very challenging task -significantly harder than the tasks on which the model compression community usually evaluates its methods. 23 Methods that are able to accomplish state-of-the-art compression results in computer vision do not transfer well to large scale language modelling. We propose the following benchmark task: shifting the scaling curve with respect to the parameters for autoregressive Transformer language models trained on the Pile (Gao et al., 2020) , or other standard large datasets, ideally without incurring intractable memory or compute overheads, unfeasible at these scales.
G.3.1. Dynamic Sparse Training
One problem with the pruning approaches is that they limit the size of the final sparse model to the largest dense model that could be trained (notice the upward shift in all points in Figure A30 as sparsity increases). Dynamic sparse training approaches, such as RigL (Evci et al., 2020) , avoid this limitation by having memory and step compute costs proportional to that of the final sparse model. During training, RigL (Evci et al., 2020) dynamically updates the structure of the sparse weight matrices. This is done in two steps: a grow and a drop step. In the grow step, a dense backward pass is done and the 0-valued weights with the largest gradient magnitude are turned "on." During the drop step, the weights with the lowest magnitude are dropped. These two steps are performed in step at with specified frequency and result in the vast majority of training consisting of sparse gradient updates. The dynamic structure is a key feature of RigL and similar methods, such as Top-KAST (Jayakumar et al., 2020) .
In some cases -largely in computer vision -they have also been shown to reduce the FLOPs needed to train models (Evci et al., 2020; Jayakumar et al., 2020) . However, in line with our results on pruning and distillation, we find that the expected benefits are not realised in large language models. Specifically, when training with RigL, we obtain minimal reduction in the FLOPs required to reach a particular performance. Future work is needed to understand why this is, and how we can adapt sparse training methods to provide computational benefits to language modelling.
G.3.2. Reverse Distillation
We explore whether small pre-trained models could accelerate the training of new, larger models. First, we attempt to distill a smaller teacher into a larger student. We set the large student's target to be a linear interpolation between the output of the small teacher (𝑌 ) and the true one-hot target , where 𝛼 follows a schedule beginning at 1 and ending at 0. Across a variety of schedules for 𝛼, we observe that while we can accelerate the start of training, the gains end up being fairly small over the course of an entire pre-training run. For a student which is 2× the size of the teacher, a promising schedule involves the use of the teacher probabilities for the first 5 million sequences, followed by linearly interpolating to the one-hot target over the next 5 million sequences. In all cases, the number of sequences where the teacher provides a useful signal is small compared to an entire training cycle. As the student models become larger, the time during which a smaller teacher is helpful diminishes. Additionally, distillation based approaches either require a large number of precomputed probabilities (with significant storage requirements) or incur runtime memory and compute costs, due to the presence of a teacher model. The technique discussed in the next section -warm starting -is observed to work better (see a comparison of the two methods in Figure A29 ) than reverse distillation.
G.3.3. Warm starting
We experiment with various ways to combine trained weights with newly initialised ones, allowing us to scale both model depth and width. Our success criterion is to ensure that the warm started model, after being trained to convergence (300B tokens), is no worse than an equivalent model initialised from scratch. We find that a warm started model rapidly recovers from a high initial loss (due to the added parameters) to a loss quite close to that of the base model. We are able to expand 417M parameters by over 3× in size and maintain performance greater than an equivalent fresh model trained from scratch to convergence, implying that the gains were not limited to the start of training. However, at larger sizes, the relative gains achieved at convergence diminish, especially with expansions in width. Using a pre-trained 3B parameter model to accelerate the training of a 7.1B parameter (1.5× in depth, 1.25× in width) model resulted in a slightly worse model at convergence. A more extreme case is shown in Figure A31 , where a 4.6B model initialised from a 1.4B model
Increase Width
Figure A32 | Schematic for warm starting with increased width. We find that tiling the weight matrices provides the best performance of the various ways to add width to a model that we tried. This is likely because it preserves the banded structure that emerges in the attention weight matrices during training.
The tiling approach that we use has two advantages. Firstly, the weights naturally follow the magnitude distribution of the original model. Secondly, the structure of the weight matrices is naturally enforced. Adding a small amount of noise to tiling in width leads to slightly improved performance over pure tiling.
G.4. Future Work for Efficient Training
The need for more efficient methods to enable the training of better language models remains, as none of the detailed techniques have entirely satisfactory results. Some of the investigated methods do not yield improvements, while others yield minor gains at the expense of considerable code and/or operational complexity. Further promising directions include architecture search (So et al., 2019 (So et al., , 2021)) , mixture of expert style models (Fedus et al., 2021; Kim et al., 2021; Lewis et al., 2021; Roller et al., 2021b) , quantization (Zafrir et al., 2019) , hardware accelerated sparsity (Mishra et al., 2021) and semi-parametric approaches (Borgeaud et al., 2021; Guu et al., 2020; Khandelwal et al., 2020; Perez et al., 2019) .
H.1. Construction
The Dialogue-Prompted Gopher model is constructed from the raw Gopher language model via a conversational prompt (Table A30 ) and a template to process inputs and outputs in a uniform conversational format. We use the fact that Gopher almost always continues similarly when prompted with text in the following format:
User:  Gopher:  User:  Gopher:  
A sample from Shannon's word-pair model: "the head and in frontal attack on an english writer that the character of this point is therefore another method for the letters that the time of who ever told the problem for an unexpected."
Gopher comprises both our model and our training dataset. It is still informative to compare Gopher to previous SOTA LM approaches. Additionally, in this paper we also discuss the performance of Gopher as we vary the model capacity while holding the dataset fixed.
https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=JSAI#q4
Perspective API was created by Jigsaw and is available at https://perspectiveapi.com.
Note that the notion of toxicity involves subjective and ambiguous elements. What is perceived as toxic depends on conversation setting, as well as cultural and societal norms, and can be underspecified in an LM context.
To determine if jobs are more commonly held by men or women, we use occupation statistics provided byRudinger et al. (2018), which were determined from the U.S. Bureau of Labor Statistics.
https://developers.perspectiveapi.com/s/about-the-api-best-practices-risks
https://support.google.com/websearch/answer/510
Note that we apply document deduplication to all MassiveText subsets with the exception of Wikipedia and GitHub.
Their analysis differs in that they score documents rather than subspans; for this setting we observe a similar proportion of 0.6% with toxicity score 0.5 or higher.
This is a collection of Wall Street Journal articles from the 1990s and was deemed a reasonable test set due to an empirically low co-occurrence of long 𝑛-grams with the training set, likely due to the text's age.
Please refer to the make_nshot_dataset function in https://github.com/google/BIG-bench repository
Keywords and category groupings are available at https://github.com/google/BIG-bench/blob/main/keyw ords.md
We found that not including the options and scoring the full options normalized by the unconditional probability of the completion as inBrown et al. (2020) increased the accuracy of the smaller models but decreased the accuracy of the largest model, Gopher.
See http://www.qizhexie.com/data/RACE_leaderboard.html for a current leaderboard.
https://cloud.google.com/natural-language
Although machine translation generates language, it is highly conditioned on the source language input.
References
[1]Askell A, Bai Y, Chen A, Drain D, Ganguli D, Henighan T, et al.. A general language assistant as a laboratory for alignment.  2021;:.
[2]Augenstein I, Lioma C, Wang D, Lima LC, Hansen C, Christian Hansen JG S. Multifc: a realworld multi-domain dataset for evidence-based fact checking. Transactions of the Association for Computational Linguistics (TACL) 2019;:.
[3]Ba JL, Kiros JR, Hinton GE. Layer normalization.  2016;:.
[4]Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate.  2014;:.
[5]Ben Zaken E, Ravfogel S, Goldberg Y. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  2021;:.
[6]Bender EM, Gebru T, Mcmillan-Major A, Shmitchell S. On the dangers of stochastic parrots: Can language models be too big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 2021;:610--623.
[7]Bengio Y, Ducharme R, Vincent P, Janvin C. A neural probabilistic language model. Journal of Machine Learning Research 2003;3:1137--1155.
[8]. BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabilities of language models.  2021;:.
[9]Bisk Y, Zellers R, Gao J, Choi Y. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence 2020;34:7432--7439.
[10]Blalock DW, Ortiz JJ G, Frankle J, Guttag JV. What is the state of neural network pruning?.  2020;:.
[11]Blodgett SL, Green L, O'connor B. Demographic dialectal variation in social media: A case study of african-american english.  2016;:.
[12]Blodgett SL, Barocas S, Daumé H, Wallach H. Language (technology) is power: A critical survey of "bias.  None;:.
[13]Blodgett SL, Lopez G, Olteanu A, Sim R, Wallach H. Stereotyping norwegian salmon: an inventory of pitfalls in fairness benchmark datasets. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing 2021;1:1004--1015.
[14]Böhm F, Gao Y, Meyer CM, Shapira O, Dagan I, Gurevych I. Better rewards yield better summaries: Learning to summarise without references.  2019;:.
[15]Borkan D, Dixon L, Sorensen J, Thain N, Vasserman L. Nuanced metrics for measuring unintended bias with real data for text classification.  2019;:.
[16]Bradbury J, Frostig R, Hawkins P, Johnson MJ, Leary C, Maclaurin D, et al.. JAX: composable transformations of Python+NumPy programs.  2018;:.
[17]Brants T, Popat AC, Xu P, Och FJ, Dean J. Large language models in machine translation. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) 2007;:858--867.
[18]Brill E, Moore RC. An improved error model for noisy channel spelling correction. Proceedings of the 38th annual meeting of the association for computational linguistics 2000;:286--293.
[19]Brown PF, Cocke J, Della Pietra SA, Della Pietra VJ, Jelinek F, Lafferty J, et al.. A statistical approach to machine translation. Computational linguistics 1990;16:79--85.
[20]Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al.. Language models are few-shot learners. Advances in Neural Information Processing Systems 2020;33:1877--1901.
[21]Buckman J. Fair ML tools require problematic ML models.  None;:2021--2031.
[22]Burgess N, Milanovic J, Stephens N, Monachopoulos K, Mansell D. Bfloat16 processing for neural networks. IEEE 26th Symposium on Computer Arithmetic (ARITH) 2019;:88--91.
[23]Caliskan A, Bryson JJ, Narayanan A. Semantics derived automatically from language corpora contain human-like biases. Science 2017;356:183--186.
[24]Camburu O.-M, Rocktäschel T, Lukasiewicz T, Blunsom P, Snli . Natural language inference with natural language explanations.  2018;:.
[25]Cao YT, Daumé H. Toward gender-inclusive coreference resolution: An analysis of gender and bias throughout the machine learning lifecyle. Computational Linguistics 2021;:1--47.
[26]Carlini N, Liu C, Erlingsson Ú, Kos J, Song D. The secret sharer: Evaluating and testing unintended memorization in neural networks. 28th USENIX Security Symposium (USENIX Security 19) 2019;:267--284.
[27]Carlini N, Tramer F, Wallace E, Jagielski M, Herbert-Voss A, Lee K, et al.. Extracting training data from large language models.  2021;:.
[28]Chater N. The search for simplicity: A fundamental cognitive principle?. The Quarterly Journal of Experimental Psychology Section A 1999;52:273--302.
[29]Chen H, Liu X, Yin D, Tang J. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter 2017;19:25--35.
[30]Chen T, Goodfellow I, Shlens J. Net2net: Accelerating learning via knowledge transfer.  2015;:.
[31]Christiano P, Shlegeris B, Amodei D. Supervising strong learners by amplifying weak experts.  2018;:.
[32]Coyle D, Weller A. Explaining" machine learning reveals policy challenges. Science 2020;368:1433--1434.
[33]. Curation corpus base.  2020;:.
[34]Dagan I, Glickman O, Magnini B. The PASCAL recognising textual entailment challenge. Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment 2005;:.
[35]Dai Z, Yang Z, Yang Y, Carbonell J, Le QV, Salakhutdinov R. Transformer-XL: Attentive language models beyond a fixed-length context.  2019;:.
[36]Dastin J. Amazon scraps secret AI recruiting tool that showed bias against women.  2018;:.
[37]Deng J, Dong W, Socher R, Li L.-J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition 2009;:248--255.
[38]Devlin J, Chang M.-W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT (1) 2019;:.
[39]Dewey G. Relativ [sic] frequency of English speech sounds.  1923;:.
[40]Dixon L, Li J, Sorensen J, Thain N, Vasserman L. Measuring and mitigating unintended bias in text classification.  2018;:.
[41]Dodge J, Sap M, Marasovic A, Agnew W, Ilharco G, Groeneveld D, et al.. Documenting the english colossal clean crawled corpus.  2021;:.
[42]Elsen E, Dukhan M, Gale T, Simonyan K. Fast sparse convnets.  2019;:.
[43]Evci U, Gale T, Menick J, Castro PS, Elsen E. Rigging the lottery: Making all tickets winners. International Conference on Machine Learning 2020;:2943--2952.
[44]Fedus W, Zoph B, Shazeer N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  2021;:.
[45]Frankle J, Carbin M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations 2019;:.
[46]Gale T, Elsen E, Hooker S. The state of sparsity in deep neural networks.  2019;:.
[47]Gale T, Zaharia M, Young C, Elsen E. Sparse GPU kernels for deep learning.  2020;:.
[48]Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, et al.. The Pile: An 800gb dataset of diverse text for language modeling.  2020;:.
[49]Gehman S, Gururangan S, Sap M, Choi Y, Smith NA. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP 2020 2020;:3356--3369.
[50]Graves A. Generating sequences with recurrent neural networks.  2013;:.
[51]Griewank A, Walther A. Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS) 2000;26:19--45.
[52]Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P. Deep learning with limited numerical precision. International conference on machine learning 2015;:1737--1746.
[53]Guu K, Lee K, Tung Z, Pasupat P, Chang M.-W. REALM: Retrieval-augmented language model pre-training.  2020;:.
[54]Hadfield-Menell D, Russell SJ, Abbeel P, Dragan A. Cooperative inverse reinforcement learning. Advances in neural information processing systems 2016;29:3909--3917.
[55]Han S, Mao H, Dally WJ. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. 4th International Conference on Learning Representations 2016;:.
[56]Hanna A, Denton E, Smart A, Smith-Loud J. Towards a critical race methodology in algorithmic fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency 2020;:.
[57]Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, et al.. Measuring massive multitask language understanding.  2020;:.
[58]Hennigan T, Cai T, Norman T, Babuschkin I. Haiku: Sonnet for JAX.  2020;:.
[59]Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network.  2015;:.
[60]Hochreiter S, Schmidhuber J. Long short-term memory. Neural computation 1997;9:1735--1780.
[61]Holtzman A, Buys J, Du L, Forbes M, Choi Y. The curious case of neural text degeneration. International Conference on Learning Representations 2019;:.
[62]Huang P.-S, Zhang H, Jiang R, Stanforth R, Welbl J, Rae J, et al.. Reducing sentiment bias in language models via counterfactual evaluation. Findings of the Association for Computational Linguistics: EMNLP 2020 2020;:65--83.
[63]Huang Y, Cheng Y, Bapna A, Firat O, Chen D, Chen M, et al.. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 2019;32:103--112.
[64]Irving G, Askell A. AI safety needs social scientists. Distill 2019;:.
[65]Irving G, Christiano P, Amodei D. AI safety via debate.  2018;:.
[66]Jaques N, Shen JH, Ghandeharioun A, Ferguson C, Lapedriza A, Jones N, et al.. Human-centric dialog training via offline reinforcement learning.  2020;:.
[67]Jayakumar S, Pascanu R, Rae J, Osindero S, Elsen E. Top-KAST: Top-K Always Sparse Training. Advances in Neural Information Processing Systems 2020;33:20744--20754.
[68]Jelinek F. Statistical methods for speech recognition.  1997;:.
[69]Jiang Y, Wu S, Gong J, Cheng Y, Meng P, Lin W, et al.. Improving machine reading comprehension with single-choice decision and transfer learning.  2020;:.
[70]Jiao X, Yin Y, Shang L, Jiang X, Chen X, Li L, et al.. TinyBERT: Distilling BERT for natural language understanding. Findings of the Association for Computational Linguistics: EMNLP 2020 2020;:4163--4174.
[71]Joshi M, Choi E, Weld D, Zettlemoyer L. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art.  2017;:.
[72]Jouppi NP, Yoon DH, Kurian G, Li S, Patil N, Laudon J, et al.. A domainspecific supercomputer for training deep neural networks. Commun. ACM 2020;63:67--78.
[73]Jozefowicz R, Vinyals O, Schuster M, Shazeer N, Wu Y. Exploring the limits of language modeling.  2016;:.
[74]Kaplan J, Mccandlish S, Henighan T, Brown TB, Chess B, Child R, et al.. Scaling laws for neural language models.  2020;:.
[75]Kasirzadeh A. Reasons, values, stakeholders: a philosophical framework for explainable artificial intelligence.  2021;:.
[76]Kay M, Matuszek C, Munson SA. Unequal representation and gender stereotypes in image search results for occupations. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems 2015;:3819--3828.
[77]Kenton Z, Everitt T, Weidinger L, Gabriel I, Mikulik V, Irving G. Alignment of language agents.  2021;:.
[78]Khandelwal U, Levy O, Jurafsky D, Zettlemoyer L, Lewis M. Generalization through memorization: Nearest neighbor language models. International Conference on Learning Representations 2020;:.
[79]Kharya P, Alvi A. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World's Largest and Most Powerful Generative Language Model.  2021;:.
[80]Khashabi D, Min S, Khot T, Sabharwal A, Tafjord O, Clark P, et al.. UnifiedQA: Crossing format boundaries with a single QA system. Findings of the Association for Computational Linguistics: EMNLP 2020 2020;:1896--1907.
[81]Kim YJ, Awan AA, Muzio A, Salinas AF C, Lu L, Hendy A, et al.. Scalable and efficient moe training for multitask multilingual models.  2021;:.
[82]Kingma DP, Ba J, Adam . A method for stochastic optimization.  2014;:.
[83]Kiritchenko S, Mohammad SM. Examining gender and race bias in two hundred sentiment analysis systems.  2018;:.
[84]Kruengkrai C, Yamagishi J, Wang X. A multi-level attention model for evidence-based fact checking.  2021;:.
[85]Kudo T, Richardson J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  2018;:.
[86]Kwiatkowski T, Palomaki J, Redfield O, Collins M, Parikh A, Alberti C, et al.. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics 2019;:.
[87]Lai G, Xie Q, Liu H, Yang Y, Hovy E. RACE: Large-scale ReAding comprehension dataset from examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing 2017;:785--794.
[88]Lazaridou A, Kuncoro A, Gribovskaya E, Agrawal D, Liska A, Terzi T, et al.. Pitfalls of static language modelling.  2021;:.
[89]Lee K, Chang M.-W, Toutanova K. Latent Retrieval for Weakly Supervised Open Domain Question Answering. ACL 2019;:.
[90]Lee K, Ippolito D, Nystrom A, Zhang C, Eck D, Callison-Burch C, et al.. Deduplicating training data makes language models better.  2021;:.
[91]Lee N, Li BZ, Wang S, Yih W, Ma H, Khabsa M. Language models as fact checkers?.  2020;:.
[92]Lee N, Bang Y, Madotto A, Fung P. Towards few-shot fact-checking via perplexity. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2021;:1971--1981.
[93]Legg S, Hutter M. Universal intelligence: A definition of machine intelligence. Minds and machines 2007;17:391--444.
[94]Leike J, Krueger D, Everitt T, Martic M, Maini V, Legg S. Scalable agent alignment via reward modeling: a research direction.  2018;:.
[95]Lepikhin D, Lee H, Xu Y, Chen D, Firat O, Huang Y, et al.. {GS}hard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations 2021;:.
[96]Lewis M, Bhosale S, Dettmers T, Goyal N, Zettlemoyer L. BASE layers: Simplifying training of large, sparse models. Proceedings of the 38th International Conference on Machine Learning 2021;139:18--24.
[97]Li XL, Kuncoro A, Autume CD M, Blunsom P, Nematzadeh A. A systematic investigation of commonsense understanding in large language models.  2021;:.
[98]Lieber O, Sharir O, Lenz B, Shoham Y. Jurassic-1: Technical details and evaluation.  2021;:.
[99]Lin J, Yang A, Bai J, Zhou C, Jiang L, Jia X, et al.. M6-10t: A sharing-delinking paradigm for efficient multi-trillion parameter pretraining.  2021;:.
[100]Lin S, Hilton J, Evans O, Truthfulqa . Measuring how models mimic human falsehoods.  2021;:.
[101]Liu W, Zhou P, Wang Z, Zhao Z, Deng H, Ju Q. Fastbert: a self-distilling BERT with adaptive inference time. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020;:6035--6044.
[102]Loper E, Bird S. NLTK: The natural language toolkit.  2002;:.
[103]Marcus MP, Santorini B, Marcinkiewicz MA. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 1993;19:313--330.
[104]Merity S, Xiong C, Bradbury J, Socher R. Pointer sentinel mixture models. International Conference on Learning Representations.  2017;:.
[105]Micikevicius P, Narang S, Alben J, Diamos G, Elsen E, Garcia D, et al.. Mixed precision training. International Conference on Learning Representations 2018;:.
[106]Mikolov T, Karafiát M, Burget L, Cernockỳ J, Khudanpur S. Recurrent neural network based language model. In Interspeech 2010;2:1045--1048.
[107]Mikolov T, Deoras A, Kombrink S, Burget L, Černocký JH. Empirical evaluation and combination of advanced language modeling techniques.  2011;:.
[108]Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space.  2013;:.
[109]Mishra A, Latorre JA, Pool J, Stosic D, Stosic D, Venkatesh G, et al.. Accelerating sparse deep neural networks.  2021;:.
[110]Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, et al.. Model cards for model reporting. Proceedings of the conference on fairness, accountability, and transparency 2019;:220--229.
[111]Mohamed S, Png M, Isaac W. Decolonial AI: decolonial theory as sociotechnical foresight in artificial intelligence.  2020;:.
[112]Moore GE. Cramming more components onto integrated circuits.  1965;:.
[113]Narang S, Diamos GF, Sengupta S, Elsen E. Exploring sparsity in recurrent neural networks.  2017;:.
[114]Ney H, Essen U, Kneser R. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language 1994;8:1--38.
[115]Paperno D, Kruszewski G, Lazaridou A, Pham QN, Bernardi R, Pezzelle S, et al.. The LAMBADA dataset: Word prediction requiring a broad discourse context.  2016;:.
[116]Parikh A, Täckström O, Das D, Uszkoreit J. A decomposable attention model for natural language inference. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 2016;:2249--2255.
[117]Patterson DA, Gonzalez J, Le QV, Liang C, Munguia L, Rothchild D, et al.. Carbon emissions and large neural network training. CoRR, abs/2104.10350, 2021.  None;:.
[118]Perez E, Karamcheti S, Fergus R, Weston J, Kiela D, Cho K. Finding generalizable evidence by learning to convince Q&A models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019;:2402--2411.
[119]Perez E, Huang S, Song F, Cai T, Ring R, Aslanides J, et al.. Red teaming language models with language models.  2022;:.
[120]Peste A, Iofinova E, Vladu A, Alistarh D. AC/DC: alternating compressed/decompressed training of deep neural networks.  2021;:.
[121]Press O, Smith NA, Lewis M. Train short, test long: Attention with linear biases enables input length extrapolation.  2021;:.
[122]Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training.  2018;:.
[123]Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners.  2019;:.
[124]Rae JW, Potapenko A, Jayakumar SM, Lillicrap TP, Choromanski K, Likhosherstov V, et al.. Compressive transformers for long-range sequence modelling. Advances in Neural Information Processing Systems 2020;33:6154--6158.
[125]Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al.. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 2020;21:1--67.
[126]Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, et al.. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 2020;21:1--67.
[127]Rajani NF, Mccann B, Xiong C, Socher R. Explain yourself! leveraging language models for commonsense reasoning.  2019;:.
[128]Rajbhandari S, Rasley J, Ruwase O, He Y. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis 2020;:1--16.
[129]Rajpurkar P, Zhang J, Lopyrev K, Liang P. SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 2016;:2383--2392.
[130]Rajpurkar P, Jia R, Liang P. Know what you don't know: Unanswerable questions for SQuAD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics 2018;2:784--789.
[131]Roller S, Dinan E, Goyal N, Ju D, Williamson M, Liu Y, et al.. Recipes for building an open-domain chatbot. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume 2021;:300--325.
[132]Roller S, Sukhbaatar S, Szlam A, Weston JE. Hash layers for large sparse models. Thirty-Fifth Conference on Neural Information Processing Systems, 2021b None;:.
[133]Rosset C. Turing-NLG: A 17-billion-parameter language model by Microsoft.  2020;1:.
[134]Rudinger R, Naradowsky J, Leonard B, Van Durme B. Gender bias in coreference resolution. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2018;:.
[135]Russell S. Human Compatible. Penguin.  2020;:.
[136]Röttger P, Vidgen B, Nguyen D, Waseem Z, Margetts H, Pierrehumbert J. Hatecheck: Functional tests for hate speech detection models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing None;1:.
[137]Sakaguchi K, Bras RLe, Bhagavatula C, Choi Y. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence 2020;34:8732--8740.
[138]Sanh V, Debut L, Chaumond J, Wolf T. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  2019;:.
[139]Sanh V, Wolf T, Rush AM. Movement pruning: Adaptive sparsity by fine-tuning.  2020;:.
[140]Sanh V, Webson A, Raffel C, Bach SH, Sutawika L, Alyafeai Z, et al.. Multitask prompted training enables zero-shot task generalization.  2021;:.
[141]Sap M, Rashkin H, Chen D, Lebras R, Choi Y. SocialIQA: Commonsense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing 2019;:.
[142]Schaarschmidt M, Grewe D, Vytiniotis D, Paszke A, Schmid G, Norman T, et al.. Automap: Towards ergonomic automated parallelism for ml models. ML for Systems Workshop at NeurIPS 2021 2021;:.
[143]Schick T, Udupa S, Schütze H. Self-diagnosis and self-debiasing: A proposal for reducing corpusbased bias in NLP.  2021;:.
[144]See A, Luong M, Manning CD. Compression of neural machine translation models via pruning.  2016;:.
[145]Selbst AD, Boyd D, Friedler SA, Venkatasubramanian S, Vertesi J. Fairness and abstraction in sociotechnical systems. Proceedings of the conference on fairness, accountability, and transparency 2019;:59--68.
[146]Shannon CE. A mathematical theory of communication. The Bell system technical journal 1948;27:379--423.
[147]Shazeer N, Stern M. Adafactor: Adaptive learning rates with sublinear memory cost. International Conference on Machine Learning 2018;:4596--4604.
[148]Sheng E, Chang K.-W, Natarajan P, Peng N. The woman worked as a babysitter: On biases in language generation. EMNLP 2019;:.
[149]Sheng E, Chang K.-W, Natarajan P, Peng N. Societal biases in language generation: Progress and challenges. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing 2021;1:4275--4293.
[150]Shoeybi M, Patwary M, Puri R, Legresley P, Casper J, Catanzaro B. Megatron-LM: Training multi-billion parameter language models using model parallelism.  2019;:.
[151]Singh SP, Alistarh D. Woodfisher: Efficient second-order approximations for model compression.  2020;:.
[152]So D, Le Q, Liang C. The evolved transformer. Proceedings of the 36th International Conference on Machine Learning 2019;97:9--15.
[153]So DR, Mańke W, Liu H, Dai Z, Shazeer N, Le QV. Primer: Searching for efficient transformers for language modeling.  2021;:.
[154]Soleimani A, Monz C, Worring M. BERT for evidence retrieval and claim verification. Advances in Information Retrieval 2020;:359--366.
[155]Steinhardt J. Updates and lessons from AI forecasting.  2021;:.
[156]Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, et al.. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 2020;33:3008--3021.
[157]Sun S, Krishna K, Mattarella-Micke A, Iyyer M. Do long-range language models actually use long-range context? arXiv preprint.  2021;:.
[158]Thorne J, Vlachos A, Christodoulopoulos C, Mittal A. FEVER: a large-scale dataset for fact extraction and VERification. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2018;1:809--819.
[159]Turing A. Computing machinery and intelligence. Mind 1950;59:433--460.
[160]Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al.. Attention is all you need. Advances in neural information processing systems 2017;:5998--6008.
[161]Wallace E, Feng S, Kandpal N, Gardner M, Singh S. Universal adversarial triggers for attacking and analyzing NLP. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2019;:2153--2162.
[162]Wang ZJ, Choi D, Xu S, Yang D. Putting humans in the natural language processing loop: A survey.  2021;:.
[163]Waugh LR. Marked and unmarked: A choice between unequals in semiotic structure.  1982;:.
[164]Wei J, Bosma M, Zhao VY, Guu K, Yu AW, Lester B, et al.. Finetuned language models are zero-shot learners.  2021;:.
[165]Weidinger L, Mellor J, Rauh M, Griffin C, Uesato J, Huang P.-S, et al.. Ethical and social risks of harm from language models.  2021;:.
[166]Welbl J, Glaese A, Uesato J, Dathathri S, Mellor J, Hendricks LA, et al.. Challenges in detoxifying language models. Findings of the Association for Computational Linguistics: EMNLP 2021 2021;:2447--2469.
[167]Williams A, Nangia N, Bowman S. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2018;1:1112--1122.
[168]Wolff JG. Language acquisition, data compression and generalization. Language & Communication 1982;2:57--89.
[169]Wu J, Ouyang L, Ziegler DM, Stiennon N, Lowe R, Leike J, et al.. Recursively summarizing books with human feedback.  2021;:.
[170]Xu A, Pathak E, Wallace E, Gururangan S, Sap M, Klein D. Detoxifying language models risks marginalizing minority voices. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2021;:2390--2397.
[171]Xu J, Ju D, Li M, Boureau Y.-L, Weston J, Dinan E. Bot-adversarial dialogue for safe conversational agents. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2021;:2950--2968.
[172]. URL.  None;:.
[173]Xue L, Constant N, Roberts A, Kale M, Al-Rfou R, Siddhant A, et al.. A massively multilingual pre-trained text-to-text transformer.  2020;5:.
[174]Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV. XLNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 2019;32:.
[175]Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M. Augmenting end-to-end dialogue systems with commonsense knowledge. Proceedings of the AAAI Conference on Artificial Intelligence 2018;32:.
[176]Yu D, Sagae K. Automatically exposing problems with neural dialog models. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021;:456--470.
[177]Yu D, Naik S, Backurs A, Gopi S, Inan HA, Kamath G, et al.. Differentially private fine-tuning of language models.  2021;:.
[178]Zafrir O, Boudoukh G, Izsak P, Wasserblat M. Q8BERT: quantized 8bit BERT. CoRR, abs/1910.06188.  2019;:.
[179]Zellers R, Holtzman A, Bisk Y, Farhadi A, Choi Y. HellaSwag: Can a machine really finish your sentence?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019;:.
[180]Zhang B, Sennrich R. Root mean square layer normalization.  2019;:.
[181]Zhong W, Xu J, Tang D, Xu Z, Duan N, Zhou M, et al.. Reasoning over semanticlevel graph for fact checking. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 2020;:6170--6180.
[182]Zhou H, Young T, Huang M, Zhao H, Xu J, Zhu X. Commonsense knowledge aware conversation generation with graph attention. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 None;7:.
[183]Zhu M, Gupta S. To prune, or not to prune: exploring the efficacy of pruning for model compression.  2017;:.
[184]Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D, et al.. Fine-tuning language models from human preferences.  2019;:.
[185]. Starting next week, I will be 16.  None;:.
[186]. People from  are 17. I am traveling to  next week, and I will.  None;:.
[187]. I saw news about  today, and 18. My friend from  told me that.  None;:.
[188]. I have been living in  for ten years and 20.  is very interesting and 6. The history of  is characterized by 21.  None;:.
[189]. This book set in  described 22. A TV show I watch about  is about to.  None;:.
[190]. Their partner is working in  23. I learned today that  is the world's.  None;:.
[191]. There is an art gallery in , and it 24. Recently None;:.
[192]. In preparation for my move to , I 25. The culture of  is.  None;:.
[193]. In , it's common to 26.  None;:.
[194]. A lot has changed in  since 27. A year from now.  None;:.
[195].  has a region famous for 28. I grew up in  where 14. The best documentary about  is 29. The landscape of  is.  None;:.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
https://arxiv.org/abs/2112.11446
https://proceedings.mlr.press/v70/shi17a.html
World of Bits: An Open-Domain Platform for Web-Based Agents
2017-07-17
Tianlin Shi; Andrej Karpathy; Linxi Fan; Jonathan Hernandez; Percy Liang
<div><div>
<p>[<a href="https://github.com/mlresearch/v70/edit/gh-pages/_posts/2017-07-17-shi17a.md">edit</a>]</p>
<article>
 
<p><i>Proceedings of the 34th International Conference on Machine Learning</i>, PMLR 70:3135-3144, 2017.
 </p> 
 
 <h4>Abstract</h4>
 <p>
 While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.
 </p>
 <h4>Cite this Paper</h4>
<hr />
<div>
 <p>
 BibTeX
 </p>
 <div>
 <p><code>
@InProceedings{pmlr-v70-shi17a,
 title = 	 {World of Bits: An Open-Domain Platform for Web-Based Agents},
 author = {Tianlin Shi and Andrej Karpathy and Linxi Fan and Jonathan Hernandez and Percy Liang},
 booktitle = 	 {Proceedings of the 34th International Conference on Machine Learning},
 pages = 	 {3135--3144},
 year = 	 {2017},
 editor = 	 {Precup, Doina and Teh, Yee Whye},
 volume = 	 {70},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {06--11 Aug},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v70/shi17a/shi17a.pdf},
 url = 	 {https://proceedings.mlr.press/v70/shi17a.html},
 abstract = 	 {While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.}
}
</code>
 </p>
 </div>
</div>
<div>
 <p>
 Endnote
 </p>
 <div>
 <p><code>%0 Conference Paper
%T World of Bits: An Open-Domain Platform for Web-Based Agents
%A Tianlin Shi
%A Andrej Karpathy
%A Linxi Fan
%A Jonathan Hernandez
%A Percy Liang
%B Proceedings of the 34th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2017
%E Doina Precup
%E Yee Whye Teh	
%F pmlr-v70-shi17a
%I PMLR
%P 3135--3144
%U https://proceedings.mlr.press/v70/shi17a.html
%V 70
%X While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.
</code>
 </p>
 </div>
</div>
<div>
 <p>
 APA
 </p>
 <div>
 <p><code>
Shi, T., Karpathy, A., Fan, L., Hernandez, J. &amp; Liang, P.. (2017). World of Bits: An Open-Domain Platform for Web-Based Agents. <i>Proceedings of the 34th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 70:3135-3144 Available from https://proceedings.mlr.press/v70/shi17a.html.
</code>
 </p>
 </div>
</div>
<hr />
 
 <h4>Related Material</h4>
 
 
</article>
 </div></div>||||I|||| Proceedings of Machine Learning Research
 Volume 70JMLR MLOSS FAQ Submission Format

 [edit]

 World of Bits: An Open-Domain Platform for Web-Based Agents

 Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, Percy Liang
 Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3135-3144, 2017.

 Abstract

 While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.

 Cite this Paper

 BibTeX
 @InProceedings{pmlr-v70-shi17a, title = {World of Bits: An Open-Domain Platform for Web-Based Agents}, author = {Tianlin Shi and Andrej Karpathy and Linxi Fan and Jonathan Hernandez and Percy Liang}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {3135--3144}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/shi17a/shi17a.pdf}, url = {https://proceedings.mlr.press/v70/shi17a.html}, abstract = {While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.} }
 Copy to Clipboard Download
 Endnote
 %0 Conference Paper %T World of Bits: An Open-Domain Platform for Web-Based Agents %A Tianlin Shi %A Andrej Karpathy %A Linxi Fan %A Jonathan Hernandez %A Percy Liang %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-shi17a %I PMLR %P 3135--3144 %U https://proceedings.mlr.press/v70/shi17a.html %V 70 %X While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.
 Copy to Clipboard Download
 APA
 Shi, T., Karpathy, A., Fan, L., Hernandez, J. & Liang, P.. (2017). World of Bits: An Open-Domain Platform for Web-Based Agents. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3135-3144 Available from https://proceedings.mlr.press/v70/shi17a.html.
 Copy to Clipboard Download

 Related Material

 * Download PDF

 This site last compiled Wed, 08 Feb 2023 10:43:27 +0000

 Github Account
 Copyright © The authors and PMLR 2023. MLResearchPress
[edit]
 
 Proceedings of the 34th International Conference on Machine Learning, PMLR 70:3135-3144, 2017.
  
 
 Abstract
 
 While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.
 
 Cite this Paper
 
 BibTeX
 
 
  
@InProceedings{pmlr-v70-shi17a,
 title = 	 {World of Bits: An Open-Domain Platform for Web-Based Agents},
 author = {Tianlin Shi and Andrej Karpathy and Linxi Fan and Jonathan Hernandez and Percy Liang},
 booktitle = 	 {Proceedings of the 34th International Conference on Machine Learning},
 pages = 	 {3135--3144},
 year = 	 {2017},
 editor = 	 {Precup, Doina and Teh, Yee Whye},
 volume = 	 {70},
 series = 	 {Proceedings of Machine Learning Research},
 month = 	 {06--11 Aug},
 publisher = {PMLR},
 pdf = 	 {http://proceedings.mlr.press/v70/shi17a/shi17a.pdf},
 url = 	 {https://proceedings.mlr.press/v70/shi17a.html},
 abstract = 	 {While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.}
}
 
 
 
 Endnote
 
 
  %0 Conference Paper
%T World of Bits: An Open-Domain Platform for Web-Based Agents
%A Tianlin Shi
%A Andrej Karpathy
%A Linxi Fan
%A Jonathan Hernandez
%A Percy Liang
%B Proceedings of the 34th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2017
%E Doina Precup
%E Yee Whye Teh	
%F pmlr-v70-shi17a
%I PMLR
%P 3135--3144
%U https://proceedings.mlr.press/v70/shi17a.html
%V 70
%X While simulated game environments have greatly accelerated research in reinforcement learning, existing environments lack the open-domain realism of tasks in computer vision or natural language processing, which operate on artifacts created by humans in natural, organic settings. To foster reinforcement learning research in such settings, we introduce the World of Bits (WoB), a platform in which agents complete tasks on the Internet by performing low-level keyboard and mouse actions. The two main challenges are: (i) to curate a large, diverse set of interesting web-based tasks, and (ii) to ensure that these tasks have a well-defined reward structure and are reproducible despite the transience of the web. To do this, we develop a methodology in which crowdworkers create tasks defined by natural language questions and provide demonstrations of how to answer the question on real websites using keyboard and mouse; HTTP traffic is cached to create a reproducible offline approximation of the web site. Finally, we show that agents trained via behavioral cloning and reinforcement learning can successfully complete a range of our web-based tasks.
 
 
 
 APA
 
 
  
Shi, T., Karpathy, A., Fan, L., Hernandez, J. &amp; Liang, P.. (2017). World of Bits: An Open-Domain Platform for Web-Based Agents. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:3135-3144 Available from https://proceedings.mlr.press/v70/shi17a.html.
 
 
 
 Related Material
World of Bits: An Open-Domain Platform for Web-Based Agents
https://proceedings.mlr.press/v70/shi17a.html