section_path: Training
section_name: Training
paragraph_text: Training
  
  ChatGPT is based on GPT foundation models that were fine-tuned for conversational assistance, including GPT-4o, GPT-4.5, o3, and o4-mini. The fine-tuning process leveraged supervised learning and reinforcement learning from human feedback (RLHF). Both approaches employed human trainers to improve model performance. In the case of supervised learning, the trainers played both sides: the user and the AI assistant. In the reinforcement learning stage, human trainers first ranked responses that the model had created in a previous conversation. These rankings were used to create "reward models" that were used to fine-tune the model further by using several iterations of proximal policy optimization.
  
  Time magazine reported that, to build a safety system against harmful content (e.g., sexual abuse, violence, racism, sexism), OpenAI used outsourced Kenyan workers earning around $1.32 to $2 per hour to label harmful content. These labels were used to train a model to detect such content in the future. The laborers were exposed to "toxic" and traumatic content; one worker described the assignment as "torture". OpenAI's outsourcing partner was Sama, a training-data company based in San Francisco, California.
  
  OpenAI collects data from ChatGPT users to train and fine-tune the service further. Users can upvote or downvote responses they receive from ChatGPT and fill in a text field with additional feedback.
  
  ChatGPT's training data includes software manual pages, information about internet phenomena such as bulletin board systems, multiple programming languages, and the text of Wikipedia.