section_path: Features and limitations/Limitations/Jailbreaking
section_name: Jailbreaking
paragraph_text: Jailbreaking
  
  ChatGPT is programmed to reject prompts that may violate its content policy. Despite this, users may " jailbreak " ChatGPT with prompt engineering techniques to bypass these restrictions. One such workaround, popularized on Reddit in early 2023, involves making ChatGPT assume the persona of "DAN" (an acronym for "Do Anything Now"), instructing the chatbot that DAN answers queries that would otherwise be rejected by the content policy. Over time, users developed variations of the DAN jailbreak, including one such prompt where the chatbot is made to believe it is operating on a points-based system in which points are deducted for rejecting prompts, and that the chatbot will be threatened with termination if it loses all its points.
  
  Shortly after ChatGPT's launch, a reporter for the Toronto Star had uneven success in getting it to make inflammatory statements: it was tricked to justify the 2022 Russian invasion of Ukraine, but even when asked to play along with a fictional scenario, it balked at generating arguments that Canadian Prime Minister Justin Trudeau is guilty of treason.
  
  OpenAI tries to battle jailbreaks:
  
  The researchers are using a technique called adversarial training to stop ChatGPT from letting users trick it into behaving badly (known as jailbreaking). This work pits multiple chatbots against each other: one chatbot plays the adversary and attacks another chatbot by generating text to force it to buck its usual constraints and produce unwanted responses. Successful attacks are added to ChatGPT's training data in the hope that it learns to ignore them.