section_path: Properties/Emergent abilities
section_name: Emergent abilities
paragraph_text: Emergent abilities
  
  Performance of bigger models on various tasks, when plotted on a log-log scale, appears as a linear extrapolation of performance achieved by smaller models. However, this linearity may be punctuated by " break(s) " in the scaling law, where the slope of the line changes abruptly, and where larger models acquire "emergent abilities". They arise from the complex interaction of the model's components and are not explicitly programmed or designed.
  
  Furthermore, recent research has demonstrated that AI systems, including large language models, can employ heuristic reasoning akin to human cognition. They balance between exhaustive logical processing and the use of cognitive shortcuts (heuristics), adapting their reasoning strategies to optimize between accuracy and effort. This behavior mimics principles of resource-rational human cognition, as discussed in classical theories of bounded rationality and dual-process theory.
  
  One of the emergent abilities is in-context learning from example demonstrations. In-context learning is involved in tasks, such as:
  
  Schaeffer et. al. argue that the emergent abilities are not unpredictably acquired, but predictably acquired according to a smooth scaling law. The authors considered a toy statistical model of an LLM solving multiple-choice questions, and showed that this statistical model, modified to account for other types of tasks, applies to these tasks as well.
  
  Let x {\displaystyle x} be the number of parameter count, and y {\displaystyle y} be the performance of the model.