In-Context Learning

Process prompting published

Also known as: ICL, Learning from Examples, Prompt-Based Learning

Definition

The ability of large language models to learn and perform new tasks from examples provided in the prompt, without any parameter updates. The model "learns" the pattern from examples and applies it to new inputs within the same context window. This emerged as a surprising capability of large transformer models and is the foundation of prompt-based AI applications.

What this is NOT

  • Not fine-tuning (in-context learning doesn't update weights)
  • Not memorization (the model generalizes to new inputs)
  • Not explicit programming (the model infers the pattern)

Alternative Interpretations

Different communities use this term differently:

academic-nlp

A phenomenon where pre-trained language models can perform tasks by conditioning on demonstrations in the input, without gradient-based learning. Studied as an emergent capability that improves with scale.

Sources: GPT-3 paper (Brown et al., 2020), In-context learning surveys and analyses

llm-practitioners

The practical ability to customize model behavior by providing examples in the prompt. This is what makes few-shot prompting work—the model infers the task from examples and generalizes.

Sources: Prompt engineering documentation, Practical LLM guides

Examples

  • Model learning to classify sentiment from 3 labeled examples
  • Model learning a new output format from demonstrations
  • Model generalizing from translation examples to new sentences
  • GPT-3 performing arithmetic after seeing a few examples

Counterexamples

Things that might seem like In-Context Learning but are not:

  • Fine-tuning a model on task data (that's gradient-based learning)
  • Model using knowledge from pre-training (that's not 'learning' from the prompt)
  • Following explicit instructions without examples (that's instruction-following)

Relations

  • requires few-shot-prompting (Few-shot prompting is how you use in-context learning)
  • overlapsWith prompt (Learning happens through the prompt)
  • inTensionWith fine-tuning (Alternative to in-context learning for task adaptation)

See Also