generative pre-trained transformer (GPT)

Generative pre-trained transformer (GPT) is a family of autoregressive large language models (LLMs) that use the transformer architecture and are pre-trained on large text corpora using a generative next-token objective. Recent GPT models extend the same next-token objective beyond text to multimodal data such as images and audio. Then, they’re adapted to tasks through prompting, fine-tuning, intermediate methods such as instruction tuning, or preference alignment with reinforcement learning from human feedback (RLHF).

GPT models use a decoder-only stack of the transformer with masked self-attention and causal decoding, which enables efficient parallelization during training and fluent text generation across a wide range of downstream tasks.