transformer architecture
Transformer architecture is a neural network design that models sequence dependencies using self-attention instead of recurrence or convolutions.
A standard transformer stacks encoder and decoder blocks, each wrapping multi-head attention and position-wise feed-forward layers in residual connections and layer normalization. Encoder blocks use two sub-layers, while decoder blocks add a third that attends over the encoder’s output, and their own self-attention is masked so each position can’t see the tokens that come after it.
Because self-attention is order-agnostic, token embeddings also carry positional encodings before they enter the stack.
Transformers can be specialized for different goals, such as encoder-only models for representation and discrimination, decoder-only models for autoregressive generation, and encoder-decoder models for sequence-to-sequence tasks. Most modern large language models are decoder-only.
Related Resources
Tutorial
Hugging Face Transformers: Leverage Open-Source AI in Python
As the AI boom continues, the Hugging Face platform stands out as the leading open-source model hub. In this tutorial, you'll get hands-on experience with Hugging Face and the Transformers library in Python.
For additional information on related topics, take a look at the following resources:
By Leodanis Pozo Ramos • Updated Aug. 31, 2026
