attention mechanism

An attention mechanism is a neural network operation that computes a weighted sum of value vectors based on the similarity between a query and a set of keys. This mechanism allows models to focus on the most relevant information in or across sequences.

Variants include self-attention for intra-sequence dependencies and cross-attention to condition a decoder on encoder outputs. Earlier sequence-to-sequence models introduced additive and global or local attention to address fixed-length bottlenecks, while transformers popularized multi-head self-attention, masking for causal generation, and encoder–decoder cross-attention.

Related Resources

Tutorial

ChromaDB: Embeddings and Vector Databases in Python

Learn what embeddings and vector databases are, then store, query, and filter text with ChromaDB in Python and use it to give an LLM context.

advanced ai databases data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Building a Neural Network & Making Predictions With Python AI (Course)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)