attention mechanism

An attention mechanism is a neural network operation that computes a weighted sum of value vectors based on the similarity between a query and a set of keys. This mechanism allows models to focus on the most relevant information in or across sequences.

Variants include self-attention for intra-sequence dependencies and cross-attention to condition a decoder on encoder outputs. Earlier sequence-to-sequence models introduced additive and global or local attention to address fixed-length bottlenecks, while transformers popularized multi-head self-attention, masking for causal generation, and encoder–decoder cross-attention.

Related Resources

Tutorial

Embeddings and Vector Databases With ChromaDB

Learn how to use ChromaDB, an open-source Python vector database, to store embeddings and run semantic search with hands-on examples.

advanced ai databases data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Building a Neural Network & Making Predictions With Python AI (Course)
• Vector Databases and Embeddings With ChromaDB (Course)
• Embeddings and Vector Databases With ChromaDB (Quiz)