long short-term memory (LSTM)

Long short-term memory (LSTM) is a recurrent neural network architecture that uses learned gates to decide what to keep, add, and discard in a running memory, which lets it learn patterns that span long sequences.

Sepp Hochreiter and Jürgen Schmidhuber introduced it in a 1997 paper to get around the vanishing gradient problem that stalls plain recurrent neural networks. Each LSTM unit carries a cell state that changes mostly through addition, so the error signal can travel back through many time steps without fading.

Three gates control that memory. Each is a small layer with a sigmoid activation that outputs values between zero and one:

• Forget gate: Sets how much of the previous cell state to keep.
• Input gate: Sets how much new information to write into the cell state.
• Output gate: Sets how much of the cell state to expose as the hidden state, the output the unit passes to the next step and the next layer.

In the widget below, a single value enters at step 1 and nothing new arrives afterward. The gates are held fixed across steps here for clarity, while a trained LSTM computes them fresh at each step. Move the sliders to see how much of that value the cell state keeps over twelve steps and how much reaches the hidden state:

The gated recurrent unit (GRU) is a lighter 2014 variant with two gates. LSTMs drove speech recognition and machine translation in the mid-2010s until the transformer took over most language tasks. They remain common in time-series forecasting and streaming workloads.

Related Resources

Tutorial

Python AI: How to Build a Neural Network & Make Predictions

In this step-by-step tutorial, you'll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You'll learn how to train your neural network and make accurate predictions based on a given dataset.

intermediate ai data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Practical Text Classification With Python and Keras (Tutorial)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Learn Text Classification With Python and Keras (Course)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)