hyperparameter

A hyperparameter is a configuration value that shapes how a machine learning model is built or trained, set by a practitioner before training starts instead of learned from data. The learning rate, the number of layers, and the batch size are all hyperparameters.

The contrast is with a model’s parameters, the weights that training adjusts through gradient descent. Hyperparameters aren’t updated by that process, but they decide what it can reach. A model advertised with seven billion parameters counts its learned weights, while the hyperparameters behind it fit in a short configuration file.

The widget below runs gradient descent on a single weight with a fixed learning rate. Sweeping that rate shows the same training loop converge, overshoot, or diverge:

Most hyperparameters fall into a few groups:

• Architecture: The number of layers, the hidden size, and the number of attention heads in a transformer
• Optimization: The learning rate, batch size, number of epochs, and weight decay
• Algorithm-specific: The maximum depth of a decision tree or the number of neighbors that k-nearest neighbors consults
• Adaptation: The rank and scaling factor of a LoRA adapter used for fine-tuning

No single value works for every dataset, so good settings are found empirically by training candidate models and scoring them on held-out data that training never saw, a search known as hyperparameter tuning. In scikit-learn, hyperparameters are the arguments passed to an estimator’s constructor, such as max_depth for a decision tree.

In LLM work, the word often stretches to cover inference-time settings like temperature and top-p. Those settings shape how a trained model picks its next token and leave what it learned untouched. Inference libraries usually group them as sampling parameters.

Related Resources

Tutorial

Practical Text Classification With Python and Keras

Learn about Python text classification with Keras. Work your way from a bag-of-words model with logistic regression to more advanced methods leading to convolutional neural networks. See why word embeddings are useful and how you can use pretrained word embeddings. Use hyperparameter optimization to squeeze more performance out of your model.

advanced data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Split Your Dataset With scikit-learn's train_test_split() (Tutorial)
• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Logistic Regression in Python (Tutorial)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Learn Text Classification With Python and Keras (Course)
• Splitting Datasets With scikit-learn and train_test_split() (Course)
• Split Your Dataset With scikit-learn's train_test_split() (Quiz)
• Building a Neural Network & Making Predictions With Python AI (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)
• Python Deep Learning: PyTorch vs Tensorflow (Course)