Word Vectors and Embedding Layers
00:00 Let’s take a look at another way to represent the words. You’ve seen the bag-of-words model that represents a sequence of words as a vector. However, next you’ll see how to represent each word as a vector.
One way to represent a word as a vector is with one-hot encoding. Take a look at the following code. It simply creates a list of strings. The
LabelEncoder from scikit-learn will create a unique integer value for each member of the list, and you do this by calling the
You can use this to create one-hot encodings for each string with the
OneHotEncoder in scikit-learn. Notice that the
OneHotEncoder needs the labels to be in a column, not in a row. The result is a NumPy array.
Again, there is an entry for each string or label. This time, the labels are represented as a vector. The vector is the same length as the vocabulary—3, in this sample. In a vector, all the values are
0 except for a single
The position of the
1 value is the same as the unique value assigned to the label. But this is not the ideal representation for text unless you are dealing with categorical values.
01:25 Next, you’ll look at the concept of an embedding layer for more efficient representations of text. The text needs to be prepared—again, up to 80% of machine learning is data preparation—to work with the embeddings.
Tokenizer utility inside of Keras will help you convert the text into integer values. The
Tokenizer will assign an integer value to the 5,000 most frequently used words in the corpus.
01:51 It will then create a vector for each sentence in the data set.
As you can see, the word
'the' was assigned
1, as it is occurring most often in the data set.
So, how is this different from using the
CountVectorizer from scikit-learn that you saw earlier? Recall that the
CountVectorizer created vectors that were the size of the vocabulary.
Tokenizer generates vectors that are the length of each text. This means that the vectors are going to be of different lengths and need to be padded with zeros to make them uniform.
It doesn’t matter if you pad the beginning or end, as long as you are consistent. In Keras, the
pad_sequences() function will take care of padding for you.
Give it the text to pad, where to pad—
'post' will pad at the end of the text—and the maximum length of the padded sequences. To create the embedding layer, you can use a pretrained model. You’ll do that later, but first, you’ll train a custom layer. Keras provides more utility classes to help out. Create a new
Sequential model and add an
The keyword arguments for the
Embedding layer will be the size of the vocabulary, the size of the vectors, and the length of each padded vector.
Then send the output to an intermediate
Dense layer of size
'relu', and finally, the output layer with the size and
Notice that the output of the
Embedding layer must be flattened before the
Dense layer can use it. Compile the model with the same hyperparameters as before.
The loss function will be
'binary_crossentropy', the optimizer will be
['accuracy'] for the metrics.
Train the model with the
.fit() method like you did before. This time, use only
20 epochs. You can see that the accuracy for training and validation are quite low.
03:57 Test the model and graph the history. Again, the accuracy is too low and the errors are too high, so let’s look at a better way to train this model. The problem is that you want to consider the order of the values in the vectors instead of each individual value alone, and one way to handle this is to use a pooling layer. In this case, you’ll add a global pooling layer after the embedding layer in the model.
Inside of the pooling layer, the maximum values of all values in each dimension will be selected. There are also average pooling layers as well. The max pooling layer will highlight large values. Add the
GlobalMaxPool1D layer to the model and train it again.
04:46 Now this is much more accurate, but training a real-world embedding layer would take a lot more time and attention. In the next video, you’ll see how to save time by using a pretrained word embedding.
04:58 There are many pretrained embedding layers available to include with your project.
Become a Member to join the conversation.
B S K Karthik on April 21, 2021
Thank you for nice explanation and content. hope to see more content in pytorch or tensorflow .Hope real python starts more sessions like this on AI/ML.