Python's "in" and "not in" Operators: Check for Membership

Python’s in Operator

To better understand the in operator, you’ll start by writing some small demonstrative examples that determine if a given value is in a list:

>>> 5 in [2, 3, 5, 9, 7]
True

>>> 8 in [2, 3, 5, 9, 7]
False

The first expression returns True because 5 appears inside your list of numbers. The second expression returns False because 8 isn’t present in the list.

According to the in operator documentation, an expression like value in collection is equivalent to the following code:

any(value is item or value == item for item in collection)

The generator expression wrapped in the call to any() builds a list of the Boolean values that result from checking if the target value has the same identity or is equal to the current item in collection. The call to any() checks if any one of the resulting Boolean values is True, in which case the function returns True. If all the values are False, then any() returns False.

Python’s not in Operator

The not in membership operator does exactly the opposite. With this operator, you can check if a given value is not in a collection of values:

>>> 5 not in [2, 3, 5, 9, 7]
False

>>> 8 not in [2, 3, 5, 9, 7]
True

In the first example, you get False because 5 is in [2, 3, 5, 9, 7]. In the second example, you get True because 8 isn’t in the list of values. This negative logic may seem like a tongue twister. To avoid confusion, remember that you’re trying to determine if the value is not part of a given collection of values.

With this quick overview of how membership operators work, you’re ready to go to the next level and learn how in and not in work with different built-in data types.

Lists, Tuples, and Ranges

So far, you’ve coded a few examples of using the in and not in operators to determine if a given value is present in an existing list of values. For these examples, you’ve explicitly used list objects. So, you’re already familiar with how membership tests work with lists.

With tuples, the membership operators work the same as they would with lists:

>>> 5 in (2, 3, 5, 9, 7)
True

>>> 5 not in (2, 3, 5, 9, 7)
False

There are no surprises here. Both examples work the same as the list-focused examples. In the first example, the in operator returns True because the target value, 5, is in the tuple. In the second example, not in returns the opposite result.

For lists and tuples, the membership operators use a search algorithm that iterates over the items in the underlying collection. Therefore, as your iterable gets longer, the search time increases in direct proportion. Using Big O notation, you’d say that membership operations on these data types have a time complexity of O(n).

If you use the in and not in operators with range objects, then you get a similar result:

>>> 5 in range(10)
True

>>> 5 not in range(10)
False

>>> 5 in range(0, 10, 2)
False

>>> 5 not in range(0, 10, 2)
True

When it comes to range objects, using membership tests may seem unnecessary at first glance. Most of the time, you’ll know the values in the resulting range beforehand. But what if you’re using range() with arguments that are determined at runtime?

Consider the following examples, which use random numbers to determine arguments at runtime:

>>> from random import randint

>>> 50 in range(0, 100, randint(1, 10))
False

>>> 50 in range(0, 100, randint(1, 10))
False

>>> 50 in range(0, 100, randint(1, 10))
True

>>> 50 in range(0, 100, randint(1, 10))
True

On your machine, you might get different results because you’re working with random ranges. In these specific examples, step is the only value that varies. In real code, you could have varying values for start and stop as well.

For range objects, the algorithm behind the membership tests computes the presence of a given value using the expression (value - start) % step) == 0, which depends on the arguments used to create the range at hand. This makes membership tests very efficient when they operate on range objects. In this case, you’d say that their time complexity is O(1).

>>> (2, 3, 5, 9, 7).index(8)
Traceback (most recent call last):
    ...
ValueError: tuple.index(x): x not in tuple

Remember that the target value in a membership test can be of any type. The test will check if that value is or isn’t in the target collection. For example, say that you have a hypothetical app where the users authenticate with a username and a password. You can have something like this:

# users.py

username = input("Username: ")
password = input("Password: ")

users = [("john", "secret"), ("jane", "secret"), ("linda", "secret")]

if (username, password) in users:
    print(f"Hi {username}, you're logged in!")
else:
    print("Wrong username or password")

This is a naive example. It’s unlikely that anyone would handle their users and passwords like this. But the example shows that the target value can be of any data type. In this case, you use a tuple of strings representing the username and the password of a given user.

Here’s how the code works in practice:

$ python users.py
Username: john
Password: secret
Hi john, you're logged in!

$ python users.py
Username: tina
Password: secret
Wrong username or password

In the first example, the username and password are correct because they’re in the users list. In the second example, the username doesn’t belong to any registered user, so the authentication fails.

In these examples, it’s important to note that the order in which the data is stored in the login tuple is critical because something like ("john", "secret") isn’t equal to ("secret", "john") in tuple comparison even if they have the same items.

In this section, you’ve explored examples that showcase the core behavior of membership operators with common Python built-in sequences. However, there’s a built-in sequence left. Yes, strings! In the next section, you’ll learn how membership operators work with this data type in Python.

Strings

Python strings are a fundamental tool in every Python developer’s tool kit. Like tuples, lists, and ranges, strings are also sequences because their items or characters are sequentially stored in memory.

You can use the in and not in operators with strings when you need to figure out if a given character is present in the target string. For example, say that you’re using strings to set and manage user permissions for a given resource:

>>> class User:
...     def __init__(self, username, permissions):
...         self.username = username
...         self.permissions = permissions
...

>>> admin = User("admin", "wrx")
>>> john = User("john", "rx")

>>> def has_permission(user, permission):
...     return permission in user.permissions
...

>>> has_permission(admin, "w")
True
>>> has_permission(john, "w")
False

The User class takes two arguments, a username and a set of permissions. To provide the permissions, you use a string in which w means that the user has write permission, r means that the user has read permission, and x implies execution permissions. Note that these letters are the same ones that you’d find in the Unix-style file-system permissions.

The membership test inside has_permission() checks whether the current user has a given permission or not, returning True or False accordingly. To do this, the in operator searches the permissions string to find a single character. In this example, you want to know if the users have write permission.

However, your permission system has a hidden issue. What would happen if you called the function with an empty string? Here’s your answer:

>>> has_permission(john, "")
True

Because an empty string is always considered a substring of any other string, an expression like "" in user.permissions will return True. Depending on who has access to your users’ permissions, this behavior of membership tests may imply a security breach in your system.

You can also use the membership operators to determine if a string contains a substring:

>>> greeting = "Hi, welcome to Real Python!"

>>> "Hi" in greeting
True
>>> "Hi" not in greeting
False

>>> "Hello" in greeting
False
>>> "Hello" not in greeting
True

For the string data type, an expression like substring in string is True if substring is part of string. Otherwise, the expression is False.

>>> greeting.find("Python")
20

>>> greeting.find("Hello")
-1

An important point to remember when using membership tests on strings is that string comparisons are case-sensitive:

>>> "PYTHON" in greeting
False

This membership test returns False because strings comparisons are case-sensitive, and "PYTHON" in uppercase isn’t present in greeting. To work around this case sensitivity, you can normalize all your strings using either the .upper() or .lower() method:

>>> "PYTHON".lower() in greeting.lower()
True

In this example, you use .lower() to convert the target substring and the original string into lowercase letters. This conversion tricks the case sensitivity in the implicit string comparison.

Generators

Generator functions and generator expressions create memory-efficient iterators known as generator iterators. To be memory efficient, these iterators yield items on demand without keeping a complete series of values in memory.

In practice, a generator function is a function that uses the yield statement in its body. For example, say that you need a generator function that takes a list of numbers and returns an iterator that yields square values from the original data. In this case, you can do something like this:

>>> def squares_of(values):
...     for value in values:
...         yield value ** 2
...

>>> squares = squares_of([1, 2, 3, 4])

>>> next(squares)
1
>>> next(squares)
4
>>> next(squares)
9
>>> next(squares)
16
>>> next(squares)
Traceback (most recent call last):
    ...
StopIteration

This function returns a generator iterator that yields square numbers on demand. You can use the built-in next() function to retrieve consecutive values from the iterator. When the generator iterator is completely consumed, it raises a StopIteration exception to communicate that no more values are left.

You can use the membership operators on a generator function like squares_of():

>>> 4 in squares_of([1, 2, 3, 4])
True
>>> 9 in squares_of([1, 2, 3, 4])
True
>>> 5 in squares_of([1, 2, 3, 4])
False

The in operator works as expected when you use it with generator iterators, returning True if the value is present in the iterator and False otherwise.

However, there’s something you need to be aware of when checking for membership on generators. A generator iterator will yield each item only once. If you consume all the items, then the iterator will be exhausted, and you won’t be able to iterate over it again. If you consume only some items from a generator iterator, then you can iterate over the remaining items only.

When you use in or not in on a generator iterator, the operator will consume it while searching for the target value. If the value is present, then the operator will consume all the values up to the target value. The rest of the values will still be available in the generator iterator:

>>> squares = squares_of([1, 2, 3, 4])

>>> 4 in squares
True

>>> next(squares)
9
>>> next(squares)
16
>>> next(squares)
Traceback (most recent call last):
    ...
StopIteration

In this example, 4 is in the generator iterator because it’s the square of 2. Therefore, in returns True. When you use next() to retrieve a value from square, you get 9, which is the square of 3. This result confirms that you no longer have access to the first two values. You can continue calling next() until you get a StopIteration exception when the generator iterator is exhausted.

Likewise, if the value isn’t present in the generator iterator, then the operator will consume the iterator completely, and you won’t have access to any of its values:

>>> squares = squares_of([1, 2, 3, 4])

>>> 5 in squares
False

>>> next(squares)
Traceback (most recent call last):
    ...
StopIteration

In this example, the in operator consumes squares completely, returning False because the target value isn’t in the input data. Because the generator iterator is now exhausted, a call to next() with squares as an argument raises StopIteration.

You can also create generator iterators using generator expressions. These expressions use the same syntax as list comprehensions but replace the square brackets ([]) with round brackets (()). You can use the in and not in operators with the result of a generator expression:

>>> squares = (value ** 2 for value in [1, 2, 3, 4])
>>> squares
<generator object <genexpr> at 0x1056f20a0>

>>> 4 in squares
True

>>> next(squares)
9
>>> next(squares)
16
>>> next(squares)
Traceback (most recent call last):
    ...
StopIteration

The squares variable now holds the iterator that results from the generator expression. This iterator yields square values from the input list of numbers. Generator iterators from generator expressions work the same as generator iterators from generator functions. So, the same rules apply when you use them in membership tests.

Another critical issue can arise when you use the in and not in operators with generator iterators. This issue can appear when you’re working with infinite iterators. The function below returns an iterator that yields infinite integers:

>>> def infinite_integers():
...     number = 0
...     while True:
...         yield number
...         number += 1
...

>>> integers = infinite_integers()
>>> integers
<generator object infinite_integers at 0x1057e8c80>

>>> next(integers)
0
>>> next(integers)
1
>>> next(integers)
2
>>> next(integers)
3
>>> next(integers)

The infinite_integers() function returns a generator iterator, which is stored in integers. This iterator yields values on demand, but remember, there will be infinite values. Because of this, it won’t be a good idea to use the membership operators with this iterator. Why? Well, if the target value isn’t in the generator iterator, then you’ll run into an infinite loop that’ll make your execution hang.

Dictionaries and Sets

Python’s membership operators also work with dictionaries and sets. If you use the in or not in operators directly on a dictionary, then it’ll check whether the dictionary has a given key or not. You can also do this check using the .keys() method, which is more explicit about your intentions.

You can also check if a given value or key-value pair is in a dictionary. To do these checks, you can use the .values() and .items() methods, respectively:

>>> likes = {"color": "blue", "fruit": "apple", "pet": "dog"}

>>> "fruit" in likes
True
>>> "hobby" in likes
False
>>> "blue" in likes
False

>>> "fruit" in likes.keys()
True
>>> "hobby" in likes.keys()
False
>>> "blue" in likes.keys()
False

>>> "dog" in likes.values()
True
>>> "drawing" in likes.values()
False

>>> ("color", "blue") in likes.items()
True
>>> ("hobby", "drawing") in likes.items()
False

In these examples, you use the in operator directly on your likes dictionary to check whether the "fruit", "hobby", and "blue" keys are in the dictionary or not. Note that even though "blue" is a value in likes, the test returns False because it only considers the keys.

Next up, you use the .keys() method to get the same results. In this case, the explicit method name makes your intentions much clearer to other programmers reading your code.

To check if a value like "dog" or "drawing" is present in likes, you use the .values() method, which returns a view object with the values in the underlying dictionary. Similarly, to check if a key-value pair is contained in likes, you use .items(). Note that the target key-value pairs must be two-item tuples with the key and value in that order.

If you’re using sets, then the membership operators work as they would with lists or tuples:

>>> fruits = {"apple", "banana", "cherry", "orange"}

>>> "banana" in fruits
True
>>> "banana" not in fruits
False

>>> "grape" in fruits
False
>>> "grape" not in fruits
True

These examples show that you can also check whether a given value is contained in a set by using the membership operators in and not in.

Now that you know how the in and not in operators work with different built-in data types, it’s time to put these operators into action with a couple of examples.

Replacing Chained or Operators

Using a membership test to replace a compound Boolean expression with several or operators is a useful technique that allows you to simplify your code and make it more readable.

To see this technique in action, say that you need to write a function that takes a color name as a string and determines whether it’s a primary color. To figure this out, you’ll use the RGB (red, green, and blue) color model:

>>> def is_primary_color(color):
...     color = color.lower()
...     return color == "red" or color == "green" or color == "blue"
...

>>> is_primary_color("yellow")
False

>>> is_primary_color("green")
True

In is_primary_color(), you use a compound Boolean expression that uses the or operator to check if the input color is either red, green, or blue. Even though this function works as expected, the condition may be confusing and difficult to read and understand.

The good news is that you can replace the above condition with a compact and readable membership test:

>>> def is_primary_color(color):
...     primary_colors = {"red", "green", "blue"}
...     return color.lower() in primary_colors
...

>>> is_primary_color("yellow")
False

>>> is_primary_color("green")
True

Now your function uses the in operator to check whether the input color is red, green, or blue. Assigning the set of primary colors to a properly named variable like primary_colors also helps to make your code more readable. The final check is pretty clear now. Anyone reading your code will immediately understand that you’re trying to determine if the input color is a primary color according to the RGB color model.

If you look at the example again, then you’ll notice that the primary colors have been stored in a set. Why? You’ll find your answer in the following section.

Writing Efficient Membership Tests

Python uses a data structure called a hash table to implement dictionaries and sets. Hash tables have a remarkable property: looking for any given value in the data structure takes about the same time, no matter how many values the table has. Using Big O notation, you’ll say that value lookups in hash tables have a time complexity of O(1), which makes them super fast.

Now, what does this feature of hash tables have to do with membership tests on dictionaries and sets? Well, it turns out that the in and not in operators work very quickly when they operate on these types. This detail allows you to optimize your code’s performance by favoring dictionaries and sets over lists and other sequences in membership tests.

To have an idea of how much more efficient than a list a set can be, go ahead and create the following script:

# performance.py

from timeit import timeit

a_list = list(range(100_000))
a_set = set(range(100_000))

list_time = timeit("-1 in a_list", number=1, globals=globals())
set_time = timeit("-1 in a_set", number=1, globals=globals())

print(f"Sets are {(list_time / set_time):.2f} times faster than Lists")

This script creates a list of integer numbers with one hundred thousand values and a set with the same number of elements. Then the script computes the time that it takes to determine if the number -1 is in the list and the set. You know up front that -1 doesn’t appear in the list or set. So, the membership operator will have to check all the values before getting a final result.

As you already know, when the in operator searches for a value in a list, it uses an algorithm with a time complexity of O(n). On the other hand, when the in operator searches for a value in a set, it uses the hash table lookup algorithm, which has a time complexity of O(1). This fact can make a big difference in terms of performance.

Go ahead and run your script from the command line using the following command:

$ python performance.py
Sets are 1563.33 times faster than Lists

Although your command’s output may be slightly different, it’ll still show a significant performance difference when you use a set instead of a list in this specific membership test. With a list, the processing time will be proportional to the number of values. With a set, the time will be pretty much the same for any number of values.

This performance test shows that when your code is doing membership checks on large collections of values, you should use sets instead of lists whenever possible. You’ll also benefit from sets when your code performs several membership tests during its execution.

However, note that it’s not a good idea to convert an existing list into a set just to perform a few membership tests. Remember that converting a list into a set is an operation with O(n) time complexity.