As filtering can be a bit tricky in pandas, you’ll learn in this lesson how pandas uses boolean operators.
How pandas Uses Boolean Operators
This is a set of rules that defines the order in which mathematical operations occur. Python has its own order of operations, and the ones we’re concerned with are listed down here in order of increasing precedence. So first we have the Boolean operators, which are
or, which were evaluated after the arithmetic operators—
<=, and so on—which are operated on after the bitwise operators, and (
&), or (
|), and the inverse or compliment operator (
~). Pandas doesn’t use these Boolean operators and instead opts for these bitwise operators. Let’s open up a terminal and see this in action.
And because this will evaluate, and then this section will evaluate, and then they’ll be compared, we should end up with
False and True, which should evaluate to
False. If you throw some parentheses into this, you can change it a little bit.
So let’s say something like
(3 and 5) in parentheses and then type the same thing out. And now this evaluates to
True. This is actually because of short-circuit evaluation, where the
3 and the
5 actually evaluate to
5, because it’s the last argument in this statement.
So then this whole section, then, is
4 < 5 > 4, which is
True. Let’s take a look at a Pandas-specific example. So take a Pandas
Series that you can set equal to
[True, True, False], and then using the bitwise
&, compare it to another
Series that is
[True, False, False]. When you run this, you should get another
Series that is
The bitwise operators actually compare the bits of a value as opposed to how truthy or falsy the value is. So as a bit of an analogy, you can think of the bits as being each piece in the
Series. So if you compare the first items, they’re both
True and True is
True and False is
False and False is
False. Don’t try to say that five times fast. Down here, however, it attempts to compare the entire
Series with the entire
Series, so if you had a
Series that was
False, how do you determine if that’s actually
so this will just be a set of numbers like so. You may think you’d be able to do an operation like
s % 2 == 0 & s > 3 as a way to find even numbers that are larger than
3. But if you run this, you end up with that same error.
03:40 Let’s try to break down exactly what’s happening here. Let’s go back to the statement and start putting some parentheses in to see what’s going on. First, we can say that this operation is occurring,
Because we didn’t change the order of anything, if you try to run this, you’ll get that same
truth value […]
is ambiguous error, and that’s because you can think of this as trying to operate each of these pieces and then compare the resulting
Series with the resulting
04:50 So let’s go back to that original statement. This bitwise operator here makes sense because we want to do an element-by-element comparison of each of these statements. The problem is, because of the order of operations, this is evaluated before these arithmetic operators.
Now if you run this, you can see that the even numbers that are greater than
3 evaluate to
True. What you should take away from here is that if you run into
ValueError while doing Boolean indexing, there’s a chance that you just need to add some more parentheses. Using these bitwise operators instead of
not changes the order in which they’re evaluated, which means that your arithmetic operations that used to have precedence now occur after the bitwise operator. So when in doubt, parentheses it out.
Become a Member to join the conversation.