How to Replace a String in Python

How to Remove or Replace a Python String or Substring

The most basic way to replace a string in Python is to use the .replace() string method:

>>> "Fake Python".replace("Fake", "Real")
'Real Python'

As you can see, you can chain .replace() onto any string and provide the method with two arguments. The first is the string that you want to replace, and the second is the replacement.

>>> name = "Fake Python"
>>> name.replace("Fake", "Real")
'Real Python'

>>> name
'Fake Python'

>>> name = name.replace("Fake", "Real")
'Real Python'

>>> name
'Real Python'

Now it’s time to apply this knowledge to the transcript:

>>> transcript = """\
... [support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
... [johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
... [support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
... [johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!"""

>>> transcript.replace("BLASTED", "😤")
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!

Loading the transcript as a triple-quoted string and then using the .replace() method on one of the swear words works fine. But there’s another swear word that’s not getting replaced because in Python, the string needs to match exactly:

>>> "Fake Python".replace("fake", "Real")
'Fake Python'

As you can see, even if the casing of one letter doesn’t match, it’ll prevent any replacements. This means that if you’re using the .replace() method, you’ll need to call it various times with the variations. In this case, you can just chain on another call to .replace():

>>> transcript.replace("BLASTED", "😤").replace("Blast", "😤")
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY 😤 ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : 😤! You're right!

Success! But you’re probably thinking that this isn’t the best way to do this for something like a general-purpose transcription sanitizer. You’ll want to move toward some way of having a list of replacements, instead of having to type out .replace() each time.

Set Up Multiple Replacement Rules

There are a few more replacements that you need to make to the transcript to get it into a format acceptable for independent review:

• Shorten or remove the time stamps
• Replace the usernames with Agent and Client

Now that you’re starting to have more strings to replace, chaining on .replace() is going to get repetitive. One idea could be to keep a list of tuples, with two items in each tuple. The two items would correspond to the arguments that you need to pass into the .replace() method—the string to replace and the replacement string:

# transcript_multiple_replace.py

REPLACEMENTS = [
    ("BLASTED", "😤"),
    ("Blast", "😤"),
    ("2022-08-24T", ""),
    ("+00:00", ""),
    ("[support_tom]", "Agent "),
    ("[johndoe]", "Client"),
]

transcript = """
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!
"""

for old, new in REPLACEMENTS:
    transcript = transcript.replace(old, new)

print(transcript)

In this version of your transcript-cleaning script, you created a list of replacement tuples, which gives you a quick way to add replacements. You could even create this list of tuples from an external CSV file if you had loads of replacements.

You then iterate over the list of replacement tuples. In each iteration, you call .replace() on the string, populating the arguments with the old and new variables that have been unpacked from each replacement tuple.

for replacement in replacements:
    new_transcript = new_transcript.replace(replacement[0], replacement[1])

With this, you’ve made a big improvement in the overall readability of the transcript. It’s also easier to add replacements if you need to. Running this script reveals a much cleaner transcript:

$ python transcript_multiple_replace.py
Agent  10:02:23 : What can I help you with?
Client 10:03:15 : I CAN'T CONNECT TO MY 😤 ACCOUNT
Agent  10:03:30 : Are you sure it's not your caps lock?
Client 10:04:03 : 😤! You're right!

That’s a pretty clean transcript. Maybe that’s all you need. But if your inner automator isn’t happy, maybe it’s because there are still some things that may be bugging you:

• Replacing the swear words won’t work if there’s another variation using -ing or a different capitalization, like BLAst.
• Removing the date from the time stamp currently only works for August 24, 2022.
• Removing the full time stamp would involve setting up replacement pairs for every possible time—not something you’re too keen on doing.
• Adding the space after Agent in order to line up your columns works but isn’t very general.

If these are your concerns, then you may want to turn your attention to regular expressions.

Leverage re.sub() to Make Complex Rules

Whenever you’re looking to do any replacing that’s slightly more complex or needs some wildcards, you’ll usually want to turn your attention toward regular expressions, also known as regex.

Regex is a sort of mini-language made up of characters that define a pattern. These patterns, or regexes, are typically used to search for strings in find and find and replace operations. Many programming languages support regex, and it’s widely used. Regex will even give you superpowers.

In Python, leveraging regex means using the re module’s sub() function and building your own regex patterns:

# transcript_regex.py

import re

REGEX_REPLACEMENTS = [
    (r"blast\w*", "😤"),
    (r" [-T:+\d]{25}", ""),
    (r"\[support\w*\]", "Agent "),
    (r"\[johndoe\]", "Client"),
]

transcript = """
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!
"""

for old, new in REGEX_REPLACEMENTS:
    transcript = re.sub(old, new, transcript, flags=re.IGNORECASE)

print(transcript)

While you can mix and match the sub() function with the .replace() method, this example only uses sub(), so you can see how it’s used. You’ll note that you can replace all variations of the swear word by using just one replacement tuple now. Similarly, you’re only using one regex for the full time stamp:

$ python transcript_regex.py
Agent  : What can I help you with?
Client : I CAN'T CONNECT TO MY 😤 ACCOUNT
Agent  : Are you sure it's not your caps lock?
Client : 😤! You're right!

Now your transcript has been completely sanitized, with all noise removed! How did that happen? That’s the magic of regex.

The first regex pattern, "blast\w*", makes use of the \w special character, which will match alphanumeric characters and underscores. Adding the * quantifier directly after it will match zero or more characters of \w.

Another vital part of the first pattern is that the re.IGNORECASE flag makes it a case-insensitive pattern. So now, any substring containing blast, regardless of capitalization, will be matched and replaced.

The second regex pattern uses character sets and quantifiers to replace the time stamp. You often use character sets and quantifiers together. A regex pattern of [abc], for example, will match one character of a, b, or c. Putting a * directly after it would match zero or more characters of a, b, or c.

There are more quantifiers, though. If you used [abc]{10}, it would match exactly ten characters of a, b or c in any order and any combination. Also note that repeating characters is redundant, so [aa] is equivalent to [a].

For the time stamp, you use an extended character set of [-T:+\d] to match all the possible characters that you might find in the time stamp. Paired with the quantifier {25}, this will match any possible time stamp, at least until the year 10,000.

The time stamp regex pattern allows you to select any possible date in the time stamp format. Seeing as the the times aren’t important for the independent reviewer of these transcripts, you replace them with an empty string. It’s possible to write a more advanced regex that preserves the time information while removing the date.

The third regex pattern is used to select any user string that starts with the keyword "support". Note that you escape (\) the square bracket ([) because otherwise the keyword would be interpreted as a character set.

Finally, the last regex pattern selects the client username string and replaces it with "Client".

With regex, you can drastically cut down the number of replacements that you have to write out. That said, you still may have to come up with many patterns. Seeing as regex isn’t the most readable of languages, having lots of patterns can quickly become hard to maintain.

Thankfully, there’s a neat trick with re.sub() that allows you to have a bit more control over how replacement works, and it offers a much more maintainable architecture.

Use a Callback With re.sub() for Even More Control

One trick that Python and sub() have up their sleeves is that you can pass in a callback function instead of the replacement string. This gives you total control over how to match and replace.

To get started building this version of the transcript-sanitizing script, you’ll use a basic regex pattern to see how using a callback with sub() works:

# transcript_regex_callback.py

import re

transcript = """
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!
"""

def sanitize_message(match):
    print(match)

re.sub(r"[-T:+\d]{25}", sanitize_message, transcript)

The regex pattern that you’re using will match the time stamps, and instead of providing a replacement string, you’re passing in a reference to the sanitize_message() function. Now, when sub() finds a match, it’ll call sanitize_message() with a match object as an argument.

Since sanitize_message() just prints the object that it’s received as an argument, when running this, you’ll see the match objects being printed to the console:

$ python transcript_regex_callback.py
<re.Match object; span=(15, 40), match='2022-08-24T10:02:23+00:00'>
<re.Match object; span=(79, 104), match='2022-08-24T10:03:15+00:00'>
<re.Match object; span=(159, 184), match='2022-08-24T10:03:30+00:00'>
<re.Match object; span=(235, 260), match='2022-08-24T10:04:03+00:00'>

A match object is one of the building blocks of the re module. The more basic re.match() function returns a match object. sub() doesn’t return any match objects but uses them behind the scenes.

Because you get this match object in the callback, you can use any of the information contained within it to build the replacement string. Once it’s built, you return the new string, and sub() will replace the match with the returned string.

Apply the Callback to the Script

In your transcript-sanitizing script, you’ll make use of the .groups() method of the match object to return the contents of the two capture groups, and then you can sanitize each part in its own function or discard it:

# transcript_regex_callback.py

import re

ENTRY_PATTERN = (
    r"\[(.+)\] "  # User string, discarding square brackets
    r"[-T:+\d]{25} "  # Time stamp
    r": "  # Separator
    r"(.+)"  # Message
)
BAD_WORDS = ["blast", "dash", "beezlebub"]
CLIENTS = ["johndoe", "janedoe"]

def censor_bad_words(message):
    for word in BAD_WORDS:
        message = re.sub(rf"{word}\w*", "😤", message, flags=re.IGNORECASE)
    return message

def censor_users(user):
    if user.startswith("support"):
        return "Agent"
    elif user in CLIENTS:
        return "Client"
    else:
        raise ValueError(f"unknown client: '{user}'")

def sanitize_message(match):
    user, message = match.groups()
    return f"{censor_users(user):<6} : {censor_bad_words(message)}"

transcript = """
[support_tom] 2022-08-24T10:02:23+00:00 : What can I help you with?
[johndoe] 2022-08-24T10:03:15+00:00 : I CAN'T CONNECT TO MY BLASTED ACCOUNT
[support_tom] 2022-08-24T10:03:30+00:00 : Are you sure it's not your caps lock?
[johndoe] 2022-08-24T10:04:03+00:00 : Blast! You're right!
"""

print(re.sub(ENTRY_PATTERN, sanitize_message, transcript))

Instead of having lots of different regexes, you can have one top level regex that can match the whole line, dividing it up into capture groups with brackets (()). The capture groups have no effect on the actual matching process, but they do affect the match object that results from the match:

• \[(.+)\] matches any sequence of characters wrapped in square brackets. The capture group picks out the username string, for instance johndoe.
• [-T:+\d]{25} matches the time stamp, which you explored in the last section. Since you won’t be using the time stamp in the final transcript, it’s not captured with brackets.
• : matches a literal colon. The colon is used as a separator between the message metadata and the message itself.
• (.+) matches any sequence of characters until the end of the line, which will be the message.

The content of the capturing groups will be available as separate items in the match object by calling the .groups() method, which returns a tuple of the matched strings.

ENTRY_PATTERN = (
    r"\[(.+)\] "  # User string, discarding square brackets
    r"[-T:+\d]{25} "  # Time stamp
    r": "  # Separator
    r"(.+)"  # Message
)

The two groups are the user string and the message. The .groups() method returns them as a tuple of strings. In the sanitize_message() function, you first use unpacking to assign the two strings to variables:

def sanitize_message(match):
    user, message = match.groups()
    return f"{censor_users(user):<6} : {censor_bad_words(message)}"

Note how this architecture allows a very broad and inclusive regex at the top level, and then lets you supplement it with more precise regexes within the replacement callback.

The sanitize_message() function makes use of two functions to clean up usernames and bad words. It additionally uses f-strings to justify the messages. Note how censor_bad_words() uses a dynamically created regex while censor_users() relies on more basic string processing.

This is now looking like a good first prototype for a transcript-sanitizing script! The output is squeaky clean:

$ python transcript_regex_callback.py
Agent  : What can I help you with?
Client : I CAN'T CONNECT TO MY 😤 ACCOUNT
Agent  : Are you sure it's not your caps lock?
Client : 😤! You're right!

Nice! Using sub() with a callback gives you far more flexibility to mix and match different methods and build regexes dynamically. This structure also gives you the most room to grow when your bosses or clients inevitably change their requirements on you!

Conclusion

In this tutorial, you’ve learned how to replace strings in Python. Along the way, you’ve gone from using the basic Python .replace() string method to using callbacks with re.sub() for absolute control. You’ve also explored some regex patterns and deconstructed them into a better architecture to manage a replacement script.

With all that knowledge, you’ve successfully cleaned a chat transcript, which is now ready for independent review. Not only that, but your transcript-sanitizing script has plenty of room to grow.