Regular Expressions

1. Introduction

Regular expressions (regex) are a powerful tool used across various programming
languages and tools to search, match, and manipulate text. They provide a concise and
flexible means for matching strings of text, such as particular characters, words, or
patterns of characters. Regular expressions are widely used for string parsing, data
validation, data extraction, and transformation.

Basic Concepts of Regular Expressions

• Literal Characters: The simplest form of regular expressions. They match the exact
character or sequence of characters. For example, the regex hello matches the
substring "hello" in the string "hello world".
• Metacharacters: Special characters that have a unique meaning and represent
more than their literal value. Some common metacharacters include:
• . (dot): Matches any single character except newline \n.
• ^: Matches the start of a string.
• $: Matches the end of a string.
• *: Matches 0 or more repetitions of the preceding character.
• +: Matches 1 or more repetitions of the preceding character.
• ?: Matches 0 or 1 repetition of the preceding character.
• \: Escapes a metacharacter, treating it as a literal character.
• []: Matches any single character contained within the brackets.
• |: Logical OR operator.
• (): Groups part of the regex together.
• Character Classes: A character class matches any one character from a set of
characters. For example, the regex [aeiou] matches any one vowel.
• Quantifiers: Specify how many instances of a character or group must be present
for a match to occur. Examples include *, +, and ?.
• Anchors: Specify the position in the text where a match must occur. The caret ^
matches the start of the text, and the dollar sign $ matches the end of the text.
Python and Regular Expressions
Python provides the re module that supports regular expression operations. Here's a brief
introduction to using regex in Python:

import re

# Compiling a pattern for reuse

pattern = re.compile(r'\bfoo\b')

# Searching for a pattern in a string

match = pattern.search("The foo bar.")
if match:
print("Match found:", match.group())

# Replacing text
replaced_text = pattern.sub("baz", "The foo bar.")
print(replaced_text)

Basic Operations
• re.match(): Determines if the regex matches at the beginning of the string.
• re.search(): Scans a string for a regex match.
• re.findall(): Finds all substrings where the regex matches and returns them as a list.
• re.sub(): Replaces occurrences of the regex pattern with another string.

Examples
• Match an email address: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
• Validate a phone number: \+?\d{1,3}?[-.\s]?\(?\d{1,3}?\)?[-.\s]?\d{1,4}[-
.\s]?\d{1,4}[-.\s]?\d{1,9}
• Find all hashtags in a tweet: #\w+
Tips
• Regular expressions are powerful but can be complex for beginners. Start with simple
patterns and gradually introduce more complexity.
• Debug and test your regex patterns using online tools like regex101.com, which provides
real-time regex matching, explanations, and a testing sandbox.
• Remember that very complex regexes can be difficult to read and maintain. Sometimes,
it's better to use several simpler expressions or other string manipulation techniques.

Regular expressions are a fundamental skill in software development, offering a versatile

approach to text processing challenges.

2. Simple Character Matches

Simple character matches in regular expressions (regex) allow you to search for
specific characters or sequences of characters within strings. These are the foundation
of regex and are incredibly useful for text processing, validation, and parsing tasks.
Let's dive into how simple character matches work in regex, primarily focusing on
Python's re module for examples.

Literal Characters
The most basic regex pattern is matching literal characters. This means you can search
for exact matches within a string.

import re

text = "Hello, World!"

pattern = "World"
# Searching for the pattern
match = re.search(pattern, text)
if match:
print("Match found:", match.group())

In this example, re.search() looks for the exact sequence "World" in the string "Hello,
World!" and prints "Match found: World" when it finds a match.

Character Sets
You can match any one of several characters using square brackets []. A character set
matches any one of the characters enclosed in the brackets.

pattern = "[Hh]ello"

match = re.search(pattern, "hello, world!")

if match:
print("Match found:", match.group()) # Match found: hello

match = re.search(pattern, "Hello, world!")

if match:
print("Match found:", match.group()) # Match found: Hello

This pattern matches both "hello" and "Hello", as either "H" or "h" is allowed by the
character set [Hh].

Ranges
Within character sets, you can specify a range of characters using a hyphen -. This is
particularly useful for matching any single letter or number within a specific range.

# Match any lowercase letter

pattern = "[a-z]"

# Match any digit

digit_pattern = "[0-9]"

Special Characters
If you need to match a character that has a special meaning in regex (like ., *, ?, etc.), you
must escape it with a backslash \.

# Match a period
pattern = "\."
match = re.search(pattern, "Hello, world.")
if match:
print("Match found:", match.group())

This pattern is searching for a literal period . in the text.

Combining Character Matches

You can combine these simple matches to form more complex patterns.

# Match a word starting with 'H' or 'h', followed by 'ello'

pattern = "[Hh]ello"

# Match any word starting with a letter followed by two digits

complex_pattern = "[a-zA-Z][0-9][0-9]"

text = "Python regex can be complex123 but rewarding."

match = re.search(complex_pattern, text)
if match:
print("Match found:", match.group()) # Outputs: complex123

These examples illustrate the flexibility of regex for text searching tasks, even
with just simple character matches. As you become more familiar with regex,
you can combine these simple building blocks to match very complex patterns
with precision and efficiency.

3. Special Characters

In regular expressions (regex), special characters are symbols that have a specific meaning
beyond their literal interpretation. They're used to define patterns for matching a wide
variety of string sequences. Understanding these characters is key to leveraging the full
power of regex for tasks like pattern matching, validation, and parsing. Here's an
overview of commonly used special characters in regex:

1. Period .
• Matches any single character except for a newline character.
• Example: a.b matches "acb", "a&b", but not "ab" or "a\nb".
2. Caret ^
• Matches the start of a string.
• Example: ^Hello matches "Hello" in "Hello, world!" but not in "He said Hello".
3. Dollar Sign $
• Matches the end of a string or the end of a line if multiline mode is enabled.
• Example: world!$ matches "world!" in "Hello, world!" but not in "Hello, world! How are
you?".
4. Asterisk *
• Matches zero or more occurrences of the preceding character.
• Example: a*b matches "b", "ab", "aab", "aaab", etc.
5. Plus Sign +
• Matches one or more occurrences of the preceding character.
• Example: a+b matches "ab", "aab", "aaab", etc., but not "b".
6. Question Mark ?
• Makes the preceding character optional (matches zero or one occurrence).
• Example: colou?r matches both "color" and "colour".
7. Braces {} (Quantifiers)
• Matches a specific number of occurrences of the preceding character.
• Example: a{2}b matches "aab"; a{2,4}b matches "aab", "aaab", or "aaaab".
8. Brackets [] (Character Class)
• Matches any single character contained within the brackets.
• Example: [aeiou] matches any vowel.
9. Pipe | (Alternation)
• Logical OR operator between patterns.
• Example: cat|dog matches "cat" or "dog".
10. Parentheses ()
• Groups multiple characters into a single unit and captures matches for use with
backreferences.
• Example: (abc)+ matches "abc", "abcabc", "abcabcabc", etc.
11. Backslash \
• Escapes special characters or denotes character classes.
• Example: \d matches any digit; \\ matches a backslash.
12. Dot and Star .*
• Together, they're often used to match any sequence of characters.
• Example: a.*b matches any string that starts with "a" and ends with "b".
Character Classes
• \d: Matches any digit (equivalent to [0-9]).
• \D: Matches any non-digit.
• \w: Matches any word character (equivalent to [a-zA-Z0-9_]).
• \W: Matches any non-word character.
• \s: Matches any whitespace character.
• \S: Matches any non-whitespace character.
Using these special characters, regular expressions can create powerful patterns for
matching and manipulating strings in a highly flexible and efficient manner.

4. Character Classes
Character classes in regular expressions (regex) provide a way to match any one out of a
set of characters, making it easier to define patterns that can match various characters in
a single position. Character classes can be predefined (also known as "shorthand
character classes") or custom-defined within square brackets []. Here's an overview:

Predefined Character Classes

• \d: Matches any digit (0-9). Equivalent to [0-9].
• Example: \d matches "2" in "Room 21".
• \D: Matches any character that is not a digit. Equivalent to [^0-9].
• Example: \D matches "A" in "A4".
• \w: Matches any word character (letters, digits, or underscore). Equivalent to [a-zA-Z0-9_].
• Example: \w matches "H" in "Hello".
• \W: Matches any non-word character. Equivalent to [^a-zA-Z0-9_].
• Example: \W matches "!" in "Hello!".
• \s: Matches any whitespace character (spaces, tabs, newlines).
• Example: \s matches the space in "Hello World".
• \S: Matches any non-whitespace character.
• Example: \S matches "H" in " Hi".

Custom Character Classes

Custom character classes can be defined by enclosing a set of characters in square
brackets []. You can specify individual characters, a range of characters, or a combination
thereof.

• [abc]: Matches any one of the characters "a", "b", or "c".

• Example: [abc] matches "a" in "apple" and "b" in "boy".
• [a-z]: Matches any lowercase letter.
• Example: [a-z] matches "m" in "monkey".
• [A-Z]: Matches any uppercase letter.
• Example: [A-Z] matches "M" in "Monkey".
• [0-9]: Matches any digit. This is equivalent to \d.
• Example: [0-9] matches "2" in "Room 2".
• [a-zA-Z]: Matches any letter regardless of case.
• Example: [a-zA-Z] matches "A" in "Apple" and "a" in "apple".
• [^abc]: Matches any character that is not "a", "b", or "c" (negation).
• Example: [^abc] matches "d" in "dog".

Combining Character Classes

You can combine predefined and custom character classes to create complex matching
patterns.
 • Example: [\s\d] matches any whitespace character or digit.
Practical Usage
Character classes are useful in various scenarios, such as validating user input, parsing
text data, or processing logs. For example, to validate an email address, one might use a
combination of character classes and other regex patterns to ensure the input matches
the general structure of an email.

Understanding and effectively utilizing character classes can significantly enhance your
capability to perform sophisticated text processing tasks using regular expressions.

Validating an email address using regular expressions and character classes involves
creating a pattern that matches the general structure of email addresses. Typically, an
email address consists of a local part, an @ symbol, and a domain part, with specific rules
for what characters can appear in each section.

Here's a basic approach to email validation using Python's re module. Keep in mind, this
example aims for simplicity and may not cover all edge cases or fully comply with the RFC
5322 standard for email addresses.

Basic Email Validation Pattern

import re

def validate_email(email):
# Basic email pattern
# \w matches any word character (alphanumeric plus underscore)
# The character class [.-] allows for dots and hyphens within the name
# {2,} in the domain part ensures at least two characters, a simple way to enforce a minimal TLD length
pattern = r'^[\w\.-]+@[\w\.-]+\.\w{2,}$'

if re.match(pattern, email):
print(f"{email} is a valid email address.")
else:
print(f"{email} is not a valid email address.")

# Examples
validate_email("[email protected]") # Valid
validate_email("[email protected]") # Valid
validate_email("invalid-email@") # Invalid
validate_email("[email protected]") # Invalid
validate_email("special@char*acters.com") # Invalid

Explanation

• ^ and $ are anchors that match the start and end of the string, respectively,
ensuring the entire string conforms to the pattern.
 • [\w\.-]+ matches one or more word characters, dots, or hyphens. This is used for
both the local part of the email and the domain but before the top-level domain
(TLD).
• @ matches the literal "@" character that separates the local part from the domain.
• \.\w{2,} matches a dot followed by two or more word characters, aiming to
validate the TLD part. Note that {2,} enforces a minimal length but doesn't cap the
maximum, allowing for longer TLDs.

Limitations and Considerations

• This regex is simplified for common use cases and might not validate all valid
email addresses according to the full specifications. For instance, it doesn't support
quoted strings or comments in the local part, which are rarely used features.
• Email validation through regex can get extremely complicated if aiming for full RFC
compliance. In many practical applications, further validation (like sending a
confirmation email) is used in conjunction with regex checks.
• Adjustments might be needed based on the specific requirements or expected
format of email addresses in your application context.

5. Quantifiers

In regular expressions (regex), quantifiers determine how many instances of a character,

group, or character class must be present in the target sequence for a match to be found.
They are crucial for defining the flexibility and specificity of your regex pattern. Here's an
overview of the most commonly used quantifiers in regex:

1. * (Asterisk)
• Meaning: Matches zero or more occurrences of the preceding element.
• Example: ab* will match "a", "ab", "abb", "abbb", etc.
2. + (Plus)
• Meaning: Matches one or more occurrences of the preceding element.
• Example: ab+ will match "ab", "abb", "abbb", etc., but not "a".
3. ? (Question Mark)
• Meaning: Makes the preceding element optional. It matches zero or one occurrence of
the preceding element.
• Example: ab? will match "a" or "ab", but not "abb".
4. {n} (Exact Number)
• Meaning: Matches exactly n occurrences of the preceding element.
• Example: a{3} will match exactly three "a" characters in a row, such as in "aaa".
5. {n,} (At Least n)
• Meaning: Matches n or more occurrences of the preceding element.
• Example: a{2,} will match "aa", "aaa", "aaaa", etc.
6. {n,m} (Between n and m)
• Meaning: Matches from n to m occurrences of the preceding element. If m is omitted, it's
considered as infinity.
• Example: a{2,4} will match "aa", "aaa", or "aaaa".
7. *?, +?, ??, {n,m}? (Non-Greedy or Lazy)
• Meaning: By default, quantifiers are greedy, meaning they match as many occurrences as
possible. Adding ? after a quantifier makes it lazy (non-greedy), meaning it matches as few
occurrences as possible.
• Example: In the string "aaaa", a+? will match a single "a", whereas a+ would match all four
"a"s.
Practical Examples
• Matching an HTML Tag: <.*?> is a lazy quantifier pattern that matches any HTML
tag. The ? after * makes the match non-greedy, ensuring it stops at the first >.
• Validating Password Strength: ^(?=.*[A-Za-z])(?=.*\d)[A-Za-z\d]{8,}$ ensures a
string has at least one letter, at least one digit, and is at least 8 characters long.
The {8,} quantifier enforces the minimum length requirement.
Usage Tips
• Use quantifiers to create flexible and powerful regex patterns that can match a wide range
of text data.
• Be mindful of greedy vs. lazy quantifiers, especially when matching patterns in large texts
or strings with multiple potential matches.
• Regular expressions can become difficult to read and maintain; consider breaking complex
patterns into simpler, well-commented parts when possible.

Quantifiers are a foundational aspect of crafting effective regular expressions, enabling

precise control over how patterns match different sequences of characters in text
processing tasks.
This example will be particularly useful to illustrate how you can enforce rules such as
minimum length, inclusion of uppercase and lowercase letters, digits, and special
characters.

Scenario: Password Validation

Suppose we want to validate passwords based on the following criteria:

• At least 8 characters long.

• Contains both uppercase and lowercase letters.
• Includes at least one digit.
• Has at least one special character (e.g., @, #, $, %).
Regular Expression Pattern
We can construct a regular expression that uses quantifiers and other constructs to
enforce these rules:

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[@#$%]).{8,}$
^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[@#$%]).{8,}$

Explanation
• ^ and $ are anchors that match the start and the end of the string, respectively,
ensuring that the entire string fits the pattern.
• (?=.*[a-z]) is a positive lookahead assertion that ensures there's at least one
lowercase letter somewhere in the string. The .* part (any character except
newline, 0 or more times) allows characters before the lowercase letter.
• (?=.*[A-Z]) ensures at least one uppercase letter is present.
• (?=.*\d) checks for at least one digit.
• (?=.*[@#$%]) requires at least one special character from the set @, #, $, %.
• .{8,} stipulates that the entire string must be at least 8 characters long. The dot .
matches any character except newline, and the quantifier {8,} denotes 8 or more
of those characters.
Python Example
Let's use this pattern in a Python function to validate passwords:

import re

def validate_password(password):
pattern = r"^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[@#$%]).{8,}$"
if re.match(pattern, password):
return "Password is valid."
else:
return "Password is invalid."

# Examples
print(validate_password("Password123@")) # Valid
print(validate_password("pass")) # Invalid - too short, lacks uppercase, digit, special
char
print(validate_password("Password123")) # Invalid - lacks special character
print(validate_password("password123@")) # Invalid - lacks uppercase

Conclusion
This example showcases how you can use quantifiers along with other regex features to
create a powerful pattern for password validation. By adjusting the pattern, you can
enforce various password policies, making your applications more secure. Regular
expressions are a versatile tool for string matching and validation tasks, allowing for
precise and efficient text processing.

6. The Dot Character

In regular expressions (regex) within Python, the dot . character holds a special
significance: it matches any single character except the newline character (\n). This
makes the dot a powerful and flexible element in creating regex patterns, allowing for
the matching of a wide variety of string sequences.

Here's how the dot character is used in Python's re module:

Basic Usage of Dot Character

To demonstrate the basic usage, consider the following example where we want to
match any three-character string where the first character is 'a' and the third character
is 'c'.

import re

pattern = r'a.c'
text = "abc aac adc"

matches = re.findall(pattern, text)

print(matches) # Output: ['abc', 'aac', 'adc']

In this example, a.c will match "abc", "aac", "adc", and so on, where the dot . matches 'b',
'a', 'd', respectively.

Dot Character with Quantifiers

Combining the dot character with quantifiers like *, +, or ? can enhance its utility:

• .* matches any number of characters (including zero characters).

• .+ matches one or more characters.
• .? matches zero or one character.
Example: Matching Anything Between Quotes
pattern = r'"(.*?)"'

text = 'She said, "Hello, World!" and walked away.'

match = re.search(pattern, text)

if match:
 print(match.group(1)) # Output: Hello, World!

Here, "(.*?)" matches the shortest sequence of any characters between double quotes.
The ? after * makes the quantifier lazy (non-greedy), so it matches as few characters as
possible, allowing it to stop at the first closing quote it encounters.

Using Dot Character Across Lines

By default, the dot does not match the newline character. To make the dot match every
character, including newlines, you can use the re.DOTALL or re.S flag when compiling your
pattern or executing search functions.

pattern = r'.*'

text = """First line.

Second line."""

match = re.search(pattern, text, re.DOTALL)

if match:

print(match.group()) # Output: The entire text, including the newline

Escaping the Dot Character

If you need to match the dot character literally, you must escape it with a backslash \..

pattern = r'\.'

text = "Find the dot. Here it is."

matches = re.findall(pattern, text)

print(matches) # Output: ['.', '.']

In this case, \. will match only the literal dot characters in the text.
Conclusion
The dot character is a versatile tool in regex patterns, enabling the matching of
variable text sequences with ease. Its utility can be further enhanced by
combining it with quantifiers and flags to suit complex matching needs.
Understanding how to effectively use the dot character and its nuances is
crucial for mastering regular expressions in Python.

7. Greedy Matches
In the context of regular expressions (regex), "greedy" matching refers to the
tendency of certain quantifiers to match as much of the input text as possible. These
quantifiers include * (zero or more), + (one or more), and ? (zero or one), along with
{n,} (at least n), and {n,m} (between n and m, inclusive). When applied, they attempt to
consume as much of the string as they can while still allowing the overall pattern to
match.

Understanding Greedy Matching

To illustrate greedy behavior, consider the regex pattern applied to a hypothetical
string:

import re

pattern = r'<.*>'
text = "<div>Hello, <span>World!</span></div>"

match = re.search(pattern, text)

print(match.group())

output:
<div>Hello, <span>World!</span></div>

The pattern <.*> is intended to match HTML-like tags. However, due to the greedy nature
of .*, it matches the entire string from the first < to the last >, instead of stopping at the
first closing >.

Non-Greedy (Lazy) Matching

To counteract greedy behavior, you can make quantifiers "lazy" (or "non-greedy") by
appending a ? to them. This modification instructs the regex engine to match as few
characters as necessary for the pattern to succeed, effectively the opposite of greedy
matching.
Modified Pattern (Lazy):

pattern = r'<.*?>'

match = re.search(pattern, text)

print(match.group())

output:
<div>

Here, <.*?> matches the shortest possible string that starts with < and ends with >, which
is the opening <div> tag.

Practical Example: HTML Parsing

While regex is generally not recommended for parsing complex HTML or XML (use
specialized parsers like BeautifulSoup or lxml for that), here’s how greedy vs. non-greedy
matching can affect extracting data from simple HTML snippets:

html_snippet = "<title>The Greedy Fox</title><body>Once upon a time...</body>"

# Greedy matching

greedy_pattern = r'<title>.*</title>'

greedy_match = re.search(greedy_pattern, html_snippet)

print("Greedy match:", greedy_match.group())

# Non-Greedy matching

lazy_pattern = r'<title>.*?</title>'

lazy_match = re.search(lazy_pattern, html_snippet)

print("Non-greedy match:", lazy_match.group())

Output:
Greedy match: <title>The Greedy Fox</title><body>Once upon a time...</body>

Non-greedy match: <title>The Greedy Fox</title>

The greedy pattern mistakenly captures beyond the intended </title> tag, including
<body> tag content, whereas the non-greedy pattern correctly stops at the first </title>
tag encountered.

Conclusion
Greedy matching is powerful but requires careful handling to avoid over-matching. In
situations where precision is crucial, or you're dealing with nested structures, switching to
non-greedy matching by appending a ? to your quantifiers can provide more control and
accuracy. Remember, the choice between greedy and non-greedy matching depends on
the specific requirements of your pattern matching task.

8. Grouping
Grouping in regular expressions is a powerful feature that allows you to
match and isolate parts of a pattern. This is achieved using parentheses ()
around the parts of the regex pattern you wish to group. Grouping serves
multiple purposes: it can be used to apply a quantifier to a part of a pattern,
to isolate data within a match, or to specify alternatives.

Basic Grouping
Here's a simple example to illustrate grouping:

import re

text = "The rain in Spain falls mainly in the plain."

# Grouping with parentheses

pattern = r"(rain|plain)"
matches = re.findall(pattern, text)

print(matches) # Output: ['rain', 'plain']

In this example, (rain|plain) matches either "rain" or "plain". The parentheses

group the alternatives together.
Capturing Groups
When a part of a regex is enclosed in parentheses, it creates a capturing
group. You can extract the content matched by each group:

pattern = r"The (rain) in (Spain)"

match = re.search(pattern, text)
if match:
print(match.group(0)) # Entire match: 'The rain in Spain'
print(match.group(1)) # First group: 'rain'
print(match.group(2)) # Second group: 'Spain'

Non-Capturing Groups
If you want to group parts of your pattern without creating a capturing
group, use (?:...):

pattern = r"The (?:rain) in (?:Spain)"

matches = re.findall(pattern, text)
print(matches) # Output: []
# No groups are captured, but you can still use re.search().group(0) to get the
entire match.

Named Groups
For complex patterns, you can name your groups for easier access:

pattern = r"The (?P<weather>rain) in (?P<country>Spain)"

match = re.search(pattern, text)
if match:
print(match.group('weather')) # 'rain'
print(match.group('country')) # 'Spain'

Grouping for Repetition

Groups can also be used to apply quantifiers to multiple characters or
patterns:

text = "Goaaaaaal!"
pattern = r"goa{2,}l!" # Without grouping, matches 'oa' repeated, but fails
to match the whole text
 match = re.search(pattern, text, re.IGNORECASE) # re.IGNORECASE makes the
match case-insensitive
if not match:
pattern = r"go(a{2,})l!" # Group 'a{2,}' to correctly apply the
quantifier
match = re.search(pattern, text, re.IGNORECASE)
if match:
print(match.group(0)) # 'Goaaaaaal!'

Grouping for Logical Separation

You can use groups to logically separate parts of your patterns, making
complex regex more readable and maintainable:

date_text = "Today's date is 2023-04-14."

date_pattern = r"(\d{4})-(\d{2})-(\d{2})" # Matches dates in YYYY-MM-DD format
match = re.search(date_pattern, date_text)
if match:
year, month, day = match.groups()
print(f"Year: {year}, Month: {month}, Day: {day}")

Grouping is a fundamental concept in constructing regular expressions,

enabling detailed pattern matching and extraction. By effectively using
groups, you can enhance the versatility and clarity of your regex operations.

9. Matching at Beginning or End

To check if a string starts with certain words or characters, you use ^.

Example: Let's see if a sentence starts with "Hello":

import re

text = "Hello, how are you?"

pattern = r"^Hello"

if re.search(pattern, text):
print("The text starts with 'Hello'.")
else:
print("The text does not start with 'Hello'.")

Matching the End of a String

To check if a string ends with certain words or characters, you use $.
Example: Let's see if a sentence ends with "you?":

pattern = r"you\?$" # Note: The '?' needs to be escaped with '\' because it's a special character in regex.

if re.search(pattern, text):
print("The text ends with 'you?'.")
else:
print("The text does not end with 'you?'.")

Key Points:
• Use ^ to match the start of a string.
• Use $ to match the end of a string.
• These symbols help us find out if our text starts or ends with specific words or characters.

10. Match Objects

In Python's regular expression (regex) module (re), when you search for patterns within
text, the search functions return a Match object if a match is found. These objects contain
information about the search and the result. Understanding how to work with these Match
objects is crucial for effectively using regex in Python.

Key Methods and Attributes of Match Objects

• .group(): Returns the part of the string where there was a match. Without any
arguments, .group() returns the entire matched text. You can pass it numbers to
get specific matched subgroups, which are parts of the regex enclosed in
parentheses ().
• .groups(): Returns a tuple containing all the subgroups of the match.
• .groupdict(): If you have named groups in your pattern (using the (?P<name>...)
syntax), this returns a dictionary with group names as keys and their corresponding
matched text as values.
• .start() and .end(): Return the start and end positions of the match in the input
string.
• .span(): Returns a tuple containing the start and end positions of the match.

Example Usage
Let's see an example to illustrate the use of these methods and attributes:

import re

text = "John Doe, email: [email protected]"

pattern = r"(?P<first_name>\w+) (?P<last_name>\w+), email: (?P<email>\S+)"

match = re.search(pattern, text)

if match:
 # Entire match
print("Matched text:", match.group())

# Specific subgroups by number

print("First name:", match.group(1))
print("Last name:", match.group(2))

# Subgroups by name
print("Email:", match.group('email'))

# All subgroups
print("All groups:", match.groups())

# Named subgroups as a dictionary

print("Group dict:", match.groupdict())

# Match position
print("Start, End:", match.span())

Output:

Matched text: John Doe, email: [email protected]

First name: John
Last name: Doe
Email: [email protected]
All groups: ('John', 'Doe', '[email protected]')
Group dict: {'first_name': 'John', 'last_name': 'Doe', 'email': '[email protected]'}
Start, End: (0, 34)
Conclusion
Match objects provide a powerful way to extract and manipulate parts of strings that
match your regex patterns. By using methods like .group(), .groups(), and .groupdict(),
you can easily access specific parts of the matches for further processing or validation.
Understanding these objects is essential for anyone looking to perform complex text
manipulation tasks in Python.

11. Substituting
Substituting text using regular expressions in Python involves replacing parts of a string
that match a specific pattern with another string. This is commonly done with the re.sub()
function from Python's re module. The re.sub() function is powerful for various tasks,
such as cleaning data, making corrections, or formatting text according to specific rules.
Basic Syntax of re.sub()
The basic syntax of re.sub() is as follows:

re.sub(pattern, replacement, string, count=0, flags=0)

• pattern: The regex pattern to search for.

• replacement: The string to replace any matches.
• string: The original string to be searched.
• count: Optional. The maximum number of pattern occurrences to replace. The default
value of 0 means replace all occurrences.
• flags: Optional. Modifiers that affect the way the pattern is interpreted, such as
re.IGNORECASE.

Example: Basic Text Substitution

import re

text = "The rain in Spain stays mainly in the plain."

# Replace 'Spain' with 'Spring'
new_text = re.sub(r"Spain", "Spring", text)

print(new_text) # Output: The rain in Spring stays mainly in the plain.

Example: Formatting Dates
Suppose you have dates in the format "YYYY-MM-DD" and you want to change them to
"DD/MM/YYYY".

date = "2023-04-01"
formatted_date = re.sub(r"(\d{4})-(\d{2})-(\d{2})", r"\3/\2/\1", date)
print(formatted_date) # Output: 01/04/2023

In the replacement string, \1, \2, and \3 refer to the contents of the first, second, and third
capturing groups in the pattern, respectively.

Example: Removing Extra Spaces

You can also use re.sub() to remove extra spaces between words in a sentence, leaving
only one space.

messy_text = "This text has too many spaces."

clean_text = re.sub(r"\s+", " ", messy_text)

print(clean_text) # Output: This text has too many spaces.

The pattern \s+ matches one or more whitespace characters, which are then replaced with
a single space.

Dynamic Replacement with a Function

For more complex substitutions, you can pass a function as the replacement argument.
This function will be called for every non-overlapping occurrence of the pattern. The
function is passed a Match object and should return the replacement string.

def censor(match):
word = match.group(0) # The matched word
return "*" * len(word) # Replace with asterisks

text = "Avoid using words like error or fail in user messages."

censored_text = re.sub(r"error|fail", censor, text)

print(censored_text) # Output: Avoid using words like ***** or **** in user messages.

Substituting text with regular expressions in Python is a versatile way to search for and
replace patterns within strings, making it a valuable tool for text processing and data
cleaning tasks.

12. Splitting a String

Splitting a string involves dividing it into multiple parts based on a specific delimiter
or pattern. Python's re module provides a powerful method, re.split(), which allows
for splitting strings based on a regular expression pattern. This is particularly useful
when the delimiter is not a fixed character but can vary or when it's defined by more
complex rules.

Basic Usage of re.split()

The basic syntax of re.split() is:

re.split(pattern, string, maxsplit=0, flags=0)

• pattern: The regex pattern to search for as the delimiter.

• string: The original string to be split.
• maxsplit: Optional. Specifies the maximum number of splits. The default (0) means no
limit.
• flags: Optional. Modifiers that affect the way the pattern is interpreted.

Example: Splitting with Fixed Delimiter

For a simple example, splitting a string by spaces can be achieved with the regular split
method, but re.split() shows its strength with more complex patterns:

import re
text = "one, two,three, four , five"
parts = re.split(r",\s*", text) # Split by comma followed by zero or more spaces

print(parts) # Output: ['one', 'two', 'three', 'four', 'five']: ['one', 'two', 'three', 'four', 'five']

Example: Splitting with Multiple Delimiters

re.split() becomes particularly handy when you need to split a string using multiple
possible delimiters.

text = "apples|oranges bananas;grapes,tomatoes"

parts = re.split(r"[| ;,]", text) # Split by pipe, space, semicolon, or comma

print(parts) # Output: ['apples', 'oranges', 'bananas', 'grapes', 'tomatoes']

Example: Keeping Delimiters

You can also keep the delimiters by including them in capturing groups ( ) within the
pattern:

text = "The rain in Spain."

parts = re.split(r"(\s)", text) # Split by spaces but keep them

print(parts) # Output: ['The', ' ', 'rain', ' ', 'in', ' ', 'Spain.']

Example: Splitting with a Limit

The maxsplit parameter controls how many splits are performed:

text = "one two three four five"

parts = re.split(r"\s", text, maxsplit=2)

print(parts) # Output: ['one', 'two', 'three four five']

In this example, the string is only split at the first two spaces, leaving the rest of the string
intact.

Conclusion
The re.split() function offers flexible and powerful options for splitting strings,
outperforming the basic str.split() method when dealing with complex patterns or
multiple delimiters. It's especially useful in data parsing, cleaning, and preprocessing tasks
where the structure of the input data might not be uniform.
 13. Compiling Regular Expressions
Compiling a regular expression in Python refers to converting a regex pattern, specified
as a string, into a re.Pattern object using the re.compile() method. This compiled object
can then be used to perform various regex operations, such as matching, searching, and
replacing, more efficiently, especially when the same pattern is used multiple times.

Why Compile Regular Expressions?

• Efficiency: Compiling a regex pattern into a Pattern object saves the time of parsing the
regex each time it's used. This is particularly beneficial when the pattern is applied
repeatedly in a program.
• Readability: It allows for separating the definition of the regex from its usage, improving
code readability.
• Reusability: Once compiled, the Pattern object can be passed around in your code and
used in multiple match/search/replace operations.
How to Compile and Use a Compiled Regular Expression
Step 1: Compile the Pattern
import re

# Compile a regex pattern

pattern = re.compile(r'\bfoo\b')

Step 2: Use the Compiled Pattern

You can use methods like .match(), .search(), .findall(), .sub(), directly on the compiled
Pattern object.

text = "bar foo baz"

# Using the compiled pattern

match = pattern.search(text)
if match:
print("Match found:", match.group())

# Find all matches

matches = pattern.findall("foo foofoo foo")
print("All matches:", matches)

# Replace
replaced = pattern.sub("bar", "foo foofoo foo")
print("After replacement:", replaced))

Named Groups in Compiled Patterns

Compiled patterns also support named groups, which can improve the readability of your
code by allowing you to refer to match groups by names instead of numbers.
# Compiling with named groups

pattern = re.compile(r'(?P<word>\bfoo\b)')

match = pattern.search("bar foo baz")

if match:

print("Matched word:", match.group('word'))

Using Flags with Compiled Patterns

Compilation also allows you to specify flags that modify the behavior of the pattern
matching. Flags are provided as an additional argument to re.compile().

# Compiling with the IGNORECASE flag

pattern = re.compile(r'foo', re.IGNORECASE)

match = pattern.search("Foo Bar")

if match:
print("Case-insensitive match found:", match.group())

Conclusion
Compiling regular expressions is a best practice when the same pattern is applied
multiple times, improving both the efficiency and readability of your code. It offers a
structured approach to performing regex operations in Python, making your pattern
matching tasks more manageable and maintainable.

14. Flags
In regular expressions (regex), flags are special settings that modify the behavior of
the pattern matching. Flags can make your regex more powerful and adaptable,
allowing for case-insensitive matching, multiline searches, and more. In Python's re
module, flags are passed as additional arguments to functions like re.compile(),
re.search(), re.match(), re.findall(), and others.

Here are some commonly used flags in Python's re module:

re.IGNORECASE or re.I
Makes the regex case-insensitive, allowing it to match letters regardless of case.
import re

pattern = r'python'
text = 'I love Python!'

# Without the IGNORECASE flag

print(re.findall(pattern, text)) # Output: []

# With the IGNORECASE flag

print(re.findall(pattern, text, re.IGNORECASE)) # Output: ['Python']

re.MULTILINE or re.M
Changes the behavior of ^ and $ so they match the beginning and end of each line,
not just the beginning and end of the entire string.

text = '''first line

second line
third line'''

# Without the MULTILINE flag

print(re.findall(r'^\w+', text)) # Output: ['first']

# With the MULTILINE flag

print(re.findall(r'^\w+', text, re.MULTILINE)) # Output: ['first', 'second', 'third']

re.DOTALL or re.S
Allows the dot . to match any character, including newline characters. By default, .
does not match newlines.
text = 'Hello\nWorld'

# Without the DOTALL flag

print(re.findall(r'.+', text)) # Output: ['Hello']

# With the DOTALL flag

print(re.findall(r'.+', text, re.DOTALL)) # Output: ['Hello\nWorld']

re.VERBOSE or re.X
Allows you to add whitespace and comments to the regex string, making it more
readable.

pattern = re.compile(r"""
\b # Word boundary
\w+ # One or more word characters
\b # Word boundary
""", re.VERBOSE)

text = 'Hello, world!'

print(re.findall(pattern, text)) # Output: ['Hello', 'world']

Combining Flags
You can combine multiple flags using the bitwise OR operator |.

text = '''Python

python

PYTHON'''

pattern = r'^python$'

# Combining IGNORECASE and MULTILINE flags

matches = re.findall(pattern, text, re.IGNORECASE | re.MULTILINE)

print(matches) # Output: ['Python', 'python', 'PYTHON']

 Flags enhance the flexibility and power of regular expressions, making them an
essential part of crafting effective regex patterns for complex text processing tasks.

15. % method
The % method, often referred to as the string formatting or interpolation operator in
Python, is a way to format strings. It allows you to insert values into a string template. The
% operator is considered somewhat outdated in modern Python versions, with the
str.format() method and formatted string literals (f-strings, introduced in Python 3.6)
being preferred for readability and flexibility. However, understanding the % method can
still be useful, especially for maintaining older codebases.

Basic Usage
The % operator is used with a format string on the left and the values to be inserted on
the right, which can be a single value, a tuple of values, or a dictionary of values.

# Single value
name = "John"
greeting = "Hello, %s!" % name
print(greeting) # Output: Hello, John!

# Multiple values
age = 30
info = "Name: %s, Age: %d" % (name, age)
print(info) # Output: Name: John, Age: 30

Format Specifiers
The % method uses format specifiers like %s for strings, %d for integers, and %f for floating-
point numbers. Each specifier can be preceded by additional parameters to specify
minimum widths, alignment, padding, decimal precision, etc.

price = 9.99
message = "Price: $%.2f" % price # 2 decimal places for a float
print(message) # Output: Price: $9.99

Dictionary-Based Formatting
You can also use a dictionary for string formatting with the % operator. This approach
uses named placeholders.
data = {'name': 'Alice', 'age': 28}
message = "Name: %(name)s, Age: %(age)d" % data
print(message) # Output: Name: Alice, Age: 28

Considerations and Alternatives

• Readability: The % operator can be less readable and more error-prone, especially with
multiple values.
• Type-Specific Placeholders: You need to use the correct type-specific placeholder (e.g.,
%s for strings, %d for integers), which adds a layer of complexity.
• Alternatives: Consider using str.format() or f-strings for a more modern, flexible
approach to string formatting. F-strings, in particular, provide a concise and readable way
to embed expressions inside string literals:

# Using f-strings (Python 3.6+)

name = "Alice"

age = 28

message = f"Name: {name}, Age: {age}"

print(message) # Output: Name: Alice, Age: 28

While the % method is part of Python's history and is still supported, the newer string
formatting options are generally recommended for their improved functionality and ease
of use.