Regular Expressions
Regular expressions are patterns that describe text, so a program can search, extract, validate, or replace matching strings in a single step. They appear in editors, command-line tools, and programming languages. This topic covers the pattern grammar, how matching engines work, and where patterns fail.
itComputer fundamentals | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Intro
Regular Expressions
A regular expression is a pattern that describes a set of strings. A program can use it to search text for anything that matches, to extract the matching parts, to validate that input has a required shape, or to replace matches. Every mainstream programming language ships a regex engine, and the same pattern language shows up in editors, command-line tools, and build scripts.
A regular expression is itself a string. Most characters match themselves: the pattern cat matches the text cat. A few characters, the metacharacters, carry special meaning instead. The pattern ^\d{4}-\d{2}-\d{2}$ matches strings with the shape of a YYYY-MM-DD date: a start-of-text anchor, four digits, a literal hyphen, two digits, another hyphen, two digits, and an end-of-text anchor. The phrase "regular expression" is usually shortened to regex or regexp; the terms are interchangeable.
This topic builds the pattern grammar from those pieces, then explains what happens inside the engine, because the engine explains the traps.
A pattern is a description, not a literal
Each character in a pattern is either a literal or a metacharacter. A literal matches itself. A metacharacter describes structure: a position, a set of characters, repetition, or a choice. The common metacharacters are:
. * + ? { } [ ] ( ) | ^ $ \
The backslash does double duty. It escapes a metacharacter so the pattern matches the literal character: \. matches a literal period. It also introduces escape sequences with their own meanings, such as \d for a digit. An escaped metacharacter and a plain letter after a backslash mean different things in different dialects, which is why the same pattern sometimes behaves differently across languages.
Character classes
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.rand.org/content/dam/rand/pubs/research_memoranda/2008/RM704.pdf
Supports
- Stephen Kleene introduced regular sets and regular expressions in the 1951 RAND memorandum "Representation of Events in Nerve Nets and Finite Automata"
- Regular expressions began as a mathematical model for finite automata and nerve nets
- https://doi.org/10.1145/363347.363387
Supports
- Ken Thompson published "Programming Techniques: Regular expression search algorithm" in Communications of the ACM in June 1968
- Thompson described compiling a regular expression into machine instructions for a fast matching algorithm
- https://www.cs.dartmouth.edu/~doug/reader.pdf
Supports
- Ken Thompson created grep in 1973 by extracting the g/re/p command from the ed editor
- https://www.gnu.org/software/gawk/manual/html_node/History.html
Supports
- Alfred Aho, Peter Weinberger, and Brian Kernighan created awk at Bell Labs in 1977
- The awk name comes from the initials of its three authors
- https://onlinelibrary.wiley.com/doi/10.1002/spe.4380090403
Supports
- The 1979 paper "Awk: a pattern scanning and processing language" documents awk's design, including its use of regular expressions
- https://perldoc.perl.org/perlhist
Supports
- Perl 1.0 was released on December 18, 1987
- Perl 5.0 was released on October 17, 1994
- Perl and Perl-derived flavors popularized extended regex features such as lazy quantifiers and lookarounds
- https://github.com/Perl/perl5/tree/perl-1.0
Supports
- The perl-1.0 tag preserves Larry Wall's 1987 original Perl source
- https://standards.ieee.org/ieee/1003.2/1408/
Supports
- IEEE 1003.2-1992 (POSIX.2) standardized the Basic Regular Expression and Extended Regular Expression syntaxes
- https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap09.html
Supports
- POSIX defines two regex flavors: Basic Regular Expressions (BRE) and Extended Regular Expressions (ERE)
- In BRE the characters + ? | ( ) and {} are ordinary unless escaped; in ERE they are metacharacters unless escaped
- POSIX regex appears in tools such as grep, sed, and awk
- https://help.uis.cam.ac.uk/system/files/documents/techlink-hazel-pcre-brief-history.pdf
Supports
- Philip Hazel began writing PCRE in the summer of 1997 so that the Exim mail server could use Perl-compatible regular expressions
- https://www.pcre.org/
Supports
- The original PCRE library, first released in 1997, is end of life at version 8.45
- PCRE2, released in 2015, is the current library and new projects should use it
- https://pcre2project.github.io/pcre2/
Supports
- Philip Hazel authored and maintained PCRE and PCRE2 from 1997 to 2024; since 2024 the maintainers are Nicholas Wilson and Zoltán Herczeg
- PCRE2 is bundled in products such as Excel, Safari, Apache, and Git and is the regex basis for languages including PHP and R
- https://ecma-international.org/wp-content/uploads/ECMA-262_3rd_edition_december_1999.pdf
Supports
- The third edition of ECMA-262, adopted in December 1999, added powerful regular expressions; RegExp objects are defined in section 15.10
- https://www.smashingmagazine.com/2024/08/history-future-regular-expressions-javascript/
Supports
- ECMAScript 3 introduced Perl-inspired regular expressions to JavaScript in 1999
- ES2018 significantly extended JavaScript regexes with dotAll, lookbehind, named groups, and Unicode property escapes
- https://swtch.com/~rsc/regexp/regexp1.html
Supports
- Backtracking engines can re-check the same characters many times, so some patterns match in exponential time
- The Thompson construction compiles a regular expression into an NFA that a machine can simulate in linear time
- Backreferences make matching NP-complete, so they cannot be handled by linear-time automata
- These performance facts motivated RE2 and the linear-time engine design
- https://swtch.com/~rsc/regexp/regexp3.html
Supports
- Google open-sourced RE2 in March 2010
- RE2 provides most of PCRE's functionality with guaranteed linear-time execution, omitting backreferences and backtracking
- https://github.com/google/re2
Supports
- RE2 is a fast, safe, thread-friendly C++ regex library that avoids catastrophic backtracking
- RE2 matches using automata with linear-time guarantees
- https://github.com/google/re2/wiki/Syntax
Supports
- RE2 documents its supported pattern syntax and lists unsupported constructs such as backreferences and lookarounds
- https://github.com/rust-lang/regex
Supports
- The Rust regex crate uses finite automata and guarantees linear-time matching
- The crate deliberately omits backreferences and lookarounds to preserve those guarantees
- ripgrep is built on the crate
- https://docs.rs/regex/
Supports
- The regex crate documentation is the reference for its syntax, API, and linear-time guarantees
- https://github.com/kkos/oniguruma
Supports
- Oniguruma is a regex library under the BSD license that supports POSIX, grep, GNU, Perl, Java, Ruby, and Emacs syntax flavors
- https://en.wikipedia.org/wiki/Oniguruma
Supports
- Ruby 1.9 and PHP's multi-byte string module use Oniguruma as their regex engine
- Oniguruma is embedded in jq, TextMate, Sublime Text, VS Code, and other products
- The final release was 6.9.10 in January 2025 and the repository is archived
- https://github.com/VectorCamp/vectorscan
Supports
- Vectorscan is an open-source fork of Intel Hyperscan that adds ARM and Power platform support
- Intel moved Hyperscan development to a proprietary license beginning with version 5.5; the last open-source release is 5.4
- https://www.gnu.org/software/grep/
Supports
- GNU grep searches input lines for text matching a pattern and prints the matching lines
- grep uses BRE by default and ERE with the -E option
- https://www.gnu.org/software/sed/
Supports
- GNU sed is a stream editor that applies editing commands, including regex substitutions, to input lines
- https://www.gnu.org/software/gawk/
Supports
- gawk is the GNU implementation of awk, which matches input records against patterns and runs associated actions
- https://github.com/BurntSushi/ripgrep
Supports
- ripgrep is a line-oriented search tool built on the Rust regex engine and designed for fast recursive search of directory trees
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions
Supports
- JavaScript regex pattern grammar: literals, metacharacters, character classes, quantifiers, groups, assertions, and flags
- Greedy quantifiers match as much as possible; lazy quantifiers match as little as possible
- Anchors, word boundaries, and lookahead and lookbehind assertions match zero characters
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions/Cheatsheet
Supports
- The shorthand character classes \d \w \s and their complements \D \W \S
- Named capturing groups, lookbehind assertions, and Unicode property escapes \p{...}
- The dot matches any character except line terminators unless the dotAll flag is set
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp
Supports
- The RegExp object and its methods test, exec, match, replace, and split
- The flags g, i, m, s, u, and y and what each changes about matching
- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Lookbehind_assertion
Supports
- JavaScript lookbehind assertions (?<=...) and (?<!...), part of ES2018
- https://mathiasbynens.be/notes/es-regexp-proposals
Supports
- ES2018 added dotAll mode (the s flag), lookbehind assertions, named capture groups, and Unicode property escapes
- https://docs.python.org/3/library/re.html
Supports
- Python's re module pattern syntax: character classes, quantifiers, groups, backreferences, and flags such as re.IGNORECASE, re.MULTILINE, and re.DOTALL
- Python's backtracking engine can be slow on certain patterns, which the docs note alongside the atomic-group workaround
- https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
Supports
- ReDoS is a denial-of-service attack that exploits regex patterns vulnerable to catastrophic backtracking
- https://github.com/aloisdg/awesome-regex
Supports
- A curated, categorized list of regex libraries, tools, frameworks, and learning resources
- It sources the ecosystem entries for the Awesome Links tab: tutorials, visualizers, generators, security tools, and books
- https://www.regular-expressions.info/
Supports
- An independent reference and tutorial covering regex syntax across many flavors, maintained by Jan Goyvaerts
- https://www.rexegg.com/
Supports
- A regex tutorial and reference that goes deep into advanced features such as lookarounds and atomic groups
- https://regexone.com/
Supports
- Interactive, example-led lessons that teach regex fundamentals through small exercises
- https://regexlearn.com/
Supports
- A step-by-step interactive regex tutorial from basics to advanced techniques, with a cheat sheet
- https://www.hackerrank.com/domains/regex
Supports
- Regex coding challenges at multiple difficulty levels for practicing pattern construction
- https://regexper.com/
Supports
- A visualizer that renders JavaScript regular expressions as railroad diagrams
- https://www.debuggex.com/
Supports
- An online visual regex tester with diagrams for JavaScript, PCRE, and Python flavors
- https://projects.lukehaas.me/regexhub/
Supports
- A collection of useful regex patterns for common cases such as emails, URLs, and passwords
- https://github.com/pemistahl/grex
Supports
- A command-line tool and library that generates a regular expression from user-provided test cases
- https://github.com/jkutner/saferegex
Supports
- A command-line tool for testing regexes for ReDoS vulnerabilities
- https://xregexp.com/
Supports
- XRegExp is a JavaScript regex library that adds extended syntax and features beyond the built-in RegExp
- https://www.oreilly.com/library/view/mastering-regular-expressions/9780596528126/
Supports
- Mastering Regular Expressions (Friedl) is the standard book-length treatment of regex engines, backtracking, and efficiency
- https://www.oreilly.com/library/view/regular-expressions-cookbook/9781449327453/
Supports
- The Regular Expressions Cookbook (Goyvaerts and Levithan) collects tested patterns and recipes for many languages and tools
- https://regex101.com/
Supports
- An online regex tester, debugger, and explainer with multi-flavor support; free with a paid Pro tier
- https://regexr.com/
Supports
- RegExr is a free web-based regex tester supporting PCRE and JavaScript flavors, created by gskinner
- https://www.regexbuddy.com/
Supports
- RegexBuddy is a Windows desktop tool for creating, testing, understanding, and reusing regexes across programming languages
