3 perlretut - Perl regular expressions tutorial
7 This page provides a basic tutorial on understanding, creating and
8 using regular expressions in Perl. It serves as a complement to the
9 reference page on regular expressions L<perlre>. Regular expressions
10 are an integral part of the C<m//>, C<s///>, C<qr//> and C<split>
11 operators and so this tutorial also overlaps with
12 L<perlop/"Regexp Quote-Like Operators"> and L<perlfunc/split>.
14 Perl is widely renowned for excellence in text processing, and regular
15 expressions are one of the big factors behind this fame. Perl regular
16 expressions display an efficiency and flexibility unknown in most
17 other computer languages. Mastering even the basics of regular
18 expressions will allow you to manipulate text with surprising ease.
20 What is a regular expression? A regular expression is simply a string
21 that describes a pattern. Patterns are in common use these days;
22 examples are the patterns typed into a search engine to find web pages
23 and the patterns used to list files in a directory, e.g., C<ls *.txt>
24 or C<dir *.*>. In Perl, the patterns described by regular expressions
25 are used to search strings, extract desired parts of strings, and to
26 do search and replace operations.
28 Regular expressions have the undeserved reputation of being abstract
29 and difficult to understand. Regular expressions are constructed using
30 simple concepts like conditionals and loops and are no more difficult
31 to understand than the corresponding C<if> conditionals and C<while>
32 loops in the Perl language itself. In fact, the main challenge in
33 learning regular expressions is just getting used to the terse
34 notation used to express these concepts.
36 This tutorial flattens the learning curve by discussing regular
37 expression concepts, along with their notation, one at a time and with
38 many examples. The first part of the tutorial will progress from the
39 simplest word searches to the basic regular expression concepts. If
40 you master the first part, you will have all the tools needed to solve
41 about 98% of your needs. The second part of the tutorial is for those
42 comfortable with the basics and hungry for more power tools. It
43 discusses the more advanced regular expression operators and
44 introduces the latest cutting edge innovations in 5.6.0.
46 A note: to save time, 'regular expression' is often abbreviated as
47 regexp or regex. Regexp is a more natural abbreviation than regex, but
48 is harder to pronounce. The Perl pod documentation is evenly split on
49 regexp vs regex; in Perl, there is more than one way to abbreviate it.
50 We'll use regexp in this tutorial.
52 =head1 Part 1: The basics
54 =head2 Simple word matching
56 The simplest regexp is simply a word, or more generally, a string of
57 characters. A regexp consisting of a word matches any string that
60 "Hello World" =~ /World/; # matches
62 What is this perl statement all about? C<"Hello World"> is a simple
63 double quoted string. C<World> is the regular expression and the
64 C<//> enclosing C</World/> tells perl to search a string for a match.
65 The operator C<=~> associates the string with the regexp match and
66 produces a true value if the regexp matched, or false if the regexp
67 did not match. In our case, C<World> matches the second word in
68 C<"Hello World">, so the expression is true. Expressions like this
69 are useful in conditionals:
71 if ("Hello World" =~ /World/) {
75 print "It doesn't match\n";
78 There are useful variations on this theme. The sense of the match can
79 be reversed by using C<!~> operator:
81 if ("Hello World" !~ /World/) {
82 print "It doesn't match\n";
88 The literal string in the regexp can be replaced by a variable:
91 if ("Hello World" =~ /$greeting/) {
95 print "It doesn't match\n";
98 If you're matching against the special default variable C<$_>, the
99 C<$_ =~> part can be omitted:
103 print "It matches\n";
106 print "It doesn't match\n";
109 And finally, the C<//> default delimiters for a match can be changed
110 to arbitrary delimiters by putting an C<'m'> out front:
112 "Hello World" =~ m!World!; # matches, delimited by '!'
113 "Hello World" =~ m{World}; # matches, note the matching '{}'
114 "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin',
115 # '/' becomes an ordinary char
117 C</World/>, C<m!World!>, and C<m{World}> all represent the
118 same thing. When, e.g., C<""> is used as a delimiter, the forward
119 slash C<'/'> becomes an ordinary character and can be used in a regexp
122 Let's consider how different regexps would match C<"Hello World">:
124 "Hello World" =~ /world/; # doesn't match
125 "Hello World" =~ /o W/; # matches
126 "Hello World" =~ /oW/; # doesn't match
127 "Hello World" =~ /World /; # doesn't match
129 The first regexp C<world> doesn't match because regexps are
130 case-sensitive. The second regexp matches because the substring
131 S<C<'o W'> > occurs in the string S<C<"Hello World"> >. The space
132 character ' ' is treated like any other character in a regexp and is
133 needed to match in this case. The lack of a space character is the
134 reason the third regexp C<'oW'> doesn't match. The fourth regexp
135 C<'World '> doesn't match because there is a space at the end of the
136 regexp, but not at the end of the string. The lesson here is that
137 regexps must match a part of the string I<exactly> in order for the
138 statement to be true.
140 If a regexp matches in more than one place in the string, perl will
141 always match at the earliest possible point in the string:
143 "Hello World" =~ /o/; # matches 'o' in 'Hello'
144 "That hat is red" =~ /hat/; # matches 'hat' in 'That'
146 With respect to character matching, there are a few more points you
147 need to know about. First of all, not all characters can be used 'as
148 is' in a match. Some characters, called B<metacharacters>, are reserved
149 for use in regexp notation. The metacharacters are
153 The significance of each of these will be explained
154 in the rest of the tutorial, but for now, it is important only to know
155 that a metacharacter can be matched by putting a backslash before it:
157 "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter
158 "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary +
159 "The interval is [0,1)." =~ /[0,1)./ # is a syntax error!
160 "The interval is [0,1)." =~ /\[0,1\)\./ # matches
161 "/usr/bin/perl" =~ /\/usr\/local\/bin\/perl/; # matches
163 In the last regexp, the forward slash C<'/'> is also backslashed,
164 because it is used to delimit the regexp. This can lead to LTS
165 (leaning toothpick syndrome), however, and it is often more readable
166 to change delimiters.
169 The backslash character C<'\'> is a metacharacter itself and needs to
172 'C:\WIN32' =~ /C:\\WIN/; # matches
174 In addition to the metacharacters, there are some ASCII characters
175 which don't have printable character equivalents and are instead
176 represented by B<escape sequences>. Common examples are C<\t> for a
177 tab, C<\n> for a newline, C<\r> for a carriage return and C<\a> for a
178 bell. If your string is better thought of as a sequence of arbitrary
179 bytes, the octal escape sequence, e.g., C<\033>, or hexadecimal escape
180 sequence, e.g., C<\x1B> may be a more natural representation for your
181 bytes. Here are some examples of escapes:
183 "1000\t2000" =~ m(0\t2) # matches
184 "1000\n2000" =~ /0\n20/ # matches
185 "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000"
186 "cat" =~ /\143\x61\x74/ # matches, but a weird way to spell cat
188 If you've been around Perl a while, all this talk of escape sequences
189 may seem familiar. Similar escape sequences are used in double-quoted
190 strings and in fact the regexps in Perl are mostly treated as
191 double-quoted strings. This means that variables can be used in
192 regexps as well. Just like double-quoted strings, the values of the
193 variables in the regexp will be substituted in before the regexp is
194 evaluated for matching purposes. So we have:
197 'housecat' =~ /$foo/; # matches
198 'cathouse' =~ /cat$foo/; # matches
199 'housecat' =~ /${foo}cat/; # matches
201 So far, so good. With the knowledge above you can already perform
202 searches with just about any literal string regexp you can dream up.
203 Here is a I<very simple> emulation of the Unix grep program:
213 % chmod +x simple_grep
215 % simple_grep abba /usr/dict/words
226 This program is easy to understand. C<#!/usr/bin/perl> is the standard
227 way to invoke a perl program from the shell.
228 S<C<$regexp = shift;> > saves the first command line argument as the
229 regexp to be used, leaving the rest of the command line arguments to
230 be treated as files. S<C<< while (<>) >> > loops over all the lines in
231 all the files. For each line, S<C<print if /$regexp/;> > prints the
232 line if the regexp matches the line. In this line, both C<print> and
233 C</$regexp/> use the default variable C<$_> implicitly.
235 With all of the regexps above, if the regexp matched anywhere in the
236 string, it was considered a match. Sometimes, however, we'd like to
237 specify I<where> in the string the regexp should try to match. To do
238 this, we would use the B<anchor> metacharacters C<^> and C<$>. The
239 anchor C<^> means match at the beginning of the string and the anchor
240 C<$> means match at the end of the string, or before a newline at the
241 end of the string. Here is how they are used:
243 "housekeeper" =~ /keeper/; # matches
244 "housekeeper" =~ /^keeper/; # doesn't match
245 "housekeeper" =~ /keeper$/; # matches
246 "housekeeper\n" =~ /keeper$/; # matches
248 The second regexp doesn't match because C<^> constrains C<keeper> to
249 match only at the beginning of the string, but C<"housekeeper"> has
250 keeper starting in the middle. The third regexp does match, since the
251 C<$> constrains C<keeper> to match only at the end of the string.
253 When both C<^> and C<$> are used at the same time, the regexp has to
254 match both the beginning and the end of the string, i.e., the regexp
255 matches the whole string. Consider
257 "keeper" =~ /^keep$/; # doesn't match
258 "keeper" =~ /^keeper$/; # matches
259 "" =~ /^$/; # ^$ matches an empty string
261 The first regexp doesn't match because the string has more to it than
262 C<keep>. Since the second regexp is exactly the string, it
263 matches. Using both C<^> and C<$> in a regexp forces the complete
264 string to match, so it gives you complete control over which strings
265 match and which don't. Suppose you are looking for a fellow named
266 bert, off in a string by himself:
268 "dogbert" =~ /bert/; # matches, but not what you want
270 "dilbert" =~ /^bert/; # doesn't match, but ..
271 "bertram" =~ /^bert/; # matches, so still not good enough
273 "bertram" =~ /^bert$/; # doesn't match, good
274 "dilbert" =~ /^bert$/; # doesn't match, good
275 "bert" =~ /^bert$/; # matches, perfect
277 Of course, in the case of a literal string, one could just as easily
278 use the string equivalence S<C<$string eq 'bert'> > and it would be
279 more efficient. The C<^...$> regexp really becomes useful when we
280 add in the more powerful regexp tools below.
282 =head2 Using character classes
284 Although one can already do quite a lot with the literal string
285 regexps above, we've only scratched the surface of regular expression
286 technology. In this and subsequent sections we will introduce regexp
287 concepts (and associated metacharacter notations) that will allow a
288 regexp to not just represent a single character sequence, but a I<whole
291 One such concept is that of a B<character class>. A character class
292 allows a set of possible characters, rather than just a single
293 character, to match at a particular point in a regexp. Character
294 classes are denoted by brackets C<[...]>, with the set of characters
295 to be possibly matched inside. Here are some examples:
297 /cat/; # matches 'cat'
298 /[bcr]at/; # matches 'bat, 'cat', or 'rat'
299 /item[0123456789]/; # matches 'item0' or ... or 'item9'
300 "abc" =~ /[cab]/; # matches 'a'
302 In the last statement, even though C<'c'> is the first character in
303 the class, C<'a'> matches because the first character position in the
304 string is the earliest point at which the regexp can match.
306 /[yY][eE][sS]/; # match 'yes' in a case-insensitive way
307 # 'yes', 'Yes', 'YES', etc.
309 This regexp displays a common task: perform a case-insensitive
310 match. Perl provides away of avoiding all those brackets by simply
311 appending an C<'i'> to the end of the match. Then C</[yY][eE][sS]/;>
312 can be rewritten as C</yes/i;>. The C<'i'> stands for
313 case-insensitive and is an example of a B<modifier> of the matching
314 operation. We will meet other modifiers later in the tutorial.
316 We saw in the section above that there were ordinary characters, which
317 represented themselves, and special characters, which needed a
318 backslash C<\> to represent themselves. The same is true in a
319 character class, but the sets of ordinary and special characters
320 inside a character class are different than those outside a character
321 class. The special characters for a character class are C<-]\^$>. C<]>
322 is special because it denotes the end of a character class. C<$> is
323 special because it denotes a scalar variable. C<\> is special because
324 it is used in escape sequences, just like above. Here is how the
325 special characters C<]$\> are handled:
327 /[\]c]def/; # matches ']def' or 'cdef'
329 /[$x]at/; # matches 'bat', 'cat', or 'rat'
330 /[\$x]at/; # matches '$at' or 'xat'
331 /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat'
333 The last two are a little tricky. in C<[\$x]>, the backslash protects
334 the dollar sign, so the character class has two members C<$> and C<x>.
335 In C<[\\$x]>, the backslash is protected, so C<$x> is treated as a
336 variable and substituted in double quote fashion.
338 The special character C<'-'> acts as a range operator within character
339 classes, so that a contiguous set of characters can be written as a
340 range. With ranges, the unwieldy C<[0123456789]> and C<[abc...xyz]>
341 become the svelte C<[0-9]> and C<[a-z]>. Some examples are
343 /item[0-9]/; # matches 'item0' or ... or 'item9'
344 /[0-9bx-z]aa/; # matches '0aa', ..., '9aa',
345 # 'baa', 'xaa', 'yaa', or 'zaa'
346 /[0-9a-fA-F]/; # matches a hexadecimal digit
347 /[0-9a-zA-Z_]/; # matches a "word" character,
348 # like those in a perl variable name
350 If C<'-'> is the first or last character in a character class, it is
351 treated as an ordinary character; C<[-ab]>, C<[ab-]> and C<[a\-b]> are
354 The special character C<^> in the first position of a character class
355 denotes a B<negated character class>, which matches any character but
356 those in the brackets. Both C<[...]> and C<[^...]> must match a
357 character, or the match fails. Then
359 /[^a]at/; # doesn't match 'aat' or 'at', but matches
360 # all other 'bat', 'cat, '0at', '%at', etc.
361 /[^0-9]/; # matches a non-numeric character
362 /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary
364 Now, even C<[0-9]> can be a bother the write multiple times, so in the
365 interest of saving keystrokes and making regexps more readable, Perl
366 has several abbreviations for common character classes:
372 \d is a digit and represents [0-9]
376 \s is a whitespace character and represents [\ \t\r\n\f]
380 \w is a word character (alphanumeric or _) and represents [0-9a-zA-Z_]
384 \D is a negated \d; it represents any character but a digit [^0-9]
388 \S is a negated \s; it represents any non-whitespace character [^\s]
392 \W is a negated \w; it represents any non-word character [^\w]
396 The period '.' matches any character but "\n"
400 The C<\d\s\w\D\S\W> abbreviations can be used both inside and outside
401 of character classes. Here are some in use:
403 /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format
404 /[\d\s]/; # matches any digit or whitespace character
405 /\w\W\w/; # matches a word char, followed by a
406 # non-word char, followed by a word char
407 /..rt/; # matches any two chars, followed by 'rt'
408 /end\./; # matches 'end.'
409 /end[.]/; # same thing, matches 'end.'
411 Because a period is a metacharacter, it needs to be escaped to match
412 as an ordinary period. Because, for example, C<\d> and C<\w> are sets
413 of characters, it is incorrect to think of C<[^\d\w]> as C<[\D\W]>; in
414 fact C<[^\d\w]> is the same as C<[^\w]>, which is the same as
415 C<[\W]>. Think DeMorgan's laws.
417 An anchor useful in basic regexps is the S<B<word anchor> >
418 C<\b>. This matches a boundary between a word character and a non-word
419 character C<\w\W> or C<\W\w>:
421 $x = "Housecat catenates house and cat";
422 $x =~ /cat/; # matches cat in 'housecat'
423 $x =~ /\bcat/; # matches cat in 'catenates'
424 $x =~ /cat\b/; # matches cat in 'housecat'
425 $x =~ /\bcat\b/; # matches 'cat' at end of string
427 Note in the last example, the end of the string is considered a word
430 You might wonder why C<'.'> matches everything but C<"\n"> - why not
431 every character? The reason is that often one is matching against
432 lines and would like to ignore the newline characters. For instance,
433 while the string C<"\n"> represents one line, we would like to think
436 "" =~ /^$/; # matches
437 "\n" =~ /^$/; # matches, "\n" is ignored
439 "" =~ /./; # doesn't match; it needs a char
440 "" =~ /^.$/; # doesn't match; it needs a char
441 "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n"
442 "a" =~ /^.$/; # matches
443 "a\n" =~ /^.$/; # matches, ignores the "\n"
445 This behavior is convenient, because we usually want to ignore
446 newlines when we count and match characters in a line. Sometimes,
447 however, we want to keep track of newlines. We might even want C<^>
448 and C<$> to anchor at the beginning and end of lines within the
449 string, rather than just the beginning and end of the string. Perl
450 allows us to choose between ignoring and paying attention to newlines
451 by using the C<//s> and C<//m> modifiers. C<//s> and C<//m> stand for
452 single line and multi-line and they determine whether a string is to
453 be treated as one continuous string, or as a set of lines. The two
454 modifiers affect two aspects of how the regexp is interpreted: 1) how
455 the C<'.'> character class is defined, and 2) where the anchors C<^>
456 and C<$> are able to match. Here are the four possible combinations:
462 no modifiers (//): Default behavior. C<'.'> matches any character
463 except C<"\n">. C<^> matches only at the beginning of the string and
464 C<$> matches only at the end or before a newline at the end.
468 s modifier (//s): Treat string as a single long line. C<'.'> matches
469 any character, even C<"\n">. C<^> matches only at the beginning of
470 the string and C<$> matches only at the end or before a newline at the
475 m modifier (//m): Treat string as a set of multiple lines. C<'.'>
476 matches any character except C<"\n">. C<^> and C<$> are able to match
477 at the start or end of I<any> line within the string.
481 both s and m modifiers (//sm): Treat string as a single long line, but
482 detect multiple lines. C<'.'> matches any character, even
483 C<"\n">. C<^> and C<$>, however, are able to match at the start or end
484 of I<any> line within the string.
488 Here are examples of C<//s> and C<//m> in action:
490 $x = "There once was a girl\nWho programmed in Perl\n";
492 $x =~ /^Who/; # doesn't match, "Who" not at start of string
493 $x =~ /^Who/s; # doesn't match, "Who" not at start of string
494 $x =~ /^Who/m; # matches, "Who" at start of second line
495 $x =~ /^Who/sm; # matches, "Who" at start of second line
497 $x =~ /girl.Who/; # doesn't match, "." doesn't match "\n"
498 $x =~ /girl.Who/s; # matches, "." matches "\n"
499 $x =~ /girl.Who/m; # doesn't match, "." doesn't match "\n"
500 $x =~ /girl.Who/sm; # matches, "." matches "\n"
502 Most of the time, the default behavior is what is want, but C<//s> and
503 C<//m> are occasionally very useful. If C<//m> is being used, the start
504 of the string can still be matched with C<\A> and the end of string
505 can still be matched with the anchors C<\Z> (matches both the end and
506 the newline before, like C<$>), and C<\z> (matches only the end):
508 $x =~ /^Who/m; # matches, "Who" at start of second line
509 $x =~ /\AWho/m; # doesn't match, "Who" is not at start of string
511 $x =~ /girl$/m; # matches, "girl" at end of first line
512 $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string
514 $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end
515 $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string
517 We now know how to create choices among classes of characters in a
518 regexp. What about choices among words or character strings? Such
519 choices are described in the next section.
521 =head2 Matching this or that
523 Sometimes we would like to our regexp to be able to match different
524 possible words or character strings. This is accomplished by using
525 the B<alternation> metacharacter C<|>. To match C<dog> or C<cat>, we
526 form the regexp C<dog|cat>. As before, perl will try to match the
527 regexp at the earliest possible point in the string. At each
528 character position, perl will first try to match the first
529 alternative, C<dog>. If C<dog> doesn't match, perl will then try the
530 next alternative, C<cat>. If C<cat> doesn't match either, then the
531 match fails and perl moves to the next position in the string. Some
534 "cats and dogs" =~ /cat|dog|bird/; # matches "cat"
535 "cats and dogs" =~ /dog|cat|bird/; # matches "cat"
537 Even though C<dog> is the first alternative in the second regexp,
538 C<cat> is able to match earlier in the string.
540 "cats" =~ /c|ca|cat|cats/; # matches "c"
541 "cats" =~ /cats|cat|ca|c/; # matches "cats"
543 Here, all the alternatives match at the first string position, so the
544 first alternative is the one that matches. If some of the
545 alternatives are truncations of the others, put the longest ones first
546 to give them a chance to match.
548 "cab" =~ /a|b|c/ # matches "c"
551 The last example points out that character classes are like
552 alternations of characters. At a given character position, the first
553 alternative that allows the regexp match to succeed will be the one
556 =head2 Grouping things and hierarchical matching
558 Alternation allows a regexp to choose among alternatives, but by
559 itself it unsatisfying. The reason is that each alternative is a whole
560 regexp, but sometime we want alternatives for just part of a
561 regexp. For instance, suppose we want to search for housecats or
562 housekeepers. The regexp C<housecat|housekeeper> fits the bill, but is
563 inefficient because we had to type C<house> twice. It would be nice to
564 have parts of the regexp be constant, like C<house>, and some
565 parts have alternatives, like C<cat|keeper>.
567 The B<grouping> metacharacters C<()> solve this problem. Grouping
568 allows parts of a regexp to be treated as a single unit. Parts of a
569 regexp are grouped by enclosing them in parentheses. Thus we could solve
570 the C<housecat|housekeeper> by forming the regexp as
571 C<house(cat|keeper)>. The regexp C<house(cat|keeper)> means match
572 C<house> followed by either C<cat> or C<keeper>. Some more examples
575 /(a|b)b/; # matches 'ab' or 'bb'
576 /(ac|b)b/; # matches 'acb' or 'bb'
577 /(^a|b)c/; # matches 'ac' at start of string or 'bc' anywhere
578 /(a|[bc])d/; # matches 'ad', 'bd', or 'cd'
580 /house(cat|)/; # matches either 'housecat' or 'house'
581 /house(cat(s|)|)/; # matches either 'housecats' or 'housecat' or
582 # 'house'. Note groups can be nested.
584 /(19|20|)\d\d/; # match years 19xx, 20xx, or the Y2K problem, xx
585 "20" =~ /(19|20|)\d\d/; # matches the null alternative '()\d\d',
586 # because '20\d\d' can't match
588 Alternations behave the same way in groups as out of them: at a given
589 string position, the leftmost alternative that allows the regexp to
590 match is taken. So in the last example at the first string position,
591 C<"20"> matches the second alternative, but there is nothing left over
592 to match the next two digits C<\d\d>. So perl moves on to the next
593 alternative, which is the null alternative and that works, since
594 C<"20"> is two digits.
596 The process of trying one alternative, seeing if it matches, and
597 moving on to the next alternative if it doesn't, is called
598 B<backtracking>. The term 'backtracking' comes from the idea that
599 matching a regexp is like a walk in the woods. Successfully matching
600 a regexp is like arriving at a destination. There are many possible
601 trailheads, one for each string position, and each one is tried in
602 order, left to right. From each trailhead there may be many paths,
603 some of which get you there, and some which are dead ends. When you
604 walk along a trail and hit a dead end, you have to backtrack along the
605 trail to an earlier point to try another trail. If you hit your
606 destination, you stop immediately and forget about trying all the
607 other trails. You are persistent, and only if you have tried all the
608 trails from all the trailheads and not arrived at your destination, do
609 you declare failure. To be concrete, here is a step-by-step analysis
610 of what perl does when it tries to match the regexp
612 "abcde" =~ /(abd|abc)(df|d|de)/;
618 Start with the first letter in the string 'a'.
622 Try the first alternative in the first group 'abd'.
626 Match 'a' followed by 'b'. So far so good.
630 'd' in the regexp doesn't match 'c' in the string - a dead
631 end. So backtrack two characters and pick the second alternative in
632 the first group 'abc'.
636 Match 'a' followed by 'b' followed by 'c'. We are on a roll
637 and have satisfied the first group. Set $1 to 'abc'.
641 Move on to the second group and pick the first alternative
650 'f' in the regexp doesn't match 'e' in the string, so a dead
651 end. Backtrack one character and pick the second alternative in the
656 'd' matches. The second grouping is satisfied, so set $2 to
661 We are at the end of the regexp, so we are done! We have
662 matched 'abcd' out of the string "abcde".
666 There are a couple of things to note about this analysis. First, the
667 third alternative in the second group 'de' also allows a match, but we
668 stopped before we got to it - at a given character position, leftmost
669 wins. Second, we were able to get a match at the first character
670 position of the string 'a'. If there were no matches at the first
671 position, perl would move to the second character position 'b' and
672 attempt the match all over again. Only when all possible paths at all
673 possible character positions have been exhausted does perl give
674 up and declare S<C<$string =~ /(abd|abc)(df|d|de)/;> > to be false.
676 Even with all this work, regexp matching happens remarkably fast. To
677 speed things up, during compilation stage, perl compiles the regexp
678 into a compact sequence of opcodes that can often fit inside a
679 processor cache. When the code is executed, these opcodes can then run
680 at full throttle and search very quickly.
682 =head2 Extracting matches
684 The grouping metacharacters C<()> also serve another completely
685 different function: they allow the extraction of the parts of a string
686 that matched. This is very useful to find out what matched and for
687 text processing in general. For each grouping, the part that matched
688 inside goes into the special variables C<$1>, C<$2>, etc. They can be
689 used just as ordinary variables:
691 # extract hours, minutes, seconds
692 $time =~ /(\d\d):(\d\d):(\d\d)/; # match hh:mm:ss format
697 Now, we know that in scalar context,
698 S<C<$time =~ /(\d\d):(\d\d):(\d\d)/> > returns a true or false
699 value. In list context, however, it returns the list of matched values
700 C<($1,$2,$3)>. So we could write the code more compactly as
702 # extract hours, minutes, seconds
703 ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/);
705 If the groupings in a regexp are nested, C<$1> gets the group with the
706 leftmost opening parenthesis, C<$2> the next opening parenthesis,
707 etc. For example, here is a complex regexp and the matching variables
710 /(ab(cd|ef)((gi)|j))/;
713 so that if the regexp matched, e.g., C<$2> would contain 'cd' or 'ef'. For
714 convenience, perl sets C<$+> to the string held by the highest numbered
715 C<$1>, C<$2>, ... that got assigned (and, somewhat related, C<$^N> to the
716 value of the C<$1>, C<$2>, ... most-recently assigned; i.e. the C<$1>,
717 C<$2>, ... associated with the rightmost closing parenthesis used in the
720 Closely associated with the matching variables C<$1>, C<$2>, ... are
721 the B<backreferences> C<\1>, C<\2>, ... . Backreferences are simply
722 matching variables that can be used I<inside> a regexp. This is a
723 really nice feature - what matches later in a regexp can depend on
724 what matched earlier in the regexp. Suppose we wanted to look
725 for doubled words in text, like 'the the'. The following regexp finds
726 all 3-letter doubles with a space in between:
730 The grouping assigns a value to \1, so that the same 3 letter sequence
731 is used for both parts. Here are some words with repeated parts:
733 % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\1$' /usr/dict/words
741 The regexp has a single grouping which considers 4-letter
742 combinations, then 3-letter combinations, etc. and uses C<\1> to look for
743 a repeat. Although C<$1> and C<\1> represent the same thing, care should be
744 taken to use matched variables C<$1>, C<$2>, ... only outside a regexp
745 and backreferences C<\1>, C<\2>, ... only inside a regexp; not doing
746 so may lead to surprising and/or undefined results.
748 In addition to what was matched, Perl 5.6.0 also provides the
749 positions of what was matched with the C<@-> and C<@+>
750 arrays. C<$-[0]> is the position of the start of the entire match and
751 C<$+[0]> is the position of the end. Similarly, C<$-[n]> is the
752 position of the start of the C<$n> match and C<$+[n]> is the position
753 of the end. If C<$n> is undefined, so are C<$-[n]> and C<$+[n]>. Then
756 $x = "Mmm...donut, thought Homer";
757 $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches
758 foreach $expr (1..$#-) {
759 print "Match $expr: '${$expr}' at position ($-[$expr],$+[$expr])\n";
764 Match 1: 'Mmm' at position (0,3)
765 Match 2: 'donut' at position (6,11)
767 Even if there are no groupings in a regexp, it is still possible to
768 find out what exactly matched in a string. If you use them, perl
769 will set C<$`> to the part of the string before the match, will set C<$&>
770 to the part of the string that matched, and will set C<$'> to the part
771 of the string after the match. An example:
773 $x = "the cat caught the mouse";
774 $x =~ /cat/; # $` = 'the ', $& = 'cat', $' = ' caught the mouse'
775 $x =~ /the/; # $` = '', $& = 'the', $' = ' cat caught the mouse'
777 In the second match, S<C<$` = ''> > because the regexp matched at the
778 first character position in the string and stopped, it never saw the
779 second 'the'. It is important to note that using C<$`> and C<$'>
780 slows down regexp matching quite a bit, and C< $& > slows it down to a
781 lesser extent, because if they are used in one regexp in a program,
782 they are generated for <all> regexps in the program. So if raw
783 performance is a goal of your application, they should be avoided.
784 If you need them, use C<@-> and C<@+> instead:
786 $` is the same as substr( $x, 0, $-[0] )
787 $& is the same as substr( $x, $-[0], $+[0]-$-[0] )
788 $' is the same as substr( $x, $+[0] )
790 =head2 Matching repetitions
792 The examples in the previous section display an annoying weakness. We
793 were only matching 3-letter words, or syllables of 4 letters or
794 less. We'd like to be able to match words or syllables of any length,
795 without writing out tedious alternatives like
796 C<\w\w\w\w|\w\w\w|\w\w|\w>.
798 This is exactly the problem the B<quantifier> metacharacters C<?>,
799 C<*>, C<+>, and C<{}> were created for. They allow us to determine the
800 number of repeats of a portion of a regexp we consider to be a
801 match. Quantifiers are put immediately after the character, character
802 class, or grouping that we want to specify. They have the following
809 C<a?> = match 'a' 1 or 0 times
813 C<a*> = match 'a' 0 or more times, i.e., any number of times
817 C<a+> = match 'a' 1 or more times, i.e., at least once
821 C<a{n,m}> = match at least C<n> times, but not more than C<m>
826 C<a{n,}> = match at least C<n> or more times
830 C<a{n}> = match exactly C<n> times
834 Here are some examples:
836 /[a-z]+\s+\d*/; # match a lowercase word, at least some space, and
837 # any number of digits
838 /(\w+)\s+\1/; # match doubled words of arbitrary length
839 /y(es)?/i; # matches 'y', 'Y', or a case-insensitive 'yes'
840 $year =~ /\d{2,4}/; # make sure year is at least 2 but not more
842 $year =~ /\d{4}|\d{2}/; # better match; throw out 3 digit dates
843 $year =~ /\d{2}(\d{2})?/; # same thing written differently. However,
844 # this produces $1 and the other does not.
846 % simple_grep '^(\w+)\1$' /usr/dict/words # isn't this easier?
854 For all of these quantifiers, perl will try to match as much of the
855 string as possible, while still allowing the regexp to succeed. Thus
856 with C</a?.../>, perl will first try to match the regexp with the C<a>
857 present; if that fails, perl will try to match the regexp without the
858 C<a> present. For the quantifier C<*>, we get the following:
860 $x = "the cat in the hat";
861 $x =~ /^(.*)(cat)(.*)$/; # matches,
866 Which is what we might expect, the match finds the only C<cat> in the
867 string and locks onto it. Consider, however, this regexp:
869 $x =~ /^(.*)(at)(.*)$/; # matches,
870 # $1 = 'the cat in the h'
872 # $3 = '' (0 matches)
874 One might initially guess that perl would find the C<at> in C<cat> and
875 stop there, but that wouldn't give the longest possible string to the
876 first quantifier C<.*>. Instead, the first quantifier C<.*> grabs as
877 much of the string as possible while still having the regexp match. In
878 this example, that means having the C<at> sequence with the final C<at>
879 in the string. The other important principle illustrated here is that
880 when there are two or more elements in a regexp, the I<leftmost>
881 quantifier, if there is one, gets to grab as much the string as
882 possible, leaving the rest of the regexp to fight over scraps. Thus in
883 our example, the first quantifier C<.*> grabs most of the string, while
884 the second quantifier C<.*> gets the empty string. Quantifiers that
885 grab as much of the string as possible are called B<maximal match> or
886 B<greedy> quantifiers.
888 When a regexp can match a string in several different ways, we can use
889 the principles above to predict which way the regexp will match:
895 Principle 0: Taken as a whole, any regexp will be matched at the
896 earliest possible position in the string.
900 Principle 1: In an alternation C<a|b|c...>, the leftmost alternative
901 that allows a match for the whole regexp will be the one used.
905 Principle 2: The maximal matching quantifiers C<?>, C<*>, C<+> and
906 C<{n,m}> will in general match as much of the string as possible while
907 still allowing the whole regexp to match.
911 Principle 3: If there are two or more elements in a regexp, the
912 leftmost greedy quantifier, if any, will match as much of the string
913 as possible while still allowing the whole regexp to match. The next
914 leftmost greedy quantifier, if any, will try to match as much of the
915 string remaining available to it as possible, while still allowing the
916 whole regexp to match. And so on, until all the regexp elements are
921 As we have seen above, Principle 0 overrides the others - the regexp
922 will be matched as early as possible, with the other principles
923 determining how the regexp matches at that earliest character
926 Here is an example of these principles in action:
928 $x = "The programming republic of Perl";
929 $x =~ /^(.+)(e|r)(.*)$/; # matches,
930 # $1 = 'The programming republic of Pe'
934 This regexp matches at the earliest string position, C<'T'>. One
935 might think that C<e>, being leftmost in the alternation, would be
936 matched, but C<r> produces the longest string in the first quantifier.
938 $x =~ /(m{1,2})(.*)$/; # matches,
940 # $2 = 'ing republic of Perl'
942 Here, The earliest possible match is at the first C<'m'> in
943 C<programming>. C<m{1,2}> is the first quantifier, so it gets to match
946 $x =~ /.*(m{1,2})(.*)$/; # matches,
948 # $2 = 'ing republic of Perl'
950 Here, the regexp matches at the start of the string. The first
951 quantifier C<.*> grabs as much as possible, leaving just a single
952 C<'m'> for the second quantifier C<m{1,2}>.
954 $x =~ /(.?)(m{1,2})(.*)$/; # matches,
957 # $3 = 'ing republic of Perl'
959 Here, C<.?> eats its maximal one character at the earliest possible
960 position in the string, C<'a'> in C<programming>, leaving C<m{1,2}>
961 the opportunity to match both C<m>'s. Finally,
963 "aXXXb" =~ /(X*)/; # matches with $1 = ''
965 because it can match zero copies of C<'X'> at the beginning of the
966 string. If you definitely want to match at least one C<'X'>, use
969 Sometimes greed is not good. At times, we would like quantifiers to
970 match a I<minimal> piece of string, rather than a maximal piece. For
971 this purpose, Larry Wall created the S<B<minimal match> > or
972 B<non-greedy> quantifiers C<??>,C<*?>, C<+?>, and C<{}?>. These are
973 the usual quantifiers with a C<?> appended to them. They have the
980 C<a??> = match 'a' 0 or 1 times. Try 0 first, then 1.
984 C<a*?> = match 'a' 0 or more times, i.e., any number of times,
985 but as few times as possible
989 C<a+?> = match 'a' 1 or more times, i.e., at least once, but
990 as few times as possible
994 C<a{n,m}?> = match at least C<n> times, not more than C<m>
995 times, as few times as possible
999 C<a{n,}?> = match at least C<n> times, but as few times as
1004 C<a{n}?> = match exactly C<n> times. Because we match exactly
1005 C<n> times, C<a{n}?> is equivalent to C<a{n}> and is just there for
1006 notational consistency.
1010 Let's look at the example above, but with minimal quantifiers:
1012 $x = "The programming republic of Perl";
1013 $x =~ /^(.+?)(e|r)(.*)$/; # matches,
1016 # $3 = ' programming republic of Perl'
1018 The minimal string that will allow both the start of the string C<^>
1019 and the alternation to match is C<Th>, with the alternation C<e|r>
1020 matching C<e>. The second quantifier C<.*> is free to gobble up the
1023 $x =~ /(m{1,2}?)(.*?)$/; # matches,
1025 # $2 = 'ming republic of Perl'
1027 The first string position that this regexp can match is at the first
1028 C<'m'> in C<programming>. At this position, the minimal C<m{1,2}?>
1029 matches just one C<'m'>. Although the second quantifier C<.*?> would
1030 prefer to match no characters, it is constrained by the end-of-string
1031 anchor C<$> to match the rest of the string.
1033 $x =~ /(.*?)(m{1,2}?)(.*)$/; # matches,
1036 # $3 = 'ming republic of Perl'
1038 In this regexp, you might expect the first minimal quantifier C<.*?>
1039 to match the empty string, because it is not constrained by a C<^>
1040 anchor to match the beginning of the word. Principle 0 applies here,
1041 however. Because it is possible for the whole regexp to match at the
1042 start of the string, it I<will> match at the start of the string. Thus
1043 the first quantifier has to match everything up to the first C<m>. The
1044 second minimal quantifier matches just one C<m> and the third
1045 quantifier matches the rest of the string.
1047 $x =~ /(.??)(m{1,2})(.*)$/; # matches,
1050 # $3 = 'ing republic of Perl'
1052 Just as in the previous regexp, the first quantifier C<.??> can match
1053 earliest at position C<'a'>, so it does. The second quantifier is
1054 greedy, so it matches C<mm>, and the third matches the rest of the
1057 We can modify principle 3 above to take into account non-greedy
1064 Principle 3: If there are two or more elements in a regexp, the
1065 leftmost greedy (non-greedy) quantifier, if any, will match as much
1066 (little) of the string as possible while still allowing the whole
1067 regexp to match. The next leftmost greedy (non-greedy) quantifier, if
1068 any, will try to match as much (little) of the string remaining
1069 available to it as possible, while still allowing the whole regexp to
1070 match. And so on, until all the regexp elements are satisfied.
1074 Just like alternation, quantifiers are also susceptible to
1075 backtracking. Here is a step-by-step analysis of the example
1077 $x = "the cat in the hat";
1078 $x =~ /^(.*)(at)(.*)$/; # matches,
1079 # $1 = 'the cat in the h'
1081 # $3 = '' (0 matches)
1087 Start with the first letter in the string 't'.
1091 The first quantifier '.*' starts out by matching the whole
1092 string 'the cat in the hat'.
1096 'a' in the regexp element 'at' doesn't match the end of the
1097 string. Backtrack one character.
1101 'a' in the regexp element 'at' still doesn't match the last
1102 letter of the string 't', so backtrack one more character.
1106 Now we can match the 'a' and the 't'.
1110 Move on to the third element '.*'. Since we are at the end of
1111 the string and '.*' can match 0 times, assign it the empty string.
1119 Most of the time, all this moving forward and backtracking happens
1120 quickly and searching is fast. There are some pathological regexps,
1121 however, whose execution time exponentially grows with the size of the
1122 string. A typical structure that blows up in your face is of the form
1126 The problem is the nested indeterminate quantifiers. There are many
1127 different ways of partitioning a string of length n between the C<+>
1128 and C<*>: one repetition with C<b+> of length n, two repetitions with
1129 the first C<b+> length k and the second with length n-k, m repetitions
1130 whose bits add up to length n, etc. In fact there are an exponential
1131 number of ways to partition a string as a function of length. A
1132 regexp may get lucky and match early in the process, but if there is
1133 no match, perl will try I<every> possibility before giving up. So be
1134 careful with nested C<*>'s, C<{n,m}>'s, and C<+>'s. The book
1135 I<Mastering regular expressions> by Jeffrey Friedl gives a wonderful
1136 discussion of this and other efficiency issues.
1138 =head2 Building a regexp
1140 At this point, we have all the basic regexp concepts covered, so let's
1141 give a more involved example of a regular expression. We will build a
1142 regexp that matches numbers.
1144 The first task in building a regexp is to decide what we want to match
1145 and what we want to exclude. In our case, we want to match both
1146 integers and floating point numbers and we want to reject any string
1147 that isn't a number.
1149 The next task is to break the problem down into smaller problems that
1150 are easily converted into a regexp.
1152 The simplest case is integers. These consist of a sequence of digits,
1153 with an optional sign in front. The digits we can represent with
1154 C<\d+> and the sign can be matched with C<[+-]>. Thus the integer
1157 /[+-]?\d+/; # matches integers
1159 A floating point number potentially has a sign, an integral part, a
1160 decimal point, a fractional part, and an exponent. One or more of these
1161 parts is optional, so we need to check out the different
1162 possibilities. Floating point numbers which are in proper form include
1163 123., 0.345, .34, -1e6, and 25.4E-72. As with integers, the sign out
1164 front is completely optional and can be matched by C<[+-]?>. We can
1165 see that if there is no exponent, floating point numbers must have a
1166 decimal point, otherwise they are integers. We might be tempted to
1167 model these with C<\d*\.\d*>, but this would also match just a single
1168 decimal point, which is not a number. So the three cases of floating
1169 point number sans exponent are
1171 /[+-]?\d+\./; # 1., 321., etc.
1172 /[+-]?\.\d+/; # .1, .234, etc.
1173 /[+-]?\d+\.\d+/; # 1.0, 30.56, etc.
1175 These can be combined into a single regexp with a three-way alternation:
1177 /[+-]?(\d+\.\d+|\d+\.|\.\d+)/; # floating point, no exponent
1179 In this alternation, it is important to put C<'\d+\.\d+'> before
1180 C<'\d+\.'>. If C<'\d+\.'> were first, the regexp would happily match that
1181 and ignore the fractional part of the number.
1183 Now consider floating point numbers with exponents. The key
1184 observation here is that I<both> integers and numbers with decimal
1185 points are allowed in front of an exponent. Then exponents, like the
1186 overall sign, are independent of whether we are matching numbers with
1187 or without decimal points, and can be 'decoupled' from the
1188 mantissa. The overall form of the regexp now becomes clear:
1190 /^(optional sign)(integer | f.p. mantissa)(optional exponent)$/;
1192 The exponent is an C<e> or C<E>, followed by an integer. So the
1195 /[eE][+-]?\d+/; # exponent
1197 Putting all the parts together, we get a regexp that matches numbers:
1199 /^[+-]?(\d+\.\d+|\d+\.|\.\d+|\d+)([eE][+-]?\d+)?$/; # Ta da!
1201 Long regexps like this may impress your friends, but can be hard to
1202 decipher. In complex situations like this, the C<//x> modifier for a
1203 match is invaluable. It allows one to put nearly arbitrary whitespace
1204 and comments into a regexp without affecting their meaning. Using it,
1205 we can rewrite our 'extended' regexp in the more pleasing form
1208 [+-]? # first, match an optional sign
1209 ( # then match integers or f.p. mantissas:
1210 \d+\.\d+ # mantissa of the form a.b
1211 |\d+\. # mantissa of the form a.
1212 |\.\d+ # mantissa of the form .b
1213 |\d+ # integer of the form a
1215 ([eE][+-]?\d+)? # finally, optionally match an exponent
1218 If whitespace is mostly irrelevant, how does one include space
1219 characters in an extended regexp? The answer is to backslash it
1220 S<C<'\ '> > or put it in a character class S<C<[ ]> >. The same thing
1221 goes for pound signs, use C<\#> or C<[#]>. For instance, Perl allows
1222 a space between the sign and the mantissa/integer, and we could add
1223 this to our regexp as follows:
1226 [+-]?\ * # first, match an optional sign *and space*
1227 ( # then match integers or f.p. mantissas:
1228 \d+\.\d+ # mantissa of the form a.b
1229 |\d+\. # mantissa of the form a.
1230 |\.\d+ # mantissa of the form .b
1231 |\d+ # integer of the form a
1233 ([eE][+-]?\d+)? # finally, optionally match an exponent
1236 In this form, it is easier to see a way to simplify the
1237 alternation. Alternatives 1, 2, and 4 all start with C<\d+>, so it
1238 could be factored out:
1241 [+-]?\ * # first, match an optional sign
1242 ( # then match integers or f.p. mantissas:
1243 \d+ # start out with a ...
1245 \.\d* # mantissa of the form a.b or a.
1246 )? # ? takes care of integers of the form a
1247 |\.\d+ # mantissa of the form .b
1249 ([eE][+-]?\d+)? # finally, optionally match an exponent
1252 or written in the compact form,
1254 /^[+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/;
1256 This is our final regexp. To recap, we built a regexp by
1262 specifying the task in detail,
1266 breaking down the problem into smaller parts,
1270 translating the small parts into regexps,
1274 combining the regexps,
1278 and optimizing the final combined regexp.
1282 These are also the typical steps involved in writing a computer
1283 program. This makes perfect sense, because regular expressions are
1284 essentially programs written a little computer language that specifies
1287 =head2 Using regular expressions in Perl
1289 The last topic of Part 1 briefly covers how regexps are used in Perl
1290 programs. Where do they fit into Perl syntax?
1292 We have already introduced the matching operator in its default
1293 C</regexp/> and arbitrary delimiter C<m!regexp!> forms. We have used
1294 the binding operator C<=~> and its negation C<!~> to test for string
1295 matches. Associated with the matching operator, we have discussed the
1296 single line C<//s>, multi-line C<//m>, case-insensitive C<//i> and
1297 extended C<//x> modifiers.
1299 There are a few more things you might want to know about matching
1300 operators. First, we pointed out earlier that variables in regexps are
1301 substituted before the regexp is evaluated:
1305 print if /$pattern/;
1308 This will print any lines containing the word C<Seuss>. It is not as
1309 efficient as it could be, however, because perl has to re-evaluate
1310 C<$pattern> each time through the loop. If C<$pattern> won't be
1311 changing over the lifetime of the script, we can add the C<//o>
1312 modifier, which directs perl to only perform variable substitutions
1316 # Improved simple_grep
1319 print if /$regexp/o; # a good deal faster
1322 If you change C<$pattern> after the first substitution happens, perl
1323 will ignore it. If you don't want any substitutions at all, use the
1324 special delimiter C<m''>:
1328 print if m'$pattern'; # matches '$pattern', not 'Seuss'
1331 C<m''> acts like single quotes on a regexp; all other C<m> delimiters
1332 act like double quotes. If the regexp evaluates to the empty string,
1333 the regexp in the I<last successful match> is used instead. So we have
1335 "dog" =~ /d/; # 'd' matches
1336 "dogbert =~ //; # this matches the 'd' regexp used before
1338 The final two modifiers C<//g> and C<//c> concern multiple matches.
1339 The modifier C<//g> stands for global matching and allows the
1340 matching operator to match within a string as many times as possible.
1341 In scalar context, successive invocations against a string will have
1342 `C<//g> jump from match to match, keeping track of position in the
1343 string as it goes along. You can get or set the position with the
1346 The use of C<//g> is shown in the following example. Suppose we have
1347 a string that consists of words separated by spaces. If we know how
1348 many words there are in advance, we could extract the words using
1351 $x = "cat dog house"; # 3 words
1352 $x =~ /^\s*(\w+)\s+(\w+)\s+(\w+)\s*$/; # matches,
1357 But what if we had an indeterminate number of words? This is the sort
1358 of task C<//g> was made for. To extract all words, form the simple
1359 regexp C<(\w+)> and loop over all matches with C</(\w+)/g>:
1361 while ($x =~ /(\w+)/g) {
1362 print "Word is $1, ends at position ", pos $x, "\n";
1367 Word is cat, ends at position 3
1368 Word is dog, ends at position 7
1369 Word is house, ends at position 13
1371 A failed match or changing the target string resets the position. If
1372 you don't want the position reset after failure to match, add the
1373 C<//c>, as in C</regexp/gc>. The current position in the string is
1374 associated with the string, not the regexp. This means that different
1375 strings have different positions and their respective positions can be
1376 set or read independently.
1378 In list context, C<//g> returns a list of matched groupings, or if
1379 there are no groupings, a list of matches to the whole regexp. So if
1380 we wanted just the words, we could use
1382 @words = ($x =~ /(\w+)/g); # matches,
1385 # $word[2] = 'house'
1387 Closely associated with the C<//g> modifier is the C<\G> anchor. The
1388 C<\G> anchor matches at the point where the previous C<//g> match left
1389 off. C<\G> allows us to easily do context-sensitive matching:
1391 $metric = 1; # use metric units
1393 $x = <FILE>; # read in measurement
1394 $x =~ /^([+-]?\d+)\s*/g; # get magnitude
1396 if ($metric) { # error checking
1397 print "Units error!" unless $x =~ /\Gkg\./g;
1400 print "Units error!" unless $x =~ /\Glbs\./g;
1402 $x =~ /\G\s+(widget|sprocket)/g; # continue processing
1404 The combination of C<//g> and C<\G> allows us to process the string a
1405 bit at a time and use arbitrary Perl logic to decide what to do next.
1406 Currently, the C<\G> anchor is only fully supported when used to anchor
1407 to the start of the pattern.
1409 C<\G> is also invaluable in processing fixed length records with
1410 regexps. Suppose we have a snippet of coding region DNA, encoded as
1411 base pair letters C<ATCGTTGAAT...> and we want to find all the stop
1412 codons C<TGA>. In a coding region, codons are 3-letter sequences, so
1413 we can think of the DNA snippet as a sequence of 3-letter records. The
1416 # expanded, this is "ATC GTT GAA TGC AAA TGA CAT GAC"
1417 $dna = "ATCGTTGAATGCAAATGACATGAC";
1420 doesn't work; it may match a C<TGA>, but there is no guarantee that
1421 the match is aligned with codon boundaries, e.g., the substring
1422 S<C<GTT GAA> > gives a match. A better solution is
1424 while ($dna =~ /(\w\w\w)*?TGA/g) { # note the minimal *?
1425 print "Got a TGA stop codon at position ", pos $dna, "\n";
1430 Got a TGA stop codon at position 18
1431 Got a TGA stop codon at position 23
1433 Position 18 is good, but position 23 is bogus. What happened?
1435 The answer is that our regexp works well until we get past the last
1436 real match. Then the regexp will fail to match a synchronized C<TGA>
1437 and start stepping ahead one character position at a time, not what we
1438 want. The solution is to use C<\G> to anchor the match to the codon
1441 while ($dna =~ /\G(\w\w\w)*?TGA/g) {
1442 print "Got a TGA stop codon at position ", pos $dna, "\n";
1447 Got a TGA stop codon at position 18
1449 which is the correct answer. This example illustrates that it is
1450 important not only to match what is desired, but to reject what is not
1453 B<search and replace>
1455 Regular expressions also play a big role in B<search and replace>
1456 operations in Perl. Search and replace is accomplished with the
1457 C<s///> operator. The general form is
1458 C<s/regexp/replacement/modifiers>, with everything we know about
1459 regexps and modifiers applying in this case as well. The
1460 C<replacement> is a Perl double quoted string that replaces in the
1461 string whatever is matched with the C<regexp>. The operator C<=~> is
1462 also used here to associate a string with C<s///>. If matching
1463 against C<$_>, the S<C<$_ =~> > can be dropped. If there is a match,
1464 C<s///> returns the number of substitutions made, otherwise it returns
1465 false. Here are a few examples:
1467 $x = "Time to feed the cat!";
1468 $x =~ s/cat/hacker/; # $x contains "Time to feed the hacker!"
1469 if ($x =~ s/^(Time.*hacker)!$/$1 now!/) {
1470 $more_insistent = 1;
1472 $y = "'quoted words'";
1473 $y =~ s/^'(.*)'$/$1/; # strip single quotes,
1474 # $y contains "quoted words"
1476 In the last example, the whole string was matched, but only the part
1477 inside the single quotes was grouped. With the C<s///> operator, the
1478 matched variables C<$1>, C<$2>, etc. are immediately available for use
1479 in the replacement expression, so we use C<$1> to replace the quoted
1480 string with just what was quoted. With the global modifier, C<s///g>
1481 will search and replace all occurrences of the regexp in the string:
1483 $x = "I batted 4 for 4";
1484 $x =~ s/4/four/; # doesn't do it all:
1485 # $x contains "I batted four for 4"
1486 $x = "I batted 4 for 4";
1487 $x =~ s/4/four/g; # does it all:
1488 # $x contains "I batted four for four"
1490 If you prefer 'regex' over 'regexp' in this tutorial, you could use
1491 the following program to replace it:
1493 % cat > simple_replace
1496 $replacement = shift;
1498 s/$regexp/$replacement/go;
1503 % simple_replace regexp regex perlretut.pod
1505 In C<simple_replace> we used the C<s///g> modifier to replace all
1506 occurrences of the regexp on each line and the C<s///o> modifier to
1507 compile the regexp only once. As with C<simple_grep>, both the
1508 C<print> and the C<s/$regexp/$replacement/go> use C<$_> implicitly.
1510 A modifier available specifically to search and replace is the
1511 C<s///e> evaluation modifier. C<s///e> wraps an C<eval{...}> around
1512 the replacement string and the evaluated result is substituted for the
1513 matched substring. C<s///e> is useful if you need to do a bit of
1514 computation in the process of replacing text. This example counts
1515 character frequencies in a line:
1517 $x = "Bill the cat";
1518 $x =~ s/(.)/$chars{$1}++;$1/eg; # final $1 replaces char with itself
1519 print "frequency of '$_' is $chars{$_}\n"
1520 foreach (sort {$chars{$b} <=> $chars{$a}} keys %chars);
1524 frequency of ' ' is 2
1525 frequency of 't' is 2
1526 frequency of 'l' is 2
1527 frequency of 'B' is 1
1528 frequency of 'c' is 1
1529 frequency of 'e' is 1
1530 frequency of 'h' is 1
1531 frequency of 'i' is 1
1532 frequency of 'a' is 1
1534 As with the match C<m//> operator, C<s///> can use other delimiters,
1535 such as C<s!!!> and C<s{}{}>, and even C<s{}//>. If single quotes are
1536 used C<s'''>, then the regexp and replacement are treated as single
1537 quoted strings and there are no substitutions. C<s///> in list context
1538 returns the same thing as in scalar context, i.e., the number of
1541 B<The split operator>
1543 The B<C<split> > function can also optionally use a matching operator
1544 C<m//> to split a string. C<split /regexp/, string, limit> splits
1545 C<string> into a list of substrings and returns that list. The regexp
1546 is used to match the character sequence that the C<string> is split
1547 with respect to. The C<limit>, if present, constrains splitting into
1548 no more than C<limit> number of strings. For example, to split a
1549 string into words, use
1551 $x = "Calvin and Hobbes";
1552 @words = split /\s+/, $x; # $word[0] = 'Calvin'
1554 # $word[2] = 'Hobbes'
1556 If the empty regexp C<//> is used, the regexp always matches and
1557 the string is split into individual characters. If the regexp has
1558 groupings, then list produced contains the matched substrings from the
1559 groupings as well. For instance,
1561 $x = "/usr/bin/perl";
1562 @dirs = split m!/!, $x; # $dirs[0] = ''
1566 @parts = split m!(/)!, $x; # $parts[0] = ''
1572 # $parts[6] = 'perl'
1574 Since the first character of $x matched the regexp, C<split> prepended
1575 an empty initial element to the list.
1577 If you have read this far, congratulations! You now have all the basic
1578 tools needed to use regular expressions to solve a wide range of text
1579 processing problems. If this is your first time through the tutorial,
1580 why not stop here and play around with regexps a while... S<Part 2>
1581 concerns the more esoteric aspects of regular expressions and those
1582 concepts certainly aren't needed right at the start.
1584 =head1 Part 2: Power tools
1586 OK, you know the basics of regexps and you want to know more. If
1587 matching regular expressions is analogous to a walk in the woods, then
1588 the tools discussed in Part 1 are analogous to topo maps and a
1589 compass, basic tools we use all the time. Most of the tools in part 2
1590 are analogous to flare guns and satellite phones. They aren't used
1591 too often on a hike, but when we are stuck, they can be invaluable.
1593 What follows are the more advanced, less used, or sometimes esoteric
1594 capabilities of perl regexps. In Part 2, we will assume you are
1595 comfortable with the basics and concentrate on the new features.
1597 =head2 More on characters, strings, and character classes
1599 There are a number of escape sequences and character classes that we
1600 haven't covered yet.
1602 There are several escape sequences that convert characters or strings
1603 between upper and lower case. C<\l> and C<\u> convert the next
1604 character to lower or upper case, respectively:
1607 $string =~ /\u$x/; # matches 'Perl' in $string
1608 $x = "M(rs?|s)\\."; # note the double backslash
1609 $string =~ /\l$x/; # matches 'mr.', 'mrs.', and 'ms.',
1611 C<\L> and C<\U> converts a whole substring, delimited by C<\L> or
1612 C<\U> and C<\E>, to lower or upper case:
1614 $x = "This word is in lower case:\L SHOUT\E";
1615 $x =~ /shout/; # matches
1616 $x = "I STILL KEYPUNCH CARDS FOR MY 360"
1617 $x =~ /\Ukeypunch/; # matches punch card string
1619 If there is no C<\E>, case is converted until the end of the
1620 string. The regexps C<\L\u$word> or C<\u\L$word> convert the first
1621 character of C<$word> to uppercase and the rest of the characters to
1624 Control characters can be escaped with C<\c>, so that a control-Z
1625 character would be matched with C<\cZ>. The escape sequence
1626 C<\Q>...C<\E> quotes, or protects most non-alphabetic characters. For
1629 $x = "\QThat !^*&%~& cat!";
1630 $x =~ /\Q!^*&%~&\E/; # check for rough language
1632 It does not protect C<$> or C<@>, so that variables can still be
1635 With the advent of 5.6.0, perl regexps can handle more than just the
1636 standard ASCII character set. Perl now supports B<Unicode>, a standard
1637 for encoding the character sets from many of the world's written
1638 languages. Unicode does this by allowing characters to be more than
1639 one byte wide. Perl uses the UTF-8 encoding, in which ASCII characters
1640 are still encoded as one byte, but characters greater than C<chr(127)>
1641 may be stored as two or more bytes.
1643 What does this mean for regexps? Well, regexp users don't need to know
1644 much about perl's internal representation of strings. But they do need
1645 to know 1) how to represent Unicode characters in a regexp and 2) when
1646 a matching operation will treat the string to be searched as a
1647 sequence of bytes (the old way) or as a sequence of Unicode characters
1648 (the new way). The answer to 1) is that Unicode characters greater
1649 than C<chr(127)> may be represented using the C<\x{hex}> notation,
1650 with C<hex> a hexadecimal integer:
1652 /\x{263a}/; # match a Unicode smiley face :)
1654 Unicode characters in the range of 128-255 use two hexadecimal digits
1655 with braces: C<\x{ab}>. Note that this is different than C<\xab>,
1656 which is just a hexadecimal byte with no Unicode significance.
1658 B<NOTE>: in Perl 5.6.0 it used to be that one needed to say C<use
1659 utf8> to use any Unicode features. This is no more the case: for
1660 almost all Unicode processing, the explicit C<utf8> pragma is not
1661 needed. (The only case where it matters is if your Perl script is in
1662 Unicode and encoded in UTF-8, then an explicit C<use utf8> is needed.)
1664 Figuring out the hexadecimal sequence of a Unicode character you want
1665 or deciphering someone else's hexadecimal Unicode regexp is about as
1666 much fun as programming in machine code. So another way to specify
1667 Unicode characters is to use the S<B<named character> > escape
1668 sequence C<\N{name}>. C<name> is a name for the Unicode character, as
1669 specified in the Unicode standard. For instance, if we wanted to
1670 represent or match the astrological sign for the planet Mercury, we
1673 use charnames ":full"; # use named chars with Unicode full names
1674 $x = "abc\N{MERCURY}def";
1675 $x =~ /\N{MERCURY}/; # matches
1677 One can also use short names or restrict names to a certain alphabet:
1679 use charnames ':full';
1680 print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n";
1682 use charnames ":short";
1683 print "\N{greek:Sigma} is an upper-case sigma.\n";
1685 use charnames qw(greek);
1686 print "\N{sigma} is Greek sigma\n";
1688 A list of full names is found in the file Names.txt in the
1689 lib/perl5/5.X.X/unicore directory.
1691 The answer to requirement 2), as of 5.6.0, is that if a regexp
1692 contains Unicode characters, the string is searched as a sequence of
1693 Unicode characters. Otherwise, the string is searched as a sequence of
1694 bytes. If the string is being searched as a sequence of Unicode
1695 characters, but matching a single byte is required, we can use the C<\C>
1696 escape sequence. C<\C> is a character class akin to C<.> except that
1697 it matches I<any> byte 0-255. So
1699 use charnames ":full"; # use named chars with Unicode full names
1701 $x =~ /\C/; # matches 'a', eats one byte
1703 $x =~ /\C/; # doesn't match, no bytes to match
1704 $x = "\N{MERCURY}"; # two-byte Unicode character
1705 $x =~ /\C/; # matches, but dangerous!
1707 The last regexp matches, but is dangerous because the string
1708 I<character> position is no longer synchronized to the string I<byte>
1709 position. This generates the warning 'Malformed UTF-8
1710 character'. The C<\C> is best used for matching the binary data in strings
1711 with binary data intermixed with Unicode characters.
1713 Let us now discuss the rest of the character classes. Just as with
1714 Unicode characters, there are named Unicode character classes
1715 represented by the C<\p{name}> escape sequence. Closely associated is
1716 the C<\P{name}> character class, which is the negation of the
1717 C<\p{name}> class. For example, to match lower and uppercase
1720 use charnames ":full"; # use named chars with Unicode full names
1722 $x =~ /^\p{IsUpper}/; # matches, uppercase char class
1723 $x =~ /^\P{IsUpper}/; # doesn't match, char class sans uppercase
1724 $x =~ /^\p{IsLower}/; # doesn't match, lowercase char class
1725 $x =~ /^\P{IsLower}/; # matches, char class sans lowercase
1727 Here is the association between some Perl named classes and the
1728 traditional Unicode classes:
1730 Perl class name Unicode class name or regular expression
1734 IsASCII $code <= 127
1736 IsBlank $code =~ /^(0020|0009)$/ || /^Z[^lp]/
1738 IsGraph /^([LMNPS]|Co)/
1740 IsPrint /^([LMNPS]|Co|Zs)/
1742 IsSpace /^Z/ || ($code =~ /^(0009|000A|000B|000C|000D)$/
1743 IsSpacePerl /^Z/ || ($code =~ /^(0009|000A|000C|000D|0085|2028|2029)$/
1745 IsWord /^[LMN]/ || $code eq "005F"
1746 IsXDigit $code =~ /^00(3[0-9]|[46][1-6])$/
1748 You can also use the official Unicode class names with the C<\p> and
1749 C<\P>, like C<\p{L}> for Unicode 'letters', or C<\p{Lu}> for uppercase
1750 letters, or C<\P{Nd}> for non-digits. If a C<name> is just one
1751 letter, the braces can be dropped. For instance, C<\pM> is the
1752 character class of Unicode 'marks', for example accent marks.
1753 For the full list see L<perlunicode>.
1755 The Unicode has also been separated into various sets of charaters
1756 which you can test with C<\p{In...}> (in) and C<\P{In...}> (not in),
1757 for example C<\p{Latin}>, C<\p{Greek}>, or C<\P{Katakana}>.
1758 For the full list see L<perlunicode>.
1760 C<\X> is an abbreviation for a character class sequence that includes
1761 the Unicode 'combining character sequences'. A 'combining character
1762 sequence' is a base character followed by any number of combining
1763 characters. An example of a combining character is an accent. Using
1764 the Unicode full names, e.g., S<C<A + COMBINING RING> > is a combining
1765 character sequence with base character C<A> and combining character
1766 S<C<COMBINING RING> >, which translates in Danish to A with the circle
1767 atop it, as in the word Angstrom. C<\X> is equivalent to C<\PM\pM*}>,
1768 i.e., a non-mark followed by one or more marks.
1770 For the full and latest information about Unicode see the latest
1771 Unicode standard, or the Unicode Consortium's website http://www.unicode.org/
1773 As if all those classes weren't enough, Perl also defines POSIX style
1774 character classes. These have the form C<[:name:]>, with C<name> the
1775 name of the POSIX class. The POSIX classes are C<alpha>, C<alnum>,
1776 C<ascii>, C<cntrl>, C<digit>, C<graph>, C<lower>, C<print>, C<punct>,
1777 C<space>, C<upper>, and C<xdigit>, and two extensions, C<word> (a Perl
1778 extension to match C<\w>), and C<blank> (a GNU extension). If C<utf8>
1779 is being used, then these classes are defined the same as their
1780 corresponding perl Unicode classes: C<[:upper:]> is the same as
1781 C<\p{IsUpper}>, etc. The POSIX character classes, however, don't
1782 require using C<utf8>. The C<[:digit:]>, C<[:word:]>, and
1783 C<[:space:]> correspond to the familiar C<\d>, C<\w>, and C<\s>
1784 character classes. To negate a POSIX class, put a C<^> in front of
1785 the name, so that, e.g., C<[:^digit:]> corresponds to C<\D> and under
1786 C<utf8>, C<\P{IsDigit}>. The Unicode and POSIX character classes can
1787 be used just like C<\d>, with the exception that POSIX character
1788 classes can only be used inside of a character class:
1790 /\s+[abc[:digit:]xyz]\s*/; # match a,b,c,x,y,z, or a digit
1791 /^=item\s[[:digit:]]/; # match '=item',
1792 # followed by a space and a digit
1793 use charnames ":full";
1794 /\s+[abc\p{IsDigit}xyz]\s+/; # match a,b,c,x,y,z, or a digit
1795 /^=item\s\p{IsDigit}/; # match '=item',
1796 # followed by a space and a digit
1798 Whew! That is all the rest of the characters and character classes.
1800 =head2 Compiling and saving regular expressions
1802 In Part 1 we discussed the C<//o> modifier, which compiles a regexp
1803 just once. This suggests that a compiled regexp is some data structure
1804 that can be stored once and used again and again. The regexp quote
1805 C<qr//> does exactly that: C<qr/string/> compiles the C<string> as a
1806 regexp and transforms the result into a form that can be assigned to a
1809 $reg = qr/foo+bar?/; # reg contains a compiled regexp
1811 Then C<$reg> can be used as a regexp:
1814 $x =~ $reg; # matches, just like /foo+bar?/
1815 $x =~ /$reg/; # same thing, alternate form
1817 C<$reg> can also be interpolated into a larger regexp:
1819 $x =~ /(abc)?$reg/; # still matches
1821 As with the matching operator, the regexp quote can use different
1822 delimiters, e.g., C<qr!!>, C<qr{}> and C<qr~~>. The single quote
1823 delimiters C<qr''> prevent any interpolation from taking place.
1825 Pre-compiled regexps are useful for creating dynamic matches that
1826 don't need to be recompiled each time they are encountered. Using
1827 pre-compiled regexps, C<simple_grep> program can be expanded into a
1828 program that matches multiple patterns:
1832 # multi_grep - match any of <number> regexps
1833 # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ...
1836 $regexp[$_] = shift foreach (0..$number-1);
1837 @compiled = map qr/$_/, @regexp;
1838 while ($line = <>) {
1839 foreach $pattern (@compiled) {
1840 if ($line =~ /$pattern/) {
1842 last; # we matched, so move onto the next line
1848 % multi_grep 2 last for multi_grep
1849 $regexp[$_] = shift foreach (0..$number-1);
1850 foreach $pattern (@compiled) {
1853 Storing pre-compiled regexps in an array C<@compiled> allows us to
1854 simply loop through the regexps without any recompilation, thus gaining
1855 flexibility without sacrificing speed.
1857 =head2 Embedding comments and modifiers in a regular expression
1859 Starting with this section, we will be discussing Perl's set of
1860 B<extended patterns>. These are extensions to the traditional regular
1861 expression syntax that provide powerful new tools for pattern
1862 matching. We have already seen extensions in the form of the minimal
1863 matching constructs C<??>, C<*?>, C<+?>, C<{n,m}?>, and C<{n,}?>. The
1864 rest of the extensions below have the form C<(?char...)>, where the
1865 C<char> is a character that determines the type of extension.
1867 The first extension is an embedded comment C<(?#text)>. This embeds a
1868 comment into the regular expression without affecting its meaning. The
1869 comment should not have any closing parentheses in the text. An
1872 /(?# Match an integer:)[+-]?\d+/;
1874 This style of commenting has been largely superseded by the raw,
1875 freeform commenting that is allowed with the C<//x> modifier.
1877 The modifiers C<//i>, C<//m>, C<//s>, and C<//x> can also embedded in
1878 a regexp using C<(?i)>, C<(?m)>, C<(?s)>, and C<(?x)>. For instance,
1880 /(?i)yes/; # match 'yes' case insensitively
1881 /yes/i; # same thing
1882 /(?x)( # freeform version of an integer regexp
1883 [+-]? # match an optional sign
1884 \d+ # match a sequence of digits
1888 Embedded modifiers can have two important advantages over the usual
1889 modifiers. Embedded modifiers allow a custom set of modifiers to
1890 I<each> regexp pattern. This is great for matching an array of regexps
1891 that must have different modifiers:
1893 $pattern[0] = '(?i)doctor';
1894 $pattern[1] = 'Johnson';
1897 foreach $patt (@pattern) {
1902 The second advantage is that embedded modifiers only affect the regexp
1903 inside the group the embedded modifier is contained in. So grouping
1904 can be used to localize the modifier's effects:
1906 /Answer: ((?i)yes)/; # matches 'Answer: yes', 'Answer: YES', etc.
1908 Embedded modifiers can also turn off any modifiers already present
1909 by using, e.g., C<(?-i)>. Modifiers can also be combined into
1910 a single expression, e.g., C<(?s-i)> turns on single line mode and
1911 turns off case insensitivity.
1913 =head2 Non-capturing groupings
1915 We noted in Part 1 that groupings C<()> had two distinct functions: 1)
1916 group regexp elements together as a single unit, and 2) extract, or
1917 capture, substrings that matched the regexp in the
1918 grouping. Non-capturing groupings, denoted by C<(?:regexp)>, allow the
1919 regexp to be treated as a single unit, but don't extract substrings or
1920 set matching variables C<$1>, etc. Both capturing and non-capturing
1921 groupings are allowed to co-exist in the same regexp. Because there is
1922 no extraction, non-capturing groupings are faster than capturing
1923 groupings. Non-capturing groupings are also handy for choosing exactly
1924 which parts of a regexp are to be extracted to matching variables:
1926 # match a number, $1-$4 are set, but we only want $1
1927 /([+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)/;
1929 # match a number faster , only $1 is set
1930 /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)/;
1932 # match a number, get $1 = whole number, $2 = exponent
1933 /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE]([+-]?\d+))?)/;
1935 Non-capturing groupings are also useful for removing nuisance
1936 elements gathered from a split operation:
1939 @num = split /(a|b)/, $x; # @num = ('12','a','34','b','5')
1940 @num = split /(?:a|b)/, $x; # @num = ('12','34','5')
1942 Non-capturing groupings may also have embedded modifiers:
1943 C<(?i-m:regexp)> is a non-capturing grouping that matches C<regexp>
1944 case insensitively and turns off multi-line mode.
1946 =head2 Looking ahead and looking behind
1948 This section concerns the lookahead and lookbehind assertions. First,
1949 a little background.
1951 In Perl regular expressions, most regexp elements 'eat up' a certain
1952 amount of string when they match. For instance, the regexp element
1953 C<[abc}]> eats up one character of the string when it matches, in the
1954 sense that perl moves to the next character position in the string
1955 after the match. There are some elements, however, that don't eat up
1956 characters (advance the character position) if they match. The examples
1957 we have seen so far are the anchors. The anchor C<^> matches the
1958 beginning of the line, but doesn't eat any characters. Similarly, the
1959 word boundary anchor C<\b> matches, e.g., if the character to the left
1960 is a word character and the character to the right is a non-word
1961 character, but it doesn't eat up any characters itself. Anchors are
1962 examples of 'zero-width assertions'. Zero-width, because they consume
1963 no characters, and assertions, because they test some property of the
1964 string. In the context of our walk in the woods analogy to regexp
1965 matching, most regexp elements move us along a trail, but anchors have
1966 us stop a moment and check our surroundings. If the local environment
1967 checks out, we can proceed forward. But if the local environment
1968 doesn't satisfy us, we must backtrack.
1970 Checking the environment entails either looking ahead on the trail,
1971 looking behind, or both. C<^> looks behind, to see that there are no
1972 characters before. C<$> looks ahead, to see that there are no
1973 characters after. C<\b> looks both ahead and behind, to see if the
1974 characters on either side differ in their 'word'-ness.
1976 The lookahead and lookbehind assertions are generalizations of the
1977 anchor concept. Lookahead and lookbehind are zero-width assertions
1978 that let us specify which characters we want to test for. The
1979 lookahead assertion is denoted by C<(?=regexp)> and the lookbehind
1980 assertion is denoted by C<< (?<=fixed-regexp) >>. Some examples are
1982 $x = "I catch the housecat 'Tom-cat' with catnip";
1983 $x =~ /cat(?=\s+)/; # matches 'cat' in 'housecat'
1984 @catwords = ($x =~ /(?<=\s)cat\w+/g); # matches,
1985 # $catwords[0] = 'catch'
1986 # $catwords[1] = 'catnip'
1987 $x =~ /\bcat\b/; # matches 'cat' in 'Tom-cat'
1988 $x =~ /(?<=\s)cat(?=\s)/; # doesn't match; no isolated 'cat' in
1991 Note that the parentheses in C<(?=regexp)> and C<< (?<=regexp) >> are
1992 non-capturing, since these are zero-width assertions. Thus in the
1993 second regexp, the substrings captured are those of the whole regexp
1994 itself. Lookahead C<(?=regexp)> can match arbitrary regexps, but
1995 lookbehind C<< (?<=fixed-regexp) >> only works for regexps of fixed
1996 width, i.e., a fixed number of characters long. Thus
1997 C<< (?<=(ab|bc)) >> is fine, but C<< (?<=(ab)*) >> is not. The
1998 negated versions of the lookahead and lookbehind assertions are
1999 denoted by C<(?!regexp)> and C<< (?<!fixed-regexp) >> respectively.
2000 They evaluate true if the regexps do I<not> match:
2003 $x =~ /foo(?!bar)/; # doesn't match, 'bar' follows 'foo'
2004 $x =~ /foo(?!baz)/; # matches, 'baz' doesn't follow 'foo'
2005 $x =~ /(?<!\s)foo/; # matches, there is no \s before 'foo'
2007 The C<\C> is unsupported in lookbehind, because the already
2008 treacherous definition of C<\C> would become even more so
2009 when going backwards.
2011 =head2 Using independent subexpressions to prevent backtracking
2013 The last few extended patterns in this tutorial are experimental as of
2014 5.6.0. Play with them, use them in some code, but don't rely on them
2015 just yet for production code.
2017 S<B<Independent subexpressions> > are regular expressions, in the
2018 context of a larger regular expression, that function independently of
2019 the larger regular expression. That is, they consume as much or as
2020 little of the string as they wish without regard for the ability of
2021 the larger regexp to match. Independent subexpressions are represented
2022 by C<< (?>regexp) >>. We can illustrate their behavior by first
2023 considering an ordinary regexp:
2026 $x =~ /a*ab/; # matches
2028 This obviously matches, but in the process of matching, the
2029 subexpression C<a*> first grabbed the C<a>. Doing so, however,
2030 wouldn't allow the whole regexp to match, so after backtracking, C<a*>
2031 eventually gave back the C<a> and matched the empty string. Here, what
2032 C<a*> matched was I<dependent> on what the rest of the regexp matched.
2034 Contrast that with an independent subexpression:
2036 $x =~ /(?>a*)ab/; # doesn't match!
2038 The independent subexpression C<< (?>a*) >> doesn't care about the rest
2039 of the regexp, so it sees an C<a> and grabs it. Then the rest of the
2040 regexp C<ab> cannot match. Because C<< (?>a*) >> is independent, there
2041 is no backtracking and the independent subexpression does not give
2042 up its C<a>. Thus the match of the regexp as a whole fails. A similar
2043 behavior occurs with completely independent regexps:
2046 $x =~ /a*/g; # matches, eats an 'a'
2047 $x =~ /\Gab/g; # doesn't match, no 'a' available
2049 Here C<//g> and C<\G> create a 'tag team' handoff of the string from
2050 one regexp to the other. Regexps with an independent subexpression are
2051 much like this, with a handoff of the string to the independent
2052 subexpression, and a handoff of the string back to the enclosing
2055 The ability of an independent subexpression to prevent backtracking
2056 can be quite useful. Suppose we want to match a non-empty string
2057 enclosed in parentheses up to two levels deep. Then the following
2060 $x = "abc(de(fg)h"; # unbalanced parentheses
2061 $x =~ /\( ( [^()]+ | \([^()]*\) )+ \)/x;
2063 The regexp matches an open parenthesis, one or more copies of an
2064 alternation, and a close parenthesis. The alternation is two-way, with
2065 the first alternative C<[^()]+> matching a substring with no
2066 parentheses and the second alternative C<\([^()]*\)> matching a
2067 substring delimited by parentheses. The problem with this regexp is
2068 that it is pathological: it has nested indeterminate quantifiers
2069 of the form C<(a+|b)+>. We discussed in Part 1 how nested quantifiers
2070 like this could take an exponentially long time to execute if there
2071 was no match possible. To prevent the exponential blowup, we need to
2072 prevent useless backtracking at some point. This can be done by
2073 enclosing the inner quantifier as an independent subexpression:
2075 $x =~ /\( ( (?>[^()]+) | \([^()]*\) )+ \)/x;
2077 Here, C<< (?>[^()]+) >> breaks the degeneracy of string partitioning
2078 by gobbling up as much of the string as possible and keeping it. Then
2079 match failures fail much more quickly.
2081 =head2 Conditional expressions
2083 A S<B<conditional expression> > is a form of if-then-else statement
2084 that allows one to choose which patterns are to be matched, based on
2085 some condition. There are two types of conditional expression:
2086 C<(?(condition)yes-regexp)> and
2087 C<(?(condition)yes-regexp|no-regexp)>. C<(?(condition)yes-regexp)> is
2088 like an S<C<'if () {}'> > statement in Perl. If the C<condition> is true,
2089 the C<yes-regexp> will be matched. If the C<condition> is false, the
2090 C<yes-regexp> will be skipped and perl will move onto the next regexp
2091 element. The second form is like an S<C<'if () {} else {}'> > statement
2092 in Perl. If the C<condition> is true, the C<yes-regexp> will be
2093 matched, otherwise the C<no-regexp> will be matched.
2095 The C<condition> can have two forms. The first form is simply an
2096 integer in parentheses C<(integer)>. It is true if the corresponding
2097 backreference C<\integer> matched earlier in the regexp. The second
2098 form is a bare zero width assertion C<(?...)>, either a
2099 lookahead, a lookbehind, or a code assertion (discussed in the next
2102 The integer form of the C<condition> allows us to choose, with more
2103 flexibility, what to match based on what matched earlier in the
2104 regexp. This searches for words of the form C<"$x$x"> or
2107 % simple_grep '^(\w+)(\w+)?(?(2)\2\1|\1)$' /usr/dict/words
2117 The lookbehind C<condition> allows, along with backreferences,
2118 an earlier part of the match to influence a later part of the
2119 match. For instance,
2121 /[ATGC]+(?(?<=AA)G|C)$/;
2123 matches a DNA sequence such that it either ends in C<AAG>, or some
2124 other base pair combination and C<C>. Note that the form is
2125 C<< (?(?<=AA)G|C) >> and not C<< (?((?<=AA))G|C) >>; for the
2126 lookahead, lookbehind or code assertions, the parentheses around the
2127 conditional are not needed.
2129 =head2 A bit of magic: executing Perl code in a regular expression
2131 Normally, regexps are a part of Perl expressions.
2132 S<B<Code evaluation> > expressions turn that around by allowing
2133 arbitrary Perl code to be a part of a regexp. A code evaluation
2134 expression is denoted C<(?{code})>, with C<code> a string of Perl
2137 Code expressions are zero-width assertions, and the value they return
2138 depends on their environment. There are two possibilities: either the
2139 code expression is used as a conditional in a conditional expression
2140 C<(?(condition)...)>, or it is not. If the code expression is a
2141 conditional, the code is evaluated and the result (i.e., the result of
2142 the last statement) is used to determine truth or falsehood. If the
2143 code expression is not used as a conditional, the assertion always
2144 evaluates true and the result is put into the special variable
2145 C<$^R>. The variable C<$^R> can then be used in code expressions later
2146 in the regexp. Here are some silly examples:
2149 $x =~ /abc(?{print "Hi Mom!";})def/; # matches,
2151 $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match,
2154 Pay careful attention to the next example:
2156 $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match,
2160 At first glance, you'd think that it shouldn't print, because obviously
2161 the C<ddd> isn't going to match the target string. But look at this
2164 $x =~ /abc(?{print "Hi Mom!";})[d]dd/; # doesn't match,
2167 Hmm. What happened here? If you've been following along, you know that
2168 the above pattern should be effectively the same as the last one --
2169 enclosing the d in a character class isn't going to change what it
2170 matches. So why does the first not print while the second one does?
2172 The answer lies in the optimizations the REx engine makes. In the first
2173 case, all the engine sees are plain old characters (aside from the
2174 C<?{}> construct). It's smart enough to realize that the string 'ddd'
2175 doesn't occur in our target string before actually running the pattern
2176 through. But in the second case, we've tricked it into thinking that our
2177 pattern is more complicated than it is. It takes a look, sees our
2178 character class, and decides that it will have to actually run the
2179 pattern to determine whether or not it matches, and in the process of
2180 running it hits the print statement before it discovers that we don't
2183 To take a closer look at how the engine does optimizations, see the
2184 section L<"Pragmas and debugging"> below.
2186 More fun with C<?{}>:
2188 $x =~ /(?{print "Hi Mom!";})/; # matches,
2190 $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches,
2192 $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches,
2195 The bit of magic mentioned in the section title occurs when the regexp
2196 backtracks in the process of searching for a match. If the regexp
2197 backtracks over a code expression and if the variables used within are
2198 localized using C<local>, the changes in the variables produced by the
2199 code expression are undone! Thus, if we wanted to count how many times
2200 a character got matched inside a group, we could use, e.g.,
2203 $count = 0; # initialize 'a' count
2204 $c = "bob"; # test if $c gets clobbered
2205 $x =~ /(?{local $c = 0;}) # initialize count
2207 (?{local $c = $c + 1;}) # increment count
2208 )* # do this any number of times,
2209 aa # but match 'aa' at the end
2210 (?{$count = $c;}) # copy local $c var into $count
2212 print "'a' count is $count, \$c variable is '$c'\n";
2216 'a' count is 2, $c variable is 'bob'
2218 If we replace the S<C< (?{local $c = $c + 1;})> > with
2219 S<C< (?{$c = $c + 1;})> >, the variable changes are I<not> undone
2220 during backtracking, and we get
2222 'a' count is 4, $c variable is 'bob'
2224 Note that only localized variable changes are undone. Other side
2225 effects of code expression execution are permanent. Thus
2228 $x =~ /(a(?{print "Yow\n";}))*aa/;
2237 The result C<$^R> is automatically localized, so that it will behave
2238 properly in the presence of backtracking.
2240 This example uses a code expression in a conditional to match the
2241 article 'the' in either English or German:
2243 $lang = 'DE'; # use German
2248 $lang eq 'EN'; # is the language English?
2250 the | # if so, then match 'the'
2251 (die|das|der) # else, match 'die|das|der'
2255 Note that the syntax here is C<(?(?{...})yes-regexp|no-regexp)>, not
2256 C<(?((?{...}))yes-regexp|no-regexp)>. In other words, in the case of a
2257 code expression, we don't need the extra parentheses around the
2260 If you try to use code expressions with interpolating variables, perl
2265 /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated
2266 /foo(?{ 1 })$bar/; # compile error!
2267 /foo${pat}bar/; # compile error!
2269 $pat = qr/(?{ $foo = 1 })/; # precompile code regexp
2270 /foo${pat}bar/; # compiles ok
2272 If a regexp has (1) code expressions and interpolating variables,or
2273 (2) a variable that interpolates a code expression, perl treats the
2274 regexp as an error. If the code expression is precompiled into a
2275 variable, however, interpolating is ok. The question is, why is this
2278 The reason is that variable interpolation and code expressions
2279 together pose a security risk. The combination is dangerous because
2280 many programmers who write search engines often take user input and
2281 plug it directly into a regexp:
2283 $regexp = <>; # read user-supplied regexp
2284 $chomp $regexp; # get rid of possible newline
2285 $text =~ /$regexp/; # search $text for the $regexp
2287 If the C<$regexp> variable contains a code expression, the user could
2288 then execute arbitrary Perl code. For instance, some joker could
2289 search for S<C<system('rm -rf *');> > to erase your files. In this
2290 sense, the combination of interpolation and code expressions B<taints>
2291 your regexp. So by default, using both interpolation and code
2292 expressions in the same regexp is not allowed. If you're not
2293 concerned about malicious users, it is possible to bypass this
2294 security check by invoking S<C<use re 'eval'> >:
2296 use re 'eval'; # throw caution out the door
2299 /foo(?{ 1 })$bar/; # compiles ok
2300 /foo${pat}bar/; # compiles ok
2302 Another form of code expression is the S<B<pattern code expression> >.
2303 The pattern code expression is like a regular code expression, except
2304 that the result of the code evaluation is treated as a regular
2305 expression and matched immediately. A simple example is
2310 $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a'
2313 This final example contains both ordinary and pattern code
2314 expressions. It detects if a binary string C<1101010010001...> has a
2315 Fibonacci spacing 0,1,1,2,3,5,... of the C<1>'s:
2317 $s0 = 0; $s1 = 1; # initial conditions
2318 $x = "1101010010001000001";
2319 print "It is a Fibonacci sequence\n"
2320 if $x =~ /^1 # match an initial '1'
2322 (??{'0' x $s0}) # match $s0 of '0'
2325 $largest = $s0; # largest seq so far
2326 $s2 = $s1 + $s0; # compute next term
2327 $s0 = $s1; # in Fibonacci sequence
2330 )+ # repeat as needed
2331 $ # that is all there is
2333 print "Largest sequence matched was $largest\n";
2337 It is a Fibonacci sequence
2338 Largest sequence matched was 5
2340 Ha! Try that with your garden variety regexp package...
2342 Note that the variables C<$s0> and C<$s1> are not substituted when the
2343 regexp is compiled, as happens for ordinary variables outside a code
2344 expression. Rather, the code expressions are evaluated when perl
2345 encounters them during the search for a match.
2347 The regexp without the C<//x> modifier is
2349 /^1((??{'0'x$s0})1(?{$largest=$s0;$s2=$s1+$s0$s0=$s1;$s1=$s2;}))+$/;
2351 and is a great start on an Obfuscated Perl entry :-) When working with
2352 code and conditional expressions, the extended form of regexps is
2353 almost necessary in creating and debugging regexps.
2355 =head2 Pragmas and debugging
2357 Speaking of debugging, there are several pragmas available to control
2358 and debug regexps in Perl. We have already encountered one pragma in
2359 the previous section, S<C<use re 'eval';> >, that allows variable
2360 interpolation and code expressions to coexist in a regexp. The other
2365 @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted
2367 The C<taint> pragma causes any substrings from a match with a tainted
2368 variable to be tainted as well. This is not normally the case, as
2369 regexps are often used to extract the safe bits from a tainted
2370 variable. Use C<taint> when you are not extracting safe bits, but are
2371 performing some other processing. Both C<taint> and C<eval> pragmas
2372 are lexically scoped, which means they are in effect only until
2373 the end of the block enclosing the pragmas.
2376 /^(.*)$/s; # output debugging info
2378 use re 'debugcolor';
2379 /^(.*)$/s; # output debugging info in living color
2381 The global C<debug> and C<debugcolor> pragmas allow one to get
2382 detailed debugging info about regexp compilation and
2383 execution. C<debugcolor> is the same as debug, except the debugging
2384 information is displayed in color on terminals that can display
2385 termcap color sequences. Here is example output:
2387 % perl -e 'use re "debug"; "abc" =~ /a*b+c/;'
2388 Compiling REx `a*b+c'
2396 floating `bc' at 0..2147483647 (checking floating) minlen 2
2397 Guessing start of match, REx `a*b+c' against `abc'...
2398 Found floating substr `bc' at offset 1...
2399 Guessed: match at offset 0
2400 Matching REx `a*b+c' against `abc'
2401 Setting an EVAL scope, savestack=3
2402 0 <> <abc> | 1: STAR
2403 EXACT <a> can match 1 times out of 32767...
2404 Setting an EVAL scope, savestack=3
2405 1 <a> <bc> | 4: PLUS
2406 EXACT <b> can match 1 times out of 32767...
2407 Setting an EVAL scope, savestack=3
2408 2 <ab> <c> | 7: EXACT <c>
2411 Freeing REx: `a*b+c'
2413 If you have gotten this far into the tutorial, you can probably guess
2414 what the different parts of the debugging output tell you. The first
2417 Compiling REx `a*b+c'
2426 describes the compilation stage. C<STAR(4)> means that there is a
2427 starred object, in this case C<'a'>, and if it matches, goto line 4,
2428 i.e., C<PLUS(7)>. The middle lines describe some heuristics and
2429 optimizations performed before a match:
2431 floating `bc' at 0..2147483647 (checking floating) minlen 2
2432 Guessing start of match, REx `a*b+c' against `abc'...
2433 Found floating substr `bc' at offset 1...
2434 Guessed: match at offset 0
2436 Then the match is executed and the remaining lines describe the
2439 Matching REx `a*b+c' against `abc'
2440 Setting an EVAL scope, savestack=3
2441 0 <> <abc> | 1: STAR
2442 EXACT <a> can match 1 times out of 32767...
2443 Setting an EVAL scope, savestack=3
2444 1 <a> <bc> | 4: PLUS
2445 EXACT <b> can match 1 times out of 32767...
2446 Setting an EVAL scope, savestack=3
2447 2 <ab> <c> | 7: EXACT <c>
2450 Freeing REx: `a*b+c'
2452 Each step is of the form S<C<< n <x> <y> >> >, with C<< <x> >> the
2453 part of the string matched and C<< <y> >> the part not yet
2454 matched. The S<C<< | 1: STAR >> > says that perl is at line number 1
2455 n the compilation list above. See
2456 L<perldebguts/"Debugging regular expressions"> for much more detail.
2458 An alternative method of debugging regexps is to embed C<print>
2459 statements within the regexp. This provides a blow-by-blow account of
2460 the backtracking in an alternation:
2462 "that this" =~ m@(?{print "Start at position ", pos, "\n";})
2472 (?{print "Done at position ", pos, "\n";})
2488 Code expressions, conditional expressions, and independent expressions
2489 are B<experimental>. Don't use them in production code. Yet.
2493 This is just a tutorial. For the full story on perl regular
2494 expressions, see the L<perlre> regular expressions reference page.
2496 For more information on the matching C<m//> and substitution C<s///>
2497 operators, see L<perlop/"Regexp Quote-Like Operators">. For
2498 information on the C<split> operation, see L<perlfunc/split>.
2500 For an excellent all-around resource on the care and feeding of
2501 regular expressions, see the book I<Mastering Regular Expressions> by
2502 Jeffrey Friedl (published by O'Reilly, ISBN 1556592-257-3).
2504 =head1 AUTHOR AND COPYRIGHT
2506 Copyright (c) 2000 Mark Kvale
2507 All rights reserved.
2509 This document may be distributed under the same terms as Perl itself.
2511 =head2 Acknowledgments
2513 The inspiration for the stop codon DNA example came from the ZIP
2514 code example in chapter 7 of I<Mastering Regular Expressions>.
2516 The author would like to thank Jeff Pinyan, Andrew Johnson, Peter
2517 Haworth, Ronald J Kimball, and Joe Smith for all their helpful