extra code in pp_concat, Take 2
[p5sagit/p5-mst-13.2.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
2
3perlre - Perl regular expressions
4
5=head1 DESCRIPTION
6
91e0c79e 7This page describes the syntax of regular expressions in Perl.
8
cc46d5f2 9If you haven't used regular expressions before, a quick-start
91e0c79e 10introduction is available in L<perlrequick>, and a longer tutorial
11introduction is available in L<perlretut>.
12
13For reference on how regular expressions are used in matching
14operations, plus various examples of the same, see discussions of
15C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
16Operators">.
cb1a09d0 17
19799a22 18Matching operations can have various modifiers. Modifiers
5a964f20 19that relate to the interpretation of the regular expression inside
19799a22 20are listed below. Modifiers that alter the way a regular expression
21is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 22L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 23
55497cff 24=over 4
25
26=item i
27
28Do case-insensitive pattern matching.
29
a034a98d 30If C<use locale> is in effect, the case map is taken from the current
31locale. See L<perllocale>.
32
54310121 33=item m
55497cff 34
35Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 36the start or end of the string to matching the start or end of any
7f761169 37line anywhere within the string.
55497cff 38
54310121 39=item s
55497cff 40
41Treat string as single line. That is, change "." to match any character
19799a22 42whatsoever, even a newline, which normally it would not match.
55497cff 43
f02c194e 44Used together, as /ms, they let the "." match any character whatsoever,
fb55449c 45while still allowing "^" and "$" to match, respectively, just after
19799a22 46and just before newlines within the string.
7b8d334a 47
54310121 48=item x
55497cff 49
50Extend your pattern's legibility by permitting whitespace and comments.
51
52=back
a0d0e21e 53
54These are usually written as "the C</x> modifier", even though the delimiter
14218588 55in question might not really be a slash. Any of these
a0d0e21e 56modifiers may also be embedded within the regular expression itself using
14218588 57the C<(?...)> construct. See below.
a0d0e21e 58
4633a7c4 59The C</x> modifier itself needs a little more explanation. It tells
55497cff 60the regular expression parser to ignore whitespace that is neither
61backslashed nor within a character class. You can use this to break up
4633a7c4 62your regular expression into (slightly) more readable parts. The C<#>
54310121 63character is also treated as a metacharacter introducing a comment,
55497cff 64just as in ordinary Perl code. This also means that if you want real
14218588 65whitespace or C<#> characters in the pattern (outside a character
5a964f20 66class, where they are unaffected by C</x>), that you'll either have to
55497cff 67escape them or encode them using octal or hex escapes. Taken together,
68these features go a long way towards making Perl's regular expressions
0c815be9 69more readable. Note that you have to be careful not to include the
70pattern delimiter in the comment--perl has no way of knowing you did
5a964f20 71not intend to close the pattern early. See the C-comment deletion code
0c815be9 72in L<perlop>.
a0d0e21e 73
74=head2 Regular Expressions
75
19799a22 76The patterns used in Perl pattern matching derive from supplied in
14218588 77the Version 8 regex routines. (The routines are derived
19799a22 78(distantly) from Henry Spencer's freely redistributable reimplementation
79of the V8 routines.) See L<Version 8 Regular Expressions> for
80details.
a0d0e21e 81
82In particular the following metacharacters have their standard I<egrep>-ish
83meanings:
84
54310121 85 \ Quote the next metacharacter
a0d0e21e 86 ^ Match the beginning of the line
87 . Match any character (except newline)
c07a80fd 88 $ Match the end of the line (or before newline at the end)
a0d0e21e 89 | Alternation
90 () Grouping
91 [] Character class
92
14218588 93By default, the "^" character is guaranteed to match only the
94beginning of the string, the "$" character only the end (or before the
95newline at the end), and Perl does certain optimizations with the
a0d0e21e 96assumption that the string contains only one line. Embedded newlines
97will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 98string as a multi-line buffer, such that the "^" will match after any
a0d0e21e 99newline within the string, and "$" will match before any newline. At the
100cost of a little more overhead, you can do this by using the /m modifier
101on the pattern match operator. (Older programs did this by setting C<$*>,
f02c194e 102but this practice has been removed in perl 5.9.)
a0d0e21e 103
14218588 104To simplify multi-line substitutions, the "." character never matches a
55497cff 105newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 106the string is a single line--even if it isn't.
a0d0e21e 107
108The following standard quantifiers are recognized:
109
110 * Match 0 or more times
111 + Match 1 or more times
112 ? Match 1 or 0 times
113 {n} Match exactly n times
114 {n,} Match at least n times
115 {n,m} Match at least n but not more than m times
116
117(If a curly bracket occurs in any other context, it is treated
b975c076 118as a regular character. In particular, the lower bound
119is not optional.) The "*" modifier is equivalent to C<{0,}>, the "+"
25f94b33 120modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited
9c79236d 121to integral values less than a preset limit defined when perl is built.
122This is usually 32766 on the most common platforms. The actual limit can
123be seen in the error message generated by code such as this:
124
820475bd 125 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 126
54310121 127By default, a quantified subpattern is "greedy", that is, it will match as
128many times as possible (given a particular starting location) while still
129allowing the rest of the pattern to match. If you want it to match the
130minimum number of times possible, follow the quantifier with a "?". Note
131that the meanings don't change, just the "greediness":
a0d0e21e 132
133 *? Match 0 or more times
134 +? Match 1 or more times
135 ?? Match 0 or 1 time
136 {n}? Match exactly n times
137 {n,}? Match at least n times
138 {n,m}? Match at least n but not more than m times
139
5f05dabc 140Because patterns are processed as double quoted strings, the following
a0d0e21e 141also work:
142
0f36ee90 143 \t tab (HT, TAB)
144 \n newline (LF, NL)
145 \r return (CR)
146 \f form feed (FF)
147 \a alarm (bell) (BEL)
148 \e escape (think troff) (ESC)
cb1a09d0 149 \033 octal char (think of a PDP-11)
150 \x1B hex char
a0ed51b3 151 \x{263a} wide hex char (Unicode SMILEY)
a0d0e21e 152 \c[ control char
4a2d328f 153 \N{name} named char
cb1a09d0 154 \l lowercase next char (think vi)
155 \u uppercase next char (think vi)
156 \L lowercase till \E (think vi)
157 \U uppercase till \E (think vi)
158 \E end case modification (think vi)
5a964f20 159 \Q quote (disable) pattern metacharacters till \E
a0d0e21e 160
a034a98d 161If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u>
423cee85 162and C<\U> is taken from the current locale. See L<perllocale>. For
4a2d328f 163documentation of C<\N{name}>, see L<charnames>.
a034a98d 164
1d2dff63 165You cannot include a literal C<$> or C<@> within a C<\Q> sequence.
166An unescaped C<$> or C<@> interpolates the corresponding variable,
167while escaping will cause the literal string C<\$> to be matched.
168You'll need to write something like C<m/\Quser\E\@\Qhost/>.
169
a0d0e21e 170In addition, Perl defines the following:
171
172 \w Match a "word" character (alphanumeric plus "_")
36bbe248 173 \W Match a non-"word" character
a0d0e21e 174 \s Match a whitespace character
175 \S Match a non-whitespace character
176 \d Match a digit character
177 \D Match a non-digit character
a0ed51b3 178 \pP Match P, named property. Use \p{Prop} for longer names.
179 \PP Match non-P
f244e06d 180 \X Match eXtended Unicode "combining character sequence",
07698885 181 equivalent to (?:\PM\pM*)
72ff2908 182 \C Match a single C char (octet) even under Unicode.
07698885 183 NOTE: breaks up characters into their UTF-8 bytes,
72ff2908 184 so you may end up with malformed pieces of UTF-8.
f14c76ed 185 Unsupported in lookbehind.
a0d0e21e 186
08ce8fc6 187A C<\w> matches a single alphanumeric character (an alphabetic
188character, or a decimal digit) or C<_>, not a whole word. Use C<\w+>
189to match a string of Perl-identifier characters (which isn't the same
190as matching an English word). If C<use locale> is in effect, the list
191of alphabetic characters generated by C<\w> is taken from the current
192locale. See L<perllocale>. You may use C<\w>, C<\W>, C<\s>, C<\S>,
1209ba90 193C<\d>, and C<\D> within character classes, but if you try to use them
08ce8fc6 194as endpoints of a range, that's not a range, the "-" is understood
195literally. If Unicode is in effect, C<\s> matches also "\x{85}",
196"\x{2028}, and "\x{2029}", see L<perlunicode> for more details about
491fd90a 197C<\pP>, C<\PP>, and C<\X>, and L<perluniintro> about Unicode in general.
fa11829f 198You can define your own C<\p> and C<\P> properties, see L<perlunicode>.
a0d0e21e 199
b8c5462f 200The POSIX character class syntax
201
820475bd 202 [:class:]
b8c5462f 203
26b44a0a 204is also available. The available classes and their backslash
205equivalents (if available) are as follows:
b8c5462f 206
207 alpha
208 alnum
209 ascii
aaa51d5e 210 blank [1]
b8c5462f 211 cntrl
212 digit \d
213 graph
214 lower
215 print
216 punct
aaa51d5e 217 space \s [2]
b8c5462f 218 upper
aaa51d5e 219 word \w [3]
b8c5462f 220 xdigit
221
07698885 222=over
223
224=item [1]
225
226A GNU extension equivalent to C<[ \t]>, `all horizontal whitespace'.
227
228=item [2]
229
230Not exactly equivalent to C<\s> since the C<[[:space:]]> includes
08ce8fc6 231also the (very rare) `vertical tabulator', "\ck", chr(11).
07698885 232
233=item [3]
234
08ce8fc6 235A Perl extension, see above.
07698885 236
237=back
aaa51d5e 238
26b44a0a 239For example use C<[:upper:]> to match all the uppercase characters.
aaa51d5e 240Note that the C<[]> are part of the C<[::]> construct, not part of the
241whole character class. For example:
b8c5462f 242
820475bd 243 [01[:alpha:]%]
b8c5462f 244
593df60c 245matches zero, one, any alphabetic character, and the percentage sign.
b8c5462f 246
72ff2908 247The following equivalences to Unicode \p{} constructs and equivalent
248backslash character classes (if available), will hold:
249
250 [:...:] \p{...} backslash
b8c5462f 251
252 alpha IsAlpha
253 alnum IsAlnum
254 ascii IsASCII
aaa51d5e 255 blank IsSpace
b8c5462f 256 cntrl IsCntrl
3bec3564 257 digit IsDigit \d
b8c5462f 258 graph IsGraph
259 lower IsLower
260 print IsPrint
261 punct IsPunct
262 space IsSpace
3bec3564 263 IsSpacePerl \s
b8c5462f 264 upper IsUpper
265 word IsWord
266 xdigit IsXDigit
267
26b44a0a 268For example C<[:lower:]> and C<\p{IsLower}> are equivalent.
b8c5462f 269
270If the C<utf8> pragma is not used but the C<locale> pragma is, the
aaa51d5e 271classes correlate with the usual isalpha(3) interface (except for
272`word' and `blank').
b8c5462f 273
274The assumedly non-obviously named classes are:
275
276=over 4
277
278=item cntrl
279
820475bd 280Any control character. Usually characters that don't produce output as
281such but instead control the terminal somehow: for example newline and
282backspace are control characters. All characters with ord() less than
593df60c 28332 are most often classified as control characters (assuming ASCII,
7be5a6cf 284the ISO Latin character sets, and Unicode), as is the character with
285the ord() value of 127 (C<DEL>).
b8c5462f 286
287=item graph
288
f1cbbd6e 289Any alphanumeric or punctuation (special) character.
b8c5462f 290
291=item print
292
f79b3095 293Any alphanumeric or punctuation (special) character or the space character.
b8c5462f 294
295=item punct
296
f1cbbd6e 297Any punctuation (special) character.
b8c5462f 298
299=item xdigit
300
593df60c 301Any hexadecimal digit. Though this may feel silly ([0-9A-Fa-f] would
820475bd 302work just fine) it is included for completeness.
b8c5462f 303
b8c5462f 304=back
305
306You can negate the [::] character classes by prefixing the class name
307with a '^'. This is a Perl extension. For example:
308
72ff2908 309 POSIX traditional Unicode
93733859 310
311 [:^digit:] \D \P{IsDigit}
312 [:^space:] \S \P{IsSpace}
313 [:^word:] \W \P{IsWord}
b8c5462f 314
54c18d04 315Perl respects the POSIX standard in that POSIX character classes are
316only supported within a character class. The POSIX character classes
317[.cc.] and [=cc=] are recognized but B<not> supported and trying to
318use them will cause an error.
b8c5462f 319
a0d0e21e 320Perl defines the following zero-width assertions:
321
322 \b Match a word boundary
323 \B Match a non-(word boundary)
b85d18e9 324 \A Match only at beginning of string
325 \Z Match only at end of string, or before newline at the end
326 \z Match only at end of string
9da458fc 327 \G Match only at pos() (e.g. at the end-of-match position
328 of prior m//g)
a0d0e21e 329
14218588 330A word boundary (C<\b>) is a spot between two characters
19799a22 331that has a C<\w> on one side of it and a C<\W> on the other side
332of it (in either order), counting the imaginary characters off the
333beginning and end of the string as matching a C<\W>. (Within
334character classes C<\b> represents backspace rather than a word
335boundary, just as it normally does in any double-quoted string.)
336The C<\A> and C<\Z> are just like "^" and "$", except that they
337won't match multiple times when the C</m> modifier is used, while
338"^" and "$" will match at every internal line boundary. To match
339the actual end of the string and not ignore an optional trailing
340newline, use C<\z>.
341
342The C<\G> assertion can be used to chain global matches (using
343C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
344It is also useful when writing C<lex>-like scanners, when you have
345several patterns that you want to match against consequent substrings
346of your string, see the previous reference. The actual location
347where C<\G> will match can also be influenced by using C<pos()> as
25cf8c22 348an lvalue: see L<perlfunc/pos>. Currently C<\G> is only fully
349supported when anchored to the start of the pattern; while it
350is permitted to use it elsewhere, as in C</(?<=\G..)./g>, some
351such uses (C</.\G/g>, for example) currently cause problems, and
352it is recommended that you avoid such usage for now.
c47ff5f1 353
14218588 354The bracketing construct C<( ... )> creates capture buffers. To
c47ff5f1 355refer to the digit'th buffer use \<digit> within the
14218588 356match. Outside the match use "$" instead of "\". (The
c47ff5f1 357\<digit> notation works in certain circumstances outside
14218588 358the match. See the warning below about \1 vs $1 for details.)
359Referring back to another part of the match is called a
360I<backreference>.
361
362There is no limit to the number of captured substrings that you may
363use. However Perl also uses \10, \11, etc. as aliases for \010,
fb55449c 364\011, etc. (Recall that 0 means octal, so \011 is the character at
365number 9 in your coded character set; which would be the 10th character,
366a horizontal tab under ASCII.) Perl resolves this
367ambiguity by interpreting \10 as a backreference only if at least 10
368left parentheses have opened before it. Likewise \11 is a
369backreference only if at least 11 left parentheses have opened
370before it. And so on. \1 through \9 are always interpreted as
371backreferences.
14218588 372
373Examples:
a0d0e21e 374
375 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
376
14218588 377 if (/(.)\1/) { # find first doubled char
378 print "'$1' is the first doubled character\n";
379 }
c47ff5f1 380
14218588 381 if (/Time: (..):(..):(..)/) { # parse out values
a0d0e21e 382 $hours = $1;
383 $minutes = $2;
384 $seconds = $3;
385 }
c47ff5f1 386
14218588 387Several special variables also refer back to portions of the previous
388match. C<$+> returns whatever the last bracket match matched.
389C<$&> returns the entire matched string. (At one point C<$0> did
390also, but now it returns the name of the program.) C<$`> returns
77ea4f6d 391everything before the matched string. C<$'> returns everything
392after the matched string. And C<$^N> contains whatever was matched by
393the most-recently closed group (submatch). C<$^N> can be used in
394extended patterns (see below), for example to assign a submatch to a
395variable.
14218588 396
665e98b9 397The numbered match variables ($1, $2, $3, etc.) and the related punctuation
77ea4f6d 398set (C<$+>, C<$&>, C<$`>, C<$'>, and C<$^N>) are all dynamically scoped
14218588 399until the end of the enclosing block or until the next successful
400match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
401
665e98b9 402B<NOTE>: failed matches in Perl do not reset the match variables,
403which makes easier to write code that tests for a series of more
404specific cases and remembers the best match.
405
14218588 406B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
407C<$'> anywhere in the program, it has to provide them for every
408pattern match. This may substantially slow your program. Perl
409uses the same mechanism to produce $1, $2, etc, so you also pay a
410price for each pattern that contains capturing parentheses. (To
411avoid this cost while retaining the grouping behaviour, use the
412extended regular expression C<(?: ... )> instead.) But if you never
413use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
414parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
415if you can, but if you can't (and some algorithms really appreciate
416them), once you've used them once, use them at will, because you've
417already paid the price. As of 5.005, C<$&> is not so costly as the
418other two.
68dc0745 419
19799a22 420Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
421C<\w>, C<\n>. Unlike some other regular expression languages, there
422are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 423that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22 424interpreted as a literal character, not a metacharacter. This was
425once used in a common idiom to disable or quote the special meanings
426of regular expression metacharacters in a string that you want to
36bbe248 427use for a pattern. Simply quote all non-"word" characters:
a0d0e21e 428
429 $pattern =~ s/(\W)/\\$1/g;
430
f1cbbd6e 431(If C<use locale> is set, then this depends on the current locale.)
14218588 432Today it is more common to use the quotemeta() function or the C<\Q>
433metaquoting escape sequence to disable all metacharacters' special
434meanings like this:
a0d0e21e 435
436 /$unquoted\Q$quoted\E$unquoted/
437
9da458fc 438Beware that if you put literal backslashes (those not inside
439interpolated variables) between C<\Q> and C<\E>, double-quotish
440backslash interpolation may lead to confusing results. If you
441I<need> to use literal backslashes within C<\Q...\E>,
442consult L<perlop/"Gory details of parsing quoted constructs">.
443
19799a22 444=head2 Extended Patterns
445
14218588 446Perl also defines a consistent extension syntax for features not
447found in standard tools like B<awk> and B<lex>. The syntax is a
448pair of parentheses with a question mark as the first thing within
449the parentheses. The character after the question mark indicates
450the extension.
19799a22 451
14218588 452The stability of these extensions varies widely. Some have been
453part of the core language for many years. Others are experimental
454and may change without warning or be completely removed. Check
455the documentation on an individual feature to verify its current
456status.
19799a22 457
14218588 458A question mark was chosen for this and for the minimal-matching
459construct because 1) question marks are rare in older regular
460expressions, and 2) whenever you see one, you should stop and
461"question" exactly what is going on. That's psychology...
a0d0e21e 462
463=over 10
464
cc6b7395 465=item C<(?#text)>
a0d0e21e 466
14218588 467A comment. The text is ignored. If the C</x> modifier enables
19799a22 468whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3 469the comment as soon as it sees a C<)>, so there is no way to put a literal
470C<)> in the comment.
a0d0e21e 471
19799a22 472=item C<(?imsx-imsx)>
473
0b6d1084 474One or more embedded pattern-match modifiers, to be turned on (or
475turned off, if preceded by C<->) for the remainder of the pattern or
476the remainder of the enclosing pattern group (if any). This is
477particularly useful for dynamic patterns, such as those read in from a
478configuration file, read in as an argument, are specified in a table
479somewhere, etc. Consider the case that some of which want to be case
480sensitive and some do not. The case insensitive ones need to include
481merely C<(?i)> at the front of the pattern. For example:
19799a22 482
483 $pattern = "foobar";
484 if ( /$pattern/i ) { }
485
486 # more flexible:
487
488 $pattern = "(?i)foobar";
489 if ( /$pattern/ ) { }
490
0b6d1084 491These modifiers are restored at the end of the enclosing group. For example,
19799a22 492
493 ( (?i) blah ) \s+ \1
494
495will match a repeated (I<including the case>!) word C<blah> in any
14218588 496case, assuming C<x> modifier, and no C<i> modifier outside this
19799a22 497group.
498
5a964f20 499=item C<(?:pattern)>
a0d0e21e 500
ca9dfc88 501=item C<(?imsx-imsx:pattern)>
502
5a964f20 503This is for clustering, not capturing; it groups subexpressions like
504"()", but doesn't make backreferences as "()" does. So
a0d0e21e 505
5a964f20 506 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e 507
508is like
509
5a964f20 510 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 511
19799a22 512but doesn't spit out extra fields. It's also cheaper not to capture
513characters if you don't need to.
a0d0e21e 514
19799a22 515Any letters between C<?> and C<:> act as flags modifiers as with
516C<(?imsx-imsx)>. For example,
ca9dfc88 517
518 /(?s-i:more.*than).*million/i
519
14218588 520is equivalent to the more verbose
ca9dfc88 521
522 /(?:(?s-i)more.*than).*million/i
523
5a964f20 524=item C<(?=pattern)>
a0d0e21e 525
19799a22 526A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e 527matches a word followed by a tab, without including the tab in C<$&>.
528
5a964f20 529=item C<(?!pattern)>
a0d0e21e 530
19799a22 531A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 532matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22 533however that look-ahead and look-behind are NOT the same thing. You cannot
534use this for look-behind.
7b8d334a 535
5a964f20 536If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a 537will not do what you want. That's because the C<(?!foo)> is just saying that
538the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
539match. You would have to do something like C</(?!foo)...bar/> for that. We
540say "like" because there's the case of your "bar" not having three characters
541before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
542Sometimes it's still easier just to say:
a0d0e21e 543
a3cb178b 544 if (/bar/ && $` !~ /foo$/)
a0d0e21e 545
19799a22 546For look-behind see below.
c277df42 547
c47ff5f1 548=item C<(?<=pattern)>
c277df42 549
c47ff5f1 550A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22 551matches a word that follows a tab, without including the tab in C<$&>.
552Works only for fixed-width look-behind.
c277df42 553
5a964f20 554=item C<(?<!pattern)>
c277df42 555
19799a22 556A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
557matches any occurrence of "foo" that does not follow "bar". Works
558only for fixed-width look-behind.
c277df42 559
cc6b7395 560=item C<(?{ code })>
c277df42 561
19799a22 562B<WARNING>: This extended regular expression feature is considered
563highly experimental, and may be changed or deleted without notice.
c277df42 564
cc46d5f2 565This zero-width assertion evaluates any embedded Perl code. It
19799a22 566always succeeds, and its C<code> is not interpolated. Currently,
567the rules to determine where the C<code> ends are somewhat convoluted.
568
77ea4f6d 569This feature can be used together with the special variable C<$^N> to
570capture the results of submatches in variables without having to keep
571track of the number of nested parentheses. For example:
572
573 $_ = "The brown fox jumps over the lazy dog";
574 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
575 print "color = $color, animal = $animal\n";
576
754091cb 577Inside the C<(?{...})> block, C<$_> refers to the string the regular
578expression is matching against. You can also use C<pos()> to know what is
fa11829f 579the current position of matching within this string.
754091cb 580
19799a22 581The C<code> is properly scoped in the following sense: If the assertion
582is backtracked (compare L<"Backtracking">), all changes introduced after
583C<local>ization are undone, so that
b9ac3b5b 584
585 $_ = 'a' x 8;
586 m<
587 (?{ $cnt = 0 }) # Initialize $cnt.
588 (
589 a
590 (?{
591 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
592 })
593 )*
594 aaaa
595 (?{ $res = $cnt }) # On success copy to non-localized
596 # location.
597 >x;
598
19799a22 599will set C<$res = 4>. Note that after the match, $cnt returns to the globally
14218588 600introduced value, because the scopes that restrict C<local> operators
b9ac3b5b 601are unwound.
602
19799a22 603This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
604switch. If I<not> used in this way, the result of evaluation of
605C<code> is put into the special variable C<$^R>. This happens
606immediately, so C<$^R> can be used from other C<(?{ code })> assertions
607inside the same regular expression.
b9ac3b5b 608
19799a22 609The assignment to C<$^R> above is properly localized, so the old
610value of C<$^R> is restored if the assertion is backtracked; compare
611L<"Backtracking">.
b9ac3b5b 612
19799a22 613For reasons of security, this construct is forbidden if the regular
614expression involves run-time interpolation of variables, unless the
615perilous C<use re 'eval'> pragma has been used (see L<re>), or the
616variables contain results of C<qr//> operator (see
617L<perlop/"qr/STRING/imosx">).
871b0233 618
14218588 619This restriction is because of the wide-spread and remarkably convenient
19799a22 620custom of using run-time determined strings as patterns. For example:
871b0233 621
622 $re = <>;
623 chomp $re;
624 $string =~ /$re/;
625
14218588 626Before Perl knew how to execute interpolated code within a pattern,
627this operation was completely safe from a security point of view,
628although it could raise an exception from an illegal pattern. If
629you turn on the C<use re 'eval'>, though, it is no longer secure,
630so you should only do so if you are also using taint checking.
631Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 632compartment. See L<perlsec> for details about both these mechanisms.
871b0233 633
14455d6c 634=item C<(??{ code })>
0f5d15d6 635
19799a22 636B<WARNING>: This extended regular expression feature is considered
637highly experimental, and may be changed or deleted without notice.
9da458fc 638A simplified version of the syntax may be introduced for commonly
639used idioms.
0f5d15d6 640
19799a22 641This is a "postponed" regular subexpression. The C<code> is evaluated
642at run time, at the moment this subexpression may match. The result
643of evaluation is considered as a regular expression and matched as
644if it were inserted instead of this construct.
0f5d15d6 645
428594d9 646The C<code> is not interpolated. As before, the rules to determine
19799a22 647where the C<code> ends are currently somewhat convoluted.
648
649The following pattern matches a parenthesized group:
0f5d15d6 650
651 $re = qr{
652 \(
653 (?:
654 (?> [^()]+ ) # Non-parens without backtracking
655 |
14455d6c 656 (??{ $re }) # Group with matching parens
0f5d15d6 657 )*
658 \)
659 }x;
660
c47ff5f1 661=item C<< (?>pattern) >>
5a964f20 662
19799a22 663B<WARNING>: This extended regular expression feature is considered
664highly experimental, and may be changed or deleted without notice.
665
666An "independent" subexpression, one which matches the substring
667that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 668position, and it matches I<nothing other than this substring>. This
19799a22 669construct is useful for optimizations of what would otherwise be
670"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc 671It may also be useful in places where the "grab all you can, and do not
672give anything back" semantic is desirable.
19799a22 673
c47ff5f1 674For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22 675(anchored at the beginning of string, as above) will match I<all>
676characters C<a> at the beginning of string, leaving no C<a> for
677C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
678since the match of the subgroup C<a*> is influenced by the following
679group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
680C<a*ab> will match fewer characters than a standalone C<a*>, since
681this makes the tail match.
682
c47ff5f1 683An effect similar to C<< (?>pattern) >> may be achieved by writing
19799a22 684C<(?=(pattern))\1>. This matches the same substring as a standalone
685C<a+>, and the following C<\1> eats the matched string; it therefore
c47ff5f1 686makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22 687(The difference between these two constructs is that the second one
688uses a capturing group, thus shifting ordinals of backreferences
689in the rest of a regular expression.)
690
691Consider this pattern:
c277df42 692
871b0233 693 m{ \(
694 (
9da458fc 695 [^()]+ # x+
871b0233 696 |
697 \( [^()]* \)
698 )+
699 \)
700 }x
5a964f20 701
19799a22 702That will efficiently match a nonempty group with matching parentheses
703two levels deep or less. However, if there is no such group, it
704will take virtually forever on a long string. That's because there
705are so many different ways to split a long string into several
706substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
707to a subpattern of the above pattern. Consider how the pattern
708above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
709seconds, but that each extra letter doubles this time. This
710exponential performance will make it appear that your program has
14218588 711hung. However, a tiny change to this pattern
5a964f20 712
871b0233 713 m{ \(
714 (
9da458fc 715 (?> [^()]+ ) # change x+ above to (?> x+ )
871b0233 716 |
717 \( [^()]* \)
718 )+
719 \)
720 }x
c277df42 721
c47ff5f1 722which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20 723this yourself would be a productive exercise), but finishes in a fourth
724the time when used on a similar string with 1000000 C<a>s. Be aware,
725however, that this pattern currently triggers a warning message under
9f1b1f2d 726the C<use warnings> pragma or B<-w> switch saying it
6bab786b 727C<"matches null string many times in regex">.
c277df42 728
c47ff5f1 729On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 730effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42 731This was only 4 times slower on a string with 1000000 C<a>s.
732
9da458fc 733The "grab all you can, and do not give anything back" semantic is desirable
734in many situations where on the first sight a simple C<()*> looks like
735the correct solution. Suppose we parse text with comments being delimited
736by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 737its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc 738the comment delimiter, because it may "give up" some whitespace if
739the remainder of the pattern can be made to match that way. The correct
740answer is either one of these:
741
742 (?>#[ \t]*)
743 #[ \t]*(?![ \t])
744
745For example, to grab non-empty comments into $1, one should use either
746one of these:
747
748 / (?> \# [ \t]* ) ( .+ ) /x;
749 / \# [ \t]* ( [^ \t] .* ) /x;
750
751Which one you pick depends on which of these expressions better reflects
752the above specification of comments.
753
5a964f20 754=item C<(?(condition)yes-pattern|no-pattern)>
c277df42 755
5a964f20 756=item C<(?(condition)yes-pattern)>
c277df42 757
19799a22 758B<WARNING>: This extended regular expression feature is considered
759highly experimental, and may be changed or deleted without notice.
760
c277df42 761Conditional expression. C<(condition)> should be either an integer in
762parentheses (which is valid if the corresponding pair of parentheses
19799a22 763matched), or look-ahead/look-behind/evaluate zero-width assertion.
c277df42 764
19799a22 765For example:
c277df42 766
5a964f20 767 m{ ( \( )?
871b0233 768 [^()]+
5a964f20 769 (?(1) \) )
871b0233 770 }x
c277df42 771
772matches a chunk of non-parentheses, possibly included in parentheses
773themselves.
a0d0e21e 774
a0d0e21e 775=back
776
c07a80fd 777=head2 Backtracking
778
35a734be 779NOTE: This section presents an abstract approximation of regular
780expression behavior. For a more rigorous (and complicated) view of
781the rules involved in selecting a match among possible alternatives,
782see L<Combining pieces together>.
783
c277df42 784A fundamental feature of regular expression matching involves the
5a964f20 785notion called I<backtracking>, which is currently used (when needed)
c277df42 786by all regular expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc 787C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
788internally, but the general principle outlined here is valid.
c07a80fd 789
790For a regular expression to match, the I<entire> regular expression must
791match, not just part of it. So if the beginning of a pattern containing a
792quantifier succeeds in a way that causes later parts in the pattern to
793fail, the matching engine backs up and recalculates the beginning
794part--that's why it's called backtracking.
795
796Here is an example of backtracking: Let's say you want to find the
797word following "foo" in the string "Food is on the foo table.":
798
799 $_ = "Food is on the foo table.";
800 if ( /\b(foo)\s+(\w+)/i ) {
801 print "$2 follows $1.\n";
802 }
803
804When the match runs, the first part of the regular expression (C<\b(foo)>)
805finds a possible match right at the beginning of the string, and loads up
806$1 with "Foo". However, as soon as the matching engine sees that there's
807no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 808mistake and starts over again one character after where it had the
c07a80fd 809tentative match. This time it goes all the way until the next occurrence
810of "foo". The complete regular expression matches this time, and you get
811the expected output of "table follows foo."
812
813Sometimes minimal matching can help a lot. Imagine you'd like to match
814everything between "foo" and "bar". Initially, you write something
815like this:
816
817 $_ = "The food is under the bar in the barn.";
818 if ( /foo(.*)bar/ ) {
819 print "got <$1>\n";
820 }
821
822Which perhaps unexpectedly yields:
823
824 got <d is under the bar in the >
825
826That's because C<.*> was greedy, so you get everything between the
14218588 827I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd 828to use minimal matching to make sure you get the text between a "foo"
829and the first "bar" thereafter.
830
831 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
832 got <d is under the >
833
834Here's another example: let's say you'd like to match a number at the end
b6e13d97 835of a string, and you also want to keep the preceding part of the match.
c07a80fd 836So you write this:
837
838 $_ = "I have 2 numbers: 53147";
839 if ( /(.*)(\d*)/ ) { # Wrong!
840 print "Beginning is <$1>, number is <$2>.\n";
841 }
842
843That won't work at all, because C<.*> was greedy and gobbled up the
844whole string. As C<\d*> can match on an empty string the complete
845regular expression matched successfully.
846
8e1088bc 847 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd 848
849Here are some variants, most of which don't work:
850
851 $_ = "I have 2 numbers: 53147";
852 @pats = qw{
853 (.*)(\d*)
854 (.*)(\d+)
855 (.*?)(\d*)
856 (.*?)(\d+)
857 (.*)(\d+)$
858 (.*?)(\d+)$
859 (.*)\b(\d+)$
860 (.*\D)(\d+)$
861 };
862
863 for $pat (@pats) {
864 printf "%-12s ", $pat;
865 if ( /$pat/ ) {
866 print "<$1> <$2>\n";
867 } else {
868 print "FAIL\n";
869 }
870 }
871
872That will print out:
873
874 (.*)(\d*) <I have 2 numbers: 53147> <>
875 (.*)(\d+) <I have 2 numbers: 5314> <7>
876 (.*?)(\d*) <> <>
877 (.*?)(\d+) <I have > <2>
878 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
879 (.*?)(\d+)$ <I have 2 numbers: > <53147>
880 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
881 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
882
883As you see, this can be a bit tricky. It's important to realize that a
884regular expression is merely a set of assertions that gives a definition
885of success. There may be 0, 1, or several different ways that the
886definition might succeed against a particular string. And if there are
5a964f20 887multiple ways it might succeed, you need to understand backtracking to
888know which variety of success you will achieve.
c07a80fd 889
19799a22 890When using look-ahead assertions and negations, this can all get even
8b19b778 891trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd 892followed by "123". You might try to write that as
893
871b0233 894 $_ = "ABC123";
895 if ( /^\D*(?!123)/ ) { # Wrong!
896 print "Yup, no 123 in $_\n";
897 }
c07a80fd 898
899But that isn't going to match; at least, not the way you're hoping. It
900claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 901why that pattern matches, contrary to popular expectations:
c07a80fd 902
903 $x = 'ABC123' ;
904 $y = 'ABC445' ;
905
906 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/ ;
907 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/ ;
908
909 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/ ;
910 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/ ;
911
912This prints
913
914 2: got ABC
915 3: got AB
916 4: got ABC
917
5f05dabc 918You might have expected test 3 to fail because it seems to a more
c07a80fd 919general purpose version of test 1. The important difference between
920them is that test 3 contains a quantifier (C<\D*>) and so can use
921backtracking, whereas test 1 will not. What's happening is
922that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 923non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 924let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 925fail.
14218588 926
c07a80fd 927The search engine will initially match C<\D*> with "ABC". Then it will
14218588 928try to match C<(?!123> with "123", which fails. But because
c07a80fd 929a quantifier (C<\D*>) has been used in the regular expression, the
930search engine can backtrack and retry the match differently
54310121 931in the hope of matching the complete regular expression.
c07a80fd 932
5a964f20 933The pattern really, I<really> wants to succeed, so it uses the
934standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 935time. Now there's indeed something following "AB" that is not
14218588 936"123". It's "C123", which suffices.
c07a80fd 937
14218588 938We can deal with this by using both an assertion and a negation.
939We'll say that the first part in $1 must be followed both by a digit
940and by something that's not "123". Remember that the look-aheads
941are zero-width expressions--they only look, but don't consume any
942of the string in their match. So rewriting this way produces what
c07a80fd 943you'd expect; that is, case 5 will fail, but case 6 succeeds:
944
945 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/ ;
946 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/ ;
947
948 6: got ABC
949
5a964f20 950In other words, the two zero-width assertions next to each other work as though
19799a22 951they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd 952matches only if you're at the beginning of the line AND the end of the
953line simultaneously. The deeper underlying truth is that juxtaposition in
954regular expressions always means AND, except when you write an explicit OR
955using the vertical bar. C</ab/> means match "a" AND (then) match "b",
956although the attempted matches are made at different positions because "a"
957is not a zero-width assertion, but a one-width assertion.
958
19799a22 959B<WARNING>: particularly complicated regular expressions can take
14218588 960exponential time to solve because of the immense number of possible
9da458fc 961ways they can use backtracking to try match. For example, without
962internal optimizations done by the regular expression engine, this will
963take a painfully long time to run:
c07a80fd 964
e1901655 965 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
966
967And if you used C<*>'s in the internal groups instead of limiting them
968to 0 through 5 matches, then it would take forever--or until you ran
969out of stack space. Moreover, these internal optimizations are not
970always applicable. For example, if you put C<{0,5}> instead of C<*>
971on the external group, no current optimization is applicable, and the
972match takes a long time to finish.
c07a80fd 973
9da458fc 974A powerful tool for optimizing such beasts is what is known as an
975"independent group",
c47ff5f1 976which does not backtrack (see L<C<< (?>pattern) >>>). Note also that
9da458fc 977zero-length look-ahead/look-behind assertions will not backtrack to make
14218588 978the tail match, since they are in "logical" context: only
979whether they match is considered relevant. For an example
9da458fc 980where side-effects of look-ahead I<might> have influenced the
c47ff5f1 981following match, see L<C<< (?>pattern) >>>.
c277df42 982
a0d0e21e 983=head2 Version 8 Regular Expressions
984
5a964f20 985In case you're not familiar with the "regular" Version 8 regex
a0d0e21e 986routines, here are the pattern-matching rules not described above.
987
54310121 988Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 989with a special meaning described here or above. You can cause
5a964f20 990characters that normally function as metacharacters to be interpreted
5f05dabc 991literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
a0d0e21e 992character; "\\" matches a "\"). A series of characters matches that
993series of characters in the target string, so the pattern C<blurfl>
994would match "blurfl" in the target string.
995
996You can specify a character class, by enclosing a list of characters
5a964f20 997in C<[]>, which will match any one character from the list. If the
a0d0e21e 998first character after the "[" is "^", the class matches any character not
14218588 999in the list. Within a list, the "-" character specifies a
5a964f20 1000range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2 1001inclusive. If you want either "-" or "]" itself to be a member of a
1002class, put it at the start of the list (possibly after a "^"), or
1003escape it with a backslash. "-" is also taken literally when it is
1004at the end of the list, just before the closing "]". (The
84850974 1005following all specify the same class of three characters: C<[-az]>,
1006C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
fb55449c 1007specifies a class containing twenty-six characters, even on EBCDIC
1008based coded character sets.) Also, if you try to use the character
1009classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1010a range, that's not a range, the "-" is understood literally.
a0d0e21e 1011
8ada0baa 1012Note also that the whole range idea is rather unportable between
1013character sets--and even within character sets they may cause results
1014you probably didn't expect. A sound principle is to use only ranges
1015that begin from and end at either alphabets of equal case ([a-e],
1016[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1017spell out the character sets in full.
1018
54310121 1019Characters may be specified using a metacharacter syntax much like that
a0d0e21e 1020used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1021"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
fb55449c 1022of octal digits, matches the character whose coded character set value
1023is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
1024matches the character whose numeric value is I<nn>. The expression \cI<x>
1025matches the character control-I<x>. Finally, the "." metacharacter
1026matches any character except "\n" (unless you use C</s>).
a0d0e21e 1027
1028You can specify a series of alternatives for a pattern using "|" to
1029separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1030or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e 1031first alternative includes everything from the last pattern delimiter
1032("(", "[", or the beginning of the pattern) up to the first "|", and
1033the last alternative contains everything from the last "|" to the next
14218588 1034pattern delimiter. That's why it's common practice to include
1035alternatives in parentheses: to minimize confusion about where they
a3cb178b 1036start and end.
1037
5a964f20 1038Alternatives are tried from left to right, so the first
a3cb178b 1039alternative found for which the entire expression matches, is the one that
1040is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1041example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b 1042part will match, as that is the first alternative tried, and it successfully
1043matches the target string. (This might not seem important, but it is
1044important when you are capturing matched text using parentheses.)
1045
5a964f20 1046Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1047so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1048
14218588 1049Within a pattern, you may designate subpatterns for later reference
1050by enclosing them in parentheses, and you may refer back to the
1051I<n>th subpattern later in the pattern using the metacharacter
1052\I<n>. Subpatterns are numbered based on the left to right order
1053of their opening parenthesis. A backreference matches whatever
1054actually matched the subpattern in the string being examined, not
1055the rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will
1056match "0x1234 0x4321", but not "0x1234 01234", because subpattern
10571 matched "0x", even though the rule C<0|0x> could potentially match
1058the leading 0 in the second number.
cb1a09d0 1059
19799a22 1060=head2 Warning on \1 vs $1
cb1a09d0 1061
5a964f20 1062Some people get too used to writing things like:
cb1a09d0 1063
1064 $pattern =~ s/(\W)/\\\1/g;
1065
1066This is grandfathered for the RHS of a substitute to avoid shocking the
1067B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1068PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0 1069the usual double-quoted string means a control-A. The customary Unix
1070meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1071of doing that, you get yourself into trouble if you then add an C</e>
1072modifier.
1073
5a964f20 1074 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0 1075
1076Or if you try to do
1077
1078 s/(\d+)/\1000/;
1079
1080You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1081C<${1}000>. The operation of interpolation should not be confused
cb1a09d0 1082with the operation of matching a backreference. Certainly they mean two
1083different things on the I<left> side of the C<s///>.
9fa51da4 1084
c84d73f1 1085=head2 Repeated patterns matching zero-length substring
1086
19799a22 1087B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1 1088
1089Regular expressions provide a terse and powerful programming language. As
1090with most other power tools, power comes together with the ability
1091to wreak havoc.
1092
1093A common abuse of this power stems from the ability to make infinite
628afcb5 1094loops using regular expressions, with something as innocuous as:
c84d73f1 1095
1096 'foo' =~ m{ ( o? )* }x;
1097
1098The C<o?> can match at the beginning of C<'foo'>, and since the position
1099in the string is not moved by the match, C<o?> would match again and again
14218588 1100because of the C<*> modifier. Another common way to create a similar cycle
c84d73f1 1101is with the looping modifier C<//g>:
1102
1103 @matches = ( 'foo' =~ m{ o? }xg );
1104
1105or
1106
1107 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
1108
1109or the loop implied by split().
1110
1111However, long experience has shown that many programming tasks may
14218588 1112be significantly simplified by using repeated subexpressions that
1113may match zero-length substrings. Here's a simple example being:
c84d73f1 1114
1115 @chars = split //, $string; # // is not magic in split
1116 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
1117
9da458fc 1118Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1 1119the infinite loop>. The rules for this are different for lower-level
1120loops given by the greedy modifiers C<*+{}>, and for higher-level
1121ones like the C</g> modifier or split() operator.
1122
19799a22 1123The lower-level loops are I<interrupted> (that is, the loop is
1124broken) when Perl detects that a repeated expression matched a
1125zero-length substring. Thus
c84d73f1 1126
1127 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
1128
1129is made equivalent to
1130
1131 m{ (?: NON_ZERO_LENGTH )*
1132 |
1133 (?: ZERO_LENGTH )?
1134 }x;
1135
1136The higher level-loops preserve an additional state between iterations:
1137whether the last match was zero-length. To break the loop, the following
1138match after a zero-length match is prohibited to have a length of zero.
1139This prohibition interacts with backtracking (see L<"Backtracking">),
1140and so the I<second best> match is chosen if the I<best> match is of
1141zero length.
1142
19799a22 1143For example:
c84d73f1 1144
1145 $_ = 'bar';
1146 s/\w??/<$&>/g;
1147
20fb949f 1148results in C<< <><b><><a><><r><> >>. At each position of the string the best
c84d73f1 1149match given by non-greedy C<??> is the zero-length match, and the I<second
1150best> match is what is matched by C<\w>. Thus zero-length matches
1151alternate with one-character-long matches.
1152
1153Similarly, for repeated C<m/()/g> the second-best match is the match at the
1154position one notch further in the string.
1155
19799a22 1156The additional state of being I<matched with zero-length> is associated with
c84d73f1 1157the matched string, and is reset by each assignment to pos().
9da458fc 1158Zero-length matches at the end of the previous match are ignored
1159during C<split>.
c84d73f1 1160
35a734be 1161=head2 Combining pieces together
1162
1163Each of the elementary pieces of regular expressions which were described
1164before (such as C<ab> or C<\Z>) could match at most one substring
1165at the given position of the input string. However, in a typical regular
1166expression these elementary pieces are combined into more complicated
1167patterns using combining operators C<ST>, C<S|T>, C<S*> etc
1168(in these examples C<S> and C<T> are regular subexpressions).
1169
1170Such combinations can include alternatives, leading to a problem of choice:
1171if we match a regular expression C<a|ab> against C<"abc">, will it match
1172substring C<"a"> or C<"ab">? One way to describe which substring is
1173actually matched is the concept of backtracking (see L<"Backtracking">).
1174However, this description is too low-level and makes you think
1175in terms of a particular implementation.
1176
1177Another description starts with notions of "better"/"worse". All the
1178substrings which may be matched by the given regular expression can be
1179sorted from the "best" match to the "worst" match, and it is the "best"
1180match which is chosen. This substitutes the question of "what is chosen?"
1181by the question of "which matches are better, and which are worse?".
1182
1183Again, for elementary pieces there is no such question, since at most
1184one match at a given position is possible. This section describes the
1185notion of better/worse for combining operators. In the description
1186below C<S> and C<T> are regular subexpressions.
1187
13a2d996 1188=over 4
35a734be 1189
1190=item C<ST>
1191
1192Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
1193substrings which can be matched by C<S>, C<B> and C<B'> are substrings
1194which can be matched by C<T>.
1195
1196If C<A> is better match for C<S> than C<A'>, C<AB> is a better
1197match than C<A'B'>.
1198
1199If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
1200C<B> is better match for C<T> than C<B'>.
1201
1202=item C<S|T>
1203
1204When C<S> can match, it is a better match than when only C<T> can match.
1205
1206Ordering of two matches for C<S> is the same as for C<S>. Similar for
1207two matches for C<T>.
1208
1209=item C<S{REPEAT_COUNT}>
1210
1211Matches as C<SSS...S> (repeated as many times as necessary).
1212
1213=item C<S{min,max}>
1214
1215Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
1216
1217=item C<S{min,max}?>
1218
1219Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
1220
1221=item C<S?>, C<S*>, C<S+>
1222
1223Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
1224
1225=item C<S??>, C<S*?>, C<S+?>
1226
1227Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
1228
c47ff5f1 1229=item C<< (?>S) >>
35a734be 1230
1231Matches the best match for C<S> and only that.
1232
1233=item C<(?=S)>, C<(?<=S)>
1234
1235Only the best match for C<S> is considered. (This is important only if
1236C<S> has capturing parentheses, and backreferences are used somewhere
1237else in the whole regular expression.)
1238
1239=item C<(?!S)>, C<(?<!S)>
1240
1241For this grouping operator there is no need to describe the ordering, since
1242only whether or not C<S> can match is important.
1243
14455d6c 1244=item C<(??{ EXPR })>
35a734be 1245
1246The ordering is the same as for the regular expression which is
1247the result of EXPR.
1248
1249=item C<(?(condition)yes-pattern|no-pattern)>
1250
1251Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
1252already determined. The ordering of the matches is the same as for the
1253chosen subexpression.
1254
1255=back
1256
1257The above recipes describe the ordering of matches I<at a given position>.
1258One more rule is needed to understand how a match is determined for the
1259whole regular expression: a match at an earlier position is always better
1260than a match at a later position.
1261
c84d73f1 1262=head2 Creating custom RE engines
1263
1264Overloaded constants (see L<overload>) provide a simple way to extend
1265the functionality of the RE engine.
1266
1267Suppose that we want to enable a new RE escape-sequence C<\Y|> which
6b0ac556 1268matches at boundary between whitespace characters and non-whitespace
c84d73f1 1269characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
1270at these positions, so we want to have each C<\Y|> in the place of the
1271more complicated version. We can create a module C<customre> to do
1272this:
1273
1274 package customre;
1275 use overload;
1276
1277 sub import {
1278 shift;
1279 die "No argument to customre::import allowed" if @_;
1280 overload::constant 'qr' => \&convert;
1281 }
1282
1283 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
1284
580a9fe1 1285 # We must also take care of not escaping the legitimate \\Y|
1286 # sequence, hence the presence of '\\' in the conversion rules.
141db969 1287 my %rules = ( '\\' => '\\\\',
c84d73f1 1288 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
1289 sub convert {
1290 my $re = shift;
1291 $re =~ s{
1292 \\ ( \\ | Y . )
1293 }
1294 { $rules{$1} or invalid($re,$1) }sgex;
1295 return $re;
1296 }
1297
1298Now C<use customre> enables the new escape in constant regular
1299expressions, i.e., those without any runtime variable interpolations.
1300As documented in L<overload>, this conversion will work only over
1301literal parts of regular expressions. For C<\Y|$re\Y|> the variable
1302part of this regular expression needs to be converted explicitly
1303(but only if the special meaning of C<\Y|> should be enabled inside $re):
1304
1305 use customre;
1306 $re = <>;
1307 chomp $re;
1308 $re = customre::convert $re;
1309 /\Y|$re\Y|/;
1310
19799a22 1311=head1 BUGS
1312
9da458fc 1313This document varies from difficult to understand to completely
1314and utterly opaque. The wandering prose riddled with jargon is
1315hard to fathom in several places.
1316
1317This document needs a rewrite that separates the tutorial content
1318from the reference content.
19799a22 1319
1320=head1 SEE ALSO
9fa51da4 1321
91e0c79e 1322L<perlrequick>.
1323
1324L<perlretut>.
1325
9b599b2a 1326L<perlop/"Regexp Quote-Like Operators">.
1327
1e66bd83 1328L<perlop/"Gory details of parsing quoted constructs">.
1329
14218588 1330L<perlfaq6>.
1331
9b599b2a 1332L<perlfunc/pos>.
1333
1334L<perllocale>.
1335
fb55449c 1336L<perlebcdic>.
1337
14218588 1338I<Mastering Regular Expressions> by Jeffrey Friedl, published
1339by O'Reilly and Associates.