ftrwrite, ftrexec, fteread, ftewrite and fteexec can all be merged
[p5sagit/p5-mst-13.2.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
d74e8afc 2X<regular expression> X<regex> X<regexp>
a0d0e21e 3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
91e0c79e 8This page describes the syntax of regular expressions in Perl.
9
cc46d5f2 10If you haven't used regular expressions before, a quick-start
91e0c79e 11introduction is available in L<perlrequick>, and a longer tutorial
12introduction is available in L<perlretut>.
13
14For reference on how regular expressions are used in matching
15operations, plus various examples of the same, see discussions of
16C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
17Operators">.
cb1a09d0 18
19799a22 19Matching operations can have various modifiers. Modifiers
5a964f20 20that relate to the interpretation of the regular expression inside
19799a22 21are listed below. Modifiers that alter the way a regular expression
22is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 23L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 24
55497cff 25=over 4
26
27=item i
d74e8afc 28X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
29X<regular expression, case-insensitive>
55497cff 30
31Do case-insensitive pattern matching.
32
a034a98d 33If C<use locale> is in effect, the case map is taken from the current
34locale. See L<perllocale>.
35
54310121 36=item m
d74e8afc 37X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
55497cff 38
39Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 40the start or end of the string to matching the start or end of any
7f761169 41line anywhere within the string.
55497cff 42
54310121 43=item s
d74e8afc 44X</s> X<regex, single-line> X<regexp, single-line>
45X<regular expression, single-line>
55497cff 46
47Treat string as single line. That is, change "." to match any character
19799a22 48whatsoever, even a newline, which normally it would not match.
55497cff 49
f02c194e 50Used together, as /ms, they let the "." match any character whatsoever,
fb55449c 51while still allowing "^" and "$" to match, respectively, just after
19799a22 52and just before newlines within the string.
7b8d334a 53
54310121 54=item x
d74e8afc 55X</x>
55497cff 56
57Extend your pattern's legibility by permitting whitespace and comments.
58
59=back
a0d0e21e 60
61These are usually written as "the C</x> modifier", even though the delimiter
14218588 62in question might not really be a slash. Any of these
a0d0e21e 63modifiers may also be embedded within the regular expression itself using
14218588 64the C<(?...)> construct. See below.
a0d0e21e 65
4633a7c4 66The C</x> modifier itself needs a little more explanation. It tells
55497cff 67the regular expression parser to ignore whitespace that is neither
68backslashed nor within a character class. You can use this to break up
4633a7c4 69your regular expression into (slightly) more readable parts. The C<#>
54310121 70character is also treated as a metacharacter introducing a comment,
55497cff 71just as in ordinary Perl code. This also means that if you want real
14218588 72whitespace or C<#> characters in the pattern (outside a character
5a964f20 73class, where they are unaffected by C</x>), that you'll either have to
55497cff 74escape them or encode them using octal or hex escapes. Taken together,
75these features go a long way towards making Perl's regular expressions
0c815be9 76more readable. Note that you have to be careful not to include the
77pattern delimiter in the comment--perl has no way of knowing you did
5a964f20 78not intend to close the pattern early. See the C-comment deletion code
0c815be9 79in L<perlop>.
d74e8afc 80X</x>
a0d0e21e 81
82=head2 Regular Expressions
83
19799a22 84The patterns used in Perl pattern matching derive from supplied in
14218588 85the Version 8 regex routines. (The routines are derived
19799a22 86(distantly) from Henry Spencer's freely redistributable reimplementation
87of the V8 routines.) See L<Version 8 Regular Expressions> for
88details.
a0d0e21e 89
90In particular the following metacharacters have their standard I<egrep>-ish
91meanings:
d74e8afc 92X<metacharacter>
93X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
94
a0d0e21e 95
54310121 96 \ Quote the next metacharacter
a0d0e21e 97 ^ Match the beginning of the line
98 . Match any character (except newline)
c07a80fd 99 $ Match the end of the line (or before newline at the end)
a0d0e21e 100 | Alternation
101 () Grouping
102 [] Character class
103
14218588 104By default, the "^" character is guaranteed to match only the
105beginning of the string, the "$" character only the end (or before the
106newline at the end), and Perl does certain optimizations with the
a0d0e21e 107assumption that the string contains only one line. Embedded newlines
108will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 109string as a multi-line buffer, such that the "^" will match after any
a0d0e21e 110newline within the string, and "$" will match before any newline. At the
111cost of a little more overhead, you can do this by using the /m modifier
112on the pattern match operator. (Older programs did this by setting C<$*>,
f02c194e 113but this practice has been removed in perl 5.9.)
d74e8afc 114X<^> X<$> X</m>
a0d0e21e 115
14218588 116To simplify multi-line substitutions, the "." character never matches a
55497cff 117newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 118the string is a single line--even if it isn't.
d74e8afc 119X<.> X</s>
a0d0e21e 120
121The following standard quantifiers are recognized:
d74e8afc 122X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
a0d0e21e 123
124 * Match 0 or more times
125 + Match 1 or more times
126 ? Match 1 or 0 times
127 {n} Match exactly n times
128 {n,} Match at least n times
129 {n,m} Match at least n but not more than m times
130
131(If a curly bracket occurs in any other context, it is treated
b975c076 132as a regular character. In particular, the lower bound
133is not optional.) The "*" modifier is equivalent to C<{0,}>, the "+"
25f94b33 134modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited
9c79236d 135to integral values less than a preset limit defined when perl is built.
136This is usually 32766 on the most common platforms. The actual limit can
137be seen in the error message generated by code such as this:
138
820475bd 139 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 140
54310121 141By default, a quantified subpattern is "greedy", that is, it will match as
142many times as possible (given a particular starting location) while still
143allowing the rest of the pattern to match. If you want it to match the
144minimum number of times possible, follow the quantifier with a "?". Note
145that the meanings don't change, just the "greediness":
d74e8afc 146X<metacharacter> X<greedy> X<greedyness>
147X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{n,m}?>
a0d0e21e 148
149 *? Match 0 or more times
150 +? Match 1 or more times
151 ?? Match 0 or 1 time
152 {n}? Match exactly n times
153 {n,}? Match at least n times
154 {n,m}? Match at least n but not more than m times
155
5f05dabc 156Because patterns are processed as double quoted strings, the following
a0d0e21e 157also work:
d74e8afc 158X<\t> X<\n> X<\r> X<\f> X<\a> X<\l> X<\u> X<\L> X<\U> X<\E> X<\Q>
159X<\0> X<\c> X<\N> X<\x>
a0d0e21e 160
0f36ee90 161 \t tab (HT, TAB)
162 \n newline (LF, NL)
163 \r return (CR)
164 \f form feed (FF)
165 \a alarm (bell) (BEL)
166 \e escape (think troff) (ESC)
cb1a09d0 167 \033 octal char (think of a PDP-11)
168 \x1B hex char
a0ed51b3 169 \x{263a} wide hex char (Unicode SMILEY)
a0d0e21e 170 \c[ control char
4a2d328f 171 \N{name} named char
cb1a09d0 172 \l lowercase next char (think vi)
173 \u uppercase next char (think vi)
174 \L lowercase till \E (think vi)
175 \U uppercase till \E (think vi)
176 \E end case modification (think vi)
5a964f20 177 \Q quote (disable) pattern metacharacters till \E
a0d0e21e 178
a034a98d 179If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u>
423cee85 180and C<\U> is taken from the current locale. See L<perllocale>. For
4a2d328f 181documentation of C<\N{name}>, see L<charnames>.
a034a98d 182
1d2dff63 183You cannot include a literal C<$> or C<@> within a C<\Q> sequence.
184An unescaped C<$> or C<@> interpolates the corresponding variable,
185while escaping will cause the literal string C<\$> to be matched.
186You'll need to write something like C<m/\Quser\E\@\Qhost/>.
187
a0d0e21e 188In addition, Perl defines the following:
d74e8afc 189X<metacharacter>
190X<\w> X<\W> X<\s> X<\S> X<\d> X<\D> X<\X> X<\p> X<\P> X<\C>
191X<word> X<whitespace>
a0d0e21e 192
193 \w Match a "word" character (alphanumeric plus "_")
36bbe248 194 \W Match a non-"word" character
a0d0e21e 195 \s Match a whitespace character
196 \S Match a non-whitespace character
197 \d Match a digit character
198 \D Match a non-digit character
a0ed51b3 199 \pP Match P, named property. Use \p{Prop} for longer names.
200 \PP Match non-P
f244e06d 201 \X Match eXtended Unicode "combining character sequence",
07698885 202 equivalent to (?:\PM\pM*)
72ff2908 203 \C Match a single C char (octet) even under Unicode.
07698885 204 NOTE: breaks up characters into their UTF-8 bytes,
72ff2908 205 so you may end up with malformed pieces of UTF-8.
f14c76ed 206 Unsupported in lookbehind.
a0d0e21e 207
08ce8fc6 208A C<\w> matches a single alphanumeric character (an alphabetic
209character, or a decimal digit) or C<_>, not a whole word. Use C<\w+>
210to match a string of Perl-identifier characters (which isn't the same
211as matching an English word). If C<use locale> is in effect, the list
212of alphabetic characters generated by C<\w> is taken from the current
213locale. See L<perllocale>. You may use C<\w>, C<\W>, C<\s>, C<\S>,
1209ba90 214C<\d>, and C<\D> within character classes, but if you try to use them
08ce8fc6 215as endpoints of a range, that's not a range, the "-" is understood
216literally. If Unicode is in effect, C<\s> matches also "\x{85}",
217"\x{2028}, and "\x{2029}", see L<perlunicode> for more details about
491fd90a 218C<\pP>, C<\PP>, and C<\X>, and L<perluniintro> about Unicode in general.
fa11829f 219You can define your own C<\p> and C<\P> properties, see L<perlunicode>.
d74e8afc 220X<\w> X<\W> X<word>
a0d0e21e 221
b8c5462f 222The POSIX character class syntax
d74e8afc 223X<character class>
b8c5462f 224
820475bd 225 [:class:]
b8c5462f 226
26b44a0a 227is also available. The available classes and their backslash
228equivalents (if available) are as follows:
d74e8afc 229X<character class>
230X<alpha> X<alnum> X<ascii> X<blank> X<cntrl> X<digit> X<graph>
231X<lower> X<print> X<punct> X<space> X<upper> X<word> X<xdigit>
b8c5462f 232
233 alpha
234 alnum
235 ascii
aaa51d5e 236 blank [1]
b8c5462f 237 cntrl
238 digit \d
239 graph
240 lower
241 print
242 punct
aaa51d5e 243 space \s [2]
b8c5462f 244 upper
aaa51d5e 245 word \w [3]
b8c5462f 246 xdigit
247
07698885 248=over
249
250=item [1]
251
b432a672 252A GNU extension equivalent to C<[ \t]>, "all horizontal whitespace".
07698885 253
254=item [2]
255
256Not exactly equivalent to C<\s> since the C<[[:space:]]> includes
b432a672 257also the (very rare) "vertical tabulator", "\ck", chr(11).
07698885 258
259=item [3]
260
08ce8fc6 261A Perl extension, see above.
07698885 262
263=back
aaa51d5e 264
26b44a0a 265For example use C<[:upper:]> to match all the uppercase characters.
aaa51d5e 266Note that the C<[]> are part of the C<[::]> construct, not part of the
267whole character class. For example:
b8c5462f 268
820475bd 269 [01[:alpha:]%]
b8c5462f 270
593df60c 271matches zero, one, any alphabetic character, and the percentage sign.
b8c5462f 272
72ff2908 273The following equivalences to Unicode \p{} constructs and equivalent
274backslash character classes (if available), will hold:
d74e8afc 275X<character class> X<\p> X<\p{}>
72ff2908 276
277 [:...:] \p{...} backslash
b8c5462f 278
279 alpha IsAlpha
280 alnum IsAlnum
281 ascii IsASCII
b432a672 282 blank IsSpace
b8c5462f 283 cntrl IsCntrl
3bec3564 284 digit IsDigit \d
b8c5462f 285 graph IsGraph
286 lower IsLower
287 print IsPrint
288 punct IsPunct
289 space IsSpace
3bec3564 290 IsSpacePerl \s
b8c5462f 291 upper IsUpper
292 word IsWord
293 xdigit IsXDigit
294
26b44a0a 295For example C<[:lower:]> and C<\p{IsLower}> are equivalent.
b8c5462f 296
297If the C<utf8> pragma is not used but the C<locale> pragma is, the
aaa51d5e 298classes correlate with the usual isalpha(3) interface (except for
b432a672 299"word" and "blank").
b8c5462f 300
301The assumedly non-obviously named classes are:
302
303=over 4
304
305=item cntrl
d74e8afc 306X<cntrl>
b8c5462f 307
820475bd 308Any control character. Usually characters that don't produce output as
309such but instead control the terminal somehow: for example newline and
310backspace are control characters. All characters with ord() less than
593df60c 31132 are most often classified as control characters (assuming ASCII,
7be5a6cf 312the ISO Latin character sets, and Unicode), as is the character with
313the ord() value of 127 (C<DEL>).
b8c5462f 314
315=item graph
d74e8afc 316X<graph>
b8c5462f 317
f1cbbd6e 318Any alphanumeric or punctuation (special) character.
b8c5462f 319
320=item print
d74e8afc 321X<print>
b8c5462f 322
f79b3095 323Any alphanumeric or punctuation (special) character or the space character.
b8c5462f 324
325=item punct
d74e8afc 326X<punct>
b8c5462f 327
f1cbbd6e 328Any punctuation (special) character.
b8c5462f 329
330=item xdigit
d74e8afc 331X<xdigit>
b8c5462f 332
593df60c 333Any hexadecimal digit. Though this may feel silly ([0-9A-Fa-f] would
820475bd 334work just fine) it is included for completeness.
b8c5462f 335
b8c5462f 336=back
337
338You can negate the [::] character classes by prefixing the class name
339with a '^'. This is a Perl extension. For example:
d74e8afc 340X<character class, negation>
b8c5462f 341
72ff2908 342 POSIX traditional Unicode
93733859 343
344 [:^digit:] \D \P{IsDigit}
345 [:^space:] \S \P{IsSpace}
346 [:^word:] \W \P{IsWord}
b8c5462f 347
54c18d04 348Perl respects the POSIX standard in that POSIX character classes are
349only supported within a character class. The POSIX character classes
350[.cc.] and [=cc=] are recognized but B<not> supported and trying to
351use them will cause an error.
b8c5462f 352
a0d0e21e 353Perl defines the following zero-width assertions:
d74e8afc 354X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
355X<regexp, zero-width assertion>
356X<regular expression, zero-width assertion>
357X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
a0d0e21e 358
359 \b Match a word boundary
360 \B Match a non-(word boundary)
b85d18e9 361 \A Match only at beginning of string
362 \Z Match only at end of string, or before newline at the end
363 \z Match only at end of string
9da458fc 364 \G Match only at pos() (e.g. at the end-of-match position
365 of prior m//g)
a0d0e21e 366
14218588 367A word boundary (C<\b>) is a spot between two characters
19799a22 368that has a C<\w> on one side of it and a C<\W> on the other side
369of it (in either order), counting the imaginary characters off the
370beginning and end of the string as matching a C<\W>. (Within
371character classes C<\b> represents backspace rather than a word
372boundary, just as it normally does in any double-quoted string.)
373The C<\A> and C<\Z> are just like "^" and "$", except that they
374won't match multiple times when the C</m> modifier is used, while
375"^" and "$" will match at every internal line boundary. To match
376the actual end of the string and not ignore an optional trailing
377newline, use C<\z>.
d74e8afc 378X<\b> X<\A> X<\Z> X<\z> X</m>
19799a22 379
380The C<\G> assertion can be used to chain global matches (using
381C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
382It is also useful when writing C<lex>-like scanners, when you have
383several patterns that you want to match against consequent substrings
384of your string, see the previous reference. The actual location
385where C<\G> will match can also be influenced by using C<pos()> as
25cf8c22 386an lvalue: see L<perlfunc/pos>. Currently C<\G> is only fully
387supported when anchored to the start of the pattern; while it
388is permitted to use it elsewhere, as in C</(?<=\G..)./g>, some
389such uses (C</.\G/g>, for example) currently cause problems, and
390it is recommended that you avoid such usage for now.
d74e8afc 391X<\G>
c47ff5f1 392
14218588 393The bracketing construct C<( ... )> creates capture buffers. To
c47ff5f1 394refer to the digit'th buffer use \<digit> within the
14218588 395match. Outside the match use "$" instead of "\". (The
c47ff5f1 396\<digit> notation works in certain circumstances outside
14218588 397the match. See the warning below about \1 vs $1 for details.)
398Referring back to another part of the match is called a
399I<backreference>.
d74e8afc 400X<regex, capture buffer> X<regexp, capture buffer>
401X<regular expression, capture buffer> X<backreference>
14218588 402
403There is no limit to the number of captured substrings that you may
404use. However Perl also uses \10, \11, etc. as aliases for \010,
fb55449c 405\011, etc. (Recall that 0 means octal, so \011 is the character at
406number 9 in your coded character set; which would be the 10th character,
407a horizontal tab under ASCII.) Perl resolves this
408ambiguity by interpreting \10 as a backreference only if at least 10
409left parentheses have opened before it. Likewise \11 is a
410backreference only if at least 11 left parentheses have opened
411before it. And so on. \1 through \9 are always interpreted as
412backreferences.
14218588 413
414Examples:
a0d0e21e 415
416 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
417
14218588 418 if (/(.)\1/) { # find first doubled char
419 print "'$1' is the first doubled character\n";
420 }
c47ff5f1 421
14218588 422 if (/Time: (..):(..):(..)/) { # parse out values
a0d0e21e 423 $hours = $1;
424 $minutes = $2;
425 $seconds = $3;
426 }
c47ff5f1 427
14218588 428Several special variables also refer back to portions of the previous
429match. C<$+> returns whatever the last bracket match matched.
430C<$&> returns the entire matched string. (At one point C<$0> did
431also, but now it returns the name of the program.) C<$`> returns
77ea4f6d 432everything before the matched string. C<$'> returns everything
433after the matched string. And C<$^N> contains whatever was matched by
434the most-recently closed group (submatch). C<$^N> can be used in
435extended patterns (see below), for example to assign a submatch to a
436variable.
d74e8afc 437X<$+> X<$^N> X<$&> X<$`> X<$'>
14218588 438
665e98b9 439The numbered match variables ($1, $2, $3, etc.) and the related punctuation
77ea4f6d 440set (C<$+>, C<$&>, C<$`>, C<$'>, and C<$^N>) are all dynamically scoped
14218588 441until the end of the enclosing block or until the next successful
442match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
d74e8afc 443X<$+> X<$^N> X<$&> X<$`> X<$'>
444X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
445
14218588 446
665e98b9 447B<NOTE>: failed matches in Perl do not reset the match variables,
5146ce24 448which makes it easier to write code that tests for a series of more
665e98b9 449specific cases and remembers the best match.
450
14218588 451B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
452C<$'> anywhere in the program, it has to provide them for every
453pattern match. This may substantially slow your program. Perl
454uses the same mechanism to produce $1, $2, etc, so you also pay a
455price for each pattern that contains capturing parentheses. (To
456avoid this cost while retaining the grouping behaviour, use the
457extended regular expression C<(?: ... )> instead.) But if you never
458use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
459parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
460if you can, but if you can't (and some algorithms really appreciate
461them), once you've used them once, use them at will, because you've
462already paid the price. As of 5.005, C<$&> is not so costly as the
463other two.
d74e8afc 464X<$&> X<$`> X<$'>
68dc0745 465
19799a22 466Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
467C<\w>, C<\n>. Unlike some other regular expression languages, there
468are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 469that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22 470interpreted as a literal character, not a metacharacter. This was
471once used in a common idiom to disable or quote the special meanings
472of regular expression metacharacters in a string that you want to
36bbe248 473use for a pattern. Simply quote all non-"word" characters:
a0d0e21e 474
475 $pattern =~ s/(\W)/\\$1/g;
476
f1cbbd6e 477(If C<use locale> is set, then this depends on the current locale.)
14218588 478Today it is more common to use the quotemeta() function or the C<\Q>
479metaquoting escape sequence to disable all metacharacters' special
480meanings like this:
a0d0e21e 481
482 /$unquoted\Q$quoted\E$unquoted/
483
9da458fc 484Beware that if you put literal backslashes (those not inside
485interpolated variables) between C<\Q> and C<\E>, double-quotish
486backslash interpolation may lead to confusing results. If you
487I<need> to use literal backslashes within C<\Q...\E>,
488consult L<perlop/"Gory details of parsing quoted constructs">.
489
19799a22 490=head2 Extended Patterns
491
14218588 492Perl also defines a consistent extension syntax for features not
493found in standard tools like B<awk> and B<lex>. The syntax is a
494pair of parentheses with a question mark as the first thing within
495the parentheses. The character after the question mark indicates
496the extension.
19799a22 497
14218588 498The stability of these extensions varies widely. Some have been
499part of the core language for many years. Others are experimental
500and may change without warning or be completely removed. Check
501the documentation on an individual feature to verify its current
502status.
19799a22 503
14218588 504A question mark was chosen for this and for the minimal-matching
505construct because 1) question marks are rare in older regular
506expressions, and 2) whenever you see one, you should stop and
507"question" exactly what is going on. That's psychology...
a0d0e21e 508
509=over 10
510
cc6b7395 511=item C<(?#text)>
d74e8afc 512X<(?#)>
a0d0e21e 513
14218588 514A comment. The text is ignored. If the C</x> modifier enables
19799a22 515whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3 516the comment as soon as it sees a C<)>, so there is no way to put a literal
517C<)> in the comment.
a0d0e21e 518
19799a22 519=item C<(?imsx-imsx)>
d74e8afc 520X<(?)>
19799a22 521
0b6d1084 522One or more embedded pattern-match modifiers, to be turned on (or
523turned off, if preceded by C<->) for the remainder of the pattern or
524the remainder of the enclosing pattern group (if any). This is
525particularly useful for dynamic patterns, such as those read in from a
526configuration file, read in as an argument, are specified in a table
527somewhere, etc. Consider the case that some of which want to be case
528sensitive and some do not. The case insensitive ones need to include
529merely C<(?i)> at the front of the pattern. For example:
19799a22 530
531 $pattern = "foobar";
532 if ( /$pattern/i ) { }
533
534 # more flexible:
535
536 $pattern = "(?i)foobar";
537 if ( /$pattern/ ) { }
538
0b6d1084 539These modifiers are restored at the end of the enclosing group. For example,
19799a22 540
541 ( (?i) blah ) \s+ \1
542
543will match a repeated (I<including the case>!) word C<blah> in any
14218588 544case, assuming C<x> modifier, and no C<i> modifier outside this
19799a22 545group.
546
5a964f20 547=item C<(?:pattern)>
d74e8afc 548X<(?:)>
a0d0e21e 549
ca9dfc88 550=item C<(?imsx-imsx:pattern)>
551
5a964f20 552This is for clustering, not capturing; it groups subexpressions like
553"()", but doesn't make backreferences as "()" does. So
a0d0e21e 554
5a964f20 555 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e 556
557is like
558
5a964f20 559 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 560
19799a22 561but doesn't spit out extra fields. It's also cheaper not to capture
562characters if you don't need to.
a0d0e21e 563
19799a22 564Any letters between C<?> and C<:> act as flags modifiers as with
565C<(?imsx-imsx)>. For example,
ca9dfc88 566
567 /(?s-i:more.*than).*million/i
568
14218588 569is equivalent to the more verbose
ca9dfc88 570
571 /(?:(?s-i)more.*than).*million/i
572
5a964f20 573=item C<(?=pattern)>
d74e8afc 574X<(?=)> X<look-ahead, positive> X<lookahead, positive>
a0d0e21e 575
19799a22 576A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e 577matches a word followed by a tab, without including the tab in C<$&>.
578
5a964f20 579=item C<(?!pattern)>
d74e8afc 580X<(?!)> X<look-ahead, negative> X<lookahead, negative>
a0d0e21e 581
19799a22 582A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 583matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22 584however that look-ahead and look-behind are NOT the same thing. You cannot
585use this for look-behind.
7b8d334a 586
5a964f20 587If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a 588will not do what you want. That's because the C<(?!foo)> is just saying that
589the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
590match. You would have to do something like C</(?!foo)...bar/> for that. We
591say "like" because there's the case of your "bar" not having three characters
592before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
593Sometimes it's still easier just to say:
a0d0e21e 594
a3cb178b 595 if (/bar/ && $` !~ /foo$/)
a0d0e21e 596
19799a22 597For look-behind see below.
c277df42 598
c47ff5f1 599=item C<(?<=pattern)>
d74e8afc 600X<(?<=)> X<look-behind, positive> X<lookbehind, positive>
c277df42 601
c47ff5f1 602A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22 603matches a word that follows a tab, without including the tab in C<$&>.
604Works only for fixed-width look-behind.
c277df42 605
5a964f20 606=item C<(?<!pattern)>
d74e8afc 607X<(?<!)> X<look-behind, negative> X<lookbehind, negative>
c277df42 608
19799a22 609A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
610matches any occurrence of "foo" that does not follow "bar". Works
611only for fixed-width look-behind.
c277df42 612
cc6b7395 613=item C<(?{ code })>
d74e8afc 614X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
c277df42 615
19799a22 616B<WARNING>: This extended regular expression feature is considered
617highly experimental, and may be changed or deleted without notice.
c277df42 618
cc46d5f2 619This zero-width assertion evaluates any embedded Perl code. It
19799a22 620always succeeds, and its C<code> is not interpolated. Currently,
621the rules to determine where the C<code> ends are somewhat convoluted.
622
77ea4f6d 623This feature can be used together with the special variable C<$^N> to
624capture the results of submatches in variables without having to keep
625track of the number of nested parentheses. For example:
626
627 $_ = "The brown fox jumps over the lazy dog";
628 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
629 print "color = $color, animal = $animal\n";
630
754091cb 631Inside the C<(?{...})> block, C<$_> refers to the string the regular
632expression is matching against. You can also use C<pos()> to know what is
fa11829f 633the current position of matching within this string.
754091cb 634
19799a22 635The C<code> is properly scoped in the following sense: If the assertion
636is backtracked (compare L<"Backtracking">), all changes introduced after
637C<local>ization are undone, so that
b9ac3b5b 638
639 $_ = 'a' x 8;
640 m<
641 (?{ $cnt = 0 }) # Initialize $cnt.
642 (
643 a
644 (?{
645 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
646 })
647 )*
648 aaaa
649 (?{ $res = $cnt }) # On success copy to non-localized
650 # location.
651 >x;
652
19799a22 653will set C<$res = 4>. Note that after the match, $cnt returns to the globally
14218588 654introduced value, because the scopes that restrict C<local> operators
b9ac3b5b 655are unwound.
656
19799a22 657This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
658switch. If I<not> used in this way, the result of evaluation of
659C<code> is put into the special variable C<$^R>. This happens
660immediately, so C<$^R> can be used from other C<(?{ code })> assertions
661inside the same regular expression.
b9ac3b5b 662
19799a22 663The assignment to C<$^R> above is properly localized, so the old
664value of C<$^R> is restored if the assertion is backtracked; compare
665L<"Backtracking">.
b9ac3b5b 666
19799a22 667For reasons of security, this construct is forbidden if the regular
668expression involves run-time interpolation of variables, unless the
669perilous C<use re 'eval'> pragma has been used (see L<re>), or the
670variables contain results of C<qr//> operator (see
671L<perlop/"qr/STRING/imosx">).
871b0233 672
14218588 673This restriction is because of the wide-spread and remarkably convenient
19799a22 674custom of using run-time determined strings as patterns. For example:
871b0233 675
676 $re = <>;
677 chomp $re;
678 $string =~ /$re/;
679
14218588 680Before Perl knew how to execute interpolated code within a pattern,
681this operation was completely safe from a security point of view,
682although it could raise an exception from an illegal pattern. If
683you turn on the C<use re 'eval'>, though, it is no longer secure,
684so you should only do so if you are also using taint checking.
685Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 686compartment. See L<perlsec> for details about both these mechanisms.
871b0233 687
14455d6c 688=item C<(??{ code })>
d74e8afc 689X<(??{})>
690X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
691X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
0f5d15d6 692
19799a22 693B<WARNING>: This extended regular expression feature is considered
694highly experimental, and may be changed or deleted without notice.
9da458fc 695A simplified version of the syntax may be introduced for commonly
696used idioms.
0f5d15d6 697
19799a22 698This is a "postponed" regular subexpression. The C<code> is evaluated
699at run time, at the moment this subexpression may match. The result
700of evaluation is considered as a regular expression and matched as
701if it were inserted instead of this construct.
0f5d15d6 702
428594d9 703The C<code> is not interpolated. As before, the rules to determine
19799a22 704where the C<code> ends are currently somewhat convoluted.
705
706The following pattern matches a parenthesized group:
0f5d15d6 707
708 $re = qr{
709 \(
710 (?:
711 (?> [^()]+ ) # Non-parens without backtracking
712 |
14455d6c 713 (??{ $re }) # Group with matching parens
0f5d15d6 714 )*
715 \)
716 }x;
717
c47ff5f1 718=item C<< (?>pattern) >>
d74e8afc 719X<backtrack> X<backtracking>
5a964f20 720
19799a22 721B<WARNING>: This extended regular expression feature is considered
722highly experimental, and may be changed or deleted without notice.
723
724An "independent" subexpression, one which matches the substring
725that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 726position, and it matches I<nothing other than this substring>. This
19799a22 727construct is useful for optimizations of what would otherwise be
728"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc 729It may also be useful in places where the "grab all you can, and do not
730give anything back" semantic is desirable.
19799a22 731
c47ff5f1 732For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22 733(anchored at the beginning of string, as above) will match I<all>
734characters C<a> at the beginning of string, leaving no C<a> for
735C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
736since the match of the subgroup C<a*> is influenced by the following
737group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
738C<a*ab> will match fewer characters than a standalone C<a*>, since
739this makes the tail match.
740
c47ff5f1 741An effect similar to C<< (?>pattern) >> may be achieved by writing
19799a22 742C<(?=(pattern))\1>. This matches the same substring as a standalone
743C<a+>, and the following C<\1> eats the matched string; it therefore
c47ff5f1 744makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22 745(The difference between these two constructs is that the second one
746uses a capturing group, thus shifting ordinals of backreferences
747in the rest of a regular expression.)
748
749Consider this pattern:
c277df42 750
871b0233 751 m{ \(
752 (
9da458fc 753 [^()]+ # x+
871b0233 754 |
755 \( [^()]* \)
756 )+
757 \)
758 }x
5a964f20 759
19799a22 760That will efficiently match a nonempty group with matching parentheses
761two levels deep or less. However, if there is no such group, it
762will take virtually forever on a long string. That's because there
763are so many different ways to split a long string into several
764substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
765to a subpattern of the above pattern. Consider how the pattern
766above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
767seconds, but that each extra letter doubles this time. This
768exponential performance will make it appear that your program has
14218588 769hung. However, a tiny change to this pattern
5a964f20 770
871b0233 771 m{ \(
772 (
9da458fc 773 (?> [^()]+ ) # change x+ above to (?> x+ )
871b0233 774 |
775 \( [^()]* \)
776 )+
777 \)
778 }x
c277df42 779
c47ff5f1 780which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20 781this yourself would be a productive exercise), but finishes in a fourth
782the time when used on a similar string with 1000000 C<a>s. Be aware,
783however, that this pattern currently triggers a warning message under
9f1b1f2d 784the C<use warnings> pragma or B<-w> switch saying it
6bab786b 785C<"matches null string many times in regex">.
c277df42 786
c47ff5f1 787On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 788effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42 789This was only 4 times slower on a string with 1000000 C<a>s.
790
9da458fc 791The "grab all you can, and do not give anything back" semantic is desirable
792in many situations where on the first sight a simple C<()*> looks like
793the correct solution. Suppose we parse text with comments being delimited
794by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 795its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc 796the comment delimiter, because it may "give up" some whitespace if
797the remainder of the pattern can be made to match that way. The correct
798answer is either one of these:
799
800 (?>#[ \t]*)
801 #[ \t]*(?![ \t])
802
803For example, to grab non-empty comments into $1, one should use either
804one of these:
805
806 / (?> \# [ \t]* ) ( .+ ) /x;
807 / \# [ \t]* ( [^ \t] .* ) /x;
808
809Which one you pick depends on which of these expressions better reflects
810the above specification of comments.
811
5a964f20 812=item C<(?(condition)yes-pattern|no-pattern)>
d74e8afc 813X<(?()>
c277df42 814
5a964f20 815=item C<(?(condition)yes-pattern)>
c277df42 816
19799a22 817B<WARNING>: This extended regular expression feature is considered
818highly experimental, and may be changed or deleted without notice.
819
c277df42 820Conditional expression. C<(condition)> should be either an integer in
821parentheses (which is valid if the corresponding pair of parentheses
19799a22 822matched), or look-ahead/look-behind/evaluate zero-width assertion.
c277df42 823
19799a22 824For example:
c277df42 825
5a964f20 826 m{ ( \( )?
871b0233 827 [^()]+
5a964f20 828 (?(1) \) )
871b0233 829 }x
c277df42 830
831matches a chunk of non-parentheses, possibly included in parentheses
832themselves.
a0d0e21e 833
a0d0e21e 834=back
835
c07a80fd 836=head2 Backtracking
d74e8afc 837X<backtrack> X<backtracking>
c07a80fd 838
35a734be 839NOTE: This section presents an abstract approximation of regular
840expression behavior. For a more rigorous (and complicated) view of
841the rules involved in selecting a match among possible alternatives,
842see L<Combining pieces together>.
843
c277df42 844A fundamental feature of regular expression matching involves the
5a964f20 845notion called I<backtracking>, which is currently used (when needed)
c277df42 846by all regular expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc 847C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
848internally, but the general principle outlined here is valid.
c07a80fd 849
850For a regular expression to match, the I<entire> regular expression must
851match, not just part of it. So if the beginning of a pattern containing a
852quantifier succeeds in a way that causes later parts in the pattern to
853fail, the matching engine backs up and recalculates the beginning
854part--that's why it's called backtracking.
855
856Here is an example of backtracking: Let's say you want to find the
857word following "foo" in the string "Food is on the foo table.":
858
859 $_ = "Food is on the foo table.";
860 if ( /\b(foo)\s+(\w+)/i ) {
861 print "$2 follows $1.\n";
862 }
863
864When the match runs, the first part of the regular expression (C<\b(foo)>)
865finds a possible match right at the beginning of the string, and loads up
866$1 with "Foo". However, as soon as the matching engine sees that there's
867no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 868mistake and starts over again one character after where it had the
c07a80fd 869tentative match. This time it goes all the way until the next occurrence
870of "foo". The complete regular expression matches this time, and you get
871the expected output of "table follows foo."
872
873Sometimes minimal matching can help a lot. Imagine you'd like to match
874everything between "foo" and "bar". Initially, you write something
875like this:
876
877 $_ = "The food is under the bar in the barn.";
878 if ( /foo(.*)bar/ ) {
879 print "got <$1>\n";
880 }
881
882Which perhaps unexpectedly yields:
883
884 got <d is under the bar in the >
885
886That's because C<.*> was greedy, so you get everything between the
14218588 887I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd 888to use minimal matching to make sure you get the text between a "foo"
889and the first "bar" thereafter.
890
891 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
892 got <d is under the >
893
894Here's another example: let's say you'd like to match a number at the end
b6e13d97 895of a string, and you also want to keep the preceding part of the match.
c07a80fd 896So you write this:
897
898 $_ = "I have 2 numbers: 53147";
899 if ( /(.*)(\d*)/ ) { # Wrong!
900 print "Beginning is <$1>, number is <$2>.\n";
901 }
902
903That won't work at all, because C<.*> was greedy and gobbled up the
904whole string. As C<\d*> can match on an empty string the complete
905regular expression matched successfully.
906
8e1088bc 907 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd 908
909Here are some variants, most of which don't work:
910
911 $_ = "I have 2 numbers: 53147";
912 @pats = qw{
913 (.*)(\d*)
914 (.*)(\d+)
915 (.*?)(\d*)
916 (.*?)(\d+)
917 (.*)(\d+)$
918 (.*?)(\d+)$
919 (.*)\b(\d+)$
920 (.*\D)(\d+)$
921 };
922
923 for $pat (@pats) {
924 printf "%-12s ", $pat;
925 if ( /$pat/ ) {
926 print "<$1> <$2>\n";
927 } else {
928 print "FAIL\n";
929 }
930 }
931
932That will print out:
933
934 (.*)(\d*) <I have 2 numbers: 53147> <>
935 (.*)(\d+) <I have 2 numbers: 5314> <7>
936 (.*?)(\d*) <> <>
937 (.*?)(\d+) <I have > <2>
938 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
939 (.*?)(\d+)$ <I have 2 numbers: > <53147>
940 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
941 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
942
943As you see, this can be a bit tricky. It's important to realize that a
944regular expression is merely a set of assertions that gives a definition
945of success. There may be 0, 1, or several different ways that the
946definition might succeed against a particular string. And if there are
5a964f20 947multiple ways it might succeed, you need to understand backtracking to
948know which variety of success you will achieve.
c07a80fd 949
19799a22 950When using look-ahead assertions and negations, this can all get even
8b19b778 951trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd 952followed by "123". You might try to write that as
953
871b0233 954 $_ = "ABC123";
955 if ( /^\D*(?!123)/ ) { # Wrong!
956 print "Yup, no 123 in $_\n";
957 }
c07a80fd 958
959But that isn't going to match; at least, not the way you're hoping. It
960claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 961why that pattern matches, contrary to popular expectations:
c07a80fd 962
963 $x = 'ABC123' ;
964 $y = 'ABC445' ;
965
966 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/ ;
967 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/ ;
968
969 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/ ;
970 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/ ;
971
972This prints
973
974 2: got ABC
975 3: got AB
976 4: got ABC
977
5f05dabc 978You might have expected test 3 to fail because it seems to a more
c07a80fd 979general purpose version of test 1. The important difference between
980them is that test 3 contains a quantifier (C<\D*>) and so can use
981backtracking, whereas test 1 will not. What's happening is
982that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 983non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 984let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 985fail.
14218588 986
c07a80fd 987The search engine will initially match C<\D*> with "ABC". Then it will
14218588 988try to match C<(?!123> with "123", which fails. But because
c07a80fd 989a quantifier (C<\D*>) has been used in the regular expression, the
990search engine can backtrack and retry the match differently
54310121 991in the hope of matching the complete regular expression.
c07a80fd 992
5a964f20 993The pattern really, I<really> wants to succeed, so it uses the
994standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 995time. Now there's indeed something following "AB" that is not
14218588 996"123". It's "C123", which suffices.
c07a80fd 997
14218588 998We can deal with this by using both an assertion and a negation.
999We'll say that the first part in $1 must be followed both by a digit
1000and by something that's not "123". Remember that the look-aheads
1001are zero-width expressions--they only look, but don't consume any
1002of the string in their match. So rewriting this way produces what
c07a80fd 1003you'd expect; that is, case 5 will fail, but case 6 succeeds:
1004
1005 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/ ;
1006 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/ ;
1007
1008 6: got ABC
1009
5a964f20 1010In other words, the two zero-width assertions next to each other work as though
19799a22 1011they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd 1012matches only if you're at the beginning of the line AND the end of the
1013line simultaneously. The deeper underlying truth is that juxtaposition in
1014regular expressions always means AND, except when you write an explicit OR
1015using the vertical bar. C</ab/> means match "a" AND (then) match "b",
1016although the attempted matches are made at different positions because "a"
1017is not a zero-width assertion, but a one-width assertion.
1018
19799a22 1019B<WARNING>: particularly complicated regular expressions can take
14218588 1020exponential time to solve because of the immense number of possible
9da458fc 1021ways they can use backtracking to try match. For example, without
1022internal optimizations done by the regular expression engine, this will
1023take a painfully long time to run:
c07a80fd 1024
e1901655 1025 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
1026
1027And if you used C<*>'s in the internal groups instead of limiting them
1028to 0 through 5 matches, then it would take forever--or until you ran
1029out of stack space. Moreover, these internal optimizations are not
1030always applicable. For example, if you put C<{0,5}> instead of C<*>
1031on the external group, no current optimization is applicable, and the
1032match takes a long time to finish.
c07a80fd 1033
9da458fc 1034A powerful tool for optimizing such beasts is what is known as an
1035"independent group",
c47ff5f1 1036which does not backtrack (see L<C<< (?>pattern) >>>). Note also that
9da458fc 1037zero-length look-ahead/look-behind assertions will not backtrack to make
14218588 1038the tail match, since they are in "logical" context: only
1039whether they match is considered relevant. For an example
9da458fc 1040where side-effects of look-ahead I<might> have influenced the
c47ff5f1 1041following match, see L<C<< (?>pattern) >>>.
c277df42 1042
a0d0e21e 1043=head2 Version 8 Regular Expressions
d74e8afc 1044X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
a0d0e21e 1045
5a964f20 1046In case you're not familiar with the "regular" Version 8 regex
a0d0e21e 1047routines, here are the pattern-matching rules not described above.
1048
54310121 1049Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 1050with a special meaning described here or above. You can cause
5a964f20 1051characters that normally function as metacharacters to be interpreted
5f05dabc 1052literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
a0d0e21e 1053character; "\\" matches a "\"). A series of characters matches that
1054series of characters in the target string, so the pattern C<blurfl>
1055would match "blurfl" in the target string.
1056
1057You can specify a character class, by enclosing a list of characters
5a964f20 1058in C<[]>, which will match any one character from the list. If the
a0d0e21e 1059first character after the "[" is "^", the class matches any character not
14218588 1060in the list. Within a list, the "-" character specifies a
5a964f20 1061range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2 1062inclusive. If you want either "-" or "]" itself to be a member of a
1063class, put it at the start of the list (possibly after a "^"), or
1064escape it with a backslash. "-" is also taken literally when it is
1065at the end of the list, just before the closing "]". (The
84850974 1066following all specify the same class of three characters: C<[-az]>,
1067C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
fb55449c 1068specifies a class containing twenty-six characters, even on EBCDIC
1069based coded character sets.) Also, if you try to use the character
1070classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1071a range, that's not a range, the "-" is understood literally.
a0d0e21e 1072
8ada0baa 1073Note also that the whole range idea is rather unportable between
1074character sets--and even within character sets they may cause results
1075you probably didn't expect. A sound principle is to use only ranges
1076that begin from and end at either alphabets of equal case ([a-e],
1077[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1078spell out the character sets in full.
1079
54310121 1080Characters may be specified using a metacharacter syntax much like that
a0d0e21e 1081used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1082"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
fb55449c 1083of octal digits, matches the character whose coded character set value
1084is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
1085matches the character whose numeric value is I<nn>. The expression \cI<x>
1086matches the character control-I<x>. Finally, the "." metacharacter
1087matches any character except "\n" (unless you use C</s>).
a0d0e21e 1088
1089You can specify a series of alternatives for a pattern using "|" to
1090separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1091or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e 1092first alternative includes everything from the last pattern delimiter
1093("(", "[", or the beginning of the pattern) up to the first "|", and
1094the last alternative contains everything from the last "|" to the next
14218588 1095pattern delimiter. That's why it's common practice to include
1096alternatives in parentheses: to minimize confusion about where they
a3cb178b 1097start and end.
1098
5a964f20 1099Alternatives are tried from left to right, so the first
a3cb178b 1100alternative found for which the entire expression matches, is the one that
1101is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1102example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b 1103part will match, as that is the first alternative tried, and it successfully
1104matches the target string. (This might not seem important, but it is
1105important when you are capturing matched text using parentheses.)
1106
5a964f20 1107Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1108so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1109
14218588 1110Within a pattern, you may designate subpatterns for later reference
1111by enclosing them in parentheses, and you may refer back to the
1112I<n>th subpattern later in the pattern using the metacharacter
1113\I<n>. Subpatterns are numbered based on the left to right order
1114of their opening parenthesis. A backreference matches whatever
1115actually matched the subpattern in the string being examined, not
1116the rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will
1117match "0x1234 0x4321", but not "0x1234 01234", because subpattern
11181 matched "0x", even though the rule C<0|0x> could potentially match
1119the leading 0 in the second number.
cb1a09d0 1120
19799a22 1121=head2 Warning on \1 vs $1
cb1a09d0 1122
5a964f20 1123Some people get too used to writing things like:
cb1a09d0 1124
1125 $pattern =~ s/(\W)/\\\1/g;
1126
1127This is grandfathered for the RHS of a substitute to avoid shocking the
1128B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1129PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0 1130the usual double-quoted string means a control-A. The customary Unix
1131meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1132of doing that, you get yourself into trouble if you then add an C</e>
1133modifier.
1134
5a964f20 1135 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0 1136
1137Or if you try to do
1138
1139 s/(\d+)/\1000/;
1140
1141You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1142C<${1}000>. The operation of interpolation should not be confused
cb1a09d0 1143with the operation of matching a backreference. Certainly they mean two
1144different things on the I<left> side of the C<s///>.
9fa51da4 1145
c84d73f1 1146=head2 Repeated patterns matching zero-length substring
1147
19799a22 1148B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1 1149
1150Regular expressions provide a terse and powerful programming language. As
1151with most other power tools, power comes together with the ability
1152to wreak havoc.
1153
1154A common abuse of this power stems from the ability to make infinite
628afcb5 1155loops using regular expressions, with something as innocuous as:
c84d73f1 1156
1157 'foo' =~ m{ ( o? )* }x;
1158
1159The C<o?> can match at the beginning of C<'foo'>, and since the position
1160in the string is not moved by the match, C<o?> would match again and again
14218588 1161because of the C<*> modifier. Another common way to create a similar cycle
c84d73f1 1162is with the looping modifier C<//g>:
1163
1164 @matches = ( 'foo' =~ m{ o? }xg );
1165
1166or
1167
1168 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
1169
1170or the loop implied by split().
1171
1172However, long experience has shown that many programming tasks may
14218588 1173be significantly simplified by using repeated subexpressions that
1174may match zero-length substrings. Here's a simple example being:
c84d73f1 1175
1176 @chars = split //, $string; # // is not magic in split
1177 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
1178
9da458fc 1179Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1 1180the infinite loop>. The rules for this are different for lower-level
1181loops given by the greedy modifiers C<*+{}>, and for higher-level
1182ones like the C</g> modifier or split() operator.
1183
19799a22 1184The lower-level loops are I<interrupted> (that is, the loop is
1185broken) when Perl detects that a repeated expression matched a
1186zero-length substring. Thus
c84d73f1 1187
1188 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
1189
1190is made equivalent to
1191
1192 m{ (?: NON_ZERO_LENGTH )*
1193 |
1194 (?: ZERO_LENGTH )?
1195 }x;
1196
1197The higher level-loops preserve an additional state between iterations:
1198whether the last match was zero-length. To break the loop, the following
1199match after a zero-length match is prohibited to have a length of zero.
1200This prohibition interacts with backtracking (see L<"Backtracking">),
1201and so the I<second best> match is chosen if the I<best> match is of
1202zero length.
1203
19799a22 1204For example:
c84d73f1 1205
1206 $_ = 'bar';
1207 s/\w??/<$&>/g;
1208
20fb949f 1209results in C<< <><b><><a><><r><> >>. At each position of the string the best
c84d73f1 1210match given by non-greedy C<??> is the zero-length match, and the I<second
1211best> match is what is matched by C<\w>. Thus zero-length matches
1212alternate with one-character-long matches.
1213
1214Similarly, for repeated C<m/()/g> the second-best match is the match at the
1215position one notch further in the string.
1216
19799a22 1217The additional state of being I<matched with zero-length> is associated with
c84d73f1 1218the matched string, and is reset by each assignment to pos().
9da458fc 1219Zero-length matches at the end of the previous match are ignored
1220during C<split>.
c84d73f1 1221
35a734be 1222=head2 Combining pieces together
1223
1224Each of the elementary pieces of regular expressions which were described
1225before (such as C<ab> or C<\Z>) could match at most one substring
1226at the given position of the input string. However, in a typical regular
1227expression these elementary pieces are combined into more complicated
1228patterns using combining operators C<ST>, C<S|T>, C<S*> etc
1229(in these examples C<S> and C<T> are regular subexpressions).
1230
1231Such combinations can include alternatives, leading to a problem of choice:
1232if we match a regular expression C<a|ab> against C<"abc">, will it match
1233substring C<"a"> or C<"ab">? One way to describe which substring is
1234actually matched is the concept of backtracking (see L<"Backtracking">).
1235However, this description is too low-level and makes you think
1236in terms of a particular implementation.
1237
1238Another description starts with notions of "better"/"worse". All the
1239substrings which may be matched by the given regular expression can be
1240sorted from the "best" match to the "worst" match, and it is the "best"
1241match which is chosen. This substitutes the question of "what is chosen?"
1242by the question of "which matches are better, and which are worse?".
1243
1244Again, for elementary pieces there is no such question, since at most
1245one match at a given position is possible. This section describes the
1246notion of better/worse for combining operators. In the description
1247below C<S> and C<T> are regular subexpressions.
1248
13a2d996 1249=over 4
35a734be 1250
1251=item C<ST>
1252
1253Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
1254substrings which can be matched by C<S>, C<B> and C<B'> are substrings
1255which can be matched by C<T>.
1256
1257If C<A> is better match for C<S> than C<A'>, C<AB> is a better
1258match than C<A'B'>.
1259
1260If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
1261C<B> is better match for C<T> than C<B'>.
1262
1263=item C<S|T>
1264
1265When C<S> can match, it is a better match than when only C<T> can match.
1266
1267Ordering of two matches for C<S> is the same as for C<S>. Similar for
1268two matches for C<T>.
1269
1270=item C<S{REPEAT_COUNT}>
1271
1272Matches as C<SSS...S> (repeated as many times as necessary).
1273
1274=item C<S{min,max}>
1275
1276Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
1277
1278=item C<S{min,max}?>
1279
1280Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
1281
1282=item C<S?>, C<S*>, C<S+>
1283
1284Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
1285
1286=item C<S??>, C<S*?>, C<S+?>
1287
1288Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
1289
c47ff5f1 1290=item C<< (?>S) >>
35a734be 1291
1292Matches the best match for C<S> and only that.
1293
1294=item C<(?=S)>, C<(?<=S)>
1295
1296Only the best match for C<S> is considered. (This is important only if
1297C<S> has capturing parentheses, and backreferences are used somewhere
1298else in the whole regular expression.)
1299
1300=item C<(?!S)>, C<(?<!S)>
1301
1302For this grouping operator there is no need to describe the ordering, since
1303only whether or not C<S> can match is important.
1304
14455d6c 1305=item C<(??{ EXPR })>
35a734be 1306
1307The ordering is the same as for the regular expression which is
1308the result of EXPR.
1309
1310=item C<(?(condition)yes-pattern|no-pattern)>
1311
1312Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
1313already determined. The ordering of the matches is the same as for the
1314chosen subexpression.
1315
1316=back
1317
1318The above recipes describe the ordering of matches I<at a given position>.
1319One more rule is needed to understand how a match is determined for the
1320whole regular expression: a match at an earlier position is always better
1321than a match at a later position.
1322
c84d73f1 1323=head2 Creating custom RE engines
1324
1325Overloaded constants (see L<overload>) provide a simple way to extend
1326the functionality of the RE engine.
1327
1328Suppose that we want to enable a new RE escape-sequence C<\Y|> which
6b0ac556 1329matches at boundary between whitespace characters and non-whitespace
c84d73f1 1330characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
1331at these positions, so we want to have each C<\Y|> in the place of the
1332more complicated version. We can create a module C<customre> to do
1333this:
1334
1335 package customre;
1336 use overload;
1337
1338 sub import {
1339 shift;
1340 die "No argument to customre::import allowed" if @_;
1341 overload::constant 'qr' => \&convert;
1342 }
1343
1344 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
1345
580a9fe1 1346 # We must also take care of not escaping the legitimate \\Y|
1347 # sequence, hence the presence of '\\' in the conversion rules.
141db969 1348 my %rules = ( '\\' => '\\\\',
c84d73f1 1349 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
1350 sub convert {
1351 my $re = shift;
1352 $re =~ s{
1353 \\ ( \\ | Y . )
1354 }
1355 { $rules{$1} or invalid($re,$1) }sgex;
1356 return $re;
1357 }
1358
1359Now C<use customre> enables the new escape in constant regular
1360expressions, i.e., those without any runtime variable interpolations.
1361As documented in L<overload>, this conversion will work only over
1362literal parts of regular expressions. For C<\Y|$re\Y|> the variable
1363part of this regular expression needs to be converted explicitly
1364(but only if the special meaning of C<\Y|> should be enabled inside $re):
1365
1366 use customre;
1367 $re = <>;
1368 chomp $re;
1369 $re = customre::convert $re;
1370 /\Y|$re\Y|/;
1371
19799a22 1372=head1 BUGS
1373
9da458fc 1374This document varies from difficult to understand to completely
1375and utterly opaque. The wandering prose riddled with jargon is
1376hard to fathom in several places.
1377
1378This document needs a rewrite that separates the tutorial content
1379from the reference content.
19799a22 1380
1381=head1 SEE ALSO
9fa51da4 1382
91e0c79e 1383L<perlrequick>.
1384
1385L<perlretut>.
1386
9b599b2a 1387L<perlop/"Regexp Quote-Like Operators">.
1388
1e66bd83 1389L<perlop/"Gory details of parsing quoted constructs">.
1390
14218588 1391L<perlfaq6>.
1392
9b599b2a 1393L<perlfunc/pos>.
1394
1395L<perllocale>.
1396
fb55449c 1397L<perlebcdic>.
1398
14218588 1399I<Mastering Regular Expressions> by Jeffrey Friedl, published
1400by O'Reilly and Associates.