Commit | Line | Data |
a0d0e21e |
1 | =head1 NAME |
2 | |
3 | perlre - Perl regular expressions |
4 | |
5 | =head1 DESCRIPTION |
6 | |
cb1a09d0 |
7 | This page describes the syntax of regular expressions in Perl. For a |
5f05dabc |
8 | description of how to I<use> regular expressions in matching |
75e14d17 |
9 | operations, plus various examples of the same, see discussion |
0a92e3a8 |
10 | of C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/Regexp Quote-Like Operators>. |
cb1a09d0 |
11 | |
68dc0745 |
12 | The matching operations can have various modifiers. The modifiers |
5a964f20 |
13 | that relate to the interpretation of the regular expression inside |
75e14d17 |
14 | are listed below. For the modifiers that alter the way regular expression |
15 | is used by Perl, see L<perlop/Regexp Quote-Like Operators>. |
a0d0e21e |
16 | |
55497cff |
17 | =over 4 |
18 | |
19 | =item i |
20 | |
21 | Do case-insensitive pattern matching. |
22 | |
a034a98d |
23 | If C<use locale> is in effect, the case map is taken from the current |
24 | locale. See L<perllocale>. |
25 | |
54310121 |
26 | =item m |
55497cff |
27 | |
28 | Treat string as multiple lines. That is, change "^" and "$" from matching |
5f05dabc |
29 | at only the very start or end of the string to the start or end of any |
55497cff |
30 | line anywhere within the string, |
31 | |
54310121 |
32 | =item s |
55497cff |
33 | |
34 | Treat string as single line. That is, change "." to match any character |
35 | whatsoever, even a newline, which it normally would not match. |
36 | |
5a964f20 |
37 | The C</s> and C</m> modifiers both override the C<$*> setting. That is, no matter |
38 | what C<$*> contains, C</s> without C</m> will force "^" to match only at the |
7b8d334a |
39 | beginning of the string and "$" to match only at the end (or just before a |
40 | newline at the end) of the string. Together, as /ms, they let the "." match |
41 | any character whatsoever, while yet allowing "^" and "$" to match, |
42 | respectively, just after and just before newlines within the string. |
43 | |
54310121 |
44 | =item x |
55497cff |
45 | |
46 | Extend your pattern's legibility by permitting whitespace and comments. |
47 | |
48 | =back |
a0d0e21e |
49 | |
50 | These are usually written as "the C</x> modifier", even though the delimiter |
51 | in question might not actually be a slash. In fact, any of these |
52 | modifiers may also be embedded within the regular expression itself using |
53 | the new C<(?...)> construct. See below. |
54 | |
4633a7c4 |
55 | The C</x> modifier itself needs a little more explanation. It tells |
55497cff |
56 | the regular expression parser to ignore whitespace that is neither |
57 | backslashed nor within a character class. You can use this to break up |
4633a7c4 |
58 | your regular expression into (slightly) more readable parts. The C<#> |
54310121 |
59 | character is also treated as a metacharacter introducing a comment, |
55497cff |
60 | just as in ordinary Perl code. This also means that if you want real |
5a964f20 |
61 | whitespace or C<#> characters in the pattern (outside of a character |
62 | class, where they are unaffected by C</x>), that you'll either have to |
55497cff |
63 | escape them or encode them using octal or hex escapes. Taken together, |
64 | these features go a long way towards making Perl's regular expressions |
0c815be9 |
65 | more readable. Note that you have to be careful not to include the |
66 | pattern delimiter in the comment--perl has no way of knowing you did |
5a964f20 |
67 | not intend to close the pattern early. See the C-comment deletion code |
0c815be9 |
68 | in L<perlop>. |
a0d0e21e |
69 | |
70 | =head2 Regular Expressions |
71 | |
72 | The patterns used in pattern matching are regular expressions such as |
5a964f20 |
73 | those supplied in the Version 8 regex routines. (In fact, the |
a0d0e21e |
74 | routines are derived (distantly) from Henry Spencer's freely |
75 | redistributable reimplementation of the V8 routines.) |
76 | See L<Version 8 Regular Expressions> for details. |
77 | |
78 | In particular the following metacharacters have their standard I<egrep>-ish |
79 | meanings: |
80 | |
54310121 |
81 | \ Quote the next metacharacter |
a0d0e21e |
82 | ^ Match the beginning of the line |
83 | . Match any character (except newline) |
c07a80fd |
84 | $ Match the end of the line (or before newline at the end) |
a0d0e21e |
85 | | Alternation |
86 | () Grouping |
87 | [] Character class |
88 | |
5f05dabc |
89 | By default, the "^" character is guaranteed to match at only the |
90 | beginning of the string, the "$" character at only the end (or before the |
a0d0e21e |
91 | newline at the end) and Perl does certain optimizations with the |
92 | assumption that the string contains only one line. Embedded newlines |
93 | will not be matched by "^" or "$". You may, however, wish to treat a |
4a6725af |
94 | string as a multi-line buffer, such that the "^" will match after any |
a0d0e21e |
95 | newline within the string, and "$" will match before any newline. At the |
96 | cost of a little more overhead, you can do this by using the /m modifier |
97 | on the pattern match operator. (Older programs did this by setting C<$*>, |
5f05dabc |
98 | but this practice is now deprecated.) |
a0d0e21e |
99 | |
4a6725af |
100 | To facilitate multi-line substitutions, the "." character never matches a |
55497cff |
101 | newline unless you use the C</s> modifier, which in effect tells Perl to pretend |
a0d0e21e |
102 | the string is a single line--even if it isn't. The C</s> modifier also |
103 | overrides the setting of C<$*>, in case you have some (badly behaved) older |
104 | code that sets it in another module. |
105 | |
106 | The following standard quantifiers are recognized: |
107 | |
108 | * Match 0 or more times |
109 | + Match 1 or more times |
110 | ? Match 1 or 0 times |
111 | {n} Match exactly n times |
112 | {n,} Match at least n times |
113 | {n,m} Match at least n but not more than m times |
114 | |
115 | (If a curly bracket occurs in any other context, it is treated |
116 | as a regular character.) The "*" modifier is equivalent to C<{0,}>, the "+" |
25f94b33 |
117 | modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited |
c07a80fd |
118 | to integral values less than 65536. |
a0d0e21e |
119 | |
54310121 |
120 | By default, a quantified subpattern is "greedy", that is, it will match as |
121 | many times as possible (given a particular starting location) while still |
122 | allowing the rest of the pattern to match. If you want it to match the |
123 | minimum number of times possible, follow the quantifier with a "?". Note |
124 | that the meanings don't change, just the "greediness": |
a0d0e21e |
125 | |
126 | *? Match 0 or more times |
127 | +? Match 1 or more times |
128 | ?? Match 0 or 1 time |
129 | {n}? Match exactly n times |
130 | {n,}? Match at least n times |
131 | {n,m}? Match at least n but not more than m times |
132 | |
5f05dabc |
133 | Because patterns are processed as double quoted strings, the following |
a0d0e21e |
134 | also work: |
135 | |
0f36ee90 |
136 | \t tab (HT, TAB) |
137 | \n newline (LF, NL) |
138 | \r return (CR) |
139 | \f form feed (FF) |
140 | \a alarm (bell) (BEL) |
141 | \e escape (think troff) (ESC) |
cb1a09d0 |
142 | \033 octal char (think of a PDP-11) |
143 | \x1B hex char |
a0d0e21e |
144 | \c[ control char |
cb1a09d0 |
145 | \l lowercase next char (think vi) |
146 | \u uppercase next char (think vi) |
147 | \L lowercase till \E (think vi) |
148 | \U uppercase till \E (think vi) |
149 | \E end case modification (think vi) |
5a964f20 |
150 | \Q quote (disable) pattern metacharacters till \E |
a0d0e21e |
151 | |
a034a98d |
152 | If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u> |
7b8d334a |
153 | and C<\U> is taken from the current locale. See L<perllocale>. |
a034a98d |
154 | |
1d2dff63 |
155 | You cannot include a literal C<$> or C<@> within a C<\Q> sequence. |
156 | An unescaped C<$> or C<@> interpolates the corresponding variable, |
157 | while escaping will cause the literal string C<\$> to be matched. |
158 | You'll need to write something like C<m/\Quser\E\@\Qhost/>. |
159 | |
a0d0e21e |
160 | In addition, Perl defines the following: |
161 | |
162 | \w Match a "word" character (alphanumeric plus "_") |
163 | \W Match a non-word character |
164 | \s Match a whitespace character |
165 | \S Match a non-whitespace character |
166 | \d Match a digit character |
167 | \D Match a non-digit character |
168 | |
5a964f20 |
169 | A C<\w> matches a single alphanumeric character, not a whole |
a034a98d |
170 | word. To match a word you'd need to say C<\w+>. If C<use locale> is in |
171 | effect, the list of alphabetic characters generated by C<\w> is taken |
172 | from the current locale. See L<perllocale>. You may use C<\w>, C<\W>, |
173 | C<\s>, C<\S>, C<\d>, and C<\D> within character classes (though not as |
174 | either end of a range). |
a0d0e21e |
175 | |
176 | Perl defines the following zero-width assertions: |
177 | |
178 | \b Match a word boundary |
179 | \B Match a non-(word boundary) |
b85d18e9 |
180 | \A Match only at beginning of string |
181 | \Z Match only at end of string, or before newline at the end |
182 | \z Match only at end of string |
a99df21c |
183 | \G Match only where previous m//g left off (works only with /g) |
a0d0e21e |
184 | |
185 | A word boundary (C<\b>) is defined as a spot between two characters that |
68dc0745 |
186 | has a C<\w> on one side of it and a C<\W> on the other side of it (in |
a0d0e21e |
187 | either order), counting the imaginary characters off the beginning and |
188 | end of the string as matching a C<\W>. (Within character classes C<\b> |
189 | represents backspace rather than a word boundary.) The C<\A> and C<\Z> are |
5a964f20 |
190 | just like "^" and "$", except that they won't match multiple times when the |
a0d0e21e |
191 | C</m> modifier is used, while "^" and "$" will match at every internal line |
c07a80fd |
192 | boundary. To match the actual end of the string, not ignoring newline, |
b85d18e9 |
193 | you can use C<\z>. The C<\G> assertion can be used to chain global |
a99df21c |
194 | matches (using C<m//g>), as described in |
e7ea3e70 |
195 | L<perlop/"Regexp Quote-Like Operators">. |
a99df21c |
196 | |
e7ea3e70 |
197 | It is also useful when writing C<lex>-like scanners, when you have several |
5a964f20 |
198 | patterns that you want to match against consequent substrings of your |
e7ea3e70 |
199 | string, see the previous reference. |
44a8e56a |
200 | The actual location where C<\G> will match can also be influenced |
201 | by using C<pos()> as an lvalue. See L<perlfunc/pos>. |
a0d0e21e |
202 | |
0f36ee90 |
203 | When the bracketing construct C<( ... )> is used, \E<lt>digitE<gt> matches the |
cb1a09d0 |
204 | digit'th substring. Outside of the pattern, always use "$" instead of "\" |
0f36ee90 |
205 | in front of the digit. (While the \E<lt>digitE<gt> notation can on rare occasion work |
cb1a09d0 |
206 | outside the current pattern, this should not be relied upon. See the |
0f36ee90 |
207 | WARNING below.) The scope of $E<lt>digitE<gt> (and C<$`>, C<$&>, and C<$'>) |
cb1a09d0 |
208 | extends to the end of the enclosing BLOCK or eval string, or to the next |
209 | successful pattern match, whichever comes first. If you want to use |
5f05dabc |
210 | parentheses to delimit a subpattern (e.g., a set of alternatives) without |
84dc3c4d |
211 | saving it as a subpattern, follow the ( with a ?:. |
cb1a09d0 |
212 | |
213 | You may have as many parentheses as you wish. If you have more |
a0d0e21e |
214 | than 9 substrings, the variables $10, $11, ... refer to the |
215 | corresponding substring. Within the pattern, \10, \11, etc. refer back |
5f05dabc |
216 | to substrings if there have been at least that many left parentheses before |
c07a80fd |
217 | the backreference. Otherwise (for backward compatibility) \10 is the |
a0d0e21e |
218 | same as \010, a backspace, and \11 the same as \011, a tab. And so |
219 | on. (\1 through \9 are always backreferences.) |
220 | |
221 | C<$+> returns whatever the last bracket match matched. C<$&> returns the |
0f36ee90 |
222 | entire matched string. (C<$0> used to return the same thing, but not any |
a0d0e21e |
223 | more.) C<$`> returns everything before the matched string. C<$'> returns |
224 | everything after the matched string. Examples: |
225 | |
226 | s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words |
227 | |
228 | if (/Time: (..):(..):(..)/) { |
229 | $hours = $1; |
230 | $minutes = $2; |
231 | $seconds = $3; |
232 | } |
233 | |
68dc0745 |
234 | Once perl sees that you need one of C<$&>, C<$`> or C<$'> anywhere in |
235 | the program, it has to provide them on each and every pattern match. |
236 | This can slow your program down. The same mechanism that handles |
237 | these provides for the use of $1, $2, etc., so you pay the same price |
5a964f20 |
238 | for each pattern that contains capturing parentheses. But if you never |
239 | use $&, etc., in your script, then patterns I<without> capturing |
68dc0745 |
240 | parentheses won't be penalized. So avoid $&, $', and $` if you can, |
241 | but if you can't (and some algorithms really appreciate them), once |
242 | you've used them once, use them at will, because you've already paid |
5a964f20 |
243 | the price. As of 5.005, $& is not so costly as the other two. |
68dc0745 |
244 | |
5a964f20 |
245 | Backslashed metacharacters in Perl are |
201ecf35 |
246 | alphanumeric, such as C<\b>, C<\w>, C<\n>. Unlike some other regular |
247 | expression languages, there are no backslashed symbols that aren't |
248 | alphanumeric. So anything that looks like \\, \(, \), \E<lt>, \E<gt>, |
249 | \{, or \} is always interpreted as a literal character, not a |
250 | metacharacter. This was once used in a common idiom to disable or |
251 | quote the special meanings of regular expression metacharacters in a |
5a964f20 |
252 | string that you want to use for a pattern. Simply quote all |
a0d0e21e |
253 | non-alphanumeric characters: |
254 | |
255 | $pattern =~ s/(\W)/\\$1/g; |
256 | |
201ecf35 |
257 | Now it is much more common to see either the quotemeta() function or |
7b8d334a |
258 | the C<\Q> escape sequence used to disable all metacharacters' special |
201ecf35 |
259 | meanings like this: |
a0d0e21e |
260 | |
261 | /$unquoted\Q$quoted\E$unquoted/ |
262 | |
5f05dabc |
263 | Perl defines a consistent extension syntax for regular expressions. |
264 | The syntax is a pair of parentheses with a question mark as the first |
265 | thing within the parentheses (this was a syntax error in older |
266 | versions of Perl). The character after the question mark gives the |
267 | function of the extension. Several extensions are already supported: |
a0d0e21e |
268 | |
269 | =over 10 |
270 | |
cc6b7395 |
271 | =item C<(?#text)> |
a0d0e21e |
272 | |
cb1a09d0 |
273 | A comment. The text is ignored. If the C</x> switch is used to enable |
259138e3 |
274 | whitespace formatting, a simple C<#> will suffice. Note that perl closes |
275 | the comment as soon as it sees a C<)>, so there is no way to put a literal |
276 | C<)> in the comment. |
a0d0e21e |
277 | |
5a964f20 |
278 | =item C<(?:pattern)> |
a0d0e21e |
279 | |
ca9dfc88 |
280 | =item C<(?imsx-imsx:pattern)> |
281 | |
5a964f20 |
282 | This is for clustering, not capturing; it groups subexpressions like |
283 | "()", but doesn't make backreferences as "()" does. So |
a0d0e21e |
284 | |
5a964f20 |
285 | @fields = split(/\b(?:a|b|c)\b/) |
a0d0e21e |
286 | |
287 | is like |
288 | |
5a964f20 |
289 | @fields = split(/\b(a|b|c)\b/) |
a0d0e21e |
290 | |
291 | but doesn't spit out extra fields. |
292 | |
ca9dfc88 |
293 | The letters between C<?> and C<:> act as flags modifiers, see |
294 | L<C<(?imsx-imsx)>>. In particular, |
295 | |
296 | /(?s-i:more.*than).*million/i |
297 | |
298 | is equivalent to more verbose |
299 | |
300 | /(?:(?s-i)more.*than).*million/i |
301 | |
5a964f20 |
302 | =item C<(?=pattern)> |
a0d0e21e |
303 | |
304 | A zero-width positive lookahead assertion. For example, C</\w+(?=\t)/> |
305 | matches a word followed by a tab, without including the tab in C<$&>. |
306 | |
5a964f20 |
307 | =item C<(?!pattern)> |
a0d0e21e |
308 | |
309 | A zero-width negative lookahead assertion. For example C</foo(?!bar)/> |
310 | matches any occurrence of "foo" that isn't followed by "bar". Note |
311 | however that lookahead and lookbehind are NOT the same thing. You cannot |
7b8d334a |
312 | use this for lookbehind. |
313 | |
5a964f20 |
314 | If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/> |
7b8d334a |
315 | will not do what you want. That's because the C<(?!foo)> is just saying that |
316 | the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will |
317 | match. You would have to do something like C</(?!foo)...bar/> for that. We |
318 | say "like" because there's the case of your "bar" not having three characters |
319 | before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>. |
320 | Sometimes it's still easier just to say: |
a0d0e21e |
321 | |
a3cb178b |
322 | if (/bar/ && $` !~ /foo$/) |
a0d0e21e |
323 | |
c277df42 |
324 | For lookbehind see below. |
325 | |
5a964f20 |
326 | =item C<(?E<lt>=pattern)> |
c277df42 |
327 | |
5a964f20 |
328 | A zero-width positive lookbehind assertion. For example, C</(?E<lt>=\t)\w+/> |
c277df42 |
329 | matches a word following a tab, without including the tab in C<$&>. |
330 | Works only for fixed-width lookbehind. |
331 | |
5a964f20 |
332 | =item C<(?<!pattern)> |
c277df42 |
333 | |
334 | A zero-width negative lookbehind assertion. For example C</(?<!bar)foo/> |
335 | matches any occurrence of "foo" that isn't following "bar". |
336 | Works only for fixed-width lookbehind. |
337 | |
cc6b7395 |
338 | =item C<(?{ code })> |
c277df42 |
339 | |
340 | Experimental "evaluate any Perl code" zero-width assertion. Always |
cc6b7395 |
341 | succeeds. C<code> is not interpolated. Currently the rules to |
342 | determine where the C<code> ends are somewhat convoluted. |
c277df42 |
343 | |
e4d48cc9 |
344 | Owing to the risks to security, this is only available when the |
345 | C<use re 'eval'> pragma is used, and then only for patterns that don't |
346 | have any variables that must be interpolated at run time. |
347 | |
b9ac3b5b |
348 | The C<code> is properly scoped in the following sense: if the assertion |
349 | is backtracked (compare L<"Backtracking">), all the changes introduced after |
350 | C<local>isation are undone, so |
351 | |
352 | $_ = 'a' x 8; |
353 | m< |
354 | (?{ $cnt = 0 }) # Initialize $cnt. |
355 | ( |
356 | a |
357 | (?{ |
358 | local $cnt = $cnt + 1; # Update $cnt, backtracking-safe. |
359 | }) |
360 | )* |
361 | aaaa |
362 | (?{ $res = $cnt }) # On success copy to non-localized |
363 | # location. |
364 | >x; |
365 | |
366 | will set C<$res = 4>. Note that after the match $cnt returns to the globally |
367 | introduced value 0, since the scopes which restrict C<local> statements |
368 | are unwound. |
369 | |
370 | This assertion may be used as L<C<(?(condition)yes-pattern|no-pattern)>> |
371 | switch. If I<not> used in this way, the result of evaluation of C<code> |
372 | is put into variable $^R. This happens immediately, so $^R can be used from |
373 | other C<(?{ code })> assertions inside the same regular expression. |
374 | |
375 | The above assignment to $^R is properly localized, thus the old value of $^R |
376 | is restored if the assertion is backtracked (compare L<"Backtracking">). |
377 | |
5a964f20 |
378 | =item C<(?E<gt>pattern)> |
379 | |
380 | An "independent" subexpression. Matches the substring that a |
381 | I<standalone> C<pattern> would match if anchored at the given position, |
c277df42 |
382 | B<and only this substring>. |
383 | |
384 | Say, C<^(?E<gt>a*)ab> will never match, since C<(?E<gt>a*)> (anchored |
5a964f20 |
385 | at the beginning of string, as above) will match I<all> characters |
c277df42 |
386 | C<a> at the beginning of string, leaving no C<a> for C<ab> to match. |
387 | In contrast, C<a*ab> will match the same as C<a+b>, since the match of |
388 | the subgroup C<a*> is influenced by the following group C<ab> (see |
389 | L<"Backtracking">). In particular, C<a*> inside C<a*ab> will match |
390 | less characters that a standalone C<a*>, since this makes the tail match. |
391 | |
5a964f20 |
392 | An effect similar to C<(?E<gt>pattern)> may be achieved by |
c277df42 |
393 | |
5a964f20 |
394 | (?=(pattern))\1 |
c277df42 |
395 | |
396 | since the lookahead is in I<"logical"> context, thus matches the same |
397 | substring as a standalone C<a+>. The following C<\1> eats the matched |
398 | string, thus making a zero-length assertion into an analogue of |
5a964f20 |
399 | C<(?>...)>. (The difference between these two constructs is that the |
400 | second one uses a catching group, thus shifting ordinals of |
c277df42 |
401 | backreferences in the rest of a regular expression.) |
402 | |
5a964f20 |
403 | This construct is useful for optimizations of "eternal" |
404 | matches, because it will not backtrack (see L<"Backtracking">). |
c277df42 |
405 | |
5a964f20 |
406 | m{ \( ( |
c277df42 |
407 | [^()]+ |
408 | | |
409 | \( [^()]* \) |
410 | )+ |
5a964f20 |
411 | \) |
412 | }x |
413 | |
414 | That will efficiently match a nonempty group with matching |
415 | two-or-less-level-deep parentheses. However, if there is no such group, |
416 | it will take virtually forever on a long string. That's because there are |
417 | so many different ways to split a long string into several substrings. |
418 | This is essentially what C<(.+)+> is doing, and this is a subpattern |
419 | of the above pattern. Consider that C<((()aaaaaaaaaaaaaaaaaa> on the |
420 | pattern above detects no-match in several seconds, but that each extra |
421 | letter doubles this time. This exponential performance will make it |
422 | appear that your program has hung. |
423 | |
424 | However, a tiny modification of this pattern |
425 | |
426 | m{ \( ( |
c277df42 |
427 | (?> [^()]+ ) |
428 | | |
429 | \( [^()]* \) |
430 | )+ |
5a964f20 |
431 | \) |
432 | }x |
c277df42 |
433 | |
5a964f20 |
434 | which uses C<(?E<gt>...)> matches exactly when the one above does (verifying |
435 | this yourself would be a productive exercise), but finishes in a fourth |
436 | the time when used on a similar string with 1000000 C<a>s. Be aware, |
437 | however, that this pattern currently triggers a warning message under |
438 | B<-w> saying it C<"matches the null string many times">): |
c277df42 |
439 | |
5a964f20 |
440 | On simple groups, such as the pattern C<(?> [^()]+ )>, a comparable |
c277df42 |
441 | effect may be achieved by negative lookahead, as in C<[^()]+ (?! [^()] )>. |
442 | This was only 4 times slower on a string with 1000000 C<a>s. |
443 | |
5a964f20 |
444 | =item C<(?(condition)yes-pattern|no-pattern)> |
c277df42 |
445 | |
5a964f20 |
446 | =item C<(?(condition)yes-pattern)> |
c277df42 |
447 | |
448 | Conditional expression. C<(condition)> should be either an integer in |
449 | parentheses (which is valid if the corresponding pair of parentheses |
450 | matched), or lookahead/lookbehind/evaluate zero-width assertion. |
451 | |
452 | Say, |
453 | |
5a964f20 |
454 | m{ ( \( )? |
c277df42 |
455 | [^()]+ |
5a964f20 |
456 | (?(1) \) ) |
457 | }x |
c277df42 |
458 | |
459 | matches a chunk of non-parentheses, possibly included in parentheses |
460 | themselves. |
a0d0e21e |
461 | |
ca9dfc88 |
462 | =item C<(?imsx-imsx)> |
a0d0e21e |
463 | |
464 | One or more embedded pattern-match modifiers. This is particularly |
465 | useful for patterns that are specified in a table somewhere, some of |
466 | which want to be case sensitive, and some of which don't. The case |
5f05dabc |
467 | insensitive ones need to include merely C<(?i)> at the front of the |
a0d0e21e |
468 | pattern. For example: |
469 | |
470 | $pattern = "foobar"; |
5a964f20 |
471 | if ( /$pattern/i ) { } |
a0d0e21e |
472 | |
473 | # more flexible: |
474 | |
475 | $pattern = "(?i)foobar"; |
5a964f20 |
476 | if ( /$pattern/ ) { } |
a0d0e21e |
477 | |
ca9dfc88 |
478 | Letters after C<-> switch modifiers off. |
479 | |
5a964f20 |
480 | These modifiers are localized inside an enclosing group (if any). Say, |
c277df42 |
481 | |
482 | ( (?i) blah ) \s+ \1 |
483 | |
484 | (assuming C<x> modifier, and no C<i> modifier outside of this group) |
485 | will match a repeated (I<including the case>!) word C<blah> in any |
486 | case. |
487 | |
a0d0e21e |
488 | =back |
489 | |
5a964f20 |
490 | A question mark was chosen for this and for the new minimal-matching |
491 | construct because 1) question mark is pretty rare in older regular |
492 | expressions, and 2) whenever you see one, you should stop and "question" |
493 | exactly what is going on. That's psychology... |
a0d0e21e |
494 | |
c07a80fd |
495 | =head2 Backtracking |
496 | |
c277df42 |
497 | A fundamental feature of regular expression matching involves the |
5a964f20 |
498 | notion called I<backtracking>, which is currently used (when needed) |
c277df42 |
499 | by all regular expression quantifiers, namely C<*>, C<*?>, C<+>, |
500 | C<+?>, C<{n,m}>, and C<{n,m}?>. |
c07a80fd |
501 | |
502 | For a regular expression to match, the I<entire> regular expression must |
503 | match, not just part of it. So if the beginning of a pattern containing a |
504 | quantifier succeeds in a way that causes later parts in the pattern to |
505 | fail, the matching engine backs up and recalculates the beginning |
506 | part--that's why it's called backtracking. |
507 | |
508 | Here is an example of backtracking: Let's say you want to find the |
509 | word following "foo" in the string "Food is on the foo table.": |
510 | |
511 | $_ = "Food is on the foo table."; |
512 | if ( /\b(foo)\s+(\w+)/i ) { |
513 | print "$2 follows $1.\n"; |
514 | } |
515 | |
516 | When the match runs, the first part of the regular expression (C<\b(foo)>) |
517 | finds a possible match right at the beginning of the string, and loads up |
518 | $1 with "Foo". However, as soon as the matching engine sees that there's |
519 | no whitespace following the "Foo" that it had saved in $1, it realizes its |
68dc0745 |
520 | mistake and starts over again one character after where it had the |
c07a80fd |
521 | tentative match. This time it goes all the way until the next occurrence |
522 | of "foo". The complete regular expression matches this time, and you get |
523 | the expected output of "table follows foo." |
524 | |
525 | Sometimes minimal matching can help a lot. Imagine you'd like to match |
526 | everything between "foo" and "bar". Initially, you write something |
527 | like this: |
528 | |
529 | $_ = "The food is under the bar in the barn."; |
530 | if ( /foo(.*)bar/ ) { |
531 | print "got <$1>\n"; |
532 | } |
533 | |
534 | Which perhaps unexpectedly yields: |
535 | |
536 | got <d is under the bar in the > |
537 | |
538 | That's because C<.*> was greedy, so you get everything between the |
539 | I<first> "foo" and the I<last> "bar". In this case, it's more effective |
540 | to use minimal matching to make sure you get the text between a "foo" |
541 | and the first "bar" thereafter. |
542 | |
543 | if ( /foo(.*?)bar/ ) { print "got <$1>\n" } |
544 | got <d is under the > |
545 | |
546 | Here's another example: let's say you'd like to match a number at the end |
547 | of a string, and you also want to keep the preceding part the match. |
548 | So you write this: |
549 | |
550 | $_ = "I have 2 numbers: 53147"; |
551 | if ( /(.*)(\d*)/ ) { # Wrong! |
552 | print "Beginning is <$1>, number is <$2>.\n"; |
553 | } |
554 | |
555 | That won't work at all, because C<.*> was greedy and gobbled up the |
556 | whole string. As C<\d*> can match on an empty string the complete |
557 | regular expression matched successfully. |
558 | |
8e1088bc |
559 | Beginning is <I have 2 numbers: 53147>, number is <>. |
c07a80fd |
560 | |
561 | Here are some variants, most of which don't work: |
562 | |
563 | $_ = "I have 2 numbers: 53147"; |
564 | @pats = qw{ |
565 | (.*)(\d*) |
566 | (.*)(\d+) |
567 | (.*?)(\d*) |
568 | (.*?)(\d+) |
569 | (.*)(\d+)$ |
570 | (.*?)(\d+)$ |
571 | (.*)\b(\d+)$ |
572 | (.*\D)(\d+)$ |
573 | }; |
574 | |
575 | for $pat (@pats) { |
576 | printf "%-12s ", $pat; |
577 | if ( /$pat/ ) { |
578 | print "<$1> <$2>\n"; |
579 | } else { |
580 | print "FAIL\n"; |
581 | } |
582 | } |
583 | |
584 | That will print out: |
585 | |
586 | (.*)(\d*) <I have 2 numbers: 53147> <> |
587 | (.*)(\d+) <I have 2 numbers: 5314> <7> |
588 | (.*?)(\d*) <> <> |
589 | (.*?)(\d+) <I have > <2> |
590 | (.*)(\d+)$ <I have 2 numbers: 5314> <7> |
591 | (.*?)(\d+)$ <I have 2 numbers: > <53147> |
592 | (.*)\b(\d+)$ <I have 2 numbers: > <53147> |
593 | (.*\D)(\d+)$ <I have 2 numbers: > <53147> |
594 | |
595 | As you see, this can be a bit tricky. It's important to realize that a |
596 | regular expression is merely a set of assertions that gives a definition |
597 | of success. There may be 0, 1, or several different ways that the |
598 | definition might succeed against a particular string. And if there are |
5a964f20 |
599 | multiple ways it might succeed, you need to understand backtracking to |
600 | know which variety of success you will achieve. |
c07a80fd |
601 | |
602 | When using lookahead assertions and negations, this can all get even |
54310121 |
603 | tricker. Imagine you'd like to find a sequence of non-digits not |
c07a80fd |
604 | followed by "123". You might try to write that as |
605 | |
606 | $_ = "ABC123"; |
607 | if ( /^\D*(?!123)/ ) { # Wrong! |
608 | print "Yup, no 123 in $_\n"; |
609 | } |
610 | |
611 | But that isn't going to match; at least, not the way you're hoping. It |
612 | claims that there is no 123 in the string. Here's a clearer picture of |
613 | why it that pattern matches, contrary to popular expectations: |
614 | |
615 | $x = 'ABC123' ; |
616 | $y = 'ABC445' ; |
617 | |
618 | print "1: got $1\n" if $x =~ /^(ABC)(?!123)/ ; |
619 | print "2: got $1\n" if $y =~ /^(ABC)(?!123)/ ; |
620 | |
621 | print "3: got $1\n" if $x =~ /^(\D*)(?!123)/ ; |
622 | print "4: got $1\n" if $y =~ /^(\D*)(?!123)/ ; |
623 | |
624 | This prints |
625 | |
626 | 2: got ABC |
627 | 3: got AB |
628 | 4: got ABC |
629 | |
5f05dabc |
630 | You might have expected test 3 to fail because it seems to a more |
c07a80fd |
631 | general purpose version of test 1. The important difference between |
632 | them is that test 3 contains a quantifier (C<\D*>) and so can use |
633 | backtracking, whereas test 1 will not. What's happening is |
634 | that you've asked "Is it true that at the start of $x, following 0 or more |
5f05dabc |
635 | non-digits, you have something that's not 123?" If the pattern matcher had |
c07a80fd |
636 | let C<\D*> expand to "ABC", this would have caused the whole pattern to |
54310121 |
637 | fail. |
c07a80fd |
638 | The search engine will initially match C<\D*> with "ABC". Then it will |
5a964f20 |
639 | try to match C<(?!123> with "123", which of course fails. But because |
c07a80fd |
640 | a quantifier (C<\D*>) has been used in the regular expression, the |
641 | search engine can backtrack and retry the match differently |
54310121 |
642 | in the hope of matching the complete regular expression. |
c07a80fd |
643 | |
5a964f20 |
644 | The pattern really, I<really> wants to succeed, so it uses the |
645 | standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this |
c07a80fd |
646 | time. Now there's indeed something following "AB" that is not |
647 | "123". It's in fact "C123", which suffices. |
648 | |
649 | We can deal with this by using both an assertion and a negation. We'll |
650 | say that the first part in $1 must be followed by a digit, and in fact, it |
651 | must also be followed by something that's not "123". Remember that the |
652 | lookaheads are zero-width expressions--they only look, but don't consume |
653 | any of the string in their match. So rewriting this way produces what |
654 | you'd expect; that is, case 5 will fail, but case 6 succeeds: |
655 | |
656 | print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/ ; |
657 | print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/ ; |
658 | |
659 | 6: got ABC |
660 | |
5a964f20 |
661 | In other words, the two zero-width assertions next to each other work as though |
c07a80fd |
662 | they're ANDed together, just as you'd use any builtin assertions: C</^$/> |
663 | matches only if you're at the beginning of the line AND the end of the |
664 | line simultaneously. The deeper underlying truth is that juxtaposition in |
665 | regular expressions always means AND, except when you write an explicit OR |
666 | using the vertical bar. C</ab/> means match "a" AND (then) match "b", |
667 | although the attempted matches are made at different positions because "a" |
668 | is not a zero-width assertion, but a one-width assertion. |
669 | |
670 | One warning: particularly complicated regular expressions can take |
671 | exponential time to solve due to the immense number of possible ways they |
672 | can use backtracking to try match. For example this will take a very long |
673 | time to run |
674 | |
675 | /((a{0,5}){0,5}){0,5}/ |
676 | |
677 | And if you used C<*>'s instead of limiting it to 0 through 5 matches, then |
678 | it would take literally forever--or until you ran out of stack space. |
679 | |
c277df42 |
680 | A powerful tool for optimizing such beasts is "independent" groups, |
5a964f20 |
681 | which do not backtrace (see L<C<(?E<gt>pattern)>>). Note also that |
c277df42 |
682 | zero-length lookahead/lookbehind assertions will not backtrace to make |
683 | the tail match, since they are in "logical" context: only the fact |
684 | whether they match or not is considered relevant. For an example |
685 | where side-effects of a lookahead I<might> have influenced the |
5a964f20 |
686 | following match, see L<C<(?E<gt>pattern)>>. |
c277df42 |
687 | |
a0d0e21e |
688 | =head2 Version 8 Regular Expressions |
689 | |
5a964f20 |
690 | In case you're not familiar with the "regular" Version 8 regex |
a0d0e21e |
691 | routines, here are the pattern-matching rules not described above. |
692 | |
54310121 |
693 | Any single character matches itself, unless it is a I<metacharacter> |
a0d0e21e |
694 | with a special meaning described here or above. You can cause |
5a964f20 |
695 | characters that normally function as metacharacters to be interpreted |
5f05dabc |
696 | literally by prefixing them with a "\" (e.g., "\." matches a ".", not any |
a0d0e21e |
697 | character; "\\" matches a "\"). A series of characters matches that |
698 | series of characters in the target string, so the pattern C<blurfl> |
699 | would match "blurfl" in the target string. |
700 | |
701 | You can specify a character class, by enclosing a list of characters |
5a964f20 |
702 | in C<[]>, which will match any one character from the list. If the |
a0d0e21e |
703 | first character after the "[" is "^", the class matches any character not |
704 | in the list. Within a list, the "-" character is used to specify a |
5a964f20 |
705 | range, so that C<a-z> represents all characters between "a" and "z", |
84850974 |
706 | inclusive. If you want "-" itself to be a member of a class, put it |
707 | at the start or end of the list, or escape it with a backslash. (The |
708 | following all specify the same class of three characters: C<[-az]>, |
709 | C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which |
710 | specifies a class containing twenty-six characters.) |
a0d0e21e |
711 | |
54310121 |
712 | Characters may be specified using a metacharacter syntax much like that |
a0d0e21e |
713 | used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return, |
714 | "\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string |
715 | of octal digits, matches the character whose ASCII value is I<nnn>. |
0f36ee90 |
716 | Similarly, \xI<nn>, where I<nn> are hexadecimal digits, matches the |
a0d0e21e |
717 | character whose ASCII value is I<nn>. The expression \cI<x> matches the |
54310121 |
718 | ASCII character control-I<x>. Finally, the "." metacharacter matches any |
a0d0e21e |
719 | character except "\n" (unless you use C</s>). |
720 | |
721 | You can specify a series of alternatives for a pattern using "|" to |
722 | separate them, so that C<fee|fie|foe> will match any of "fee", "fie", |
5a964f20 |
723 | or "foe" in the target string (as would C<f(e|i|o)e>). The |
a0d0e21e |
724 | first alternative includes everything from the last pattern delimiter |
725 | ("(", "[", or the beginning of the pattern) up to the first "|", and |
726 | the last alternative contains everything from the last "|" to the next |
727 | pattern delimiter. For this reason, it's common practice to include |
728 | alternatives in parentheses, to minimize confusion about where they |
a3cb178b |
729 | start and end. |
730 | |
5a964f20 |
731 | Alternatives are tried from left to right, so the first |
a3cb178b |
732 | alternative found for which the entire expression matches, is the one that |
733 | is chosen. This means that alternatives are not necessarily greedy. For |
734 | example: when mathing C<foo|foot> against "barefoot", only the "foo" |
735 | part will match, as that is the first alternative tried, and it successfully |
736 | matches the target string. (This might not seem important, but it is |
737 | important when you are capturing matched text using parentheses.) |
738 | |
5a964f20 |
739 | Also remember that "|" is interpreted as a literal within square brackets, |
a3cb178b |
740 | so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>. |
a0d0e21e |
741 | |
54310121 |
742 | Within a pattern, you may designate subpatterns for later reference by |
a0d0e21e |
743 | enclosing them in parentheses, and you may refer back to the I<n>th |
54310121 |
744 | subpattern later in the pattern using the metacharacter \I<n>. |
745 | Subpatterns are numbered based on the left to right order of their |
5a964f20 |
746 | opening parenthesis. A backreference matches whatever |
54310121 |
747 | actually matched the subpattern in the string being examined, not the |
748 | rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will |
5a964f20 |
749 | match "0x1234 0x4321", but not "0x1234 01234", because subpattern 1 |
748a9306 |
750 | actually matched "0x", even though the rule C<0|0x> could |
a0d0e21e |
751 | potentially match the leading 0 in the second number. |
cb1a09d0 |
752 | |
753 | =head2 WARNING on \1 vs $1 |
754 | |
5a964f20 |
755 | Some people get too used to writing things like: |
cb1a09d0 |
756 | |
757 | $pattern =~ s/(\W)/\\\1/g; |
758 | |
759 | This is grandfathered for the RHS of a substitute to avoid shocking the |
760 | B<sed> addicts, but it's a dirty habit to get into. That's because in |
5f05dabc |
761 | PerlThink, the righthand side of a C<s///> is a double-quoted string. C<\1> in |
cb1a09d0 |
762 | the usual double-quoted string means a control-A. The customary Unix |
763 | meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit |
764 | of doing that, you get yourself into trouble if you then add an C</e> |
765 | modifier. |
766 | |
5a964f20 |
767 | s/(\d+)/ \1 + 1 /eg; # causes warning under -w |
cb1a09d0 |
768 | |
769 | Or if you try to do |
770 | |
771 | s/(\d+)/\1000/; |
772 | |
773 | You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with |
774 | C<${1}000>. Basically, the operation of interpolation should not be confused |
775 | with the operation of matching a backreference. Certainly they mean two |
776 | different things on the I<left> side of the C<s///>. |
9fa51da4 |
777 | |
c84d73f1 |
778 | =head2 Repeated patterns matching zero-length substring |
779 | |
780 | WARNING: Difficult material (and prose) ahead. This section needs a rewrite. |
781 | |
782 | Regular expressions provide a terse and powerful programming language. As |
783 | with most other power tools, power comes together with the ability |
784 | to wreak havoc. |
785 | |
786 | A common abuse of this power stems from the ability to make infinite |
787 | loops using regular expressions, with something as innocous as: |
788 | |
789 | 'foo' =~ m{ ( o? )* }x; |
790 | |
791 | The C<o?> can match at the beginning of C<'foo'>, and since the position |
792 | in the string is not moved by the match, C<o?> would match again and again |
793 | due to the C<*> modifier. Another common way to create a similar cycle |
794 | is with the looping modifier C<//g>: |
795 | |
796 | @matches = ( 'foo' =~ m{ o? }xg ); |
797 | |
798 | or |
799 | |
800 | print "match: <$&>\n" while 'foo' =~ m{ o? }xg; |
801 | |
802 | or the loop implied by split(). |
803 | |
804 | However, long experience has shown that many programming tasks may |
805 | be significantly simplified by using repeated subexpressions which |
806 | may match zero-length substrings, with a simple example being: |
807 | |
808 | @chars = split //, $string; # // is not magic in split |
809 | ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// / |
810 | |
811 | Thus Perl allows the C</()/> construct, which I<forcefully breaks |
812 | the infinite loop>. The rules for this are different for lower-level |
813 | loops given by the greedy modifiers C<*+{}>, and for higher-level |
814 | ones like the C</g> modifier or split() operator. |
815 | |
816 | The lower-level loops are I<interrupted> when it is detected that a |
817 | repeated expression did match a zero-length substring, thus |
818 | |
819 | m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x; |
820 | |
821 | is made equivalent to |
822 | |
823 | m{ (?: NON_ZERO_LENGTH )* |
824 | | |
825 | (?: ZERO_LENGTH )? |
826 | }x; |
827 | |
828 | The higher level-loops preserve an additional state between iterations: |
829 | whether the last match was zero-length. To break the loop, the following |
830 | match after a zero-length match is prohibited to have a length of zero. |
831 | This prohibition interacts with backtracking (see L<"Backtracking">), |
832 | and so the I<second best> match is chosen if the I<best> match is of |
833 | zero length. |
834 | |
835 | Say, |
836 | |
837 | $_ = 'bar'; |
838 | s/\w??/<$&>/g; |
839 | |
840 | results in C<"<><b><><a><><r><>">. At each position of the string the best |
841 | match given by non-greedy C<??> is the zero-length match, and the I<second |
842 | best> match is what is matched by C<\w>. Thus zero-length matches |
843 | alternate with one-character-long matches. |
844 | |
845 | Similarly, for repeated C<m/()/g> the second-best match is the match at the |
846 | position one notch further in the string. |
847 | |
848 | The additional state of being I<matched with zero-length> is associated to |
849 | the matched string, and is reset by each assignment to pos(). |
850 | |
851 | =head2 Creating custom RE engines |
852 | |
853 | Overloaded constants (see L<overload>) provide a simple way to extend |
854 | the functionality of the RE engine. |
855 | |
856 | Suppose that we want to enable a new RE escape-sequence C<\Y|> which |
857 | matches at boundary between white-space characters and non-whitespace |
858 | characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly |
859 | at these positions, so we want to have each C<\Y|> in the place of the |
860 | more complicated version. We can create a module C<customre> to do |
861 | this: |
862 | |
863 | package customre; |
864 | use overload; |
865 | |
866 | sub import { |
867 | shift; |
868 | die "No argument to customre::import allowed" if @_; |
869 | overload::constant 'qr' => \&convert; |
870 | } |
871 | |
872 | sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"} |
873 | |
874 | my %rules = ( '\\' => '\\', |
875 | 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ ); |
876 | sub convert { |
877 | my $re = shift; |
878 | $re =~ s{ |
879 | \\ ( \\ | Y . ) |
880 | } |
881 | { $rules{$1} or invalid($re,$1) }sgex; |
882 | return $re; |
883 | } |
884 | |
885 | Now C<use customre> enables the new escape in constant regular |
886 | expressions, i.e., those without any runtime variable interpolations. |
887 | As documented in L<overload>, this conversion will work only over |
888 | literal parts of regular expressions. For C<\Y|$re\Y|> the variable |
889 | part of this regular expression needs to be converted explicitly |
890 | (but only if the special meaning of C<\Y|> should be enabled inside $re): |
891 | |
892 | use customre; |
893 | $re = <>; |
894 | chomp $re; |
895 | $re = customre::convert $re; |
896 | /\Y|$re\Y|/; |
897 | |
9fa51da4 |
898 | =head2 SEE ALSO |
899 | |
9b599b2a |
900 | L<perlop/"Regexp Quote-Like Operators">. |
901 | |
902 | L<perlfunc/pos>. |
903 | |
904 | L<perllocale>. |
905 | |
5a964f20 |
906 | I<Mastering Regular Expressions> (see L<perlbook>) by Jeffrey Friedl. |