X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=blobdiff_plain;f=pod%2Fperlfaq6.pod;h=9bbf80a0189fe870dd89bd7f1e346f93b06e715a;hb=f3f8427d8eb74488a7768102783b300690126cdc;hp=ed6c01b31b1812e067434aebd15fe727a8ef2ae7;hpb=3e2f796abf3f79e1ab191424e6c63f33a4ee4497;p=p5sagit%2Fp5-mst-13.2.git diff --git a/pod/perlfaq6.pod b/pod/perlfaq6.pod index ed6c01b..9bbf80a 100644 --- a/pod/perlfaq6.pod +++ b/pod/perlfaq6.pod @@ -1,6 +1,6 @@ =head1 NAME -perlfaq6 - Regexes ($Revision: 1.27 $, $Date: 1999/05/23 16:08:30 $) +perlfaq6 - Regular Expressions ($Revision: 1.20 $, $Date: 2003/01/03 20:05:28 $) =head1 DESCRIPTION @@ -8,8 +8,8 @@ This section is surprisingly small because the rest of the FAQ is littered with answers involving regular expressions. For example, decoding a URL and checking whether something is a number are handled with regular expressions, but those answers are found elsewhere in -this document (in L: ``How do I decode or create those %-encodings -on the web'' and L: ``How do I determine whether a scalar is +this document (in L: ``How do I decode or create those %-encodings +on the web'' and L: ``How do I determine whether a scalar is a number/whole/integer/float'', to be precise). =head2 How can I hope to use regular expressions without creating illegible and unmaintainable code? @@ -70,9 +70,9 @@ delimiter within the pattern: =head2 I'm having trouble matching over more than one line. What's wrong? -Either you don't have more than one line in the string you're looking at -(probably), or else you aren't using the correct modifier(s) on your -pattern (possibly). +Either you don't have more than one line in the string you're looking +at (probably), or else you aren't using the correct modifier(s) on +your pattern (possibly). There are many ways to get multiline data into a string. If you want it to happen automatically while reading input, you'll want to set $/ @@ -115,7 +115,7 @@ Here's code that finds everything between START and END in a paragraph: undef $/; # read in whole file, not just one line or paragraph while ( <> ) { - while ( /START(.*?)END/sm ) { # /s makes . cross line boundaries + while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries print "$1\n"; } } @@ -143,38 +143,38 @@ Here's another example of using C<..>: # now choose between them } continue { reset if eof(); # fix $. - } + } =head2 I put a regular expression into $/ but it didn't work. What's wrong? -$/ must be a string, not a regular expression. Awk has to be better -for something. :-) - -Actually, you could do this if you don't mind reading the whole file -into memory: +Up to Perl 5.8.0, $/ has to be a string. This may change in 5.10, +but don't get your hopes up. Until then, you can use these examples +if you really need to do this. - undef $/; - @records = split /your_pattern/, ; +Use the four argument form of sysread to continually add to +a buffer. After you add to the buffer, you check if you have a +complete line (using your regular expression). -The Net::Telnet module (available from CPAN) has the capability to -wait for a pattern in the input stream, or timeout if it doesn't -appear within a certain time. + local $_ = ""; + while( sysread FH, $_, 8192, length ) { + while( s/^((?s).*?)your_pattern/ ) { + my $record = $1; + # do stuff here. + } + } - ## Create a file with three lines. - open FH, ">file"; - print FH "The first line\nThe second line\nThe third line\n"; - close FH; + You can do the same thing with foreach and a match using the + c flag and the \G anchor, if you do not mind your entire file + being in memory at the end. - ## Get a read/write filehandle to it. - $fh = new FileHandle "+ $fh); - - ## Search for the second line and print out the third. - $file->waitfor('/second line\n/'); - print $file->getline; =head2 How do I substitute case insensitively on the LHS while preserving case on the RHS? @@ -194,14 +194,14 @@ properties of bitwise xor on ASCII strings. print; -And here it is as a subroutine, modelled after the above: +And here it is as a subroutine, modeled after the above: sub preserve_case($$) { my ($old, $new) = @_; my $mask = uc $old ^ $old; uc $new | $mask . - substr($mask, -1) x (length($new) - length($old)) + substr($mask, -1) x (length($new) - length($old)) } $a = "this is a TEsT case"; @@ -212,6 +212,21 @@ This prints: this is a SUcCESS case +As an alternative, to keep the case of the replacement word if it is +longer than the original, you can use this code, by Jeff Pinyan: + + sub preserve_case { + my ($from, $to) = @_; + my ($lf, $lt) = map length, @_; + + if ($lt < $lf) { $from = substr $from, 0, $lt } + else { $from .= substr $to, $lf } + + return uc $to | ($from ^ uc $from); + } + +This changes the sentence to "this is a SUcCess case." + Just to show that C programmers can write C in any programming language, if you prefer a more C-like solution, the following script makes the substitution have the same case, letter by letter, as the original. @@ -252,13 +267,21 @@ the case of the last character is used for the rest of the substitution. =head2 How can I make C<\w> match national character sets? -See L. +Put C in your script. The \w character class is taken +from the current locale. + +See L for details. =head2 How can I match a locale-smart version of C? -One alphabetic character would be C, no matter what locale -you're in. Non-alphabetics would be C (assuming you don't -consider an underscore a letter). +You can use the POSIX character class syntax C +documented in L. + +No matter which locale you are in, the alphabetic characters are +the characters in \w without the digits and the underscore. +As a regex, that looks like C. Its complement, +the non-alphabetics, is then everything in \W along with +the digits and the underscore, or C. =head2 How can I quote a variable to use in a regex? @@ -368,20 +391,30 @@ A slight modification also removes C++ comments: =head2 Can I use Perl regular expressions to match balanced text? -Although Perl regular expressions are more powerful than "mathematical" -regular expressions because they feature conveniences like backreferences -(C<\1> and its ilk), they still aren't powerful enough--with -the possible exception of bizarre and experimental features in the -development-track releases of Perl. You still need to use non-regex -techniques to parse balanced text, such as the text enclosed between -matching parentheses or braces, for example. +Historically, Perl regular expressions were not capable of matching +balanced text. As of more recent versions of perl including 5.6.1 +experimental features have been added that make it possible to do this. +Look at the documentation for the (??{ }) construct in recent perlre manual +pages to see an example of matching balanced parentheses. Be sure to take +special notice of the warnings present in the manual before making use +of this feature. + +CPAN contains many modules that can be useful for matching text +depending on the context. Damian Conway provides some useful +patterns in Regexp::Common. The module Text::Balanced provides a +general solution to this problem. + +One of the common applications of balanced text matching is working +with XML and HTML. There are many modules available that support +these needs. Two examples are HTML::Parser and XML::Parser. There +are many others. An elaborate subroutine (for 7-bit ASCII only) to pull out balanced and possibly nested single chars, like C<`> and C<'>, C<{> and C<}>, or C<(> and C<)> can be found in -http://www.perl.com/CPAN/authors/id/TOMC/scripts/pull_quotes.gz . +http://www.cpan.org/authors/id/TOMC/scripts/pull_quotes.gz . -The C::Scan module from CPAN contains such subs for internal use, +The C::Scan module from CPAN also contains such subs for internal use, but they are undocumented. =head2 What does it mean that regexes are greedy? How can I get around it? @@ -409,9 +442,9 @@ playing hot potato. Use the split function: while (<>) { - foreach $word ( split ) { + foreach $word ( split ) { # do something with $word here - } + } } Note that this isn't really a word in the English sense; it's just @@ -445,7 +478,7 @@ in the previous question: If you wanted to do the same thing for lines, you wouldn't need a regular expression: - while (<>) { + while (<>) { $seen{$_}++; } while ( ($line, $count) = each %seen ) { @@ -467,12 +500,12 @@ The following is extremely inefficient: @popstates = qw(CO ON MI WI MN); while (defined($line = <>)) { for $state (@popstates) { - if ($line =~ /\b$state\b/i) { + if ($line =~ /\b$state\b/i) { print $line; last; } } - } + } That's because Perl has to recompile all those patterns for each of the lines of the file. As of the 5.005 release, there's a much better @@ -530,69 +563,96 @@ variable is no longer "expensive" the way the other two are. =head2 What good is C<\G> in a regular expression? -The notation C<\G> is used in a match or substitution in conjunction with -the C modifier to anchor the regular expression to the point just past -where the last match occurred, i.e. the pos() point. A failed match resets -the position of C<\G> unless the C modifier is in effect. C<\G> can be -used in a match without the C modifier; it acts the same (i.e. still -anchors at the pos() point) but of course only matches once and does not -update pos(), as non-C expressions never do. C<\G> in an expression -applied to a target string that has never been matched against a C -expression before or has had its pos() reset is functionally equivalent to -C<\A>, which matches at the beginning of the string. - -For example, suppose you had a line of text quoted in standard mail -and Usenet notation, (that is, with leading C<< > >> characters), and -you want change each leading C<< > >> into a corresponding C<:>. You -could do so in this way: - - s/^(>+)/':' x length($1)/gem; - -Or, using C<\G>, the much simpler (and faster): - - s/\G>/:/g; - -A more sophisticated use might involve a tokenizer. The following -lex-like example is courtesy of Jeffrey Friedl. It did not work in -5.003 due to bugs in that release, but does work in 5.004 or better. -(Note the use of C, which prevents a failed match with C from -resetting the search position back to the beginning of the string.) +You use the C<\G> anchor to start the next match on the same +string where the last match left off. The regular +expression engine cannot skip over any characters to find +the next match with this anchor, so C<\G> is similar to the +beginning of string anchor, C<^>. The C<\G> anchor is typically +used with the C flag. It uses the value of pos() +as the position to start the next match. As the match +operator makes successive matches, it updates pos() with the +position of the next character past the last match (or the +first character of the next match, depending on how you like +to look at it). Each string has its own pos() value. + +Suppose you want to match all of consective pairs of digits +in a string like "1122a44" and stop matching when you +encounter non-digits. You want to match C<11> and C<22> but +the letter shows up between C<22> and C<44> and you want +to stop at C. Simply matching pairs of digits skips over +the C and still matches C<44>. + + $_ = "1122a44"; + my @pairs = m/(\d\d)/g; # qw( 11 22 44 ) + +If you use the \G anchor, you force the match after C<22> to +start with the C. The regular expression cannot match +there since it does not find a digit, so the next match +fails and the match operator returns the pairs it already +found. + + $_ = "1122a44"; + my @pairs = m/\G(\d\d)/g; # qw( 11 22 ) + +You can also use the C<\G> anchor in scalar context. You +still need the C flag. + + $_ = "1122a44"; + while( m/\G(\d\d)/g ) + { + print "Found $1\n"; + } + +After the match fails at the letter C, perl resets pos() +and the next match on the same string starts at the beginning. + + $_ = "1122a44"; + while( m/\G(\d\d)/g ) + { + print "Found $1\n"; + } + + print "Found $1 after while" if m/(\d\d)/g; # finds "11" + +You can disable pos() resets on fail with the C flag. +Subsequent matches start where the last successful match +ended (the value of pos()) even if a match on the same +string as failed in the meantime. In this case, the match +after the while() loop starts at the C (where the last +match stopped), and since it does not use any anchor it can +skip over the C to find "44". + + $_ = "1122a44"; + while( m/\G(\d\d)/gc ) + { + print "Found $1\n"; + } + + print "Found $1 after while" if m/(\d\d)/g; # finds "44" + +Typically you use the C<\G> anchor with the C flag +when you want to try a different match if one fails, +such as in a tokenizer. Jeffrey Friedl offers this example +which works in 5.004 or later. while (<>) { chomp; PARSER: { - m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; }; - m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; }; - m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; }; - m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; }; + m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; }; + m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; }; + m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; }; + m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; }; } } -Of course, that could have been written as - - while (<>) { - chomp; - PARSER: { - if ( /\G( \d+\b )/gcx { - print "number: $1\n"; - redo PARSER; - } - if ( /\G( \w+ )/gcx { - print "word: $1\n"; - redo PARSER; - } - if ( /\G( \s+ )/gcx { - print "space: $1\n"; - redo PARSER; - } - if ( /\G( [^\w\d]+ )/gcx { - print "other: $1\n"; - redo PARSER; - } - } - } - -but then you lose the vertical alignment of the regular expressions. +For each line, the PARSER loop first tries to match a series +of digits followed by a word boundary. This match has to +start at the place the last match left off (or the beginning +of the string on the first match). Since C uses the C flag, if the string does not match that +regular expression, perl does not reset pos() and the next +match starts at the same position to try a different +pattern. =head2 Are Perl regexes DFAs or NFAs? Are they POSIX compliant? @@ -609,18 +669,29 @@ L). =head2 What's wrong with using grep or map in a void context? -Both grep and map build a return list, regardless of their context. -This means you're making Perl go to the trouble of building up a -return list that you then just ignore. That's no way to treat a -programming language, you insensitive scoundrel! +The problem is that both grep and map build a return list, +regardless of the context. This means you're making Perl go +to the trouble of building a list that you then just throw away. +If the list is large, you waste both time and space. If your +intent is to iterate over the list then use a for loop for this +purpose. =head2 How can I match strings with multibyte characters? -This is hard, and there's no good way. Perl does not directly support -wide characters. It pretends that a byte and a character are -synonymous. The following set of approaches was offered by Jeffrey -Friedl, whose article in issue #5 of The Perl Journal talks about this -very matter. +Starting from Perl 5.6 Perl has had some level of multibyte character +support. Perl 5.8 or later is recommended. Supported multibyte +character repertoires include Unicode, and legacy encodings +through the Encode module. See L, L, +and L. + +If you are stuck with older Perls, you can do Unicode with the +C module, and character conversions using the +C and C modules. If you are using +Japanese encodings, you might try using the jperl 5.005_03. + +Finally, the following set of approaches was offered by Jeffrey +Friedl, whose article in issue #5 of The Perl Journal talks about +this very matter. Let's suppose you have some weird Martian encoding where pairs of ASCII uppercase letters encode single Martian letters (i.e. the two @@ -639,8 +710,8 @@ looks like it is because "SG" is next to "XX", but there's no real Here are a few ways, all painful, to deal with it: - $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian'' bytes - # are no longer adjacent. + $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian'' + # bytes are no longer adjacent. print "found GX!\n" if $martian =~ /GX/; Or like this: @@ -658,13 +729,21 @@ Or like this: print "found GX!\n", last if $1 eq 'GX'; } -Or like this: +Here's another, slightly less painful, way to do it from Benjamin +Goldberg: + + $martian =~ m/ + (?!<[A-Z]) + (?:[A-Z][A-Z])*? + GX + /x; - die "sorry, Perl doesn't (yet) have Martian support )-:\n"; +This succeeds if the "martian" character GX is in the string, and fails +otherwise. If you don't like using (?!<), you can replace (?!<[A-Z]) +with (?:^|[^A-Z]). -There are many double- (and multi-) byte encodings commonly used these -days. Some versions of these have 1-, 2-, 3-, and 4-byte characters, -all mixed. +It does have the drawback of putting the wrong thing in $-[0] and $+[0], +but this usually can be worked around. =head2 How do I match a pattern that is supplied by the user? @@ -694,15 +773,11 @@ in L. =head1 AUTHOR AND COPYRIGHT -Copyright (c) 1997-1999 Tom Christiansen and Nathan Torkington. +Copyright (c) 1997-2002 Tom Christiansen and Nathan Torkington. All rights reserved. -When included as part of the Standard Version of Perl, or as part of -its complete documentation whether printed or otherwise, this work -may be distributed only under the terms of Perl's Artistic License. -Any distribution of this file or derivatives thereof I -of that package require that special arrangements be made with -copyright holder. +This documentation is free; you can redistribute it and/or modify it +under the same terms as Perl itself. Irrespective of its distribution, all code examples in this file are hereby placed into the public domain. You are permitted and