=head1 NAME
-perlfaq6 - Regexes ($Revision: 1.27 $, $Date: 1999/05/23 16:08:30 $)
+perlfaq6 - Regular Expressions ($Revision: 1.20 $, $Date: 2003/01/03 20:05:28 $)
=head1 DESCRIPTION
littered with answers involving regular expressions. For example,
decoding a URL and checking whether something is a number are handled
with regular expressions, but those answers are found elsewhere in
-this document (in L<perlfaq9>: ``How do I decode or create those %-encodings
-on the web'' and L<perfaq4>: ``How do I determine whether a scalar is
+this document (in L<perlfaq9>: ``How do I decode or create those %-encodings
+on the web'' and L<perlfaq4>: ``How do I determine whether a scalar is
a number/whole/integer/float'', to be precise).
=head2 How can I hope to use regular expressions without creating illegible and unmaintainable code?
=head2 I'm having trouble matching over more than one line. What's wrong?
-Either you don't have more than one line in the string you're looking at
-(probably), or else you aren't using the correct modifier(s) on your
-pattern (possibly).
+Either you don't have more than one line in the string you're looking
+at (probably), or else you aren't using the correct modifier(s) on
+your pattern (possibly).
There are many ways to get multiline data into a string. If you want
it to happen automatically while reading input, you'll want to set $/
undef $/; # read in whole file, not just one line or paragraph
while ( <> ) {
- while ( /START(.*?)END/sm ) { # /s makes . cross line boundaries
+ while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries
print "$1\n";
}
}
# now choose between them
} continue {
reset if eof(); # fix $.
- }
+ }
=head2 I put a regular expression into $/ but it didn't work. What's wrong?
-$/ must be a string, not a regular expression. Awk has to be better
-for something. :-)
-
-Actually, you could do this if you don't mind reading the whole file
-into memory:
+Up to Perl 5.8.0, $/ has to be a string. This may change in 5.10,
+but don't get your hopes up. Until then, you can use these examples
+if you really need to do this.
- undef $/;
- @records = split /your_pattern/, <FH>;
+Use the four argument form of sysread to continually add to
+a buffer. After you add to the buffer, you check if you have a
+complete line (using your regular expression).
-The Net::Telnet module (available from CPAN) has the capability to
-wait for a pattern in the input stream, or timeout if it doesn't
-appear within a certain time.
+ local $_ = "";
+ while( sysread FH, $_, 8192, length ) {
+ while( s/^((?s).*?)your_pattern/ ) {
+ my $record = $1;
+ # do stuff here.
+ }
+ }
- ## Create a file with three lines.
- open FH, ">file";
- print FH "The first line\nThe second line\nThe third line\n";
- close FH;
+ You can do the same thing with foreach and a match using the
+ c flag and the \G anchor, if you do not mind your entire file
+ being in memory at the end.
- ## Get a read/write filehandle to it.
- $fh = new FileHandle "+<file";
+ local $_ = "";
+ while( sysread FH, $_, 8192, length ) {
+ foreach my $record ( m/\G((?s).*?)your_pattern/gc ) {
+ # do stuff here.
+ }
+ substr( $_, 0, pos ) = "" if pos;
+ }
- ## Attach it to a "stream" object.
- use Net::Telnet;
- $file = new Net::Telnet (-fhopen => $fh);
-
- ## Search for the second line and print out the third.
- $file->waitfor('/second line\n/');
- print $file->getline;
=head2 How do I substitute case insensitively on the LHS while preserving case on the RHS?
print;
-And here it is as a subroutine, modelled after the above:
+And here it is as a subroutine, modeled after the above:
sub preserve_case($$) {
my ($old, $new) = @_;
my $mask = uc $old ^ $old;
uc $new | $mask .
- substr($mask, -1) x (length($new) - length($old))
+ substr($mask, -1) x (length($new) - length($old))
}
$a = "this is a TEsT case";
this is a SUcCESS case
+As an alternative, to keep the case of the replacement word if it is
+longer than the original, you can use this code, by Jeff Pinyan:
+
+ sub preserve_case {
+ my ($from, $to) = @_;
+ my ($lf, $lt) = map length, @_;
+
+ if ($lt < $lf) { $from = substr $from, 0, $lt }
+ else { $from .= substr $to, $lf }
+
+ return uc $to | ($from ^ uc $from);
+ }
+
+This changes the sentence to "this is a SUcCess case."
+
Just to show that C programmers can write C in any programming language,
if you prefer a more C-like solution, the following script makes the
substitution have the same case, letter by letter, as the original.
=head2 How can I make C<\w> match national character sets?
-See L<perllocale>.
+Put C<use locale;> in your script. The \w character class is taken
+from the current locale.
+
+See L<perllocale> for details.
=head2 How can I match a locale-smart version of C</[a-zA-Z]/>?
-One alphabetic character would be C</[^\W\d_]/>, no matter what locale
-you're in. Non-alphabetics would be C</[\W\d_]/> (assuming you don't
-consider an underscore a letter).
+You can use the POSIX character class syntax C</[[:alpha:]]/>
+documented in L<perlre>.
+
+No matter which locale you are in, the alphabetic characters are
+the characters in \w without the digits and the underscore.
+As a regex, that looks like C</[^\W\d_]/>. Its complement,
+the non-alphabetics, is then everything in \W along with
+the digits and the underscore, or C</[\W\d_]/>.
=head2 How can I quote a variable to use in a regex?
=head2 Can I use Perl regular expressions to match balanced text?
-Although Perl regular expressions are more powerful than "mathematical"
-regular expressions because they feature conveniences like backreferences
-(C<\1> and its ilk), they still aren't powerful enough--with
-the possible exception of bizarre and experimental features in the
-development-track releases of Perl. You still need to use non-regex
-techniques to parse balanced text, such as the text enclosed between
-matching parentheses or braces, for example.
+Historically, Perl regular expressions were not capable of matching
+balanced text. As of more recent versions of perl including 5.6.1
+experimental features have been added that make it possible to do this.
+Look at the documentation for the (??{ }) construct in recent perlre manual
+pages to see an example of matching balanced parentheses. Be sure to take
+special notice of the warnings present in the manual before making use
+of this feature.
+
+CPAN contains many modules that can be useful for matching text
+depending on the context. Damian Conway provides some useful
+patterns in Regexp::Common. The module Text::Balanced provides a
+general solution to this problem.
+
+One of the common applications of balanced text matching is working
+with XML and HTML. There are many modules available that support
+these needs. Two examples are HTML::Parser and XML::Parser. There
+are many others.
An elaborate subroutine (for 7-bit ASCII only) to pull out balanced
and possibly nested single chars, like C<`> and C<'>, C<{> and C<}>,
or C<(> and C<)> can be found in
-http://www.perl.com/CPAN/authors/id/TOMC/scripts/pull_quotes.gz .
+http://www.cpan.org/authors/id/TOMC/scripts/pull_quotes.gz .
-The C::Scan module from CPAN contains such subs for internal use,
+The C::Scan module from CPAN also contains such subs for internal use,
but they are undocumented.
=head2 What does it mean that regexes are greedy? How can I get around it?
Use the split function:
while (<>) {
- foreach $word ( split ) {
+ foreach $word ( split ) {
# do something with $word here
- }
+ }
}
Note that this isn't really a word in the English sense; it's just
If you wanted to do the same thing for lines, you wouldn't need a
regular expression:
- while (<>) {
+ while (<>) {
$seen{$_}++;
}
while ( ($line, $count) = each %seen ) {
@popstates = qw(CO ON MI WI MN);
while (defined($line = <>)) {
for $state (@popstates) {
- if ($line =~ /\b$state\b/i) {
+ if ($line =~ /\b$state\b/i) {
print $line;
last;
}
}
- }
+ }
That's because Perl has to recompile all those patterns for each of
the lines of the file. As of the 5.005 release, there's a much better
=head2 What good is C<\G> in a regular expression?
-The notation C<\G> is used in a match or substitution in conjunction with
-the C</g> modifier to anchor the regular expression to the point just past
-where the last match occurred, i.e. the pos() point. A failed match resets
-the position of C<\G> unless the C</c> modifier is in effect. C<\G> can be
-used in a match without the C</g> modifier; it acts the same (i.e. still
-anchors at the pos() point) but of course only matches once and does not
-update pos(), as non-C</g> expressions never do. C<\G> in an expression
-applied to a target string that has never been matched against a C</g>
-expression before or has had its pos() reset is functionally equivalent to
-C<\A>, which matches at the beginning of the string.
-
-For example, suppose you had a line of text quoted in standard mail
-and Usenet notation, (that is, with leading C<< > >> characters), and
-you want change each leading C<< > >> into a corresponding C<:>. You
-could do so in this way:
-
- s/^(>+)/':' x length($1)/gem;
-
-Or, using C<\G>, the much simpler (and faster):
-
- s/\G>/:/g;
-
-A more sophisticated use might involve a tokenizer. The following
-lex-like example is courtesy of Jeffrey Friedl. It did not work in
-5.003 due to bugs in that release, but does work in 5.004 or better.
-(Note the use of C</c>, which prevents a failed match with C</g> from
-resetting the search position back to the beginning of the string.)
+You use the C<\G> anchor to start the next match on the same
+string where the last match left off. The regular
+expression engine cannot skip over any characters to find
+the next match with this anchor, so C<\G> is similar to the
+beginning of string anchor, C<^>. The C<\G> anchor is typically
+used with the C<g> flag. It uses the value of pos()
+as the position to start the next match. As the match
+operator makes successive matches, it updates pos() with the
+position of the next character past the last match (or the
+first character of the next match, depending on how you like
+to look at it). Each string has its own pos() value.
+
+Suppose you want to match all of consective pairs of digits
+in a string like "1122a44" and stop matching when you
+encounter non-digits. You want to match C<11> and C<22> but
+the letter <a> shows up between C<22> and C<44> and you want
+to stop at C<a>. Simply matching pairs of digits skips over
+the C<a> and still matches C<44>.
+
+ $_ = "1122a44";
+ my @pairs = m/(\d\d)/g; # qw( 11 22 44 )
+
+If you use the \G anchor, you force the match after C<22> to
+start with the C<a>. The regular expression cannot match
+there since it does not find a digit, so the next match
+fails and the match operator returns the pairs it already
+found.
+
+ $_ = "1122a44";
+ my @pairs = m/\G(\d\d)/g; # qw( 11 22 )
+
+You can also use the C<\G> anchor in scalar context. You
+still need the C<g> flag.
+
+ $_ = "1122a44";
+ while( m/\G(\d\d)/g )
+ {
+ print "Found $1\n";
+ }
+
+After the match fails at the letter C<a>, perl resets pos()
+and the next match on the same string starts at the beginning.
+
+ $_ = "1122a44";
+ while( m/\G(\d\d)/g )
+ {
+ print "Found $1\n";
+ }
+
+ print "Found $1 after while" if m/(\d\d)/g; # finds "11"
+
+You can disable pos() resets on fail with the C<c> flag.
+Subsequent matches start where the last successful match
+ended (the value of pos()) even if a match on the same
+string as failed in the meantime. In this case, the match
+after the while() loop starts at the C<a> (where the last
+match stopped), and since it does not use any anchor it can
+skip over the C<a> to find "44".
+
+ $_ = "1122a44";
+ while( m/\G(\d\d)/gc )
+ {
+ print "Found $1\n";
+ }
+
+ print "Found $1 after while" if m/(\d\d)/g; # finds "44"
+
+Typically you use the C<\G> anchor with the C<c> flag
+when you want to try a different match if one fails,
+such as in a tokenizer. Jeffrey Friedl offers this example
+which works in 5.004 or later.
while (<>) {
chomp;
PARSER: {
- m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; };
- m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; };
- m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; };
- m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; };
+ m/ \G( \d+\b )/gcx && do { print "number: $1\n"; redo; };
+ m/ \G( \w+ )/gcx && do { print "word: $1\n"; redo; };
+ m/ \G( \s+ )/gcx && do { print "space: $1\n"; redo; };
+ m/ \G( [^\w\d]+ )/gcx && do { print "other: $1\n"; redo; };
}
}
-Of course, that could have been written as
-
- while (<>) {
- chomp;
- PARSER: {
- if ( /\G( \d+\b )/gcx {
- print "number: $1\n";
- redo PARSER;
- }
- if ( /\G( \w+ )/gcx {
- print "word: $1\n";
- redo PARSER;
- }
- if ( /\G( \s+ )/gcx {
- print "space: $1\n";
- redo PARSER;
- }
- if ( /\G( [^\w\d]+ )/gcx {
- print "other: $1\n";
- redo PARSER;
- }
- }
- }
-
-but then you lose the vertical alignment of the regular expressions.
+For each line, the PARSER loop first tries to match a series
+of digits followed by a word boundary. This match has to
+start at the place the last match left off (or the beginning
+of the string on the first match). Since C<m/ \G( \d+\b
+)/gcx> uses the C<c> flag, if the string does not match that
+regular expression, perl does not reset pos() and the next
+match starts at the same position to try a different
+pattern.
=head2 Are Perl regexes DFAs or NFAs? Are they POSIX compliant?
=head2 What's wrong with using grep or map in a void context?
-Both grep and map build a return list, regardless of their context.
-This means you're making Perl go to the trouble of building up a
-return list that you then just ignore. That's no way to treat a
-programming language, you insensitive scoundrel!
+The problem is that both grep and map build a return list,
+regardless of the context. This means you're making Perl go
+to the trouble of building a list that you then just throw away.
+If the list is large, you waste both time and space. If your
+intent is to iterate over the list then use a for loop for this
+purpose.
=head2 How can I match strings with multibyte characters?
-This is hard, and there's no good way. Perl does not directly support
-wide characters. It pretends that a byte and a character are
-synonymous. The following set of approaches was offered by Jeffrey
-Friedl, whose article in issue #5 of The Perl Journal talks about this
-very matter.
+Starting from Perl 5.6 Perl has had some level of multibyte character
+support. Perl 5.8 or later is recommended. Supported multibyte
+character repertoires include Unicode, and legacy encodings
+through the Encode module. See L<perluniintro>, L<perlunicode>,
+and L<Encode>.
+
+If you are stuck with older Perls, you can do Unicode with the
+C<Unicode::String> module, and character conversions using the
+C<Unicode::Map8> and C<Unicode::Map> modules. If you are using
+Japanese encodings, you might try using the jperl 5.005_03.
+
+Finally, the following set of approaches was offered by Jeffrey
+Friedl, whose article in issue #5 of The Perl Journal talks about
+this very matter.
Let's suppose you have some weird Martian encoding where pairs of
ASCII uppercase letters encode single Martian letters (i.e. the two
Here are a few ways, all painful, to deal with it:
- $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian'' bytes
- # are no longer adjacent.
+ $martian =~ s/([A-Z][A-Z])/ $1 /g; # Make sure adjacent ``martian''
+ # bytes are no longer adjacent.
print "found GX!\n" if $martian =~ /GX/;
Or like this:
print "found GX!\n", last if $1 eq 'GX';
}
-Or like this:
+Here's another, slightly less painful, way to do it from Benjamin
+Goldberg:
+
+ $martian =~ m/
+ (?!<[A-Z])
+ (?:[A-Z][A-Z])*?
+ GX
+ /x;
- die "sorry, Perl doesn't (yet) have Martian support )-:\n";
+This succeeds if the "martian" character GX is in the string, and fails
+otherwise. If you don't like using (?!<), you can replace (?!<[A-Z])
+with (?:^|[^A-Z]).
-There are many double- (and multi-) byte encodings commonly used these
-days. Some versions of these have 1-, 2-, 3-, and 4-byte characters,
-all mixed.
+It does have the drawback of putting the wrong thing in $-[0] and $+[0],
+but this usually can be worked around.
=head2 How do I match a pattern that is supplied by the user?
=head1 AUTHOR AND COPYRIGHT
-Copyright (c) 1997-1999 Tom Christiansen and Nathan Torkington.
+Copyright (c) 1997-2002 Tom Christiansen and Nathan Torkington.
All rights reserved.
-When included as part of the Standard Version of Perl, or as part of
-its complete documentation whether printed or otherwise, this work
-may be distributed only under the terms of Perl's Artistic License.
-Any distribution of this file or derivatives thereof I<outside>
-of that package require that special arrangements be made with
-copyright holder.
+This documentation is free; you can redistribute it and/or modify it
+under the same terms as Perl itself.
Irrespective of its distribution, all code examples in this file
are hereby placed into the public domain. You are permitted and
projects / p5sagit/p5-mst-13.2.git / blobdiff