3 perlreapi - perl regular expression plugin interface
7 As of Perl 5.9.5 there is a new interface for using other regexp
8 engines than the default one. Each engine is supposed to provide
9 access to a constant structure of the following format:
11 typedef struct regexp_engine {
12 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
13 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
14 char* strbeg, I32 minend, SV* screamer,
15 void* data, U32 flags);
16 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
17 char *strend, U32 flags,
18 struct re_scream_pos_data_s *data);
19 SV* (*checkstr) (pTHX_ REGEXP * const rx);
20 void (*free) (pTHX_ REGEXP * const rx);
21 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
23 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
24 SV const * const value);
25 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
27 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
28 SV * const value, U32 flags);
29 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
31 SV* (*qr_package)(pTHX_ REGEXP * const rx);
33 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
36 When a regexp is compiled, its C<engine> field is then set to point at
37 the appropriate structure so that when it needs to be used Perl can find
38 the right routines to do so.
40 In order to install a new regexp handler, C<$^H{regcomp}> is set
41 to an integer which (when casted appropriately) resolves to one of these
42 structures. When compiling, the C<comp> method is executed, and the
43 resulting regexp structure's engine field is expected to point back at
46 The pTHX_ symbol in the definition is a macro used by perl under threading
47 to provide an extra argument to the routine holding a pointer back to
48 the interpreter that is executing the regexp. So under threading all
49 routines get an extra argument.
55 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
57 Compile the pattern stored in C<pattern> using the given C<flags> and
58 return a pointer to a prepared C<REGEXP> structure that can perform
59 the match. See L</The REGEXP structure> below for an explanation of
60 the individual fields in the REGEXP struct.
62 The C<pattern> parameter is the scalar that was used as the
63 pattern. previous versions of perl would pass two C<char*> indicating
64 the start and end of the stringifed pattern, the following snippet can
65 be used to get the old parameters:
68 char* exp = SvPV(pattern, plen);
69 char* xend = exp + plen;
71 Since any scalar can be passed as a pattern it's possible to implement
72 an engine that does something with an array (C<< "ook" =~ [ qw/ eek
73 hlagh / ] >>) or with the non-stringified form of a compiled regular
74 expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
75 stringify everything using the snippet above but that doesn't mean
76 other engines have to.
78 The C<flags> paramater is a bitfield which indicates which of the
79 C<msixp> flags the regex was compiled with. It also contains
80 additional info such as whether C<use locale> is in effect.
82 The C<eogc> flags are stripped out before being passed to the comp
83 routine. The regex engine does not need to know whether any of these
84 are set as those flags should only affect what perl does with the
85 pattern and its match variables, not how it gets compiled and
88 By the time the comp callback is called, some of these flags have
89 already had effect (noted below where applicable). However most of
90 their effect occurs after the comp callback has run in routines that
91 read the C<< rx->extflags >> field which it populates.
93 In general the flags should be preserved in C<< rx->extflags >> after
94 compilation, although the regex engine might want to add or delete
95 some of them to invoke or disable some special behavior in perl. The
96 flags along with any special behavior they cause are documented below:
98 The pattern modifiers:
102 =item C</m> - RXf_PMf_MULTILINE
104 If this is in C<< rx->extflags >> it will be passed to
105 C<Perl_fbm_instr> by C<pp_split> which will treat the subject string
106 as a multi-line string.
108 =item C</s> - RXf_PMf_SINGLELINE
110 =item C</i> - RXf_PMf_FOLD
112 =item C</x> - RXf_PMf_EXTENDED
114 If present on a regex C<#> comments will be handled differently by the
115 tokenizer in some cases.
117 TODO: Document those cases.
119 =item C</p> - RXf_PMf_KEEPCOPY
129 Set if C<use locale> is in effect. If present in C<< rx->extflags >>
130 C<split> will use the locale dependant definition of whitespace under
131 when RXf_SKIPWHITE or RXf_WHITE are in effect. Under ASCII whitespace
132 is defined as per L<isSPACE|perlapi/ISSPACE>, and by the internal
133 macros C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use
138 Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime.
140 A regex engine may want to set or disable this flag during
141 compilation. The perl engine for instance may upgrade non-UTF-8
142 strings to UTF-8 if the pattern includes constructs such as C<\x{...}>
143 that can only match Unicode values.
147 If C<split> is invoked as C<split ' '> or with no arguments (which
148 really means C<split(' ', $_>, see L<split|perlfunc/split>), perl will
149 set this flag. The regex engine can then check for it and set the
150 SKIPWHITE and WHITE extflags. To do this the perl engine does:
152 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
153 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
157 These flags can be set during compilation to enable optimizations in
158 the C<split> operator.
164 If the flag is present in C<< rx->extflags >> C<split> will delete
165 whitespace from the start of the subject string before it's operated
166 on. What is considered whitespace depends on whether the subject is a
167 UTF-8 string and whether the C<RXf_PMf_LOCALE> flag is set.
169 If RXf_WHITE is set in addition to this flag C<split> will behave like
170 C<split " "> under the perl engine.
174 Tells the split operator to split the target string on newlines
175 (C<\n>) without invoking the regex engine.
177 Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp
178 == '^'>), even under C</^/s>, see L<split|perlfunc>. Of course a
179 different regex engine might want to use the same optimizations
180 with a different syntax.
184 Tells the split operator to split the target string on whitespace
185 without invoking the regex engine. The definition of whitespace varies
186 depending on whether the target string is a UTF-8 string and on
187 whether RXf_PMf_LOCALE is set.
189 Perl's engine sets this flag if the pattern is C<\s+>.
195 I32 exec(pTHX_ REGEXP * const rx,
196 char *stringarg, char* strend, char* strbeg,
197 I32 minend, SV* screamer,
198 void* data, U32 flags);
204 char* intuit(pTHX_ REGEXP * const rx,
205 SV *sv, char *strpos, char *strend,
206 const U32 flags, struct re_scream_pos_data_s *data);
208 Find the start position where a regex match should be attempted,
209 or possibly whether the regex engine should not be run because the
210 pattern can't match. This is called as appropriate by the core
211 depending on the values of the extflags member of the regexp
216 SV* checkstr(pTHX_ REGEXP * const rx);
218 Return a SV containing a string that must appear in the pattern. Used
219 by C<split> for optimising matches.
223 void free(pTHX_ REGEXP * const rx);
225 Called by perl when it is freeing a regexp pattern so that the engine
226 can release any resources pointed to by the C<pprivate> member of the
227 regexp structure. This is only responsible for freeing private data;
228 perl will handle releasing anything else contained in the regexp structure.
230 =head2 Numbered capture callbacks
232 Called to get/set the value of C<$`>, C<$'>, C<$&> and their named
233 equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
234 numbered capture buffers (C<$1>, C<$2>, ...).
236 The C<paren> paramater will be C<-2> for C<$`>, C<-1> for C<$'>, C<0>
237 for C<$&>, C<1> for C<$1> and so forth.
239 The names have been chosen by analogy with L<Tie::Scalar> methods
240 names with an additional B<LENGTH> callback for efficiency. However
241 named capture variables are currently not tied internally but
242 implemented via magic.
244 =head3 numbered_buff_FETCH
246 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
249 Fetch a specified numbered capture. C<sv> should be set to the scalar
250 to return, the scalar is passed as an argument rather than being
251 returned from the function because when it's called perl already has a
252 scalar to store the value, creating another one would be
253 redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and
254 friends, see L<perlapi>.
256 This callback is where perl untaints its own capture variables under
257 taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch>
258 function in F<regcomp.c> for how to untaint capture variables if
259 that's something you'd like your engine to do as well.
261 =head3 numbered_buff_STORE
263 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
264 SV const * const value);
266 Set the value of a numbered capture variable. C<value> is the scalar
267 that is to be used as the new value. It's up to the engine to make
268 sure this is used as the new value (or reject it).
272 if ("ook" =~ /(o*)/) {
273 # `paren' will be `1' and `value' will be `ee'
277 Perl's own engine will croak on any attempt to modify the capture
278 variables, to do this in another engine use the following callack
279 (copied from C<Perl_reg_numbered_buff_store>):
282 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
283 SV const * const value)
286 PERL_UNUSED_ARG(paren);
287 PERL_UNUSED_ARG(value);
290 Perl_croak(aTHX_ PL_no_modify);
293 Actually perl 5.10 will not I<always> croak in a statement that looks
294 like it would modify a numbered capture variable. This is because the
295 STORE callback will not be called if perl can determine that it
296 doesn't have to modify the value. This is exactly how tied variables
297 behave in the same situation:
300 use base 'Tie::Scalar';
302 sub TIESCALAR { bless [] }
304 sub STORE { die "This doesn't get called" }
308 tie my $sv => "CatptureVar";
311 Because C<$sv> is C<undef> when the C<y///> operator is applied to it
312 the transliteration won't actually execute and the program won't
313 C<die>. This is different to how 5.8 and earlier versions behaved
314 since the capture variables were READONLY variables then, now they'll
315 just die when assigned to in the default engine.
317 =head3 numbered_buff_LENGTH
319 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
322 Get the C<length> of a capture variable. There's a special callback
323 for this so that perl doesn't have to do a FETCH and run C<length> on
324 the result, since the length is (in perl's case) known from an offset
325 stored in C<<rx->offs> this is much more efficient:
327 I32 s1 = rx->offs[paren].start;
328 I32 s2 = rx->offs[paren].end;
331 This is a little bit more complex in the case of UTF-8, see what
332 C<Perl_reg_numbered_buff_length> does with
333 L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
335 =head2 Named capture callbacks
337 Called to get/set the value of C<%+> and C<%-> as well as by some
338 utility functions in L<re>.
340 There are two callbacks, C<named_buff> is called in all the cases the
341 FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks
342 would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the
343 same cases as FIRSTKEY and NEXTKEY.
345 The C<flags> parameter can be used to determine which of these
346 operations the callbacks should respond to, the following flags are
349 Which L<Tie::Hash> operation is being performed from the Perl level on
350 C<%+> or C<%+>, if any:
361 Whether C<%+> or C<%-> is being operated on, if any.
366 Whether this is being called as C<re::regname>, C<re::regnames> or
367 C<re::regnames_count>, if any. The first two will be combined with
368 C<RXapif_ONE> or C<RXapif_ALL>.
372 RXapif_REGNAMES_COUNT
374 Internally C<%+> and C<%-> are implemented with a real tied interface
375 via L<Tie::Hash::NamedCapture>. The methods in that package will call
376 back into these functions. However the usage of
377 L<Tie::Hash::NamedCapture> for this purpose might change in future
378 releases. For instance this might be implemented by magic instead
379 (would need an extension to mgvtbl).
383 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
384 SV * const value, U32 flags);
386 =head3 named_buff_iter
388 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
393 SV* qr_package(pTHX_ REGEXP * const rx);
395 The package the qr// magic object is blessed into (as seen by C<ref
396 qr//>). It is recommended that engines change this to their package
397 name for identification regardless of whether they implement methods
400 The package this method returns should also have the internal
401 C<Regexp> package in its C<@ISA>. C<qr//->isa("Regexp")> should always
402 be true regardless of what engine is being used.
404 Example implementation might be:
407 Example_qr_package(pTHX_ REGEXP * const rx)
410 return newSVpvs("re::engine::Example");
413 Any method calls on an object created with C<qr//> will be dispatched to the
414 package as a normal object.
416 use re::engine::Example;
418 $re->meth; # dispatched to re::engine::Example::meth()
420 To retrieve the C<REGEXP> object from the scalar in an XS function use
421 the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP
426 REGEXP * re = SvRX(sv);
430 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
432 On threaded builds a regexp may need to be duplicated so that the pattern
433 can be used by mutiple threads. This routine is expected to handle the
434 duplication of any private data pointed to by the C<pprivate> member of
435 the regexp structure. It will be called with the preconstructed new
436 regexp structure as an argument, the C<pprivate> member will point at
437 the B<old> private structue, and it is this routine's responsibility to
438 construct a copy and return a pointer to it (which perl will then use to
439 overwrite the field as passed to this routine.)
441 This allows the engine to dupe its private data but also if necessary
442 modify the final structure if it really must.
444 On unthreaded builds this field doesn't exist.
446 =head1 The REGEXP structure
448 The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
449 correctly build such a structure in their L</comp> routine.
451 The REGEXP structure contains all the data that perl needs to be aware of
452 to properly work with the regular expression. It includes data about
453 optimisations that perl can use to determine if the regex engine should
454 really be used, and various other control info that is needed to properly
455 execute patterns in various contexts such as is the pattern anchored in
456 some way, or what flags were used during the compile, or whether the
457 program contains special constructs that perl needs to be aware of.
459 In addition it contains two fields that are intended for the private
460 use of the regex engine that compiled the pattern. These are the
461 C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
462 an arbitrary structure whose use and management is the responsibility
463 of the compiling engine. perl will never modify either of these
466 typedef struct regexp {
467 /* what engine created this regexp? */
468 const struct regexp_engine* engine;
470 /* what re is this a lightweight copy of? */
471 struct regexp* mother_re;
473 /* Information about the match that the perl core uses to manage things */
474 U32 extflags; /* Flags used both externally and internally */
475 I32 minlen; /* mininum possible length of string to match */
476 I32 minlenret; /* mininum possible length of $& */
477 U32 gofs; /* chars left of pos that we search from */
479 /* substring data about strings that must appear
480 in the final match, used for optimisations */
481 struct reg_substr_data *substrs;
483 U32 nparens; /* number of capture buffers */
485 /* private engine specific data */
486 U32 intflags; /* Engine Specific Internal flags */
487 void *pprivate; /* Data private to the regex engine which
488 created this object. */
490 /* Data about the last/current match. These are modified during matching*/
491 U32 lastparen; /* last open paren matched */
492 U32 lastcloseparen; /* last close paren matched */
493 regexp_paren_pair *swap; /* Swap copy of *offs */
494 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
496 char *subbeg; /* saved or original string so \digit works forever. */
497 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
498 I32 sublen; /* Length of string pointed by subbeg */
500 /* Information about the match that isn't often used */
501 I32 prelen; /* length of precomp */
502 const char *precomp; /* pre-compilation regular expression */
504 char *wrapped; /* wrapped version of the pattern */
505 I32 wraplen; /* length of wrapped */
507 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
508 HV *paren_names; /* Optional hash of paren names */
510 /* Refcount of this regexp */
511 I32 refcnt; /* Refcount of this regexp */
514 The fields are discussed in more detail below:
518 This field points at a regexp_engine structure which contains pointers
519 to the subroutines that are to be used for performing a match. It
520 is the compiling routine's responsibility to populate this field before
521 returning the regexp object.
523 Internally this is set to C<NULL> unless a custom engine is specified in
524 C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
525 pointed to by C<RE_ENGINE_PTR>.
529 TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
533 This will be used by perl to see what flags the regexp was compiled
534 with, this will normally be set to the value of the flags parameter by
535 the L<comp|/comp> callback. See the L<comp|/comp> documentation for
538 =head2 C<minlen> C<minlenret>
540 The minimum string length required for the pattern to match. This is used to
541 prune the search space by not bothering to match any closer to the end of a
542 string than would allow a match. For instance there is no point in even
543 starting the regex engine if the minlen is 10 but the string is only 5
544 characters long. There is no way that the pattern can match.
546 C<minlenret> is the minimum length of the string that would be found
549 The difference between C<minlen> and C<minlenret> can be seen in the
554 where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
555 required to match but is not actually included in the matched content. This
556 distinction is particularly important as the substitution logic uses the
557 C<minlenret> to tell whether it can do in-place substition which can result in
558 considerable speedup.
562 Left offset from pos() to start match at.
566 Substring data about strings that must appear in the final match. This
567 is currently only used internally by perl's engine for but might be
568 used in the future for all engines for optimisations.
570 =head2 C<nparens>, C<lasparen>, and C<lastcloseparen>
572 These fields are used to keep track of how many paren groups could be matched
573 in the pattern, which was the last open paren to be entered, and which was
574 the last close paren to be entered.
578 The engine's private copy of the flags the pattern was compiled with. Usually
579 this is the same as C<extflags> unless the engine chose to modify one of them.
583 A void* pointing to an engine-defined data structure. The perl engine uses the
584 C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
585 engine should use something else.
593 A C<regexp_paren_pair> structure which defines offsets into the string being
594 matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
595 C<regexp_paren_pair> struct is defined as follows:
597 typedef struct regexp_paren_pair {
602 If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
603 capture buffer did not match. C<< ->offs[0].start/end >> represents C<$&> (or
604 C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
607 =head2 C<precomp> C<prelen>
609 Used for optimisations. C<precomp> holds a copy of the pattern that
610 was compiled and C<prelen> its length. When a new pattern is to be
611 compiled (such as inside a loop) the internal C<regcomp> operator
612 checks whether the last compiled C<REGEXP>'s C<precomp> and C<prelen>
613 are equivalent to the new one, and if so uses the old pattern instead
614 of compiling a new one.
616 The relevant snippet from C<Perl_pp_regcomp>:
618 if (!re || !re->precomp || re->prelen != (I32)len ||
619 memNE(re->precomp, t, len))
620 /* Compile a new pattern */
622 =head2 C<paren_names>
624 This is a hash used internally to track named capture buffers and their
625 offsets. The keys are the names of the buffers the values are dualvars,
626 with the IV slot holding the number of buffers with the given name and the
627 pv being an embedded array of I32. The values may also be contained
628 independently in the data array in cases where named backreferences are
633 Holds information on the longest string that must occur at a fixed
634 offset from the start of the pattern, and the longest string that must
635 occur at a floating offset from the start of the pattern. Used to do
636 Fast-Boyer-Moore searches on the string to find out if its worth using
637 the regex engine at all, and if so where in the string to search.
639 =head2 C<subbeg> C<sublen> C<saved_copy>
641 Used during execution phase for managing search and replace patterns.
643 =head2 C<wrapped> C<wraplen>
645 Stores the string C<qr//> stringifies to. The perl engine for example
646 stores C<(?-xism:eek)> in the case of C<qr/eek/>.
648 When using a custom engine that doesn't support the C<(?:)> construct
649 for inline modifiers, it's probably best to have C<qr//> stringify to
650 the supplied pattern, note that this will create undesired patterns in
653 my $x = qr/a|b/; # "a|b"
654 my $y = qr/c/i; # "c"
655 my $z = qr/$x$y/; # "a|bc"
657 There's no solution for this problem other than making the custom
658 engine understand a construct like C<(?:)>.
662 This stores the number of eval groups in the pattern. This is used for security
663 purposes when embedding compiled regexes into larger patterns with C<qr//>.
667 The number of times the structure is referenced. When this falls to 0 the
668 regexp is automatically freed by a call to pregfree. This should be set to 1 in
669 each engine's L</comp> routine.
673 Originally part of L<perlreguts>.
677 Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
682 Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
684 This program is free software; you can redistribute it and/or modify it under
685 the same terms as Perl itself.