From: Simon Cozens Date: Fri, 14 Jul 2000 06:49:21 +0000 (+0000) Subject: Merge perlhacktut into perlhack, update perlguts. X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=commitdiff_plain;h=a422fd2daf7c8932b983f20168fdc22a95c19633;p=p5sagit%2Fp5-mst-13.2.git Merge perlhacktut into perlhack, update perlguts. Subject: Re: Perlhacktut Date: 14 Jul 2000 06:49:21 GMT Message-ID: Subject: Re: Perlhacktut From: simon@brecon.co.uk (Simon Cozens) Date: 14 Jul 2000 07:09:45 GMT Message-ID: p4raw-id: //depot/perl@6406 --- diff --git a/pod/perlguts.pod b/pod/perlguts.pod index 160850c..5255519 100644 --- a/pod/perlguts.pod +++ b/pod/perlguts.pod @@ -1556,17 +1556,11 @@ First problem: deciding which functions will be public API functions and which will be private. All functions whose names begin C are private (think "S" for "secret" or "static"). All other functions begin with "Perl_", but just because a function begins with "Perl_" does not mean it is -part of the API. The easiest way to be B a function is part of the API -is to find its entry in L. If it exists in L, it's part -of the API. If it doesn't, and you think it should be (i.e., you need it for -your extension), send mail via L explaining why you think it -should be. - -(L itself is generated by embed.pl, a Perl script that generates -significant portions of the Perl source code. It has a list of almost -all the functions defined by the Perl interpreter along with their calling -characteristics and some flags. Functions that are part of the public API -are marked with an 'A' in its flags.) +part of the API. (See L.) The easiest way to be B a +function is part of the API is to find its entry in L. +If it exists in L, it's part of the API. If it doesn't, and you +think it should be (i.e., you need it for your extension), send mail via +L explaining why you think it should be. Second problem: there must be a syntax so that the same subroutine declarations and calls can pass a structure as their first argument, @@ -1781,6 +1775,320 @@ The Perl engine/interpreter and the host are orthogonal entities. There could be one or more interpreters in a process, and one or more "hosts", with free association between them. +=head1 Internal Functions + +All of Perl's internal functions which will be exposed to the outside +world are be prefixed by C so that they will not conflict with XS +functions or functions used in a program in which Perl is embedded. +Similarly, all global variables begin with C. (By convention, +static functions start with C) + +Inside the Perl core, you can get at the functions either with or +without the C prefix, thanks to a bunch of defines that live in +F. This header file is generated automatically from +F. F also creates the prototyping header files for +the internal functions, generates the documentation and a lot of other +bits and pieces. It's important that when you add a new function to the +core or change an existing one, you change the data in the table at the +end of F as well. Here's a sample entry from that table: + + Apd |SV** |av_fetch |AV* ar|I32 key|I32 lval + +The second column is the return type, the third column the name. Columns +after that are the arguments. The first column is a set of flags: + +=over 3 + +=item A + +This function is a part of the public API. + +=item p + +This function has a C prefix; ie, it is defined as C + +=item d + +This function has documentation using the C feature which we'll +look at in a second. + +=back + +Other available flags are: + +=over 3 + +=item s + +This is a static function and is defined as C. + +=item n + +This does not use C and C to pass interpreter context. (See +L.) + +=item r + +This function never returns; C, C and friends. + +=item f + +This function takes a variable number of arguments, C style. +The argument list should end with C<...>, like this: + + Afprd |void |croak |const char* pat|... + +=item m + +This function is part of the experimental development API, and may change +or disappear without notice. + +=item o + +This function should not have a compatibility macro to define, say, +C to C. It must be called as C. + +=item j + +This function is not a member of C. If you don't know +what this means, don't use it. + +=item x + +This function isn't exported out of the Perl core. + +=back + +If you edit F, you will need to run C to +force a rebuild of F and other auto-generated files. + +=head2 Source Documentation + +There's an effort going on to document the internal functions and +automatically produce reference manuals from them - L is one +such manual which details all the functions which are available to XS +writers. L is the autogenerated manual for the functions +which are not part of the API and are supposedly for internal use only. + +Source documentation is created by putting POD comments into the C +source, like this: + + /* + =for apidoc sv_setiv + + Copies an integer into the given SV. Does not handle 'set' magic. See + C. + + =cut + */ + +Please try and supply some documentation if you add functions to the +Perl core. + +=head1 Unicode Support + +Perl 5.6.0 introduced Unicode support. It's important for porters and XS +writers to understand this support and make sure that the code they +write does not corrupt Unicode data. + +=head2 What B Unicode, anyway? + +In the olden, less enlightened times, we all used to use ASCII. Most of +us did, anyway. The big problem with ASCII is that it's American. Well, +no, that's not actually the problem; the problem is that it's not +particularly useful for people who don't use the Roman alphabet. What +used to happen was that particular languages would stick their own +alphabet in the upper range of the sequence, between 128 and 255. Of +course, we then ended up with plenty of variants that weren't quite +ASCII, and the whole point of it being a standard was lost. + +Worse still, if you've got a language like Chinese or +Japanese that has hundreds or thousands of characters, then you really +can't fit them into a mere 256, so they had to forget about ASCII +altogether, and build their own systems using pairs of numbers to refer +to one character. + +To fix this, some people formed Unicode, Inc. and +produced a new character set containing all the characters you can +possibly think of and more. There are several ways of representing these +characters, and the one Perl uses is called UTF8. UTF8 uses +a variable number of bytes to represent a character, instead of just +one. You can learn more about Unicode at +L + +=head2 How can I recognise a UTF8 string? + +You can't. This is because UTF8 data is stored in bytes just like +non-UTF8 data. The Unicode character 200, (C<0xC8> for you hex types) +capital E with a grave accent, is represented by the two bytes +C. Unfortunately, the non-Unicode string C +has that byte sequence as well. So you can't tell just by looking - this +is what makes Unicode input an interesting problem. + +The API function C can help; it'll tell you if a string +contains only valid UTF8 characters. However, it can't do the work for +you. On a character-by-character basis, C will tell you +whether the current character in a string is valid UTF8. + +=head2 How does UTF8 represent Unicode characters? + +As mentioned above, UTF8 uses a variable number of bytes to store a +character. Characters with values 1...128 are stored in one byte, just +like good ol' ASCII. Character 129 is stored as C; this +contines up to character 191, which is C. Now we've run out of +bits (191 is binary C<10111111>) so we move on; 192 is C. And +so it goes on, moving to three bytes at character 2048. + +Assuming you know you're dealing with a UTF8 string, you can find out +how long the first character in it is with the C macro: + + char *utf = "\305\233\340\240\201"; + I32 len; + + len = UTF8SKIP(utf); /* len is 2 here */ + utf += len; + len = UTF8SKIP(utf); /* len is 3 here */ + +Another way to skip over characters in a UTF8 string is to use +C, which takes a string and a number of characters to skip +over. You're on your own about bounds checking, though, so don't use it +lightly. + +All bytes in a multi-byte UTF8 character will have the high bit set, so +you can test if you need to do something special with this character +like this: + + UV uv; + + if (utf & 0x80) + /* Must treat this as UTF8 */ + uv = utf8_to_uv(utf); + else + /* OK to treat this character as a byte */ + uv = *utf; + +You can also see in that example that we use C to get the +value of the character; the inverse function C is available +for putting a UV into UTF8: + + if (uv > 0x80) + /* Must treat this as UTF8 */ + utf8 = uv_to_utf8(utf8, uv); + else + /* OK to treat this character as a byte */ + *utf8++ = uv; + +You B convert characters to UVs using the above functions if +you're ever in a situation where you have to match UTF8 and non-UTF8 +characters. You may not skip over UTF8 characters in this case. If you +do this, you'll lose the ability to match hi-bit non-UTF8 characters; +for instance, if your UTF8 string contains C, and you skip +that character, you can never match a C in a non-UTF8 string. +So don't do that! + +=head2 How does Perl store UTF8 strings? + +Currently, Perl deals with Unicode strings and non-Unicode strings +slightly differently. If a string has been identified as being UTF-8 +encoded, Perl will set a flag in the SV, C. You can check and +manipulate this flag with the following macros: + + SvUTF8(sv) + SvUTF8_on(sv) + SvUTF8_off(sv) + +This flag has an important effect on Perl's treatment of the string: if +Unicode data is not properly distinguished, regular expressions, +C, C and other string handling operations will have +undesirable results. + +The problem comes when you have, for instance, a string that isn't +flagged is UTF8, and contains a byte sequence that could be UTF8 - +especially when combining non-UTF8 and UTF8 strings. + +Never forget that the C flag is separate to the PV value; you +need be sure you don't accidentally knock it off while you're +manipulating SVs. More specifically, you cannot expect to do this: + + SV *sv; + SV *nsv; + STRLEN len; + char *p; + + p = SvPV(sv, len); + frobnicate(p); + nsv = newSVpvn(p, len); + +The C string does not tell you the whole story, and you can't +copy or reconstruct an SV just by copying the string value. Check if the +old SV has the UTF8 flag set, and act accordingly: + + p = SvPV(sv, len); + frobnicate(p); + nsv = newSVpvn(p, len); + if (SvUTF8(sv)) + SvUTF8_on(nsv); + +In fact, your C function should be made aware of whether or +not it's dealing with UTF8 data, so that it can handle the string +appropriately. + +=head2 How do I convert a string to UTF8? + +If you're mixing UTF8 and non-UTF8 strings, you might find it necessary +to upgrade one of the strings to UTF8. If you've got an SV, the easiest +way to do this is: + + sv_utf8_upgrade(sv); + +However, you must not do this, for example: + + if (!SvUTF8(left)) + sv_utf8_upgrade(left); + +If you do this in a binary operator, you will actually change one of the +strings that came into the operator, and, while it shouldn't be noticable +by the end user, it can cause problems. + +Instead, C will give you a UTF8-encoded B of its +string argument. This is useful for having the data available for +comparisons and so on, without harming the orginal SV. There's also +C to go the other way, but naturally, this will fail if +the string contains any characters above 255 that can't be represented +in a single byte. + +=head2 Is there anything else I need to know? + +Not really. Just remember these things: + +=over 3 + +=item * + +There's no way to tell if a string is UTF8 or not. You can tell if an SV +is UTF8 by looking at is C flag. Don't forget to set the flag if +something should be UTF8. Treat the flag as part of the PV, even though +it's not - if you pass on the PV to somewhere, pass on the flag too. + +=item * + +If a string is UTF8, B use C to get at the value, +unless C in which case you can use C<*s>. + +=item * + +When writing to a UTF8 string, B use C, unless +C in which case you can use C<*s = uv>. + +=item * + +Mixing UTF8 and non-UTF8 strings is tricky. Use C to get +a new string which is UTF8 encoded. There are tricks you can use to +delay deciding whether you need to use a UTF8 string until you get to a +high character - C is one of those. + +=back + =head1 AUTHORS Until May 1997, this document was maintained by Jeff Okamoto diff --git a/pod/perlhack.pod b/pod/perlhack.pod index c640870..4d2545d 100644 --- a/pod/perlhack.pod +++ b/pod/perlhack.pod @@ -251,31 +251,6 @@ volunteers who test CPAN modules on a variety of platforms. Perl Labs platforms and gives feedback to the CPAN testers mailing list. Both efforts welcome volunteers. -To become an active and patching Perl porter, you'll need to learn how -Perl works on the inside. Chip Salzenberg, a pumpking, has written -articles on Perl internals for The Perl Journal -(I) which explain how various parts of the Perl -interpreter work. The C manpage explains the internal data -structures. And, of course, the C source code (sometimes sparsely -commented, sometimes commented well) is a great place to start (begin -with C and see where it goes from there). A lot of the style -of the Perl source is explained in the I file in -the source distribution. - -It is essential that you be comfortable using a good debugger -(e.g. gdb, dbx) before you can patch perl. Stepping through perl -as it executes a script is perhaps the best (if sometimes tedious) -way to gain a precise understanding of the overall architecture of -the language. - -If you build a version of the Perl interpreter with C<-DDEBUGGING>, -Perl's B<-D> command line flag will cause copious debugging information -to be emitted (see the C manpage). If you build a version of -Perl with compiler debugging information (e.g. with the C compiler's -C<-g> option instead of C<-O>) then you can step through the execution -of the interpreter with your favourite C symbolic debugger, setting -breakpoints on particular functions. - It's a good idea to read and lurk for a while before chipping in. That way you'll get to see the dynamic of the conversations, learn the personalities of the players, and hopefully be better prepared to make @@ -285,6 +260,1074 @@ If after all this you still think you want to join the perl5-porters mailing list, send mail to I. To unsubscribe, send mail to I. +To hack on the Perl guts, you'll need to read the following things: + +=over 3 + +=item L + +This is of paramount importance, since it's the documentation of what +goes where in the Perl source. Read it over a couple of times and it +might start to make sense - don't worry if it doesn't yet, because the +best way to study it is to read it in conjunction with poking at Perl +source, and we'll do that later on. + +You might also want to look at Gisle Aas's illustrated perlguts - +there's no guarantee that this will be absolutely up-to-date with the +latest documentation in the Perl core, but the fundamentals will be +right. (http://gisle.aas.no/perl/illguts/) + +=item L and L + +A working knowledge of XSUB programming is incredibly useful for core +hacking; XSUBs use techniques drawn from the PP code, the portion of the +guts that actually executes a Perl program. It's a lot gentler to learn +those techniques from simple examples and explanation than from the core +itself. + +=item L + +The documentation for the Perl API explains what some of the internal +functions do, as well as the many macros used in the source. + +=item F + +This is a collection of words of wisdom for a Perl porter; some of it is +only useful to the pumpkin holder, but most of it applies to anyone +wanting to go about Perl development. + +=item The perl5-porters FAQ + +This is posted to perl5-porters at the beginning on every month, and +should be available from http://perlhacker.org/p5p-faq; alternatively, +you can get the FAQ emailed to you by sending mail to +C. It contains hints on reading +perl5-porters, information on how perl5-porters works and how Perl +development in general works. + +=back + +=head2 Finding Your Way Around + +Perl maintenance can be split into a number of areas, and certain people +(pumpkins) will have responsibility for each area. These areas sometimes +correspond to files or directories in the source kit. Among the areas are: + +=over 3 + +=item Core modules + +Modules shipped as part of the Perl core live in the F and F +subdirectories: F is for the pure-Perl modules, and F +contains the core XS modules. + +=item Documentation + +Documentation maintenance includes looking after everything in the +F directory, (as well as contributing new documentation) and +the documentation to the modules in core. + +=item Configure + +The configure process is the way we make Perl portable across the +myriad of operating systems it supports. Responsibility for the +configure, build and installation process, as well as the overall +portability of the core code rests with the configure pumpkin - others +help out with individual operating systems. + +The files involved are the operating system directories, (F, +F, F and so on) the shell scripts which generate F +and F, as well as the metaconfig files which generate +F. (metaconfig isn't included in the core distribution.) + +=item Interpreter + +And of course, there's the core of the Perl interpreter itself. Let's +have a look at that in a little more detail. + +=back + +Before we leave looking at the layout, though, don't forget that +F contains not only the file names in the Perl distribution, +but short descriptions of what's in them, too. For an overview of the +important files, try this: + + perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST + +=head2 Elements of the interpreter + +The work of the interpreter has two main stages: compiling the code +into the internal representation, or bytecode, and then executing it. +L explains exactly how the compilation stage +happens. + +Here is a short breakdown of perl's operation: + +=over 3 + +=item Startup + +The action begins in F. (or F for miniperl) +This is very high-level code, enough to fit on a single screen, and it +resembles the code found in L; most of the real action takes +place in F + +First, F allocates some memory and constructs a Perl +interpreter: + + 1 PERL_SYS_INIT3(&argc,&argv,&env); + 2 + 3 if (!PL_do_undump) { + 4 my_perl = perl_alloc(); + 5 if (!my_perl) + 6 exit(1); + 7 perl_construct(my_perl); + 8 PL_perl_destruct_level = 0; + 9 } + +Line 1 is a macro, and its definition is dependent on your operating +system. Line 3 references C, a global variable - all +global variables in Perl start with C. This tells you whether the +current running program was created with the C<-u> flag to perl and then +F, which means it's going to be false in any sane context. + +Line 4 calls a function in F to allocate memory for a Perl +interpreter. It's quite a simple function, and the guts of it looks like +this: + + my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter)); + +Here you see an example of Perl's system abstraction, which we'll see +later: C is either your system's C, or Perl's +own C as defined in F if you selected that option at +configure time. + +Next, in line 7, we construct the interpreter; this sets up all the +special variables that Perl needs, the stacks, and so on. + +Now we pass Perl the command line options, and tell it to go: + + exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL); + if (!exitstatus) { + exitstatus = perl_run(my_perl); + } + + +C is actually a wrapper around C, as defined +in F, which processes the command line options, sets up any +statically linked XS modules, opens the program and calls C to +parse it. + +=item Parsing + +The aim of this stage is to take the Perl source, and turn it into an op +tree. We'll see what one of those looks like later. Strictly speaking, +there's three things going on here. + +C, the parser, lives in F, although you're better off +reading the original YACC input in F. (Yes, Virginia, there +B a YACC grammar for Perl!) The job of the parser is to take your +code and `understand' it, splitting it into sentences, deciding which +operands go with which operators and so on. + +The parser is nobly assisted by the lexer, which chunks up your input +into tokens, and decides what type of thing each token is: a variable +name, an operator, a bareword, a subroutine, a core function, and so on. +The main point of entry to the lexer is C, and that and its +associated routines can be found in F. Perl isn't much like +other computer languages; it's highly context sensitive at times, it can +be tricky to work out what sort of token something is, or where a token +ends. As such, there's a lot of interplay between the tokeniser and the +parser, which can get pretty frightening if you're not used to it. + +As the parser understands a Perl program, it builds up a tree of +operations for the interpreter to perform during execution. The routines +which construct and link together the various operations are to be found +in F, and will be examined later. + +=item Optimization + +Now the parsing stage is complete, and the finished tree represents +the operations that the Perl interpreter needs to perform to execute our +program. Next, Perl does a dry run over the tree looking for +optimisations: constant expressions such as C<3 + 4> will be computed +now, and the optimizer will also see if any multiple operations can be +replaced with a single one. For instance, to fetch the variable C<$foo>, +instead of grabbing the glob C<*foo> and looking at the scalar +component, the optimizer fiddles the op tree to use a function which +directly looks up the scalar in question. The main optimizer is C +in F, and many ops have their own optimizing functions. + +=item Running + +Now we're finally ready to go: we have compiled Perl byte code, and all +that's left to do is run it. The actual execution is done by the +C function in F; more specifically, it's done by +these three innocent looking lines: + + while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) { + PERL_ASYNC_CHECK(); + } + +You may be more comfortable with the Perl version of that: + + PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}}; + +Well, maybe not. Anyway, each op contains a function pointer, which +stipulates the function which will actually carry out the operation. +This function will return the next op in the sequence - this allows for +things like C which choose the next op dynamically at run time. +The C makes sure that things like signals interrupt +execution if required. + +The actual functions called are known as PP code, and they're spread +between four files: F contains the `hot' code, which is most +often used and highly optimized, F contains all the +system-specific functions, F contains the functions which +implement control structures (C, C and the like) and F +contains everything else. These are, if you like, the C code for Perl's +built-in functions and operators. + +=back + +=head2 Internal Variable Types + +You should by now have had a look at L, which tells you about +Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do +that now. + +These variables are used not only to represent Perl-space variables, but +also any constants in the code, as well as some structures completely +internal to Perl. The symbol table, for instance, is an ordinary Perl +hash. Your code is represented by an SV as it's read into the parser; +any program files you call are opened via ordinary Perl filehandles, and +so on. + +The core L module lets us examine SVs from a +Perl program. Let's see, for instance, how Perl treats the constant +C<"hello">. + + % perl -MDevel::Peek -e 'Dump("hello")' + 1 SV = PV(0xa041450) at 0xa04ecbc + 2 REFCNT = 1 + 3 FLAGS = (POK,READONLY,pPOK) + 4 PV = 0xa0484e0 "hello"\0 + 5 CUR = 5 + 6 LEN = 6 + +Reading C output takes a bit of practise, so let's go +through it line by line. + +Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in +memory. SVs themselves are very simple structures, but they contain a +pointer to a more complex structure. In this case, it's a PV, a +structure which holds a string value, at location C<0xa041450>. Line 2 +is the reference count; there are no other references to this data, so +it's 1. + +Line 3 are the flags for this SV - it's OK to use it as a PV, it's a +read-only SV (because it's a constant) and the data is a PV internally. +Next we've got the contents of the string, starting at location +C<0xa0484e0>. + +Line 5 gives us the current length of the string - note that this does +B include the null terminator. Line 6 is not the length of the +string, but the length of the currently allocated buffer; as the string +grows, Perl automatically extends the available storage via a routine +called C. + +You can get at any of these quantities from C very easily; just add +C to the name of the field shown in the snippet, and you've got a +macro which will return the value: C returns the current +length of the string, C returns the reference count, +C returns the string itself with its length, and so on. +More macros to manipulate these properties can be found in L. + +Let's take an example of manipulating a PV, from C, in F + + 1 void + 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len) + 3 { + 4 STRLEN tlen; + 5 char *junk; + + 6 junk = SvPV_force(sv, tlen); + 7 SvGROW(sv, tlen + len + 1); + 8 if (ptr == junk) + 9 ptr = SvPVX(sv); + 10 Move(ptr,SvPVX(sv)+tlen,len,char); + 11 SvCUR(sv) += len; + 12 *SvEND(sv) = '\0'; + 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */ + 14 SvTAINT(sv); + 15 } + +This is a function which adds a string, C, of length C onto +the end of the PV stored in C. The first thing we do in line 6 is +make sure that the SV B a valid PV, by calling the C +macro to force a PV. As a side effect, C gets set to the current +value of the PV, and the PV itself is returned to C. + +In line 7, we make sure that the SV will have enough room to accomodate +the old string, the new string and the null terminator. If C isn't +big enough, C will reallocate space for us. + +Now, if C is the same as the string we're trying to add, we can +grab the string directly from the SV; C is the address of the PV +in the SV. + +Line 10 does the actual catenation: the C macro moves a chunk of +memory around: we move the string C to the end of the PV - that's +the start of the PV plus its current length. We're moving C bytes +of type C. After doing so, we need to tell Perl we've extended the +string, by altering C to reflect the new length. C is a +macro which gives us the end of the string, so that needs to be a +C<"\0">. + +Line 13 manipulates the flags; since we've changed the PV, any IV or NV +values will no longer be valid: if we have C<$a=10; $a.="6";> we don't +want to use the old IV of 10. C is a special UTF8-aware +version of C, a macro which turns off the IOK and NOK flags +and turns on POK. The final C is a macro which launders tainted +data if taint mode is turned on. + +AVs and HVs are more complicated, but SVs are by far the most common +variable type being thrown around. Having seen something of how we +manipulate these, let's go on and look at how the op tree is +constructed. + +=head2 Op Trees + +First, what is the op tree, anyway? The op tree is the parsed +representation of your program, as we saw in our section on parsing, and +it's the sequence of operations that Perl goes through to execute your +program, as we saw in L. + +An op is a fundamental operation that Perl can perform: all the built-in +functions and operators are ops, and there are a series of ops which +deal with concepts the interpreter needs internally - entering and +leaving a block, ending a statement, fetching a variable, and so on. + +The op tree is connected in two ways: you can imagine that there are two +"routes" through it, two orders in which you can traverse the tree. +First, parse order reflects how the parser understood the code, and +secondly, execution order tells perl what order to perform the +operations in. + +The easiest way to examine the op tree is to stop Perl after it has +finished parsing, and get it to dump out the tree. This is exactly what +the compiler backends L and L do. + +Let's have a look at how Perl sees C<$a = $b + $c>: + + % perl -MO=Terse -e '$a=$b+$c' + 1 LISTOP (0x8179888) leave + 2 OP (0x81798b0) enter + 3 COP (0x8179850) nextstate + 4 BINOP (0x8179828) sassign + 5 BINOP (0x8179800) add [1] + 6 UNOP (0x81796e0) null [15] + 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b + 8 UNOP (0x81797e0) null [15] + 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c + 10 UNOP (0x816b4f0) null [15] + 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a + +Let's start in the middle, at line 4. This is a BINOP, a binary +operator, which is at location C<0x8179828>. The specific operator in +question is C - scalar assignment - and you can find the code +which implements it in the function C in F. As a +binary operator, it has two children: the add operator, providing the +result of C<$b+$c>, is uppermost on line 5, and the left hand side is on +line 10. + +Line 10 is the null op: this does exactly nothing. What is that doing +there? If you see the null op, it's a sign that something has been +optimized away after parsing. As we mentioned in L, +the optimization stage sometimes converts two operations into one, for +example when fetching a scalar variable. When this happens, instead of +rewriting the op tree and cleaning up the dangling pointers, it's easier +just to replace the redundant operation with the null op. Originally, +the tree would have looked like this: + + 10 SVOP (0x816b4f0) rv2sv [15] + 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a + +That is, fetch the C entry from the main symbol table, and then look +at the scalar component of it: C (C into F) +happens to do both these things. + +The right hand side, starting at line 5 is similar to what we've just +seen: we have the C op (C also in F) add together +two Cs. + +Now, what's this about? + + 1 LISTOP (0x8179888) leave + 2 OP (0x81798b0) enter + 3 COP (0x8179850) nextstate + +C and C are scoping ops, and their job is to perform any +housekeeping every time you enter and leave a block: lexical variables +are tidied up, unreferenced variables are destroyed, and so on. Every +program will have those first three lines: C is a list, and its +children are all the statements in the block. Statements are delimited +by C, so a block is a collection of C ops, with +the ops to be performed for each statement being the children of +C. C is a single op which functions as a marker. + +That's how Perl parsed the program, from top to bottom: + + Program + | + Statement + | + = + / \ + / \ + $a + + / \ + $b $c + +However, it's impossible to B the operations in this order: +you have to find the values of C<$b> and C<$c> before you add them +together, for instance. So, the other thread that runs through the op +tree is the execution order: each op has a field C which points +to the next op to be run, so following these pointers tells us how perl +executes the code. We can traverse the tree in this order using +the C option to C: + + % perl -MO=Terse,exec -e '$a=$b+$c' + 1 OP (0x8179928) enter + 2 COP (0x81798c8) nextstate + 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b + 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c + 5 BINOP (0x8179878) add [1] + 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a + 7 BINOP (0x81798a0) sassign + 8 LISTOP (0x8179900) leave + +This probably makes more sense for a human: enter a block, start a +statement. Get the values of C<$b> and C<$c>, and add them together. +Find C<$a>, and assign one to the other. Then leave. + +The way Perl builds up these op trees in the parsing process can be +unravelled by examining F, the YACC grammar. Let's take the +piece we need to construct the tree for C<$a = $b + $c> + + 1 term : term ASSIGNOP term + 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); } + 3 | term ADDOP term + 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } + +If you're not used to reading BNF grammars, this is how it works: You're +fed certain things by the tokeniser, which generally end up in upper +case. Here, C, is provided when the tokeniser sees C<+> in your +code. C is provided when C<=> is used for assigning. These are +`terminal symbols', because you can't get any simpler than them. + +The grammar, lines one and three of the snippet above, tells you how to +build up more complex forms. These complex forms, `non-terminal symbols' +are generally placed in lower case. C here is a non-terminal +symbol, representing a single expression. + +The grammar gives you the following rule: you can make the thing on the +left of the colon if you see all the things on the right in sequence. +This is called a "reduction", and the aim of parsing is to completely +reduce the input. There are several different ways you can perform a +reduction, separated by vertical bars: so, C followed by C<=> +followed by C makes a C, and C followed by C<+> +followed by C can also make a C. + +So, if you see two terms with an C<=> or C<+>, between them, you can +turn them into a single expression. When you do this, you execute the +code in the block on the next line: if you see C<=>, you'll do the code +in line 2. If you see C<+>, you'll do the code in line 4. It's this code +which contributes to the op tree. + + | term ADDOP term + { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } + +What this does is creates a new binary op, and feeds it a number of +variables. The variables refer to the tokens: C<$1> is the first token in +the input, C<$2> the second, and so on - think regular expression +backreferences. C<$$> is the op returned from this reduction. So, we +call C to create a new binary operator. The first parameter to +C, a function in F, is the op type. It's an addition +operator, so we want the type to be C. We could specify this +directly, but it's right there as the second token in the input, so we +use C<$2>. The second parameter is the op's flags: 0 means `nothing +special'. Then the things to add: the left and right hand side of our +expression, in scalar context. + +=head2 Stacks + +When perl executes something like C, how does it pass on its +results to the next op? The answer is, through the use of stacks. Perl +has a number of stacks to store things it's currently working on, and +we'll look at the three most important ones here. + +=over 3 + +=item Argument stack + +Arguments are passed to PP code and returned from PP code using the +argument stack, C. The typical way to handle arguments is to pop +them off the stack, deal with them how you wish, and then push the result +back onto the stack. This is how, for instance, the cosine operator +works: + + NV value; + value = POPn; + value = Perl_cos(value); + XPUSHn(value); + +We'll see a more tricky example of this when we consider Perl's macros +below. C gives you the NV (floating point value) of the top SV on +the stack: the C<$x> in C. Then we compute the cosine, and push +the result back as an NV. The C in C means that the stack +should be extended if necessary - it can't be necessary here, because we +know there's room for one more item on the stack, since we've just +removed one! The C macros at least guarantee safety. + +Alternatively, you can fiddle with the stack directly: C gives you +the first element in your portion of the stack, and C gives you +the top SV/IV/NV/etc. on the stack. So, for instance, to do unary +negation of an integer: + + SETi(-TOPi); + +Just set the integer value of the top stack entry to its negation. + +Argument stack manipulation in the core is exactly the same as it is in +XSUBs - see L, L and L for a longer +description of the macros used in stack manipulation. + +=item Mark stack + +I say `your portion of the stack' above because PP code doesn't +necessarily get the whole stack to itself: if your function calls +another function, you'll only want to expose the arguments aimed for the +called function, and not (necessarily) let it get at your own data. The +way we do this is to have a `virtual' bottom-of-stack, exposed to each +function. The mark stack keeps bookmarks to locations in the argument +stack usable by each function. For instance, when dealing with a tied +variable, (internally, something with `P' magic) Perl has to call +methods for accesses to the tied variables. However, we need to separate +the arguments exposed to the method to the argument exposed to the +original function - the store or fetch or whatever it may be. Here's how +the tied C is implemented; see C in F: + + 1 PUSHMARK(SP); + 2 EXTEND(SP,2); + 3 PUSHs(SvTIED_obj((SV*)av, mg)); + 4 PUSHs(val); + 5 PUTBACK; + 6 ENTER; + 7 call_method("PUSH", G_SCALAR|G_DISCARD); + 8 LEAVE; + 9 POPSTACK; + +The lines which concern the mark stack are the first, fifth and last +lines: they save away, restore and remove the current position of the +argument stack. + +Let's examine the whole implementation, for practice: + + 1 PUSHMARK(SP); + +Push the current state of the stack pointer onto the mark stack. This is +so that when we've finished adding items to the argument stack, Perl +knows how many things we've added recently. + + 2 EXTEND(SP,2); + 3 PUSHs(SvTIED_obj((SV*)av, mg)); + 4 PUSHs(val); + +We're going to add two more items onto the argument stack: when you have +a tied array, the C subroutine receives the object and the value +to be pushed, and that's exactly what we have here - the tied object, +retrieved with C, and the value, the SV C. + + 5 PUTBACK; + +Next we tell Perl to make the change to the global stack pointer: C +only gave us a local copy, not a reference to the global. + + 6 ENTER; + 7 call_method("PUSH", G_SCALAR|G_DISCARD); + 8 LEAVE; + +C and C localise a block of code - they make sure that all +variables are tidied up, everything that has been localised gets +its previous value returned, and so on. Think of them as the C<{> and +C<}> of a Perl block. + +To actually do the magic method call, we have to call a subroutine in +Perl space: C takes care of that, and it's described in +L. We call the C method in scalar context, and we're +going to discard its return value. + + 9 POPSTACK; + +Finally, we remove the value we placed on the mark stack, since we +don't need it any more. + +=item Save stack + +C doesn't have a concept of local scope, so perl provides one. We've +seen that C and C are used as scoping braces; the save +stack implements the C equivalent of, for example: + + { + local $foo = 42; + ... + } + +See L for how to use the save stack. + +=back + +=head2 Millions of Macros + +One thing you'll notice about the Perl source is that it's full of +macros. Some have called the pervasive use of macros the hardest thing +to understand, others find it adds to clarity. Let's take an example, +the code which implements the addition operator: + + 1 PP(pp_add) + 2 { + 3 djSP; dATARGET; tryAMAGICbin(add,opASSIGN); + 4 { + 5 dPOPTOPnnrl_ul; + 6 SETn( left + right ); + 7 RETURN; + 8 } + 9 } + +Every line here (apart from the braces, of course) contains a macro. The +first line sets up the function declaration as Perl expects for PP code; +line 3 sets up variable declarations for the argument stack and the +target, the return value of the operation. Finally, it tries to see if +the addition operation is overloaded; if so, the appropriate subroutine +is called. + +Line 5 is another variable declaration - all variable declarations start +with C - which pops from the top of the argument stack two NVs (hence +C) and puts them into the variables C and C, hence the +C. These are the two operands to the addition operator. Next, we +call C to set the NV of the return value to the result of adding +the two values. This done, we return - the C macro makes sure +that our return value is properly handled, and we pass the next operator +to run back to the main run loop. + +Most of these macros are explained in L, and some of the more +important ones are explained in L as well. Pay special attention +to L for information on +the C<[pad]THX_?> macros. + + +=head2 Poking at Perl + +To really poke around with Perl, you'll probably want to build Perl for +debugging, like this: + + ./Configure -d -D optimize=-g + make + +C<-g> is a flag to the C compiler to have it produce debugging +information which will allow us to step through a running program. +F will also turn on the C compilation symbol which +enables all the internal debugging code in Perl. There are a whole bunch +of things you can debug with this: L lists them all, and the +best way to find out about them is to play about with them. The most +useful options are probably + + l Context (loop) stack processing + t Trace execution + o Method and overloading resolution + c String/numeric conversions + +Some of the functionality of the debugging code can be achieved using XS +modules. + + -Dr => use re 'debug' + -Dx => use O 'Debug' + +=head2 Using a source-level debugger + +If the debugging output of C<-D> doesn't help you, it's time to step +through perl's execution with a source-level debugger. + +=over 3 + +=item * + +We'll use C for our examples here; the principles will apply to any +debugger, but check the manual of the one you're using. + +=back + +To fire up the debugger, type + + gdb ./perl + +You'll want to do that in your Perl source tree so the debugger can read +the source code. You should see the copyright message, followed by the +prompt. + + (gdb) + +C will get you into the documentation, but here are the most +useful commands: + +=over 3 + +=item run [args] + +Run the program with the given arguments. + +=item break function_name + +=item break source.c:xxx + +Tells the debugger that we'll want to pause execution when we reach +either the named function (but see L!) or the given +line in the named source file. + +=item step + +Steps through the program a line at a time. + +=item next + +Steps through the program a line at a time, without descending into +functions. + +=item continue + +Run until the next breakpoint. + +=item finish + +Run until the end of the current function, then stop again. + +=item + +Just pressing Enter will do the most recent operation again - it's a +blessing when stepping through miles of source code. + +=item print + +Execute the given C code and print its results. B: Perl makes +heavy use of macros, and F is not aware of macros. You'll have to +substitute them yourself. So, for instance, you can't say + + print SvPV_nolen(sv) + +but you have to say + + print Perl_sv_2pv_nolen(sv) + +You may find it helpful to have a "macro dictionary", which you can +produce by saying C. Even then, F won't +recursively apply the macros for you. + +=back + +=head2 Dumping Perl Data Structures + +One way to get around this macro hell is to use the dumping functions in +F; these work a little like an internal +L, but they also cover OPs and other structures +that you can't get at from Perl. Let's take an example. We'll use the +C<$a = $b + $c> we used before, but give it a bit of context: +C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around? + +What about C, the function we examined earlier to implement the +C<+> operator: + + (gdb) break Perl_pp_add + Breakpoint 1 at 0x46249f: file pp_hot.c, line 309. + +Notice we use C and not C - see L. +With the breakpoint in place, we can run our program: + + (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c' + +Lots of junk will go past as gdb reads in the relevant source files and +libraries, and then: + + Breakpoint 1, Perl_pp_add () at pp_hot.c:309 + 309 djSP; dATARGET; tryAMAGICbin(add,opASSIGN); + (gdb) step + 311 dPOPTOPnnrl_ul; + (gdb) + +We looked at this bit of code before, and we said that C +arranges for two Cs to be placed into C and C - let's +slightly expand it: + + #define dPOPTOPnnrl_ul NV right = POPn; \ + SV *leftsv = TOPs; \ + NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0 + +C takes the SV from the top of the stack and obtains its NV either +directly (if C is set) or by calling the C function. +C takes the next SV from the top of the stack - yes, C uses +C - but doesn't remove it. We then use C to get the NV from +C in the same way as before - yes, C uses C. + +Since we don't have an NV for C<$b>, we'll have to use C to +convert it. If we step again, we'll find ourselves there: + + Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669 + 1669 if (!sv) + (gdb) + +We can now use C to investigate the SV: + + SV = PV(0xa057cc0) at 0xa0675d0 + REFCNT = 1 + FLAGS = (POK,pPOK) + PV = 0xa06a510 "6XXXX"\0 + CUR = 5 + LEN = 6 + $1 = void + +We know we're going to get C<6> from this, so let's finish the +subroutine: + + (gdb) finish + Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671 + 0x462669 in Perl_pp_add () at pp_hot.c:311 + 311 dPOPTOPnnrl_ul; + +We can also dump out this op: the current op is always stored in +C, and we can dump it with C. This'll give us +similar output to L. + + { + 13 TYPE = add ===> 14 + TARG = 1 + FLAGS = (SCALAR,KIDS) + { + TYPE = null ===> (12) + (was rv2sv) + FLAGS = (SCALAR,KIDS) + { + 11 TYPE = gvsv ===> 12 + FLAGS = (SCALAR) + GV = main::b + } + } + +< finish this later > + +=head2 Patching + +All right, we've now had a look at how to navigate the Perl sources and +some things you'll need to know when fiddling with them. Let's now get +on and create a simple patch. Here's something Larry suggested: if a +C is the first active format during a C, (for example, +C) then the resulting string should be treated as +UTF8 encoded. + +How do we prepare to fix this up? First we locate the code in question - +the C happens at runtime, so it's going to be in one of the F +files. Sure enough, C is in F. Since we're going to be +altering this file, let's copy it to F. + +Now let's look over C: we take a pattern into C, and then +loop over the pattern, taking each format character in turn into +C. Then for each possible format character, we swallow up +the other arguments in the pattern (a field width, an asterisk, and so +on) and convert the next chunk input into the specified format, adding +it onto the output SV C. + +How do we know if the C is the first format in the C? Well, if +we have a pointer to the start of C then, if we see a C we can +test whether we're still at the start of the string. So, here's where +C is set up: + + STRLEN fromlen; + register char *pat = SvPVx(*++MARK, fromlen); + register char *patend = pat + fromlen; + register I32 len; + I32 datumtype; + SV *fromstr; + +We'll have another string pointer in there: + + STRLEN fromlen; + register char *pat = SvPVx(*++MARK, fromlen); + register char *patend = pat + fromlen; + + char *patcopy; + register I32 len; + I32 datumtype; + SV *fromstr; + +And just before we start the loop, we'll set C to be the start +of C: + + items = SP - MARK; + MARK++; + sv_setpvn(cat, "", 0); + + patcopy = pat; + while (pat < patend) { + +Now if we see a C which was at the start of the string, we turn on +the UTF8 flag for the output SV, C: + + + if (datumtype == 'U' && pat==patcopy+1) + + SvUTF8_on(cat); + if (datumtype == '#') { + while (pat < patend && *pat != '\n') + pat++; + +Remember that it has to be C because the first character of +the string is the C which has been swallowed into C + +Oops, we forgot one thing: what if there are spaces at the start of the +pattern? C will have C as the first active +character, even though it's not the first thing in the pattern. In this +case, we have to advance C along with C when we see spaces: + + if (isSPACE(datumtype)) + continue; + +needs to become + + if (isSPACE(datumtype)) { + patcopy++; + continue; + } + +OK. That's the C part done. Now we must do two additional things before +this patch is ready to go: we've changed the behaviour of Perl, and so +we must document that change. We must also provide some more regression +tests to make sure our patch works and doesn't create a bug somewhere +else along the line. + +The regression tests for each operator live in F, and so we make +a copy of F to F. Now we can add our tests +to the end. First, we'll test that the C does indeed create Unicode +strings: + + print 'not ' unless "1.20.300.4000" eq sprintf "%vd", pack("U*",1,20,300,4000); + print "ok $test\n"; $test++; + +Now we'll test that we got that space-at-the-beginning business right: + + print 'not ' unless "1.20.300.4000" eq + sprintf "%vd", pack(" U*",1,20,300,4000); + print "ok $test\n"; $test++; + +And finally we'll test that we don't make Unicode strings if C is B +the first active format: + + print 'not ' unless v1.20.300.4000 ne + sprintf "%vd", pack("C0U*",1,20,300,4000); + print "ok $test\n"; $test++; + +Musn't forget to change the number of tests which appears at the top, or +else the automated tester will get confused: + + -print "1..156\n"; + +print "1..159\n"; + +We now compile up Perl, and run it through the test suite. Our new +tests pass, hooray! + +Finally, the documentation. The job is never done until the paperwork is +over, so let's describe the change we've just made. The relevant place +is F; again, we make a copy, and then we'll insert +this text in the description of C: + + =item * + + If the pattern begins with a C, the resulting string will be treated + as Unicode-encoded. You can force UTF8 encoding on in a string with an + initial C, and the bytes that follow will be interpreted as Unicode + characters. If you don't want this to happen, you can begin your pattern + with C (or anything else) to force Perl not to UTF8 encode your + string, and then follow this with a C somewhere in your pattern. + +All done. Now let's create the patch. F tells us +that if we're making major changes, we should copy the entire directory +to somewhere safe before we begin fiddling, and then do + + diff -ruN old new > patch + +However, we know which files we've changed, and we can simply do this: + + diff -u pp.c~ pp.c > patch + diff -u t/op/pack.t~ t/op/pack.t >> patch + diff -u pod/perlfunc.pod~ pod/perlfunc.pod >> patch + +We end up with a patch looking a little like this: + + --- pp.c~ Fri Jun 02 04:34:10 2000 + +++ pp.c Fri Jun 16 11:37:25 2000 + @@ -4375,6 +4375,7 @@ + register I32 items; + STRLEN fromlen; + register char *pat = SvPVx(*++MARK, fromlen); + + char *patcopy; + register char *patend = pat + fromlen; + register I32 len; + I32 datumtype; + @@ -4405,6 +4406,7 @@ + ... + +And finally, we submit it, with our rationale, to perl5-porters. Job +done! + +=head2 CONCLUSION + +We've had a brief look around the Perl source, an overview of the stages +F goes through when it's running your code, and how to use a +debugger to poke at the Perl guts. Finally, we took a very simple +problem and demonstrated how to solve it fully - with documentation, +regression tests, and finally a patch for submission to p5p. + +I'd now suggest you read over those references again, and then, as soon +as possible, get your hands dirty. The best way to learn is by doing, +so: + +=over 3 + +=item * + +Subscribe to perl5-porters, follow the patches and try and understand +them; don't be afraid to ask if there's a portion you're not clear on - +who knows, you may unearth a bug in the patch... + +=item * + +Keep up to date with the bleeding edge Perl distributions and get +familiar with the changes. Try and get an idea of what areas people are +working on and the changes they're making. + +=item * + +Find an area of Perl that seems interesting to you, and see if you can +work out how it works. Scan through the source, and step over it in the +debugger. Play, poke, investigate, fiddle! You'll probably get to +understand not just your chosen area but a much wider range of F's +activity as well, and probably sooner than you'd think. + +=back + +=over 3 + +=item I + +=back + +If you can do these things, you've started on the long road to Perl porting. +Thanks for wanting to help make Perl better - and happy hacking! + =head1 AUTHOR This document was written by Nathan Torkington, and is maintained by