Commit | Line | Data |
e8cd7eae |
1 | =head1 NAME |
2 | |
3 | perlhack - How to hack at the Perl internals |
4 | |
5 | =head1 DESCRIPTION |
6 | |
7 | This document attempts to explain how Perl development takes place, |
8 | and ends with some suggestions for people wanting to become bona fide |
9 | porters. |
10 | |
11 | The perl5-porters mailing list is where the Perl standard distribution |
12 | is maintained and developed. The list can get anywhere from 10 to 150 |
13 | messages a day, depending on the heatedness of the debate. Most days |
14 | there are two or three patches, extensions, features, or bugs being |
15 | discussed at a time. |
16 | |
f8e3975a |
17 | A searchable archive of the list is at either: |
e8cd7eae |
18 | |
19 | http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/ |
20 | |
f8e3975a |
21 | or |
22 | |
23 | http://archive.develooper.com/perl5-porters@perl.org/ |
24 | |
e8cd7eae |
25 | List subscribers (the porters themselves) come in several flavours. |
26 | Some are quiet curious lurkers, who rarely pitch in and instead watch |
27 | the ongoing development to ensure they're forewarned of new changes or |
28 | features in Perl. Some are representatives of vendors, who are there |
29 | to make sure that Perl continues to compile and work on their |
30 | platforms. Some patch any reported bug that they know how to fix, |
31 | some are actively patching their pet area (threads, Win32, the regexp |
32 | engine), while others seem to do nothing but complain. In other |
33 | words, it's your usual mix of technical people. |
34 | |
35 | Over this group of porters presides Larry Wall. He has the final word |
f6c51b38 |
36 | in what does and does not change in the Perl language. Various |
37 | releases of Perl are shepherded by a ``pumpking'', a porter |
38 | responsible for gathering patches, deciding on a patch-by-patch |
39 | feature-by-feature basis what will and will not go into the release. |
caf100c0 |
40 | For instance, Gurusamy Sarathy was the pumpking for the 5.6 release of |
41 | Perl, and Jarkko Hietaniemi is the pumpking for the 5.8 release, and |
42 | Hugo van der Sanden will be the pumpking for the 5.10 release. |
e8cd7eae |
43 | |
44 | In addition, various people are pumpkings for different things. For |
45 | instance, Andy Dougherty and Jarkko Hietaniemi share the I<Configure> |
caf100c0 |
46 | pumpkin. |
e8cd7eae |
47 | |
48 | Larry sees Perl development along the lines of the US government: |
49 | there's the Legislature (the porters), the Executive branch (the |
50 | pumpkings), and the Supreme Court (Larry). The legislature can |
51 | discuss and submit patches to the executive branch all they like, but |
52 | the executive branch is free to veto them. Rarely, the Supreme Court |
53 | will side with the executive branch over the legislature, or the |
54 | legislature over the executive branch. Mostly, however, the |
55 | legislature and the executive branch are supposed to get along and |
56 | work out their differences without impeachment or court cases. |
57 | |
58 | You might sometimes see reference to Rule 1 and Rule 2. Larry's power |
59 | as Supreme Court is expressed in The Rules: |
60 | |
61 | =over 4 |
62 | |
63 | =item 1 |
64 | |
65 | Larry is always by definition right about how Perl should behave. |
66 | This means he has final veto power on the core functionality. |
67 | |
68 | =item 2 |
69 | |
70 | Larry is allowed to change his mind about any matter at a later date, |
71 | regardless of whether he previously invoked Rule 1. |
72 | |
73 | =back |
74 | |
75 | Got that? Larry is always right, even when he was wrong. It's rare |
76 | to see either Rule exercised, but they are often alluded to. |
77 | |
78 | New features and extensions to the language are contentious, because |
79 | the criteria used by the pumpkings, Larry, and other porters to decide |
80 | which features should be implemented and incorporated are not codified |
81 | in a few small design goals as with some other languages. Instead, |
82 | the heuristics are flexible and often difficult to fathom. Here is |
83 | one person's list, roughly in decreasing order of importance, of |
84 | heuristics that new features have to be weighed against: |
85 | |
86 | =over 4 |
87 | |
88 | =item Does concept match the general goals of Perl? |
89 | |
90 | These haven't been written anywhere in stone, but one approximation |
91 | is: |
92 | |
93 | 1. Keep it fast, simple, and useful. |
94 | 2. Keep features/concepts as orthogonal as possible. |
95 | 3. No arbitrary limits (platforms, data sizes, cultures). |
96 | 4. Keep it open and exciting to use/patch/advocate Perl everywhere. |
97 | 5. Either assimilate new technologies, or build bridges to them. |
98 | |
99 | =item Where is the implementation? |
100 | |
101 | All the talk in the world is useless without an implementation. In |
102 | almost every case, the person or people who argue for a new feature |
103 | will be expected to be the ones who implement it. Porters capable |
104 | of coding new features have their own agendas, and are not available |
105 | to implement your (possibly good) idea. |
106 | |
107 | =item Backwards compatibility |
108 | |
109 | It's a cardinal sin to break existing Perl programs. New warnings are |
110 | contentious--some say that a program that emits warnings is not |
111 | broken, while others say it is. Adding keywords has the potential to |
112 | break programs, changing the meaning of existing token sequences or |
113 | functions might break programs. |
114 | |
115 | =item Could it be a module instead? |
116 | |
117 | Perl 5 has extension mechanisms, modules and XS, specifically to avoid |
118 | the need to keep changing the Perl interpreter. You can write modules |
119 | that export functions, you can give those functions prototypes so they |
120 | can be called like built-in functions, you can even write XS code to |
121 | mess with the runtime data structures of the Perl interpreter if you |
122 | want to implement really complicated things. If it can be done in a |
123 | module instead of in the core, it's highly unlikely to be added. |
124 | |
125 | =item Is the feature generic enough? |
126 | |
127 | Is this something that only the submitter wants added to the language, |
128 | or would it be broadly useful? Sometimes, instead of adding a feature |
129 | with a tight focus, the porters might decide to wait until someone |
130 | implements the more generalized feature. For instance, instead of |
131 | implementing a ``delayed evaluation'' feature, the porters are waiting |
132 | for a macro system that would permit delayed evaluation and much more. |
133 | |
134 | =item Does it potentially introduce new bugs? |
135 | |
136 | Radical rewrites of large chunks of the Perl interpreter have the |
137 | potential to introduce new bugs. The smaller and more localized the |
138 | change, the better. |
139 | |
140 | =item Does it preclude other desirable features? |
141 | |
142 | A patch is likely to be rejected if it closes off future avenues of |
143 | development. For instance, a patch that placed a true and final |
144 | interpretation on prototypes is likely to be rejected because there |
145 | are still options for the future of prototypes that haven't been |
146 | addressed. |
147 | |
148 | =item Is the implementation robust? |
149 | |
150 | Good patches (tight code, complete, correct) stand more chance of |
151 | going in. Sloppy or incorrect patches might be placed on the back |
152 | burner until the pumpking has time to fix, or might be discarded |
153 | altogether without further notice. |
154 | |
155 | =item Is the implementation generic enough to be portable? |
156 | |
157 | The worst patches make use of a system-specific features. It's highly |
158 | unlikely that nonportable additions to the Perl language will be |
159 | accepted. |
160 | |
a936dd3c |
161 | =item Is the implementation tested? |
162 | |
163 | Patches which change behaviour (fixing bugs or introducing new features) |
164 | must include regression tests to verify that everything works as expected. |
165 | Without tests provided by the original author, how can anyone else changing |
166 | perl in the future be sure that they haven't unwittingly broken the behaviour |
167 | the patch implements? And without tests, how can the patch's author be |
9d077eaa |
168 | confident that his/her hard work put into the patch won't be accidentally |
a936dd3c |
169 | thrown away by someone in the future? |
170 | |
e8cd7eae |
171 | =item Is there enough documentation? |
172 | |
173 | Patches without documentation are probably ill-thought out or |
174 | incomplete. Nothing can be added without documentation, so submitting |
175 | a patch for the appropriate manpages as well as the source code is |
a936dd3c |
176 | always a good idea. |
e8cd7eae |
177 | |
178 | =item Is there another way to do it? |
179 | |
180 | Larry said ``Although the Perl Slogan is I<There's More Than One Way |
181 | to Do It>, I hesitate to make 10 ways to do something''. This is a |
182 | tricky heuristic to navigate, though--one man's essential addition is |
183 | another man's pointless cruft. |
184 | |
185 | =item Does it create too much work? |
186 | |
187 | Work for the pumpking, work for Perl programmers, work for module |
188 | authors, ... Perl is supposed to be easy. |
189 | |
f6c51b38 |
190 | =item Patches speak louder than words |
191 | |
192 | Working code is always preferred to pie-in-the-sky ideas. A patch to |
193 | add a feature stands a much higher chance of making it to the language |
194 | than does a random feature request, no matter how fervently argued the |
195 | request might be. This ties into ``Will it be useful?'', as the fact |
196 | that someone took the time to make the patch demonstrates a strong |
197 | desire for the feature. |
198 | |
e8cd7eae |
199 | =back |
200 | |
201 | If you're on the list, you might hear the word ``core'' bandied |
202 | around. It refers to the standard distribution. ``Hacking on the |
203 | core'' means you're changing the C source code to the Perl |
204 | interpreter. ``A core module'' is one that ships with Perl. |
205 | |
a1f349fd |
206 | =head2 Keeping in sync |
207 | |
e8cd7eae |
208 | The source code to the Perl interpreter, in its different versions, is |
f224927c |
209 | kept in a repository managed by a revision control system ( which is |
210 | currently the Perforce program, see http://perforce.com/ ). The |
e8cd7eae |
211 | pumpkings and a few others have access to the repository to check in |
212 | changes. Periodically the pumpking for the development version of Perl |
213 | will release a new version, so the rest of the porters can see what's |
2be4c08b |
214 | changed. The current state of the main trunk of repository, and patches |
215 | that describe the individual changes that have happened since the last |
216 | public release are available at this location: |
217 | |
0cfb3454 |
218 | http://public.activestate.com/gsar/APC/ |
2be4c08b |
219 | ftp://ftp.linux.activestate.com/pub/staff/gsar/APC/ |
220 | |
0cfb3454 |
221 | If you're looking for a particular change, or a change that affected |
222 | a particular set of files, you may find the B<Perl Repository Browser> |
223 | useful: |
224 | |
225 | http://public.activestate.com/cgi-bin/perlbrowse |
226 | |
227 | You may also want to subscribe to the perl5-changes mailing list to |
228 | receive a copy of each patch that gets submitted to the maintenance |
229 | and development "branches" of the perl repository. See |
230 | http://lists.perl.org/ for subscription information. |
231 | |
a1f349fd |
232 | If you are a member of the perl5-porters mailing list, it is a good |
233 | thing to keep in touch with the most recent changes. If not only to |
234 | verify if what you would have posted as a bug report isn't already |
235 | solved in the most recent available perl development branch, also |
236 | known as perl-current, bleading edge perl, bleedperl or bleadperl. |
2be4c08b |
237 | |
238 | Needless to say, the source code in perl-current is usually in a perpetual |
239 | state of evolution. You should expect it to be very buggy. Do B<not> use |
240 | it for any purpose other than testing and development. |
e8cd7eae |
241 | |
3e148164 |
242 | Keeping in sync with the most recent branch can be done in several ways, |
243 | but the most convenient and reliable way is using B<rsync>, available at |
244 | ftp://rsync.samba.org/pub/rsync/ . (You can also get the most recent |
245 | branch by FTP.) |
a1f349fd |
246 | |
247 | If you choose to keep in sync using rsync, there are two approaches |
3e148164 |
248 | to doing so: |
a1f349fd |
249 | |
250 | =over 4 |
251 | |
252 | =item rsync'ing the source tree |
253 | |
3e148164 |
254 | Presuming you are in the directory where your perl source resides |
a1f349fd |
255 | and you have rsync installed and available, you can `upgrade' to |
256 | the bleadperl using: |
257 | |
258 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current/ . |
259 | |
260 | This takes care of updating every single item in the source tree to |
261 | the latest applied patch level, creating files that are new (to your |
262 | distribution) and setting date/time stamps of existing files to |
263 | reflect the bleadperl status. |
264 | |
c6d0653e |
265 | Note that this will not delete any files that were in '.' before |
266 | the rsync. Once you are sure that the rsync is running correctly, |
267 | run it with the --delete and the --dry-run options like this: |
268 | |
269 | # rsync -avz --delete --dry-run rsync://ftp.linux.activestate.com/perl-current/ . |
270 | |
271 | This will I<simulate> an rsync run that also deletes files not |
272 | present in the bleadperl master copy. Observe the results from |
273 | this run closely. If you are sure that the actual run would delete |
274 | no files precious to you, you could remove the '--dry-run' option. |
275 | |
a1f349fd |
276 | You can than check what patch was the latest that was applied by |
277 | looking in the file B<.patch>, which will show the number of the |
278 | latest patch. |
279 | |
280 | If you have more than one machine to keep in sync, and not all of |
281 | them have access to the WAN (so you are not able to rsync all the |
282 | source trees to the real source), there are some ways to get around |
283 | this problem. |
284 | |
285 | =over 4 |
286 | |
287 | =item Using rsync over the LAN |
288 | |
289 | Set up a local rsync server which makes the rsynced source tree |
3e148164 |
290 | available to the LAN and sync the other machines against this |
a1f349fd |
291 | directory. |
292 | |
1577cd80 |
293 | From http://rsync.samba.org/README.html : |
a1f349fd |
294 | |
295 | "Rsync uses rsh or ssh for communication. It does not need to be |
296 | setuid and requires no special privileges for installation. It |
3958b146 |
297 | does not require an inetd entry or a daemon. You must, however, |
a1f349fd |
298 | have a working rsh or ssh system. Using ssh is recommended for |
299 | its security features." |
300 | |
301 | =item Using pushing over the NFS |
302 | |
303 | Having the other systems mounted over the NFS, you can take an |
3e148164 |
304 | active pushing approach by checking the just updated tree against |
305 | the other not-yet synced trees. An example would be |
306 | |
307 | #!/usr/bin/perl -w |
308 | |
309 | use strict; |
310 | use File::Copy; |
311 | |
312 | my %MF = map { |
313 | m/(\S+)/; |
314 | $1 => [ (stat $1)[2, 7, 9] ]; # mode, size, mtime |
315 | } `cat MANIFEST`; |
316 | |
317 | my %remote = map { $_ => "/$_/pro/3gl/CPAN/perl-5.7.1" } qw(host1 host2); |
318 | |
319 | foreach my $host (keys %remote) { |
320 | unless (-d $remote{$host}) { |
321 | print STDERR "Cannot Xsync for host $host\n"; |
322 | next; |
323 | } |
324 | foreach my $file (keys %MF) { |
325 | my $rfile = "$remote{$host}/$file"; |
326 | my ($mode, $size, $mtime) = (stat $rfile)[2, 7, 9]; |
327 | defined $size or ($mode, $size, $mtime) = (0, 0, 0); |
328 | $size == $MF{$file}[1] && $mtime == $MF{$file}[2] and next; |
329 | printf "%4s %-34s %8d %9d %8d %9d\n", |
330 | $host, $file, $MF{$file}[1], $MF{$file}[2], $size, $mtime; |
331 | unlink $rfile; |
332 | copy ($file, $rfile); |
333 | utime time, $MF{$file}[2], $rfile; |
334 | chmod $MF{$file}[0], $rfile; |
335 | } |
336 | } |
337 | |
338 | though this is not perfect. It could be improved with checking |
a1f349fd |
339 | file checksums before updating. Not all NFS systems support |
340 | reliable utime support (when used over the NFS). |
341 | |
342 | =back |
343 | |
344 | =item rsync'ing the patches |
345 | |
346 | The source tree is maintained by the pumpking who applies patches to |
347 | the files in the tree. These patches are either created by the |
348 | pumpking himself using C<diff -c> after updating the file manually or |
349 | by applying patches sent in by posters on the perl5-porters list. |
350 | These patches are also saved and rsync'able, so you can apply them |
351 | yourself to the source files. |
352 | |
353 | Presuming you are in a directory where your patches reside, you can |
3e148164 |
354 | get them in sync with |
a1f349fd |
355 | |
356 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . |
357 | |
358 | This makes sure the latest available patch is downloaded to your |
359 | patch directory. |
360 | |
3e148164 |
361 | It's then up to you to apply these patches, using something like |
a1f349fd |
362 | |
df3477ff |
363 | # last=`ls -t *.gz | sed q` |
a1f349fd |
364 | # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ . |
365 | # find . -name '*.gz' -newer $last -exec gzcat {} \; >blead.patch |
366 | # cd ../perl-current |
367 | # patch -p1 -N <../perl-current-diffs/blead.patch |
368 | |
369 | or, since this is only a hint towards how it works, use CPAN-patchaperl |
370 | from Andreas König to have better control over the patching process. |
371 | |
372 | =back |
373 | |
f7e1e956 |
374 | =head2 Why rsync the source tree |
a1f349fd |
375 | |
376 | =over 4 |
377 | |
10f58044 |
378 | =item It's easier to rsync the source tree |
a1f349fd |
379 | |
380 | Since you don't have to apply the patches yourself, you are sure all |
381 | files in the source tree are in the right state. |
382 | |
a1f349fd |
383 | =item It's more reliable |
384 | |
0cfb3454 |
385 | While both the rsync-able source and patch areas are automatically |
386 | updated every few minutes, keep in mind that applying patches may |
387 | sometimes mean careful hand-holding, especially if your version of |
388 | the C<patch> program does not understand how to deal with new files, |
389 | files with 8-bit characters, or files without trailing newlines. |
a1f349fd |
390 | |
391 | =back |
392 | |
f7e1e956 |
393 | =head2 Why rsync the patches |
a1f349fd |
394 | |
395 | =over 4 |
396 | |
10f58044 |
397 | =item It's easier to rsync the patches |
a1f349fd |
398 | |
399 | If you have more than one machine that you want to keep in track with |
3e148164 |
400 | bleadperl, it's easier to rsync the patches only once and then apply |
a1f349fd |
401 | them to all the source trees on the different machines. |
402 | |
403 | In case you try to keep in pace on 5 different machines, for which |
404 | only one of them has access to the WAN, rsync'ing all the source |
3e148164 |
405 | trees should than be done 5 times over the NFS. Having |
a1f349fd |
406 | rsync'ed the patches only once, I can apply them to all the source |
3e148164 |
407 | trees automatically. Need you say more ;-) |
a1f349fd |
408 | |
409 | =item It's a good reference |
410 | |
411 | If you do not only like to have the most recent development branch, |
412 | but also like to B<fix> bugs, or extend features, you want to dive |
413 | into the sources. If you are a seasoned perl core diver, you don't |
414 | need no manuals, tips, roadmaps, perlguts.pod or other aids to find |
415 | your way around. But if you are a starter, the patches may help you |
416 | in finding where you should start and how to change the bits that |
417 | bug you. |
418 | |
419 | The file B<Changes> is updated on occasions the pumpking sees as his |
420 | own little sync points. On those occasions, he releases a tar-ball of |
421 | the current source tree (i.e. perl@7582.tar.gz), which will be an |
422 | excellent point to start with when choosing to use the 'rsync the |
423 | patches' scheme. Starting with perl@7582, which means a set of source |
424 | files on which the latest applied patch is number 7582, you apply all |
f18956b7 |
425 | succeeding patches available from then on (7583, 7584, ...). |
a1f349fd |
426 | |
427 | You can use the patches later as a kind of search archive. |
428 | |
429 | =over 4 |
430 | |
431 | =item Finding a start point |
432 | |
433 | If you want to fix/change the behaviour of function/feature Foo, just |
434 | scan the patches for patches that mention Foo either in the subject, |
3e148164 |
435 | the comments, or the body of the fix. A good chance the patch shows |
a1f349fd |
436 | you the files that are affected by that patch which are very likely |
437 | to be the starting point of your journey into the guts of perl. |
438 | |
439 | =item Finding how to fix a bug |
440 | |
441 | If you've found I<where> the function/feature Foo misbehaves, but you |
442 | don't know how to fix it (but you do know the change you want to |
443 | make), you can, again, peruse the patches for similar changes and |
444 | look how others apply the fix. |
445 | |
446 | =item Finding the source of misbehaviour |
447 | |
448 | When you keep in sync with bleadperl, the pumpking would love to |
3958b146 |
449 | I<see> that the community efforts really work. So after each of his |
a1f349fd |
450 | sync points, you are to 'make test' to check if everything is still |
451 | in working order. If it is, you do 'make ok', which will send an OK |
452 | report to perlbug@perl.org. (If you do not have access to a mailer |
3e148164 |
453 | from the system you just finished successfully 'make test', you can |
a1f349fd |
454 | do 'make okfile', which creates the file C<perl.ok>, which you can |
455 | than take to your favourite mailer and mail yourself). |
456 | |
3958b146 |
457 | But of course, as always, things will not always lead to a success |
a1f349fd |
458 | path, and one or more test do not pass the 'make test'. Before |
459 | sending in a bug report (using 'make nok' or 'make nokfile'), check |
460 | the mailing list if someone else has reported the bug already and if |
461 | so, confirm it by replying to that message. If not, you might want to |
462 | trace the source of that misbehaviour B<before> sending in the bug, |
463 | which will help all the other porters in finding the solution. |
464 | |
3e148164 |
465 | Here the saved patches come in very handy. You can check the list of |
466 | patches to see which patch changed what file and what change caused |
467 | the misbehaviour. If you note that in the bug report, it saves the |
468 | one trying to solve it, looking for that point. |
a1f349fd |
469 | |
470 | =back |
471 | |
472 | If searching the patches is too bothersome, you might consider using |
473 | perl's bugtron to find more information about discussions and |
474 | ramblings on posted bugs. |
475 | |
3e148164 |
476 | If you want to get the best of both worlds, rsync both the source |
477 | tree for convenience, reliability and ease and rsync the patches |
478 | for reference. |
479 | |
52315700 |
480 | =back |
481 | |
482 | |
483 | =head2 Perlbug remote interface |
484 | |
485 | =over 4 |
486 | |
f224927c |
487 | There are three (3) remote administrative interfaces for modifying bug |
488 | status, category, etc. In all cases an admin must be first registered |
489 | with the Perlbug database by sending an email request to |
490 | richard@perl.org or bugmongers@perl.org. |
52315700 |
491 | |
f224927c |
492 | The main requirement is the willingness to classify, (with the |
493 | emphasis on closing where possible :), outstanding bugs. Further |
494 | explanation can be garnered from the web at http://bugs.perl.org/ , or |
495 | by asking on the admin mailing list at: bugmongers@perl.org |
52315700 |
496 | |
497 | For more info on the web see |
498 | |
499 | http://bugs.perl.org/perlbug.cgi?req=spec |
500 | |
52315700 |
501 | =item 1 http://bugs.perl.org |
502 | |
503 | Login via the web, (remove B<admin/> if only browsing), where interested Cc's, tests, patches and change-ids, etc. may be assigned. |
504 | |
505 | http://bugs.perl.org/admin/index.html |
506 | |
507 | |
508 | =item 2 bugdb@perl.org |
509 | |
510 | Where the subject line is used for commands: |
511 | |
512 | To: bugdb@perl.org |
513 | Subject: -a close bugid1 bugid2 aix install |
514 | |
515 | To: bugdb@perl.org |
516 | Subject: -h |
517 | |
518 | |
519 | =item 3 commands_and_bugdids@bugs.perl.org |
520 | |
521 | Where the address itself is the source for the commands: |
522 | |
523 | To: close_bugid1_bugid2_aix@bugs.perl.org |
524 | |
525 | To: help@bugs.perl.org |
526 | |
527 | |
528 | =item notes, patches, tests |
529 | |
530 | For patches and tests, the message body is assigned to the appropriate bug/s and forwarded to p5p for their attention. |
531 | |
532 | To: test_<bugid1>_aix_close@bugs.perl.org |
533 | Subject: this is a test for the (now closed) aix bug |
534 | |
535 | Test is the body of the mail |
536 | |
537 | =back |
538 | |
a1f349fd |
539 | =head2 Submitting patches |
540 | |
f7e1e956 |
541 | Always submit patches to I<perl5-porters@perl.org>. If you're |
542 | patching a core module and there's an author listed, send the author a |
543 | copy (see L<Patching a core module>). This lets other porters review |
544 | your patch, which catches a surprising number of errors in patches. |
545 | Either use the diff program (available in source code form from |
f224927c |
546 | ftp://ftp.gnu.org/pub/gnu/ , or use Johan Vromans' I<makepatch> |
f7e1e956 |
547 | (available from I<CPAN/authors/id/JV/>). Unified diffs are preferred, |
548 | but context diffs are accepted. Do not send RCS-style diffs or diffs |
549 | without context lines. More information is given in the |
550 | I<Porting/patching.pod> file in the Perl source distribution. Please |
551 | patch against the latest B<development> version (e.g., if you're |
552 | fixing a bug in the 5.005 track, patch against the latest 5.005_5x |
553 | version). Only patches that survive the heat of the development |
554 | branch get applied to maintenance versions. |
555 | |
556 | Your patch should update the documentation and test suite. See |
557 | L<Writing a test>. |
e8cd7eae |
558 | |
559 | To report a bug in Perl, use the program I<perlbug> which comes with |
560 | Perl (if you can't get Perl to work, send mail to the address |
f18956b7 |
561 | I<perlbug@perl.org> or I<perlbug@perl.com>). Reporting bugs through |
e8cd7eae |
562 | I<perlbug> feeds into the automated bug-tracking system, access to |
f224927c |
563 | which is provided through the web at http://bugs.perl.org/ . It |
e8cd7eae |
564 | often pays to check the archives of the perl5-porters mailing list to |
565 | see whether the bug you're reporting has been reported before, and if |
566 | so whether it was considered a bug. See above for the location of |
567 | the searchable archives. |
568 | |
f224927c |
569 | The CPAN testers ( http://testers.cpan.org/ ) are a group of |
ba139f7d |
570 | volunteers who test CPAN modules on a variety of platforms. Perl |
571 | Smokers ( http://archives.develooper.com/daily-build@perl.org/ ) |
572 | automatically tests Perl source releases on platforms with various |
573 | configurations. Both efforts welcome volunteers. |
e8cd7eae |
574 | |
e8cd7eae |
575 | It's a good idea to read and lurk for a while before chipping in. |
576 | That way you'll get to see the dynamic of the conversations, learn the |
577 | personalities of the players, and hopefully be better prepared to make |
578 | a useful contribution when do you speak up. |
579 | |
580 | If after all this you still think you want to join the perl5-porters |
f6c51b38 |
581 | mailing list, send mail to I<perl5-porters-subscribe@perl.org>. To |
582 | unsubscribe, send mail to I<perl5-porters-unsubscribe@perl.org>. |
e8cd7eae |
583 | |
a422fd2d |
584 | To hack on the Perl guts, you'll need to read the following things: |
585 | |
586 | =over 3 |
587 | |
588 | =item L<perlguts> |
589 | |
590 | This is of paramount importance, since it's the documentation of what |
591 | goes where in the Perl source. Read it over a couple of times and it |
592 | might start to make sense - don't worry if it doesn't yet, because the |
593 | best way to study it is to read it in conjunction with poking at Perl |
594 | source, and we'll do that later on. |
595 | |
596 | You might also want to look at Gisle Aas's illustrated perlguts - |
597 | there's no guarantee that this will be absolutely up-to-date with the |
598 | latest documentation in the Perl core, but the fundamentals will be |
1577cd80 |
599 | right. ( http://gisle.aas.no/perl/illguts/ ) |
a422fd2d |
600 | |
601 | =item L<perlxstut> and L<perlxs> |
602 | |
603 | A working knowledge of XSUB programming is incredibly useful for core |
604 | hacking; XSUBs use techniques drawn from the PP code, the portion of the |
605 | guts that actually executes a Perl program. It's a lot gentler to learn |
606 | those techniques from simple examples and explanation than from the core |
607 | itself. |
608 | |
609 | =item L<perlapi> |
610 | |
611 | The documentation for the Perl API explains what some of the internal |
612 | functions do, as well as the many macros used in the source. |
613 | |
614 | =item F<Porting/pumpkin.pod> |
615 | |
616 | This is a collection of words of wisdom for a Perl porter; some of it is |
617 | only useful to the pumpkin holder, but most of it applies to anyone |
618 | wanting to go about Perl development. |
619 | |
620 | =item The perl5-porters FAQ |
621 | |
622 | This is posted to perl5-porters at the beginning on every month, and |
f224927c |
623 | should be available from http://perlhacker.org/p5p-faq ; alternatively, |
a422fd2d |
624 | you can get the FAQ emailed to you by sending mail to |
625 | C<perl5-porters-faq@perl.org>. It contains hints on reading |
626 | perl5-porters, information on how perl5-porters works and how Perl |
627 | development in general works. |
628 | |
629 | =back |
630 | |
631 | =head2 Finding Your Way Around |
632 | |
633 | Perl maintenance can be split into a number of areas, and certain people |
634 | (pumpkins) will have responsibility for each area. These areas sometimes |
635 | correspond to files or directories in the source kit. Among the areas are: |
636 | |
637 | =over 3 |
638 | |
639 | =item Core modules |
640 | |
641 | Modules shipped as part of the Perl core live in the F<lib/> and F<ext/> |
642 | subdirectories: F<lib/> is for the pure-Perl modules, and F<ext/> |
643 | contains the core XS modules. |
644 | |
f7e1e956 |
645 | =item Tests |
646 | |
647 | There are tests for nearly all the modules, built-ins and major bits |
648 | of functionality. Test files all have a .t suffix. Module tests live |
649 | in the F<lib/> and F<ext/> directories next to the module being |
650 | tested. Others live in F<t/>. See L<Writing a test> |
651 | |
a422fd2d |
652 | =item Documentation |
653 | |
654 | Documentation maintenance includes looking after everything in the |
655 | F<pod/> directory, (as well as contributing new documentation) and |
656 | the documentation to the modules in core. |
657 | |
658 | =item Configure |
659 | |
660 | The configure process is the way we make Perl portable across the |
661 | myriad of operating systems it supports. Responsibility for the |
662 | configure, build and installation process, as well as the overall |
663 | portability of the core code rests with the configure pumpkin - others |
664 | help out with individual operating systems. |
665 | |
666 | The files involved are the operating system directories, (F<win32/>, |
667 | F<os2/>, F<vms/> and so on) the shell scripts which generate F<config.h> |
668 | and F<Makefile>, as well as the metaconfig files which generate |
669 | F<Configure>. (metaconfig isn't included in the core distribution.) |
670 | |
671 | =item Interpreter |
672 | |
673 | And of course, there's the core of the Perl interpreter itself. Let's |
674 | have a look at that in a little more detail. |
675 | |
676 | =back |
677 | |
678 | Before we leave looking at the layout, though, don't forget that |
679 | F<MANIFEST> contains not only the file names in the Perl distribution, |
680 | but short descriptions of what's in them, too. For an overview of the |
681 | important files, try this: |
682 | |
683 | perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST |
684 | |
685 | =head2 Elements of the interpreter |
686 | |
687 | The work of the interpreter has two main stages: compiling the code |
688 | into the internal representation, or bytecode, and then executing it. |
689 | L<perlguts/Compiled code> explains exactly how the compilation stage |
690 | happens. |
691 | |
692 | Here is a short breakdown of perl's operation: |
693 | |
694 | =over 3 |
695 | |
696 | =item Startup |
697 | |
698 | The action begins in F<perlmain.c>. (or F<miniperlmain.c> for miniperl) |
699 | This is very high-level code, enough to fit on a single screen, and it |
700 | resembles the code found in L<perlembed>; most of the real action takes |
701 | place in F<perl.c> |
702 | |
703 | First, F<perlmain.c> allocates some memory and constructs a Perl |
704 | interpreter: |
705 | |
706 | 1 PERL_SYS_INIT3(&argc,&argv,&env); |
707 | 2 |
708 | 3 if (!PL_do_undump) { |
709 | 4 my_perl = perl_alloc(); |
710 | 5 if (!my_perl) |
711 | 6 exit(1); |
712 | 7 perl_construct(my_perl); |
713 | 8 PL_perl_destruct_level = 0; |
714 | 9 } |
715 | |
716 | Line 1 is a macro, and its definition is dependent on your operating |
717 | system. Line 3 references C<PL_do_undump>, a global variable - all |
718 | global variables in Perl start with C<PL_>. This tells you whether the |
719 | current running program was created with the C<-u> flag to perl and then |
720 | F<undump>, which means it's going to be false in any sane context. |
721 | |
722 | Line 4 calls a function in F<perl.c> to allocate memory for a Perl |
723 | interpreter. It's quite a simple function, and the guts of it looks like |
724 | this: |
725 | |
726 | my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter)); |
727 | |
728 | Here you see an example of Perl's system abstraction, which we'll see |
729 | later: C<PerlMem_malloc> is either your system's C<malloc>, or Perl's |
730 | own C<malloc> as defined in F<malloc.c> if you selected that option at |
731 | configure time. |
732 | |
733 | Next, in line 7, we construct the interpreter; this sets up all the |
734 | special variables that Perl needs, the stacks, and so on. |
735 | |
736 | Now we pass Perl the command line options, and tell it to go: |
737 | |
738 | exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL); |
739 | if (!exitstatus) { |
740 | exitstatus = perl_run(my_perl); |
741 | } |
742 | |
743 | |
744 | C<perl_parse> is actually a wrapper around C<S_parse_body>, as defined |
745 | in F<perl.c>, which processes the command line options, sets up any |
746 | statically linked XS modules, opens the program and calls C<yyparse> to |
747 | parse it. |
748 | |
749 | =item Parsing |
750 | |
751 | The aim of this stage is to take the Perl source, and turn it into an op |
752 | tree. We'll see what one of those looks like later. Strictly speaking, |
753 | there's three things going on here. |
754 | |
755 | C<yyparse>, the parser, lives in F<perly.c>, although you're better off |
756 | reading the original YACC input in F<perly.y>. (Yes, Virginia, there |
757 | B<is> a YACC grammar for Perl!) The job of the parser is to take your |
758 | code and `understand' it, splitting it into sentences, deciding which |
759 | operands go with which operators and so on. |
760 | |
761 | The parser is nobly assisted by the lexer, which chunks up your input |
762 | into tokens, and decides what type of thing each token is: a variable |
763 | name, an operator, a bareword, a subroutine, a core function, and so on. |
764 | The main point of entry to the lexer is C<yylex>, and that and its |
765 | associated routines can be found in F<toke.c>. Perl isn't much like |
766 | other computer languages; it's highly context sensitive at times, it can |
767 | be tricky to work out what sort of token something is, or where a token |
768 | ends. As such, there's a lot of interplay between the tokeniser and the |
769 | parser, which can get pretty frightening if you're not used to it. |
770 | |
771 | As the parser understands a Perl program, it builds up a tree of |
772 | operations for the interpreter to perform during execution. The routines |
773 | which construct and link together the various operations are to be found |
774 | in F<op.c>, and will be examined later. |
775 | |
776 | =item Optimization |
777 | |
778 | Now the parsing stage is complete, and the finished tree represents |
779 | the operations that the Perl interpreter needs to perform to execute our |
780 | program. Next, Perl does a dry run over the tree looking for |
781 | optimisations: constant expressions such as C<3 + 4> will be computed |
782 | now, and the optimizer will also see if any multiple operations can be |
783 | replaced with a single one. For instance, to fetch the variable C<$foo>, |
784 | instead of grabbing the glob C<*foo> and looking at the scalar |
785 | component, the optimizer fiddles the op tree to use a function which |
786 | directly looks up the scalar in question. The main optimizer is C<peep> |
787 | in F<op.c>, and many ops have their own optimizing functions. |
788 | |
789 | =item Running |
790 | |
791 | Now we're finally ready to go: we have compiled Perl byte code, and all |
792 | that's left to do is run it. The actual execution is done by the |
793 | C<runops_standard> function in F<run.c>; more specifically, it's done by |
794 | these three innocent looking lines: |
795 | |
796 | while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) { |
797 | PERL_ASYNC_CHECK(); |
798 | } |
799 | |
800 | You may be more comfortable with the Perl version of that: |
801 | |
802 | PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}}; |
803 | |
804 | Well, maybe not. Anyway, each op contains a function pointer, which |
805 | stipulates the function which will actually carry out the operation. |
806 | This function will return the next op in the sequence - this allows for |
807 | things like C<if> which choose the next op dynamically at run time. |
808 | The C<PERL_ASYNC_CHECK> makes sure that things like signals interrupt |
809 | execution if required. |
810 | |
811 | The actual functions called are known as PP code, and they're spread |
812 | between four files: F<pp_hot.c> contains the `hot' code, which is most |
813 | often used and highly optimized, F<pp_sys.c> contains all the |
814 | system-specific functions, F<pp_ctl.c> contains the functions which |
815 | implement control structures (C<if>, C<while> and the like) and F<pp.c> |
816 | contains everything else. These are, if you like, the C code for Perl's |
817 | built-in functions and operators. |
818 | |
819 | =back |
820 | |
821 | =head2 Internal Variable Types |
822 | |
823 | You should by now have had a look at L<perlguts>, which tells you about |
824 | Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do |
825 | that now. |
826 | |
827 | These variables are used not only to represent Perl-space variables, but |
828 | also any constants in the code, as well as some structures completely |
829 | internal to Perl. The symbol table, for instance, is an ordinary Perl |
830 | hash. Your code is represented by an SV as it's read into the parser; |
831 | any program files you call are opened via ordinary Perl filehandles, and |
832 | so on. |
833 | |
834 | The core L<Devel::Peek|Devel::Peek> module lets us examine SVs from a |
835 | Perl program. Let's see, for instance, how Perl treats the constant |
836 | C<"hello">. |
837 | |
838 | % perl -MDevel::Peek -e 'Dump("hello")' |
839 | 1 SV = PV(0xa041450) at 0xa04ecbc |
840 | 2 REFCNT = 1 |
841 | 3 FLAGS = (POK,READONLY,pPOK) |
842 | 4 PV = 0xa0484e0 "hello"\0 |
843 | 5 CUR = 5 |
844 | 6 LEN = 6 |
845 | |
846 | Reading C<Devel::Peek> output takes a bit of practise, so let's go |
847 | through it line by line. |
848 | |
849 | Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in |
850 | memory. SVs themselves are very simple structures, but they contain a |
851 | pointer to a more complex structure. In this case, it's a PV, a |
852 | structure which holds a string value, at location C<0xa041450>. Line 2 |
853 | is the reference count; there are no other references to this data, so |
854 | it's 1. |
855 | |
856 | Line 3 are the flags for this SV - it's OK to use it as a PV, it's a |
857 | read-only SV (because it's a constant) and the data is a PV internally. |
858 | Next we've got the contents of the string, starting at location |
859 | C<0xa0484e0>. |
860 | |
861 | Line 5 gives us the current length of the string - note that this does |
862 | B<not> include the null terminator. Line 6 is not the length of the |
863 | string, but the length of the currently allocated buffer; as the string |
864 | grows, Perl automatically extends the available storage via a routine |
865 | called C<SvGROW>. |
866 | |
867 | You can get at any of these quantities from C very easily; just add |
868 | C<Sv> to the name of the field shown in the snippet, and you've got a |
869 | macro which will return the value: C<SvCUR(sv)> returns the current |
870 | length of the string, C<SvREFCOUNT(sv)> returns the reference count, |
871 | C<SvPV(sv, len)> returns the string itself with its length, and so on. |
872 | More macros to manipulate these properties can be found in L<perlguts>. |
873 | |
874 | Let's take an example of manipulating a PV, from C<sv_catpvn>, in F<sv.c> |
875 | |
876 | 1 void |
877 | 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len) |
878 | 3 { |
879 | 4 STRLEN tlen; |
880 | 5 char *junk; |
881 | |
882 | 6 junk = SvPV_force(sv, tlen); |
883 | 7 SvGROW(sv, tlen + len + 1); |
884 | 8 if (ptr == junk) |
885 | 9 ptr = SvPVX(sv); |
886 | 10 Move(ptr,SvPVX(sv)+tlen,len,char); |
887 | 11 SvCUR(sv) += len; |
888 | 12 *SvEND(sv) = '\0'; |
889 | 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */ |
890 | 14 SvTAINT(sv); |
891 | 15 } |
892 | |
893 | This is a function which adds a string, C<ptr>, of length C<len> onto |
894 | the end of the PV stored in C<sv>. The first thing we do in line 6 is |
895 | make sure that the SV B<has> a valid PV, by calling the C<SvPV_force> |
896 | macro to force a PV. As a side effect, C<tlen> gets set to the current |
897 | value of the PV, and the PV itself is returned to C<junk>. |
898 | |
b1866b2d |
899 | In line 7, we make sure that the SV will have enough room to accommodate |
a422fd2d |
900 | the old string, the new string and the null terminator. If C<LEN> isn't |
901 | big enough, C<SvGROW> will reallocate space for us. |
902 | |
903 | Now, if C<junk> is the same as the string we're trying to add, we can |
904 | grab the string directly from the SV; C<SvPVX> is the address of the PV |
905 | in the SV. |
906 | |
907 | Line 10 does the actual catenation: the C<Move> macro moves a chunk of |
908 | memory around: we move the string C<ptr> to the end of the PV - that's |
909 | the start of the PV plus its current length. We're moving C<len> bytes |
910 | of type C<char>. After doing so, we need to tell Perl we've extended the |
911 | string, by altering C<CUR> to reflect the new length. C<SvEND> is a |
912 | macro which gives us the end of the string, so that needs to be a |
913 | C<"\0">. |
914 | |
915 | Line 13 manipulates the flags; since we've changed the PV, any IV or NV |
916 | values will no longer be valid: if we have C<$a=10; $a.="6";> we don't |
917 | want to use the old IV of 10. C<SvPOK_only_utf8> is a special UTF8-aware |
918 | version of C<SvPOK_only>, a macro which turns off the IOK and NOK flags |
919 | and turns on POK. The final C<SvTAINT> is a macro which launders tainted |
920 | data if taint mode is turned on. |
921 | |
922 | AVs and HVs are more complicated, but SVs are by far the most common |
923 | variable type being thrown around. Having seen something of how we |
924 | manipulate these, let's go on and look at how the op tree is |
925 | constructed. |
926 | |
927 | =head2 Op Trees |
928 | |
929 | First, what is the op tree, anyway? The op tree is the parsed |
930 | representation of your program, as we saw in our section on parsing, and |
931 | it's the sequence of operations that Perl goes through to execute your |
932 | program, as we saw in L</Running>. |
933 | |
934 | An op is a fundamental operation that Perl can perform: all the built-in |
935 | functions and operators are ops, and there are a series of ops which |
936 | deal with concepts the interpreter needs internally - entering and |
937 | leaving a block, ending a statement, fetching a variable, and so on. |
938 | |
939 | The op tree is connected in two ways: you can imagine that there are two |
940 | "routes" through it, two orders in which you can traverse the tree. |
941 | First, parse order reflects how the parser understood the code, and |
942 | secondly, execution order tells perl what order to perform the |
943 | operations in. |
944 | |
945 | The easiest way to examine the op tree is to stop Perl after it has |
946 | finished parsing, and get it to dump out the tree. This is exactly what |
947 | the compiler backends L<B::Terse|B::Terse> and L<B::Debug|B::Debug> do. |
948 | |
949 | Let's have a look at how Perl sees C<$a = $b + $c>: |
950 | |
951 | % perl -MO=Terse -e '$a=$b+$c' |
952 | 1 LISTOP (0x8179888) leave |
953 | 2 OP (0x81798b0) enter |
954 | 3 COP (0x8179850) nextstate |
955 | 4 BINOP (0x8179828) sassign |
956 | 5 BINOP (0x8179800) add [1] |
957 | 6 UNOP (0x81796e0) null [15] |
958 | 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b |
959 | 8 UNOP (0x81797e0) null [15] |
960 | 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c |
961 | 10 UNOP (0x816b4f0) null [15] |
962 | 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a |
963 | |
964 | Let's start in the middle, at line 4. This is a BINOP, a binary |
965 | operator, which is at location C<0x8179828>. The specific operator in |
966 | question is C<sassign> - scalar assignment - and you can find the code |
967 | which implements it in the function C<pp_sassign> in F<pp_hot.c>. As a |
968 | binary operator, it has two children: the add operator, providing the |
969 | result of C<$b+$c>, is uppermost on line 5, and the left hand side is on |
970 | line 10. |
971 | |
972 | Line 10 is the null op: this does exactly nothing. What is that doing |
973 | there? If you see the null op, it's a sign that something has been |
974 | optimized away after parsing. As we mentioned in L</Optimization>, |
975 | the optimization stage sometimes converts two operations into one, for |
976 | example when fetching a scalar variable. When this happens, instead of |
977 | rewriting the op tree and cleaning up the dangling pointers, it's easier |
978 | just to replace the redundant operation with the null op. Originally, |
979 | the tree would have looked like this: |
980 | |
981 | 10 SVOP (0x816b4f0) rv2sv [15] |
982 | 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a |
983 | |
984 | That is, fetch the C<a> entry from the main symbol table, and then look |
985 | at the scalar component of it: C<gvsv> (C<pp_gvsv> into F<pp_hot.c>) |
986 | happens to do both these things. |
987 | |
988 | The right hand side, starting at line 5 is similar to what we've just |
989 | seen: we have the C<add> op (C<pp_add> also in F<pp_hot.c>) add together |
990 | two C<gvsv>s. |
991 | |
992 | Now, what's this about? |
993 | |
994 | 1 LISTOP (0x8179888) leave |
995 | 2 OP (0x81798b0) enter |
996 | 3 COP (0x8179850) nextstate |
997 | |
998 | C<enter> and C<leave> are scoping ops, and their job is to perform any |
999 | housekeeping every time you enter and leave a block: lexical variables |
1000 | are tidied up, unreferenced variables are destroyed, and so on. Every |
1001 | program will have those first three lines: C<leave> is a list, and its |
1002 | children are all the statements in the block. Statements are delimited |
1003 | by C<nextstate>, so a block is a collection of C<nextstate> ops, with |
1004 | the ops to be performed for each statement being the children of |
1005 | C<nextstate>. C<enter> is a single op which functions as a marker. |
1006 | |
1007 | That's how Perl parsed the program, from top to bottom: |
1008 | |
1009 | Program |
1010 | | |
1011 | Statement |
1012 | | |
1013 | = |
1014 | / \ |
1015 | / \ |
1016 | $a + |
1017 | / \ |
1018 | $b $c |
1019 | |
1020 | However, it's impossible to B<perform> the operations in this order: |
1021 | you have to find the values of C<$b> and C<$c> before you add them |
1022 | together, for instance. So, the other thread that runs through the op |
1023 | tree is the execution order: each op has a field C<op_next> which points |
1024 | to the next op to be run, so following these pointers tells us how perl |
1025 | executes the code. We can traverse the tree in this order using |
1026 | the C<exec> option to C<B::Terse>: |
1027 | |
1028 | % perl -MO=Terse,exec -e '$a=$b+$c' |
1029 | 1 OP (0x8179928) enter |
1030 | 2 COP (0x81798c8) nextstate |
1031 | 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b |
1032 | 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c |
1033 | 5 BINOP (0x8179878) add [1] |
1034 | 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a |
1035 | 7 BINOP (0x81798a0) sassign |
1036 | 8 LISTOP (0x8179900) leave |
1037 | |
1038 | This probably makes more sense for a human: enter a block, start a |
1039 | statement. Get the values of C<$b> and C<$c>, and add them together. |
1040 | Find C<$a>, and assign one to the other. Then leave. |
1041 | |
1042 | The way Perl builds up these op trees in the parsing process can be |
1043 | unravelled by examining F<perly.y>, the YACC grammar. Let's take the |
1044 | piece we need to construct the tree for C<$a = $b + $c> |
1045 | |
1046 | 1 term : term ASSIGNOP term |
1047 | 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); } |
1048 | 3 | term ADDOP term |
1049 | 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } |
1050 | |
1051 | If you're not used to reading BNF grammars, this is how it works: You're |
1052 | fed certain things by the tokeniser, which generally end up in upper |
1053 | case. Here, C<ADDOP>, is provided when the tokeniser sees C<+> in your |
1054 | code. C<ASSIGNOP> is provided when C<=> is used for assigning. These are |
1055 | `terminal symbols', because you can't get any simpler than them. |
1056 | |
1057 | The grammar, lines one and three of the snippet above, tells you how to |
1058 | build up more complex forms. These complex forms, `non-terminal symbols' |
1059 | are generally placed in lower case. C<term> here is a non-terminal |
1060 | symbol, representing a single expression. |
1061 | |
1062 | The grammar gives you the following rule: you can make the thing on the |
1063 | left of the colon if you see all the things on the right in sequence. |
1064 | This is called a "reduction", and the aim of parsing is to completely |
1065 | reduce the input. There are several different ways you can perform a |
1066 | reduction, separated by vertical bars: so, C<term> followed by C<=> |
1067 | followed by C<term> makes a C<term>, and C<term> followed by C<+> |
1068 | followed by C<term> can also make a C<term>. |
1069 | |
1070 | So, if you see two terms with an C<=> or C<+>, between them, you can |
1071 | turn them into a single expression. When you do this, you execute the |
1072 | code in the block on the next line: if you see C<=>, you'll do the code |
1073 | in line 2. If you see C<+>, you'll do the code in line 4. It's this code |
1074 | which contributes to the op tree. |
1075 | |
1076 | | term ADDOP term |
1077 | { $$ = newBINOP($2, 0, scalar($1), scalar($3)); } |
1078 | |
1079 | What this does is creates a new binary op, and feeds it a number of |
1080 | variables. The variables refer to the tokens: C<$1> is the first token in |
1081 | the input, C<$2> the second, and so on - think regular expression |
1082 | backreferences. C<$$> is the op returned from this reduction. So, we |
1083 | call C<newBINOP> to create a new binary operator. The first parameter to |
1084 | C<newBINOP>, a function in F<op.c>, is the op type. It's an addition |
1085 | operator, so we want the type to be C<ADDOP>. We could specify this |
1086 | directly, but it's right there as the second token in the input, so we |
1087 | use C<$2>. The second parameter is the op's flags: 0 means `nothing |
1088 | special'. Then the things to add: the left and right hand side of our |
1089 | expression, in scalar context. |
1090 | |
1091 | =head2 Stacks |
1092 | |
1093 | When perl executes something like C<addop>, how does it pass on its |
1094 | results to the next op? The answer is, through the use of stacks. Perl |
1095 | has a number of stacks to store things it's currently working on, and |
1096 | we'll look at the three most important ones here. |
1097 | |
1098 | =over 3 |
1099 | |
1100 | =item Argument stack |
1101 | |
1102 | Arguments are passed to PP code and returned from PP code using the |
1103 | argument stack, C<ST>. The typical way to handle arguments is to pop |
1104 | them off the stack, deal with them how you wish, and then push the result |
1105 | back onto the stack. This is how, for instance, the cosine operator |
1106 | works: |
1107 | |
1108 | NV value; |
1109 | value = POPn; |
1110 | value = Perl_cos(value); |
1111 | XPUSHn(value); |
1112 | |
1113 | We'll see a more tricky example of this when we consider Perl's macros |
1114 | below. C<POPn> gives you the NV (floating point value) of the top SV on |
1115 | the stack: the C<$x> in C<cos($x)>. Then we compute the cosine, and push |
1116 | the result back as an NV. The C<X> in C<XPUSHn> means that the stack |
1117 | should be extended if necessary - it can't be necessary here, because we |
1118 | know there's room for one more item on the stack, since we've just |
1119 | removed one! The C<XPUSH*> macros at least guarantee safety. |
1120 | |
1121 | Alternatively, you can fiddle with the stack directly: C<SP> gives you |
1122 | the first element in your portion of the stack, and C<TOP*> gives you |
1123 | the top SV/IV/NV/etc. on the stack. So, for instance, to do unary |
1124 | negation of an integer: |
1125 | |
1126 | SETi(-TOPi); |
1127 | |
1128 | Just set the integer value of the top stack entry to its negation. |
1129 | |
1130 | Argument stack manipulation in the core is exactly the same as it is in |
1131 | XSUBs - see L<perlxstut>, L<perlxs> and L<perlguts> for a longer |
1132 | description of the macros used in stack manipulation. |
1133 | |
1134 | =item Mark stack |
1135 | |
1136 | I say `your portion of the stack' above because PP code doesn't |
1137 | necessarily get the whole stack to itself: if your function calls |
1138 | another function, you'll only want to expose the arguments aimed for the |
1139 | called function, and not (necessarily) let it get at your own data. The |
1140 | way we do this is to have a `virtual' bottom-of-stack, exposed to each |
1141 | function. The mark stack keeps bookmarks to locations in the argument |
1142 | stack usable by each function. For instance, when dealing with a tied |
1143 | variable, (internally, something with `P' magic) Perl has to call |
1144 | methods for accesses to the tied variables. However, we need to separate |
1145 | the arguments exposed to the method to the argument exposed to the |
1146 | original function - the store or fetch or whatever it may be. Here's how |
1147 | the tied C<push> is implemented; see C<av_push> in F<av.c>: |
1148 | |
1149 | 1 PUSHMARK(SP); |
1150 | 2 EXTEND(SP,2); |
1151 | 3 PUSHs(SvTIED_obj((SV*)av, mg)); |
1152 | 4 PUSHs(val); |
1153 | 5 PUTBACK; |
1154 | 6 ENTER; |
1155 | 7 call_method("PUSH", G_SCALAR|G_DISCARD); |
1156 | 8 LEAVE; |
1157 | 9 POPSTACK; |
13a2d996 |
1158 | |
a422fd2d |
1159 | The lines which concern the mark stack are the first, fifth and last |
1160 | lines: they save away, restore and remove the current position of the |
1161 | argument stack. |
1162 | |
1163 | Let's examine the whole implementation, for practice: |
1164 | |
1165 | 1 PUSHMARK(SP); |
1166 | |
1167 | Push the current state of the stack pointer onto the mark stack. This is |
1168 | so that when we've finished adding items to the argument stack, Perl |
1169 | knows how many things we've added recently. |
1170 | |
1171 | 2 EXTEND(SP,2); |
1172 | 3 PUSHs(SvTIED_obj((SV*)av, mg)); |
1173 | 4 PUSHs(val); |
1174 | |
1175 | We're going to add two more items onto the argument stack: when you have |
1176 | a tied array, the C<PUSH> subroutine receives the object and the value |
1177 | to be pushed, and that's exactly what we have here - the tied object, |
1178 | retrieved with C<SvTIED_obj>, and the value, the SV C<val>. |
1179 | |
1180 | 5 PUTBACK; |
1181 | |
1182 | Next we tell Perl to make the change to the global stack pointer: C<dSP> |
1183 | only gave us a local copy, not a reference to the global. |
1184 | |
1185 | 6 ENTER; |
1186 | 7 call_method("PUSH", G_SCALAR|G_DISCARD); |
1187 | 8 LEAVE; |
1188 | |
1189 | C<ENTER> and C<LEAVE> localise a block of code - they make sure that all |
1190 | variables are tidied up, everything that has been localised gets |
1191 | its previous value returned, and so on. Think of them as the C<{> and |
1192 | C<}> of a Perl block. |
1193 | |
1194 | To actually do the magic method call, we have to call a subroutine in |
1195 | Perl space: C<call_method> takes care of that, and it's described in |
1196 | L<perlcall>. We call the C<PUSH> method in scalar context, and we're |
1197 | going to discard its return value. |
1198 | |
1199 | 9 POPSTACK; |
1200 | |
1201 | Finally, we remove the value we placed on the mark stack, since we |
1202 | don't need it any more. |
1203 | |
1204 | =item Save stack |
1205 | |
1206 | C doesn't have a concept of local scope, so perl provides one. We've |
1207 | seen that C<ENTER> and C<LEAVE> are used as scoping braces; the save |
1208 | stack implements the C equivalent of, for example: |
1209 | |
1210 | { |
1211 | local $foo = 42; |
1212 | ... |
1213 | } |
1214 | |
1215 | See L<perlguts/Localising Changes> for how to use the save stack. |
1216 | |
1217 | =back |
1218 | |
1219 | =head2 Millions of Macros |
1220 | |
1221 | One thing you'll notice about the Perl source is that it's full of |
1222 | macros. Some have called the pervasive use of macros the hardest thing |
1223 | to understand, others find it adds to clarity. Let's take an example, |
1224 | the code which implements the addition operator: |
1225 | |
1226 | 1 PP(pp_add) |
1227 | 2 { |
39644a26 |
1228 | 3 dSP; dATARGET; tryAMAGICbin(add,opASSIGN); |
a422fd2d |
1229 | 4 { |
1230 | 5 dPOPTOPnnrl_ul; |
1231 | 6 SETn( left + right ); |
1232 | 7 RETURN; |
1233 | 8 } |
1234 | 9 } |
1235 | |
1236 | Every line here (apart from the braces, of course) contains a macro. The |
1237 | first line sets up the function declaration as Perl expects for PP code; |
1238 | line 3 sets up variable declarations for the argument stack and the |
1239 | target, the return value of the operation. Finally, it tries to see if |
1240 | the addition operation is overloaded; if so, the appropriate subroutine |
1241 | is called. |
1242 | |
1243 | Line 5 is another variable declaration - all variable declarations start |
1244 | with C<d> - which pops from the top of the argument stack two NVs (hence |
1245 | C<nn>) and puts them into the variables C<right> and C<left>, hence the |
1246 | C<rl>. These are the two operands to the addition operator. Next, we |
1247 | call C<SETn> to set the NV of the return value to the result of adding |
1248 | the two values. This done, we return - the C<RETURN> macro makes sure |
1249 | that our return value is properly handled, and we pass the next operator |
1250 | to run back to the main run loop. |
1251 | |
1252 | Most of these macros are explained in L<perlapi>, and some of the more |
1253 | important ones are explained in L<perlxs> as well. Pay special attention |
1254 | to L<perlguts/Background and PERL_IMPLICIT_CONTEXT> for information on |
1255 | the C<[pad]THX_?> macros. |
1256 | |
a422fd2d |
1257 | =head2 Poking at Perl |
1258 | |
1259 | To really poke around with Perl, you'll probably want to build Perl for |
1260 | debugging, like this: |
1261 | |
1262 | ./Configure -d -D optimize=-g |
1263 | make |
1264 | |
1265 | C<-g> is a flag to the C compiler to have it produce debugging |
1266 | information which will allow us to step through a running program. |
1267 | F<Configure> will also turn on the C<DEBUGGING> compilation symbol which |
1268 | enables all the internal debugging code in Perl. There are a whole bunch |
1269 | of things you can debug with this: L<perlrun> lists them all, and the |
1270 | best way to find out about them is to play about with them. The most |
1271 | useful options are probably |
1272 | |
1273 | l Context (loop) stack processing |
1274 | t Trace execution |
1275 | o Method and overloading resolution |
1276 | c String/numeric conversions |
1277 | |
1278 | Some of the functionality of the debugging code can be achieved using XS |
1279 | modules. |
13a2d996 |
1280 | |
a422fd2d |
1281 | -Dr => use re 'debug' |
1282 | -Dx => use O 'Debug' |
1283 | |
1284 | =head2 Using a source-level debugger |
1285 | |
1286 | If the debugging output of C<-D> doesn't help you, it's time to step |
1287 | through perl's execution with a source-level debugger. |
1288 | |
1289 | =over 3 |
1290 | |
1291 | =item * |
1292 | |
1293 | We'll use C<gdb> for our examples here; the principles will apply to any |
1294 | debugger, but check the manual of the one you're using. |
1295 | |
1296 | =back |
1297 | |
1298 | To fire up the debugger, type |
1299 | |
1300 | gdb ./perl |
1301 | |
1302 | You'll want to do that in your Perl source tree so the debugger can read |
1303 | the source code. You should see the copyright message, followed by the |
1304 | prompt. |
1305 | |
1306 | (gdb) |
1307 | |
1308 | C<help> will get you into the documentation, but here are the most |
1309 | useful commands: |
1310 | |
1311 | =over 3 |
1312 | |
1313 | =item run [args] |
1314 | |
1315 | Run the program with the given arguments. |
1316 | |
1317 | =item break function_name |
1318 | |
1319 | =item break source.c:xxx |
1320 | |
1321 | Tells the debugger that we'll want to pause execution when we reach |
cea6626f |
1322 | either the named function (but see L<perlguts/Internal Functions>!) or the given |
a422fd2d |
1323 | line in the named source file. |
1324 | |
1325 | =item step |
1326 | |
1327 | Steps through the program a line at a time. |
1328 | |
1329 | =item next |
1330 | |
1331 | Steps through the program a line at a time, without descending into |
1332 | functions. |
1333 | |
1334 | =item continue |
1335 | |
1336 | Run until the next breakpoint. |
1337 | |
1338 | =item finish |
1339 | |
1340 | Run until the end of the current function, then stop again. |
1341 | |
13a2d996 |
1342 | =item 'enter' |
a422fd2d |
1343 | |
1344 | Just pressing Enter will do the most recent operation again - it's a |
1345 | blessing when stepping through miles of source code. |
1346 | |
1347 | =item print |
1348 | |
1349 | Execute the given C code and print its results. B<WARNING>: Perl makes |
1350 | heavy use of macros, and F<gdb> is not aware of macros. You'll have to |
1351 | substitute them yourself. So, for instance, you can't say |
1352 | |
1353 | print SvPV_nolen(sv) |
1354 | |
1355 | but you have to say |
1356 | |
1357 | print Perl_sv_2pv_nolen(sv) |
1358 | |
1359 | You may find it helpful to have a "macro dictionary", which you can |
1360 | produce by saying C<cpp -dM perl.c | sort>. Even then, F<cpp> won't |
1361 | recursively apply the macros for you. |
1362 | |
1363 | =back |
1364 | |
1365 | =head2 Dumping Perl Data Structures |
1366 | |
1367 | One way to get around this macro hell is to use the dumping functions in |
1368 | F<dump.c>; these work a little like an internal |
1369 | L<Devel::Peek|Devel::Peek>, but they also cover OPs and other structures |
1370 | that you can't get at from Perl. Let's take an example. We'll use the |
1371 | C<$a = $b + $c> we used before, but give it a bit of context: |
1372 | C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around? |
1373 | |
1374 | What about C<pp_add>, the function we examined earlier to implement the |
1375 | C<+> operator: |
1376 | |
1377 | (gdb) break Perl_pp_add |
1378 | Breakpoint 1 at 0x46249f: file pp_hot.c, line 309. |
1379 | |
cea6626f |
1380 | Notice we use C<Perl_pp_add> and not C<pp_add> - see L<perlguts/Internal Functions>. |
a422fd2d |
1381 | With the breakpoint in place, we can run our program: |
1382 | |
1383 | (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c' |
1384 | |
1385 | Lots of junk will go past as gdb reads in the relevant source files and |
1386 | libraries, and then: |
1387 | |
1388 | Breakpoint 1, Perl_pp_add () at pp_hot.c:309 |
39644a26 |
1389 | 309 dSP; dATARGET; tryAMAGICbin(add,opASSIGN); |
a422fd2d |
1390 | (gdb) step |
1391 | 311 dPOPTOPnnrl_ul; |
1392 | (gdb) |
1393 | |
1394 | We looked at this bit of code before, and we said that C<dPOPTOPnnrl_ul> |
1395 | arranges for two C<NV>s to be placed into C<left> and C<right> - let's |
1396 | slightly expand it: |
1397 | |
1398 | #define dPOPTOPnnrl_ul NV right = POPn; \ |
1399 | SV *leftsv = TOPs; \ |
1400 | NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0 |
1401 | |
1402 | C<POPn> takes the SV from the top of the stack and obtains its NV either |
1403 | directly (if C<SvNOK> is set) or by calling the C<sv_2nv> function. |
1404 | C<TOPs> takes the next SV from the top of the stack - yes, C<POPn> uses |
1405 | C<TOPs> - but doesn't remove it. We then use C<SvNV> to get the NV from |
1406 | C<leftsv> in the same way as before - yes, C<POPn> uses C<SvNV>. |
1407 | |
1408 | Since we don't have an NV for C<$b>, we'll have to use C<sv_2nv> to |
1409 | convert it. If we step again, we'll find ourselves there: |
1410 | |
1411 | Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669 |
1412 | 1669 if (!sv) |
1413 | (gdb) |
1414 | |
1415 | We can now use C<Perl_sv_dump> to investigate the SV: |
1416 | |
1417 | SV = PV(0xa057cc0) at 0xa0675d0 |
1418 | REFCNT = 1 |
1419 | FLAGS = (POK,pPOK) |
1420 | PV = 0xa06a510 "6XXXX"\0 |
1421 | CUR = 5 |
1422 | LEN = 6 |
1423 | $1 = void |
1424 | |
1425 | We know we're going to get C<6> from this, so let's finish the |
1426 | subroutine: |
1427 | |
1428 | (gdb) finish |
1429 | Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671 |
1430 | 0x462669 in Perl_pp_add () at pp_hot.c:311 |
1431 | 311 dPOPTOPnnrl_ul; |
1432 | |
1433 | We can also dump out this op: the current op is always stored in |
1434 | C<PL_op>, and we can dump it with C<Perl_op_dump>. This'll give us |
1435 | similar output to L<B::Debug|B::Debug>. |
1436 | |
1437 | { |
1438 | 13 TYPE = add ===> 14 |
1439 | TARG = 1 |
1440 | FLAGS = (SCALAR,KIDS) |
1441 | { |
1442 | TYPE = null ===> (12) |
1443 | (was rv2sv) |
1444 | FLAGS = (SCALAR,KIDS) |
1445 | { |
1446 | 11 TYPE = gvsv ===> 12 |
1447 | FLAGS = (SCALAR) |
1448 | GV = main::b |
1449 | } |
1450 | } |
1451 | |
10f58044 |
1452 | # finish this later # |
a422fd2d |
1453 | |
1454 | =head2 Patching |
1455 | |
1456 | All right, we've now had a look at how to navigate the Perl sources and |
1457 | some things you'll need to know when fiddling with them. Let's now get |
1458 | on and create a simple patch. Here's something Larry suggested: if a |
1459 | C<U> is the first active format during a C<pack>, (for example, |
1460 | C<pack "U3C8", @stuff>) then the resulting string should be treated as |
1461 | UTF8 encoded. |
1462 | |
1463 | How do we prepare to fix this up? First we locate the code in question - |
1464 | the C<pack> happens at runtime, so it's going to be in one of the F<pp> |
1465 | files. Sure enough, C<pp_pack> is in F<pp.c>. Since we're going to be |
1466 | altering this file, let's copy it to F<pp.c~>. |
1467 | |
a6ec74c1 |
1468 | [Well, it was in F<pp.c> when this tutorial was written. It has now been |
1469 | split off with C<pp_unpack> to its own file, F<pp_pack.c>] |
1470 | |
a422fd2d |
1471 | Now let's look over C<pp_pack>: we take a pattern into C<pat>, and then |
1472 | loop over the pattern, taking each format character in turn into |
1473 | C<datum_type>. Then for each possible format character, we swallow up |
1474 | the other arguments in the pattern (a field width, an asterisk, and so |
1475 | on) and convert the next chunk input into the specified format, adding |
1476 | it onto the output SV C<cat>. |
1477 | |
1478 | How do we know if the C<U> is the first format in the C<pat>? Well, if |
1479 | we have a pointer to the start of C<pat> then, if we see a C<U> we can |
1480 | test whether we're still at the start of the string. So, here's where |
1481 | C<pat> is set up: |
1482 | |
1483 | STRLEN fromlen; |
1484 | register char *pat = SvPVx(*++MARK, fromlen); |
1485 | register char *patend = pat + fromlen; |
1486 | register I32 len; |
1487 | I32 datumtype; |
1488 | SV *fromstr; |
1489 | |
1490 | We'll have another string pointer in there: |
1491 | |
1492 | STRLEN fromlen; |
1493 | register char *pat = SvPVx(*++MARK, fromlen); |
1494 | register char *patend = pat + fromlen; |
1495 | + char *patcopy; |
1496 | register I32 len; |
1497 | I32 datumtype; |
1498 | SV *fromstr; |
1499 | |
1500 | And just before we start the loop, we'll set C<patcopy> to be the start |
1501 | of C<pat>: |
1502 | |
1503 | items = SP - MARK; |
1504 | MARK++; |
1505 | sv_setpvn(cat, "", 0); |
1506 | + patcopy = pat; |
1507 | while (pat < patend) { |
1508 | |
1509 | Now if we see a C<U> which was at the start of the string, we turn on |
1510 | the UTF8 flag for the output SV, C<cat>: |
1511 | |
1512 | + if (datumtype == 'U' && pat==patcopy+1) |
1513 | + SvUTF8_on(cat); |
1514 | if (datumtype == '#') { |
1515 | while (pat < patend && *pat != '\n') |
1516 | pat++; |
1517 | |
1518 | Remember that it has to be C<patcopy+1> because the first character of |
1519 | the string is the C<U> which has been swallowed into C<datumtype!> |
1520 | |
1521 | Oops, we forgot one thing: what if there are spaces at the start of the |
1522 | pattern? C<pack(" U*", @stuff)> will have C<U> as the first active |
1523 | character, even though it's not the first thing in the pattern. In this |
1524 | case, we have to advance C<patcopy> along with C<pat> when we see spaces: |
1525 | |
1526 | if (isSPACE(datumtype)) |
1527 | continue; |
1528 | |
1529 | needs to become |
1530 | |
1531 | if (isSPACE(datumtype)) { |
1532 | patcopy++; |
1533 | continue; |
1534 | } |
1535 | |
1536 | OK. That's the C part done. Now we must do two additional things before |
1537 | this patch is ready to go: we've changed the behaviour of Perl, and so |
1538 | we must document that change. We must also provide some more regression |
1539 | tests to make sure our patch works and doesn't create a bug somewhere |
1540 | else along the line. |
1541 | |
b23b8711 |
1542 | The regression tests for each operator live in F<t/op/>, and so we |
1543 | make a copy of F<t/op/pack.t> to F<t/op/pack.t~>. Now we can add our |
1544 | tests to the end. First, we'll test that the C<U> does indeed create |
1545 | Unicode strings. |
1546 | |
1547 | t/op/pack.t has a sensible ok() function, but if it didn't we could |
35c336e6 |
1548 | use the one from t/test.pl. |
b23b8711 |
1549 | |
35c336e6 |
1550 | require './test.pl'; |
1551 | plan( tests => 159 ); |
b23b8711 |
1552 | |
1553 | so instead of this: |
a422fd2d |
1554 | |
1555 | print 'not ' unless "1.20.300.4000" eq sprintf "%vd", pack("U*",1,20,300,4000); |
1556 | print "ok $test\n"; $test++; |
1557 | |
35c336e6 |
1558 | we can write the more sensible (see L<Test::More> for a full |
1559 | explanation of is() and other testing functions). |
b23b8711 |
1560 | |
35c336e6 |
1561 | is( "1.20.300.4000", sprintf "%vd", pack("U*",1,20,300,4000), |
812f5127 |
1562 | "U* produces unicode" ); |
b23b8711 |
1563 | |
a422fd2d |
1564 | Now we'll test that we got that space-at-the-beginning business right: |
1565 | |
35c336e6 |
1566 | is( "1.20.300.4000", sprintf "%vd", pack(" U*",1,20,300,4000), |
812f5127 |
1567 | " with spaces at the beginning" ); |
a422fd2d |
1568 | |
1569 | And finally we'll test that we don't make Unicode strings if C<U> is B<not> |
1570 | the first active format: |
1571 | |
35c336e6 |
1572 | isnt( v1.20.300.4000, sprintf "%vd", pack("C0U*",1,20,300,4000), |
812f5127 |
1573 | "U* not first isn't unicode" ); |
a422fd2d |
1574 | |
35c336e6 |
1575 | Mustn't forget to change the number of tests which appears at the top, |
1576 | or else the automated tester will get confused. This will either look |
1577 | like this: |
a422fd2d |
1578 | |
35c336e6 |
1579 | print "1..156\n"; |
1580 | |
1581 | or this: |
1582 | |
1583 | plan( tests => 156 ); |
a422fd2d |
1584 | |
1585 | We now compile up Perl, and run it through the test suite. Our new |
1586 | tests pass, hooray! |
1587 | |
1588 | Finally, the documentation. The job is never done until the paperwork is |
1589 | over, so let's describe the change we've just made. The relevant place |
1590 | is F<pod/perlfunc.pod>; again, we make a copy, and then we'll insert |
1591 | this text in the description of C<pack>: |
1592 | |
1593 | =item * |
1594 | |
1595 | If the pattern begins with a C<U>, the resulting string will be treated |
1596 | as Unicode-encoded. You can force UTF8 encoding on in a string with an |
1597 | initial C<U0>, and the bytes that follow will be interpreted as Unicode |
1598 | characters. If you don't want this to happen, you can begin your pattern |
1599 | with C<C0> (or anything else) to force Perl not to UTF8 encode your |
1600 | string, and then follow this with a C<U*> somewhere in your pattern. |
1601 | |
1602 | All done. Now let's create the patch. F<Porting/patching.pod> tells us |
1603 | that if we're making major changes, we should copy the entire directory |
1604 | to somewhere safe before we begin fiddling, and then do |
13a2d996 |
1605 | |
a422fd2d |
1606 | diff -ruN old new > patch |
1607 | |
1608 | However, we know which files we've changed, and we can simply do this: |
1609 | |
1610 | diff -u pp.c~ pp.c > patch |
1611 | diff -u t/op/pack.t~ t/op/pack.t >> patch |
1612 | diff -u pod/perlfunc.pod~ pod/perlfunc.pod >> patch |
1613 | |
1614 | We end up with a patch looking a little like this: |
1615 | |
1616 | --- pp.c~ Fri Jun 02 04:34:10 2000 |
1617 | +++ pp.c Fri Jun 16 11:37:25 2000 |
1618 | @@ -4375,6 +4375,7 @@ |
1619 | register I32 items; |
1620 | STRLEN fromlen; |
1621 | register char *pat = SvPVx(*++MARK, fromlen); |
1622 | + char *patcopy; |
1623 | register char *patend = pat + fromlen; |
1624 | register I32 len; |
1625 | I32 datumtype; |
1626 | @@ -4405,6 +4406,7 @@ |
1627 | ... |
1628 | |
1629 | And finally, we submit it, with our rationale, to perl5-porters. Job |
1630 | done! |
1631 | |
f7e1e956 |
1632 | =head2 Patching a core module |
1633 | |
1634 | This works just like patching anything else, with an extra |
1635 | consideration. Many core modules also live on CPAN. If this is so, |
1636 | patch the CPAN version instead of the core and send the patch off to |
1637 | the module maintainer (with a copy to p5p). This will help the module |
1638 | maintainer keep the CPAN version in sync with the core version without |
1639 | constantly scanning p5p. |
1640 | |
acbe17fc |
1641 | =head2 Adding a new function to the core |
1642 | |
1643 | If, as part of a patch to fix a bug, or just because you have an |
1644 | especially good idea, you decide to add a new function to the core, |
1645 | discuss your ideas on p5p well before you start work. It may be that |
1646 | someone else has already attempted to do what you are considering and |
1647 | can give lots of good advice or even provide you with bits of code |
1648 | that they already started (but never finished). |
1649 | |
1650 | You have to follow all of the advice given above for patching. It is |
1651 | extremely important to test any addition thoroughly and add new tests |
1652 | to explore all boundary conditions that your new function is expected |
1653 | to handle. If your new function is used only by one module (e.g. toke), |
1654 | then it should probably be named S_your_function (for static); on the |
210b36aa |
1655 | other hand, if you expect it to accessible from other functions in |
acbe17fc |
1656 | Perl, you should name it Perl_your_function. See L<perlguts/Internal Functions> |
1657 | for more details. |
1658 | |
1659 | The location of any new code is also an important consideration. Don't |
1660 | just create a new top level .c file and put your code there; you would |
1661 | have to make changes to Configure (so the Makefile is created properly), |
1662 | as well as possibly lots of include files. This is strictly pumpking |
1663 | business. |
1664 | |
1665 | It is better to add your function to one of the existing top level |
1666 | source code files, but your choice is complicated by the nature of |
1667 | the Perl distribution. Only the files that are marked as compiled |
1668 | static are located in the perl executable. Everything else is located |
1669 | in the shared library (or DLL if you are running under WIN32). So, |
1670 | for example, if a function was only used by functions located in |
1671 | toke.c, then your code can go in toke.c. If, however, you want to call |
1672 | the function from universal.c, then you should put your code in another |
1673 | location, for example util.c. |
1674 | |
1675 | In addition to writing your c-code, you will need to create an |
1676 | appropriate entry in embed.pl describing your function, then run |
1677 | 'make regen_headers' to create the entries in the numerous header |
1678 | files that perl needs to compile correctly. See L<perlguts/Internal Functions> |
1679 | for information on the various options that you can set in embed.pl. |
1680 | You will forget to do this a few (or many) times and you will get |
1681 | warnings during the compilation phase. Make sure that you mention |
1682 | this when you post your patch to P5P; the pumpking needs to know this. |
1683 | |
1684 | When you write your new code, please be conscious of existing code |
884bad00 |
1685 | conventions used in the perl source files. See L<perlstyle> for |
acbe17fc |
1686 | details. Although most of the guidelines discussed seem to focus on |
1687 | Perl code, rather than c, they all apply (except when they don't ;). |
1688 | See also I<Porting/patching.pod> file in the Perl source distribution |
1689 | for lots of details about both formatting and submitting patches of |
1690 | your changes. |
1691 | |
1692 | Lastly, TEST TEST TEST TEST TEST any code before posting to p5p. |
1693 | Test on as many platforms as you can find. Test as many perl |
1694 | Configure options as you can (e.g. MULTIPLICITY). If you have |
1695 | profiling or memory tools, see L<EXTERNAL TOOLS FOR DEBUGGING PERL> |
210b36aa |
1696 | below for how to use them to further test your code. Remember that |
acbe17fc |
1697 | most of the people on P5P are doing this on their own time and |
1698 | don't have the time to debug your code. |
f7e1e956 |
1699 | |
1700 | =head2 Writing a test |
1701 | |
1702 | Every module and built-in function has an associated test file (or |
1703 | should...). If you add or change functionality, you have to write a |
1704 | test. If you fix a bug, you have to write a test so that bug never |
1705 | comes back. If you alter the docs, it would be nice to test what the |
1706 | new documentation says. |
1707 | |
1708 | In short, if you submit a patch you probably also have to patch the |
1709 | tests. |
1710 | |
1711 | For modules, the test file is right next to the module itself. |
1712 | F<lib/strict.t> tests F<lib/strict.pm>. This is a recent innovation, |
1713 | so there are some snags (and it would be wonderful for you to brush |
1714 | them out), but it basically works that way. Everything else lives in |
1715 | F<t/>. |
1716 | |
1717 | =over 3 |
1718 | |
1719 | =item F<t/base/> |
1720 | |
1721 | Testing of the absolute basic functionality of Perl. Things like |
1722 | C<if>, basic file reads and writes, simple regexes, etc. These are |
1723 | run first in the test suite and if any of them fail, something is |
1724 | I<really> broken. |
1725 | |
1726 | =item F<t/cmd/> |
1727 | |
1728 | These test the basic control structures, C<if/else>, C<while>, |
35c336e6 |
1729 | subroutines, etc. |
f7e1e956 |
1730 | |
1731 | =item F<t/comp/> |
1732 | |
1733 | Tests basic issues of how Perl parses and compiles itself. |
1734 | |
1735 | =item F<t/io/> |
1736 | |
1737 | Tests for built-in IO functions, including command line arguments. |
1738 | |
1739 | =item F<t/lib/> |
1740 | |
1741 | The old home for the module tests, you shouldn't put anything new in |
1742 | here. There are still some bits and pieces hanging around in here |
1743 | that need to be moved. Perhaps you could move them? Thanks! |
1744 | |
1745 | =item F<t/op/> |
1746 | |
1747 | Tests for perl's built in functions that don't fit into any of the |
1748 | other directories. |
1749 | |
1750 | =item F<t/pod/> |
1751 | |
1752 | Tests for POD directives. There are still some tests for the Pod |
1753 | modules hanging around in here that need to be moved out into F<lib/>. |
1754 | |
1755 | =item F<t/run/> |
1756 | |
1757 | Testing features of how perl actually runs, including exit codes and |
1758 | handling of PERL* environment variables. |
1759 | |
1760 | =back |
1761 | |
1762 | The core uses the same testing style as the rest of Perl, a simple |
1763 | "ok/not ok" run through Test::Harness, but there are a few special |
1764 | considerations. |
1765 | |
35c336e6 |
1766 | There are three ways to write a test in the core. Test::More, |
1767 | t/test.pl and ad hoc C<print $test ? "ok 42\n" : "not ok 42\n">. The |
1768 | decision of which to use depends on what part of the test suite you're |
1769 | working on. This is a measure to prevent a high-level failure (such |
1770 | as Config.pm breaking) from causing basic functionality tests to fail. |
1771 | |
1772 | =over 4 |
1773 | |
1774 | =item t/base t/comp |
1775 | |
1776 | Since we don't know if require works, or even subroutines, use ad hoc |
1777 | tests for these two. Step carefully to avoid using the feature being |
1778 | tested. |
1779 | |
1780 | =item t/cmd t/run t/io t/op |
1781 | |
1782 | Now that basic require() and subroutines are tested, you can use the |
1783 | t/test.pl library which emulates the important features of Test::More |
1784 | while using a minimum of core features. |
1785 | |
1786 | You can also conditionally use certain libraries like Config, but be |
1787 | sure to skip the test gracefully if it's not there. |
1788 | |
1789 | =item t/lib ext lib |
1790 | |
1791 | Now that the core of Perl is tested, Test::More can be used. You can |
1792 | also use the full suite of core modules in the tests. |
1793 | |
1794 | =back |
f7e1e956 |
1795 | |
1796 | When you say "make test" Perl uses the F<t/TEST> program to run the |
1797 | test suite. All tests are run from the F<t/> directory, B<not> the |
1798 | directory which contains the test. This causes some problems with the |
1799 | tests in F<lib/>, so here's some opportunity for some patching. |
1800 | |
1801 | You must be triply conscious of cross-platform concerns. This usually |
1802 | boils down to using File::Spec and avoiding things like C<fork()> and |
1803 | C<system()> unless absolutely necessary. |
1804 | |
e018f8be |
1805 | =head2 Special Make Test Targets |
1806 | |
1807 | There are various special make targets that can be used to test Perl |
1808 | slightly differently than the standard "test" target. Not all them |
1809 | are expected to give a 100% success rate. Many of them have several |
1810 | aliases. |
1811 | |
1812 | =over 4 |
1813 | |
1814 | =item coretest |
1815 | |
1816 | Run F<perl> on all but F<lib/*> tests. |
1817 | |
1818 | =item test.deparse |
1819 | |
1820 | Run all the tests through the B::Deparse. Not all tests will succeed. |
1821 | |
1822 | =item minitest |
1823 | |
1824 | Run F<miniperl> on F<t/base>, F<t/comp>, F<t/cmd>, F<t/run>, F<t/io>, |
1825 | F<t/op>, and F<t/uni> tests. |
1826 | |
1827 | =item test.third check.third utest.third ucheck.third |
1828 | |
1829 | (Only in Tru64) Run all the tests using the memory leak + naughty |
1830 | memory access tool "Third Degree". The log files will be named |
1831 | F<perl3.log.testname>. |
1832 | |
1833 | =item test.torture torturetest |
1834 | |
1835 | Run all the usual tests and some extra tests. As of Perl 5.8.0 the |
1836 | only extra tests are Abigail's JAPHs, t/japh/abigail.t. |
1837 | |
1838 | You can also run the torture test with F<t/harness> by giving |
1839 | C<-torture> argument to F<t/harness>. |
1840 | |
1841 | =item utest ucheck test.utf8 check.utf8 |
1842 | |
1843 | Run all the tests with -Mutf8. Not all tests will succeed. |
1844 | |
1845 | =back |
f7e1e956 |
1846 | |
902b9dbf |
1847 | =head1 EXTERNAL TOOLS FOR DEBUGGING PERL |
1848 | |
1849 | Sometimes it helps to use external tools while debugging and |
1850 | testing Perl. This section tries to guide you through using |
1851 | some common testing and debugging tools with Perl. This is |
1852 | meant as a guide to interfacing these tools with Perl, not |
1853 | as any kind of guide to the use of the tools themselves. |
1854 | |
1855 | =head2 Rational Software's Purify |
1856 | |
1857 | Purify is a commercial tool that is helpful in identifying |
1858 | memory overruns, wild pointers, memory leaks and other such |
1859 | badness. Perl must be compiled in a specific way for |
1860 | optimal testing with Purify. Purify is available under |
1861 | Windows NT, Solaris, HP-UX, SGI, and Siemens Unix. |
1862 | |
1863 | The only currently known leaks happen when there are |
1864 | compile-time errors within eval or require. (Fixing these |
1865 | is non-trivial, unfortunately, but they must be fixed |
1866 | eventually.) |
1867 | |
1868 | =head2 Purify on Unix |
1869 | |
1870 | On Unix, Purify creates a new Perl binary. To get the most |
1871 | benefit out of Purify, you should create the perl to Purify |
1872 | using: |
1873 | |
1874 | sh Configure -Accflags=-DPURIFY -Doptimize='-g' \ |
1875 | -Uusemymalloc -Dusemultiplicity |
1876 | |
1877 | where these arguments mean: |
1878 | |
1879 | =over 4 |
1880 | |
1881 | =item -Accflags=-DPURIFY |
1882 | |
1883 | Disables Perl's arena memory allocation functions, as well as |
1884 | forcing use of memory allocation functions derived from the |
1885 | system malloc. |
1886 | |
1887 | =item -Doptimize='-g' |
1888 | |
1889 | Adds debugging information so that you see the exact source |
1890 | statements where the problem occurs. Without this flag, all |
1891 | you will see is the source filename of where the error occurred. |
1892 | |
1893 | =item -Uusemymalloc |
1894 | |
1895 | Disable Perl's malloc so that Purify can more closely monitor |
1896 | allocations and leaks. Using Perl's malloc will make Purify |
1897 | report most leaks in the "potential" leaks category. |
1898 | |
1899 | =item -Dusemultiplicity |
1900 | |
1901 | Enabling the multiplicity option allows perl to clean up |
1902 | thoroughly when the interpreter shuts down, which reduces the |
1903 | number of bogus leak reports from Purify. |
1904 | |
1905 | =back |
1906 | |
1907 | Once you've compiled a perl suitable for Purify'ing, then you |
1908 | can just: |
1909 | |
1910 | make pureperl |
1911 | |
1912 | which creates a binary named 'pureperl' that has been Purify'ed. |
1913 | This binary is used in place of the standard 'perl' binary |
1914 | when you want to debug Perl memory problems. |
1915 | |
1f56d61a |
1916 | To minimize the number of memory leak false alarms |
1917 | (see L</PERL_DESTRUCT_LEVEL>), set environment variable |
1918 | PERL_DESTRUCT_LEVEL to 2. |
1919 | |
1920 | setenv PERL_DESTRUCT_LEVEL 2 |
1921 | |
1922 | In Bourne-type shells: |
1923 | |
1924 | PERL_DESTRUCT_LEVEL=2 |
1925 | export PERL_DESTRUCT_LEVEL |
1926 | |
902b9dbf |
1927 | As an example, to show any memory leaks produced during the |
1928 | standard Perl testset you would create and run the Purify'ed |
1929 | perl as: |
1930 | |
1931 | make pureperl |
1932 | cd t |
1933 | ../pureperl -I../lib harness |
1934 | |
1935 | which would run Perl on test.pl and report any memory problems. |
1936 | |
1937 | Purify outputs messages in "Viewer" windows by default. If |
1938 | you don't have a windowing environment or if you simply |
1939 | want the Purify output to unobtrusively go to a log file |
1940 | instead of to the interactive window, use these following |
1941 | options to output to the log file "perl.log": |
1942 | |
1943 | setenv PURIFYOPTIONS "-chain-length=25 -windows=no \ |
1944 | -log-file=perl.log -append-logfile=yes" |
1945 | |
1946 | If you plan to use the "Viewer" windows, then you only need this option: |
1947 | |
1948 | setenv PURIFYOPTIONS "-chain-length=25" |
1949 | |
c406981e |
1950 | In Bourne-type shells: |
1951 | |
98631ff8 |
1952 | PURIFYOPTIONS="..." |
1953 | export PURIFYOPTIONS |
c406981e |
1954 | |
1955 | or if you have the "env" utility: |
1956 | |
98631ff8 |
1957 | env PURIFYOPTIONS="..." ../pureperl ... |
c406981e |
1958 | |
902b9dbf |
1959 | =head2 Purify on NT |
1960 | |
1961 | Purify on Windows NT instruments the Perl binary 'perl.exe' |
1962 | on the fly. There are several options in the makefile you |
1963 | should change to get the most use out of Purify: |
1964 | |
1965 | =over 4 |
1966 | |
1967 | =item DEFINES |
1968 | |
1969 | You should add -DPURIFY to the DEFINES line so the DEFINES |
1970 | line looks something like: |
1971 | |
1972 | DEFINES = -DWIN32 -D_CONSOLE -DNO_STRICT $(CRYPT_FLAG) -DPURIFY=1 |
1973 | |
1974 | to disable Perl's arena memory allocation functions, as |
1975 | well as to force use of memory allocation functions derived |
1976 | from the system malloc. |
1977 | |
1978 | =item USE_MULTI = define |
1979 | |
1980 | Enabling the multiplicity option allows perl to clean up |
1981 | thoroughly when the interpreter shuts down, which reduces the |
1982 | number of bogus leak reports from Purify. |
1983 | |
1984 | =item #PERL_MALLOC = define |
1985 | |
1986 | Disable Perl's malloc so that Purify can more closely monitor |
1987 | allocations and leaks. Using Perl's malloc will make Purify |
1988 | report most leaks in the "potential" leaks category. |
1989 | |
1990 | =item CFG = Debug |
1991 | |
1992 | Adds debugging information so that you see the exact source |
1993 | statements where the problem occurs. Without this flag, all |
1994 | you will see is the source filename of where the error occurred. |
1995 | |
1996 | =back |
1997 | |
1998 | As an example, to show any memory leaks produced during the |
1999 | standard Perl testset you would create and run Purify as: |
2000 | |
2001 | cd win32 |
2002 | make |
2003 | cd ../t |
2004 | purify ../perl -I../lib harness |
2005 | |
2006 | which would instrument Perl in memory, run Perl on test.pl, |
2007 | then finally report any memory problems. |
2008 | |
f134cc4e |
2009 | B<NOTE>: as of Perl 5.8.0, the ext/Encode/t/Unicode.t takes |
2010 | extraordinarily long (hours?) to complete under Purify. It has been |
2011 | theorized that it would eventually finish, but nobody has so far been |
2012 | patient enough :-) (This same extreme slowdown has been seen also with |
2013 | the Third Degree tool, so the said test must be doing something that |
2014 | is quite unfriendly for memory debuggers.) It is suggested that you |
2015 | simply kill away that testing process. |
2016 | |
2017 | =head2 Compaq's/Digital's/HP's Third Degree |
09187cb1 |
2018 | |
2019 | Third Degree is a tool for memory leak detection and memory access checks. |
2020 | It is one of the many tools in the ATOM toolkit. The toolkit is only |
2021 | available on Tru64 (formerly known as Digital UNIX formerly known as |
2022 | DEC OSF/1). |
2023 | |
2024 | When building Perl, you must first run Configure with -Doptimize=-g |
2025 | and -Uusemymalloc flags, after that you can use the make targets |
51a35ef1 |
2026 | "perl.third" and "test.third". (What is required is that Perl must be |
2027 | compiled using the C<-g> flag, you may need to re-Configure.) |
09187cb1 |
2028 | |
64cea5fd |
2029 | The short story is that with "atom" you can instrument the Perl |
83f0ef60 |
2030 | executable to create a new executable called F<perl.third>. When the |
4ae3d70a |
2031 | instrumented executable is run, it creates a log of dubious memory |
83f0ef60 |
2032 | traffic in file called F<perl.3log>. See the manual pages of atom and |
4ae3d70a |
2033 | third for more information. The most extensive Third Degree |
2034 | documentation is available in the Compaq "Tru64 UNIX Programmer's |
2035 | Guide", chapter "Debugging Programs with Third Degree". |
64cea5fd |
2036 | |
9c54ecba |
2037 | The "test.third" leaves a lot of files named F<foo_bar.3log> in the t/ |
64cea5fd |
2038 | subdirectory. There is a problem with these files: Third Degree is so |
2039 | effective that it finds problems also in the system libraries. |
9c54ecba |
2040 | Therefore you should used the Porting/thirdclean script to cleanup |
2041 | the F<*.3log> files. |
64cea5fd |
2042 | |
2043 | There are also leaks that for given certain definition of a leak, |
2044 | aren't. See L</PERL_DESTRUCT_LEVEL> for more information. |
2045 | |
2046 | =head2 PERL_DESTRUCT_LEVEL |
2047 | |
2048 | If you want to run any of the tests yourself manually using the |
2049 | pureperl or perl.third executables, please note that by default |
2050 | perl B<does not> explicitly cleanup all the memory it has allocated |
2051 | (such as global memory arenas) but instead lets the exit() of |
2052 | the whole program "take care" of such allocations, also known |
2053 | as "global destruction of objects". |
2054 | |
2055 | There is a way to tell perl to do complete cleanup: set the |
2056 | environment variable PERL_DESTRUCT_LEVEL to a non-zero value. |
2057 | The t/TEST wrapper does set this to 2, and this is what you |
2058 | need to do too, if you don't want to see the "global leaks": |
1f56d61a |
2059 | For example, for "third-degreed" Perl: |
64cea5fd |
2060 | |
1f56d61a |
2061 | env PERL_DESTRUCT_LEVEL=2 ./perl.third -Ilib t/foo/bar.t |
09187cb1 |
2062 | |
414f2397 |
2063 | (Note: the mod_perl apache module uses also this environment variable |
2064 | for its own purposes and extended its semantics. Refer to the mod_perl |
2065 | documentation for more information.) |
2066 | |
51a35ef1 |
2067 | =head2 Profiling |
2068 | |
2069 | Depending on your platform there are various of profiling Perl. |
2070 | |
2071 | There are two commonly used techniques of profiling executables: |
10f58044 |
2072 | I<statistical time-sampling> and I<basic-block counting>. |
51a35ef1 |
2073 | |
2074 | The first method takes periodically samples of the CPU program |
2075 | counter, and since the program counter can be correlated with the code |
2076 | generated for functions, we get a statistical view of in which |
2077 | functions the program is spending its time. The caveats are that very |
2078 | small/fast functions have lower probability of showing up in the |
2079 | profile, and that periodically interrupting the program (this is |
2080 | usually done rather frequently, in the scale of milliseconds) imposes |
2081 | an additional overhead that may skew the results. The first problem |
2082 | can be alleviated by running the code for longer (in general this is a |
2083 | good idea for profiling), the second problem is usually kept in guard |
2084 | by the profiling tools themselves. |
2085 | |
10f58044 |
2086 | The second method divides up the generated code into I<basic blocks>. |
51a35ef1 |
2087 | Basic blocks are sections of code that are entered only in the |
2088 | beginning and exited only at the end. For example, a conditional jump |
2089 | starts a basic block. Basic block profiling usually works by |
10f58044 |
2090 | I<instrumenting> the code by adding I<enter basic block #nnnn> |
51a35ef1 |
2091 | book-keeping code to the generated code. During the execution of the |
2092 | code the basic block counters are then updated appropriately. The |
2093 | caveat is that the added extra code can skew the results: again, the |
2094 | profiling tools usually try to factor their own effects out of the |
2095 | results. |
2096 | |
83f0ef60 |
2097 | =head2 Gprof Profiling |
2098 | |
51a35ef1 |
2099 | gprof is a profiling tool available in many UNIX platforms, |
2100 | it uses F<statistical time-sampling>. |
83f0ef60 |
2101 | |
2102 | You can build a profiled version of perl called "perl.gprof" by |
51a35ef1 |
2103 | invoking the make target "perl.gprof" (What is required is that Perl |
2104 | must be compiled using the C<-pg> flag, you may need to re-Configure). |
2105 | Running the profiled version of Perl will create an output file called |
2106 | F<gmon.out> is created which contains the profiling data collected |
2107 | during the execution. |
83f0ef60 |
2108 | |
2109 | The gprof tool can then display the collected data in various ways. |
2110 | Usually gprof understands the following options: |
2111 | |
2112 | =over 4 |
2113 | |
2114 | =item -a |
2115 | |
2116 | Suppress statically defined functions from the profile. |
2117 | |
2118 | =item -b |
2119 | |
2120 | Suppress the verbose descriptions in the profile. |
2121 | |
2122 | =item -e routine |
2123 | |
2124 | Exclude the given routine and its descendants from the profile. |
2125 | |
2126 | =item -f routine |
2127 | |
2128 | Display only the given routine and its descendants in the profile. |
2129 | |
2130 | =item -s |
2131 | |
2132 | Generate a summary file called F<gmon.sum> which then may be given |
2133 | to subsequent gprof runs to accumulate data over several runs. |
2134 | |
2135 | =item -z |
2136 | |
2137 | Display routines that have zero usage. |
2138 | |
2139 | =back |
2140 | |
2141 | For more detailed explanation of the available commands and output |
2142 | formats, see your own local documentation of gprof. |
2143 | |
51a35ef1 |
2144 | =head2 GCC gcov Profiling |
2145 | |
10f58044 |
2146 | Starting from GCC 3.0 I<basic block profiling> is officially available |
51a35ef1 |
2147 | for the GNU CC. |
2148 | |
2149 | You can build a profiled version of perl called F<perl.gcov> by |
2150 | invoking the make target "perl.gcov" (what is required that Perl must |
2151 | be compiled using gcc with the flags C<-fprofile-arcs |
2152 | -ftest-coverage>, you may need to re-Configure). |
2153 | |
2154 | Running the profiled version of Perl will cause profile output to be |
2155 | generated. For each source file an accompanying ".da" file will be |
2156 | created. |
2157 | |
2158 | To display the results you use the "gcov" utility (which should |
2159 | be installed if you have gcc 3.0 or newer installed). F<gcov> is |
2160 | run on source code files, like this |
2161 | |
2162 | gcov sv.c |
2163 | |
2164 | which will cause F<sv.c.gcov> to be created. The F<.gcov> files |
2165 | contain the source code annotated with relative frequencies of |
2166 | execution indicated by "#" markers. |
2167 | |
2168 | Useful options of F<gcov> include C<-b> which will summarise the |
2169 | basic block, branch, and function call coverage, and C<-c> which |
2170 | instead of relative frequencies will use the actual counts. For |
2171 | more information on the use of F<gcov> and basic block profiling |
2172 | with gcc, see the latest GNU CC manual, as of GCC 3.0 see |
2173 | |
2174 | http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc.html |
2175 | |
2176 | and its section titled "8. gcov: a Test Coverage Program" |
2177 | |
2178 | http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html#SEC132 |
2179 | |
4ae3d70a |
2180 | =head2 Pixie Profiling |
2181 | |
51a35ef1 |
2182 | Pixie is a profiling tool available on IRIX and Tru64 (aka Digital |
2183 | UNIX aka DEC OSF/1) platforms. Pixie does its profiling using |
10f58044 |
2184 | I<basic-block counting>. |
4ae3d70a |
2185 | |
83f0ef60 |
2186 | You can build a profiled version of perl called F<perl.pixie> by |
51a35ef1 |
2187 | invoking the make target "perl.pixie" (what is required is that Perl |
2188 | must be compiled using the C<-g> flag, you may need to re-Configure). |
2189 | |
2190 | In Tru64 a file called F<perl.Addrs> will also be silently created, |
2191 | this file contains the addresses of the basic blocks. Running the |
2192 | profiled version of Perl will create a new file called "perl.Counts" |
2193 | which contains the counts for the basic block for that particular |
2194 | program execution. |
4ae3d70a |
2195 | |
51a35ef1 |
2196 | To display the results you use the F<prof> utility. The exact |
4ae3d70a |
2197 | incantation depends on your operating system, "prof perl.Counts" in |
2198 | IRIX, and "prof -pixie -all -L. perl" in Tru64. |
2199 | |
6c41479b |
2200 | In IRIX the following prof options are available: |
2201 | |
2202 | =over 4 |
2203 | |
2204 | =item -h |
2205 | |
2206 | Reports the most heavily used lines in descending order of use. |
6e36760b |
2207 | Useful for finding the hotspot lines. |
6c41479b |
2208 | |
2209 | =item -l |
2210 | |
2211 | Groups lines by procedure, with procedures sorted in descending order of use. |
2212 | Within a procedure, lines are listed in source order. |
6e36760b |
2213 | Useful for finding the hotspots of procedures. |
6c41479b |
2214 | |
2215 | =back |
2216 | |
2217 | In Tru64 the following options are available: |
2218 | |
2219 | =over 4 |
2220 | |
3958b146 |
2221 | =item -p[rocedures] |
6c41479b |
2222 | |
3958b146 |
2223 | Procedures sorted in descending order by the number of cycles executed |
6e36760b |
2224 | in each procedure. Useful for finding the hotspot procedures. |
6c41479b |
2225 | (This is the default option.) |
2226 | |
24000d2f |
2227 | =item -h[eavy] |
6c41479b |
2228 | |
6e36760b |
2229 | Lines sorted in descending order by the number of cycles executed in |
2230 | each line. Useful for finding the hotspot lines. |
6c41479b |
2231 | |
24000d2f |
2232 | =item -i[nvocations] |
6c41479b |
2233 | |
6e36760b |
2234 | The called procedures are sorted in descending order by number of calls |
2235 | made to the procedures. Useful for finding the most used procedures. |
6c41479b |
2236 | |
24000d2f |
2237 | =item -l[ines] |
6c41479b |
2238 | |
2239 | Grouped by procedure, sorted by cycles executed per procedure. |
6e36760b |
2240 | Useful for finding the hotspots of procedures. |
6c41479b |
2241 | |
2242 | =item -testcoverage |
2243 | |
2244 | The compiler emitted code for these lines, but the code was unexecuted. |
2245 | |
24000d2f |
2246 | =item -z[ero] |
6c41479b |
2247 | |
2248 | Unexecuted procedures. |
2249 | |
aa500c9e |
2250 | =back |
6c41479b |
2251 | |
2252 | For further information, see your system's manual pages for pixie and prof. |
4ae3d70a |
2253 | |
b8ddf6b3 |
2254 | =head2 Miscellaneous tricks |
2255 | |
2256 | =over 4 |
2257 | |
2258 | =item * |
2259 | |
cc177e1a |
2260 | Those debugging perl with the DDD frontend over gdb may find the |
b8ddf6b3 |
2261 | following useful: |
2262 | |
2263 | You can extend the data conversion shortcuts menu, so for example you |
2264 | can display an SV's IV value with one click, without doing any typing. |
2265 | To do that simply edit ~/.ddd/init file and add after: |
2266 | |
2267 | ! Display shortcuts. |
2268 | Ddd*gdbDisplayShortcuts: \ |
2269 | /t () // Convert to Bin\n\ |
2270 | /d () // Convert to Dec\n\ |
2271 | /x () // Convert to Hex\n\ |
2272 | /o () // Convert to Oct(\n\ |
2273 | |
2274 | the following two lines: |
2275 | |
2276 | ((XPV*) (())->sv_any )->xpv_pv // 2pvx\n\ |
2277 | ((XPVIV*) (())->sv_any )->xiv_iv // 2ivx |
2278 | |
2279 | so now you can do ivx and pvx lookups or you can plug there the |
2280 | sv_peek "conversion": |
2281 | |
2282 | Perl_sv_peek(my_perl, (SV*)()) // sv_peek |
2283 | |
2284 | (The my_perl is for threaded builds.) |
2285 | Just remember that every line, but the last one, should end with \n\ |
2286 | |
2287 | Alternatively edit the init file interactively via: |
2288 | 3rd mouse button -> New Display -> Edit Menu |
2289 | |
2290 | Note: you can define up to 20 conversion shortcuts in the gdb |
2291 | section. |
2292 | |
9965345d |
2293 | =item * |
2294 | |
2295 | If you see in a debugger a memory area mysteriously full of 0xabababab, |
2296 | you may be seeing the effect of the Poison() macro, see L<perlclib>. |
2297 | |
b8ddf6b3 |
2298 | =back |
2299 | |
a422fd2d |
2300 | =head2 CONCLUSION |
2301 | |
2302 | We've had a brief look around the Perl source, an overview of the stages |
2303 | F<perl> goes through when it's running your code, and how to use a |
902b9dbf |
2304 | debugger to poke at the Perl guts. We took a very simple problem and |
2305 | demonstrated how to solve it fully - with documentation, regression |
2306 | tests, and finally a patch for submission to p5p. Finally, we talked |
2307 | about how to use external tools to debug and test Perl. |
a422fd2d |
2308 | |
2309 | I'd now suggest you read over those references again, and then, as soon |
2310 | as possible, get your hands dirty. The best way to learn is by doing, |
2311 | so: |
2312 | |
2313 | =over 3 |
2314 | |
2315 | =item * |
2316 | |
2317 | Subscribe to perl5-porters, follow the patches and try and understand |
2318 | them; don't be afraid to ask if there's a portion you're not clear on - |
2319 | who knows, you may unearth a bug in the patch... |
2320 | |
2321 | =item * |
2322 | |
2323 | Keep up to date with the bleeding edge Perl distributions and get |
2324 | familiar with the changes. Try and get an idea of what areas people are |
2325 | working on and the changes they're making. |
2326 | |
2327 | =item * |
2328 | |
3e148164 |
2329 | Do read the README associated with your operating system, e.g. README.aix |
a1f349fd |
2330 | on the IBM AIX OS. Don't hesitate to supply patches to that README if |
2331 | you find anything missing or changed over a new OS release. |
2332 | |
2333 | =item * |
2334 | |
a422fd2d |
2335 | Find an area of Perl that seems interesting to you, and see if you can |
2336 | work out how it works. Scan through the source, and step over it in the |
2337 | debugger. Play, poke, investigate, fiddle! You'll probably get to |
2338 | understand not just your chosen area but a much wider range of F<perl>'s |
2339 | activity as well, and probably sooner than you'd think. |
2340 | |
2341 | =back |
2342 | |
2343 | =over 3 |
2344 | |
2345 | =item I<The Road goes ever on and on, down from the door where it began.> |
2346 | |
2347 | =back |
2348 | |
2349 | If you can do these things, you've started on the long road to Perl porting. |
2350 | Thanks for wanting to help make Perl better - and happy hacking! |
2351 | |
e8cd7eae |
2352 | =head1 AUTHOR |
2353 | |
2354 | This document was written by Nathan Torkington, and is maintained by |
2355 | the perl5-porters mailing list. |
2356 | |