=head1 DESCRIPTION
-This is a list of wishes for Perl. The tasks we think are smaller or easier
-are listed first. Anyone is welcome to work on any of these, but it's a good
-idea to first contact I<perl5-porters@perl.org> to avoid duplication of
-effort. By all means contact a pumpking privately first if you prefer.
+This is a list of wishes for Perl. The most up to date version of this file
+is at http://perl5.git.perl.org/perl.git/blob_plain/HEAD:/pod/perltodo.pod
+
+The tasks we think are smaller or easier are listed first. Anyone is welcome
+to work on any of these, but it's a good idea to first contact
+I<perl5-porters@perl.org> to avoid duplication of effort, and to learn from
+any previous attempts. By all means contact a pumpking privately first if you
+prefer.
Whilst patches to make the list shorter are most welcome, ideas to add to
the list are also encouraged. Check the perl5-porters archives for past
F<AUTHORS> file, which ships in the official distribution. How many other
programming languages offer you 1 line of immortality?
-=head1 The roadmap to 5.10
+=head1 Tasks that only need Perl knowledge
-The roadmap to 5.10 envisages feature based releases, as various items in this
-TODO are completed.
+=head2 Improve Porting/cmpVERSION.pl to work from git tags
-=head2 Needed for a 5.9.4 release
+See F<Porting/release_managers_guide.pod> for a bit more detail.
-=over
+=head2 Migrate t/ from custom TAP generation
-=item *
+Many tests below F<t/> still generate TAP by "hand", rather than using library
+functions. As explained in L<perlhack/Writing a test>, tests in F<t/> are
+written in a particular way to test that more complex constructions actually
+work before using them routinely. Hence they don't use C<Test::More>, but
+instead there is an intentionally simpler library, F<t/test.pl>. However,
+quite a few tests in F<t/> have not been refactored to use it. Refactoring
+any of these tests, one at a time, is a useful thing TODO.
-Review assertions. Review syntax to combine assertions. Assertions could take
-advantage of the lexical pragmas work. L</What hooks would assertions need?>
+The subdirectories F<base>, F<cmd> and F<comp>, that contain the most
+basic tests, should be excluded from this task.
-=item *
+=head2 Test that regen.pl was run
-C<encoding::warnings> should be turned into a lexical pragma.
-C<encoding> should, too (probably).
+There are various generated files shipped with the perl distribution, for
+things like header files generate from data. The generation scripts are
+written in perl, and all can be run by F<regen.pl>. However, because they're
+written in perl, we can't run them before we've built perl. We can't run them
+as part of the F<Makefile>, because changing files underneath F<make> confuses
+it completely, and we don't want to run them automatically anyway, as they
+change files shipped by the distribution, something we seek not do to.
-=back
+If someone changes the data, but forgets to re-run F<regen.pl> then the
+generated files are out of sync. It would be good to have a test in
+F<t/porting> that checks that the generated files are in sync, and fails
+otherwise, to alert someone before they make a poor commit. I suspect that this
+would require adapting the scripts run from F<regen.pl> to have dry-run
+options, and invoking them with these, or by refactoring them into a library
+that does the generation, which can be called by the scripts, and by the test.
-=head2 Needed for a 5.9.5 release
+=head2 Automate perldelta generation
-=over
+The perldelta file accompanying each release summaries the major changes.
+It's mostly manually generated currently, but some of that could be
+automated with a bit of perl, specifically the generation of
-=item *
-Implement L</_ prototype character>
+=over
-=item *
-Implement L</state variables>
+=item Modules and Pragmata
-=back
+=item New Documentation
-=head2 Needed for a 5.9.6 release
+=item New Tests
-Stabilisation. If all goes well, this will be the equivalent of a 5.10-beta.
+=back
-=head1 Tasks that only need Perl knowledge
+See F<Porting/how_to_write_a_perldelta.pod> for details.
-=head2 common test code for timed bail out
+=head2 Remove duplication of test setup.
-Write portable self destruct code for tests to stop them burning CPU in
-infinite loops. This needs to avoid using alarm, as some of the tests are
-testing alarm/sleep or timers.
+Schwern notes, that there's duplication of code - lots and lots of tests have
+some variation on the big block of C<$Is_Foo> checks. We can safely put this
+into a file, change it to build an C<%Is> hash and require it. Maybe just put
+it into F<test.pl>. Throw in the handy tainting subroutines.
-=head2 POD -> HTML conversion in the core still sucks
+=head2 POD -E<gt> HTML conversion in the core still sucks
Which is crazy given just how simple POD purports to be, and how simple HTML
can be. It's not actually I<as> simple as it sounds, particularly with the
The addition of C<Pod::Simple> and its related modules may make this task
easier to complete.
-=head2 Parallel testing
+=head2 Make ExtUtils::ParseXS use strict;
-The core regression test suite is getting ever more comprehensive, which has
-the side effect that it takes longer to run. This isn't so good. Investigate
-whether it would be feasible to give the harness script the B<option> of
-running sets of tests in parallel. This would be useful for tests in
-F<t/op/*.t> and F<t/uni/*.t> and maybe some sets of tests in F<lib/>.
+F<lib/ExtUtils/ParseXS.pm> contains this line
-Questions to answer
+ # use strict; # One of these days...
-=over 4
+Simply uncomment it, and fix all the resulting issues :-)
-=item 1
-
-How does screen layout work when you're running more than one test?
-
-=item 2
-
-How does the caller of test specify how many tests to run in parallel?
-
-=item 3
-
-How do setup/teardown tests identify themselves?
-
-=back
-
-Pugs already does parallel testing - can their approach be re-used?
+The more practical approach, to break the task down into manageable chunks, is
+to work your way though the code from bottom to top, or if necessary adding
+extra C<{ ... }> blocks, and turning on strict within them.
=head2 Make Schwern poorer
hold him upside down and shake vigorously in order to actually extract the
cash.
-See F<t/lib/1_compile.t> for the 3 remaining modules that need tests.
-
=head2 Improve the coverage of the core tests
-Use Devel::Cover to ascertain the core's test coverage, then add tests that
-are currently missing.
+Use Devel::Cover to ascertain the core modules' test coverage, then add
+tests that are currently missing.
=head2 test B
changes would be needed to package that module and its tests up for CPAN, and
do so. Test it with older perl releases, and fix the problems you find.
-=head2 Improving C<threads::shared>
+To make a minimal perl distribution, it's useful to look at
+F<t/lib/commonsense.t>.
+
+=head2 Move dual-life pod/*.PL into ext
-Investigate whether C<threads::shared> could share aggregates properly with
-only Perl level changes to shared.pm
+Nearly all the dual-life modules have been moved to F<ext>. However, we
+still need to move F<pod/*.PL> into their respective directories
+in F<ext/>. They're referenced by (at least) C<plextract> in F<Makefile.SH>
+and C<utils> in F<win32/Makefile> and F<win32/makefile.ml>, and listed
+explicitly in F<win32/pod.mak>, F<vms/descrip_mms.template> and F<utils.lst>
=head2 POSIX memory footprint
There is a script F<embed.pl> that generates several header files to prefix
all of Perl's symbols in a consistent way, to provide some semblance of
namespace support in C<C>. Functions are declared in F<embed.fnc>, variables
-in F<interpvar.h> and F<thrdvar.h>. Quite a few of the functions and variables
+in F<interpvar.h>. Quite a few of the functions and variables
are conditionally declared there, using C<#ifdef>. However, F<embed.pl>
doesn't understand the C macros, so the rules about which symbols are present
when is duplicated in F<makedef.pl>. Writing things twice is bad, m'kay.
It would be good to teach C<embed.pl> to understand the conditional
compilation, and hence remove the duplication, and the mistakes it has caused.
+=head2 use strict; and AutoLoad
+Currently if you write
+ package Whack;
+ use AutoLoader 'AUTOLOAD';
+ use strict;
+ 1;
+ __END__
+ sub bloop {
+ print join (' ', No, strict, here), "!\n";
+ }
+then C<use strict;> isn't in force within the autoloaded subroutines. It would
+be more consistent (and less surprising) to arrange for all lexical pragmas
+in force at the __END__ block to be in force within each autoloaded subroutine.
+There's a similar problem with SelfLoader.
+
+=head2 profile installman
+
+The F<installman> script is slow. All it is doing text processing, which we're
+told is something Perl is good at. So it would be nice to know what it is doing
+that is taking so much CPU, and where possible address it.
+
+=head2 enable lexical enabling/disabling of inidvidual warnings
+
+Currently, warnings can only be enabled or disabled by category. There
+are times when it would be useful to quash a single warning, not a
+whole category.
=head1 Tasks that need a little sysadmin-type knowledge
coverage, and have C<Configure> and the F<Makefile> do all the right things
automatically.
-=head2 Make Config.pm cope with differences between build and installed perl
+=head2 Make Config.pm cope with differences between built and installed perl
Quite often vendors ship a perl binary compiled with their (pay-for)
compilers. People install a free compiler, such as gcc. To work out how to
export list is correct, and to build a perl that does not pollute the global
namespace with private symbols.
+=head2 Cross-compile support
+
+Currently C<Configure> understands C<-Dusecrosscompile> option. This option
+arranges for building C<miniperl> for TARGET machine, so this C<miniperl> is
+assumed then to be copied to TARGET machine and used as a replacement of full
+C<perl> executable.
+
+This could be done little differently. Namely C<miniperl> should be built for
+HOST and then full C<perl> with extensions should be compiled for TARGET.
+This, however, might require extra trickery for %Config: we have one config
+first for HOST and then another for TARGET. Tools like MakeMaker will be
+mightily confused. Having around two different types of executables and
+libraries (HOST and TARGET) makes life interesting for Makefiles and
+shell (and Perl) scripts. There is $Config{run}, normally empty, which
+can be used as an execution wrapper. Also note that in some
+cross-compilation/execution environments the HOST and the TARGET do
+not see the same filesystem(s), the $Config{run} may need to do some
+file/directory copying back and forth.
+
+=head2 roffitall
+
+Make F<pod/roffitall> be updated by F<pod/buildtoc>.
+
+=head2 Split "linker" from "compiler"
+
+Right now, Configure probes for two commands, and sets two variables:
+
+=over 4
+
+=item * C<cc> (in F<cc.U>)
+
+This variable holds the name of a command to execute a C compiler which
+can resolve multiple global references that happen to have the same
+name. Usual values are F<cc> and F<gcc>.
+Fervent ANSI compilers may be called F<c89>. AIX has F<xlc>.
+=item * C<ld> (in F<dlsrc.U>)
+This variable indicates the program to be used to link
+libraries for dynamic loading. On some systems, it is F<ld>.
+On ELF systems, it should be C<$cc>. Mostly, we'll try to respect
+the hint file setting.
+
+=back
+
+There is an implicit historical assumption from around Perl5.000alpha
+something, that C<$cc> is also the correct command for linking object files
+together to make an executable. This may be true on Unix, but it's not true
+on other platforms, and there are a maze of work arounds in other places (such
+as F<Makefile.SH>) to cope with this.
+
+Ideally, we should create a new variable to hold the name of the executable
+linker program, probe for it in F<Configure>, and centralise all the special
+case logic there or in hints files.
+
+A small bikeshed issue remains - what to call it, given that C<$ld> is already
+taken (arguably for the wrong thing now, but on SunOS 4.1 it is the command
+for creating dynamically-loadable modules) and C<$link> could be confused with
+the Unix command line executable of the same name, which does something
+completely different. Andy Dougherty makes the counter argument "In parrot, I
+tried to call the command used to link object files and libraries into an
+executable F<link>, since that's what my vaguely-remembered DOS and VMS
+experience suggested. I don't think any real confusion has ensued, so it's
+probably a reasonable name for perl5 to use."
+
+"Alas, I've always worried that introducing it would make things worse,
+since now the module building utilities would have to look for
+C<$Config{link}> and institute a fall-back plan if it weren't found."
+Although I can see that as confusing, given that C<$Config{d_link}> is true
+when (hard) links are available.
+
+=head2 Configure Windows using PowerShell
+
+Currently, Windows uses hard-coded config files based to build the
+config.h for compiling Perl. Makefiles are also hard-coded and need to be
+hand edited prior to building Perl. While this makes it easy to create a perl.exe
+that works across multiple Windows versions, being able to accurately
+configure a perl.exe for a specific Windows versions and VS C++ would be
+a nice enhancement. With PowerShell available on Windows XP and up, this
+may now be possible. Step 1 might be to investigate whether this is possible
+and use this to clean up our current makefile situation. Step 2 would be to
+see if there would be a way to use our existing metaconfig units to configure a
+Windows Perl or whether we go in a separate direction and make it so. Of
+course, we all know what step 3 is.
+
+=head2 decouple -g and -DDEBUGGING
+
+Currently F<Configure> automatically adds C<-DDEBUGGING> to the C compiler
+flags if it spots C<-g> in the optimiser flags. The pre-processor directive
+C<DEBUGGING> enables F<perl>'s command line C<-D> options, but in the process
+makes F<perl> slower. It would be good to disentangle this logic, so that
+C-level debugging with C<-g> and Perl level debugging with C<-D> can easily
+be enabled independently.
=head1 Tasks that need a little C knowledge
These tasks would need a little C knowledge, but don't need any specific
background or experience with XS, or how the Perl interpreter works
-=head2 Make it clear from -v if this is the exact official release
-
-Currently perl from C<p4>/C<rsync> ships with a F<patchlevel.h> file that
-usually defines one local patch, of the form "MAINT12345" or "RC1". The output
-of perl -v doesn't report that a perl isn't an official release, and this
-information can get lost in bugs reports. Because of this, the minor version
-isn't bumped up until RC time, to minimise the possibility of versions of perl
-escaping that believe themselves to be newer than they actually are.
-
-It would be useful to find an elegant way to have the "this is an interim
-maintenance release" or "this is a release candidate" in the terse -v output,
-and have it so that it's easy for the pumpking to remove this just as the
-release tarball is rolled up. This way the version pulled out of rsync would
-always say "I'm a development release" and it would be safe to bump the
-reported minor version as soon as a release ships, which would aid perl
-developers.
-
-This task is really about thinking of an elegant way to arrange the C source
-such that it's trivial for the Pumpking to flag "this is an official release"
-when making a tarball, yet leave the default source saying "I'm not the
-official release".
-
-=head2 Ordering of "global" variables.
-
-F<thrdvar.h> and F<intrpvarh> define the "global" variables that need to be
-per-thread under ithreads, where the variables are actually elements in a
-structure. As C dictates, the variables must be laid out in order of
-declaration. There is a comment
-C</* Important ones in the first cache line (if alignment is done right) */>
-which implies that at some point in the past the ordering was carefully chosen
-(at least in part). However, it's clear that the ordering is less than perfect,
-as currently there are things such as 7 C<bool>s in a row, then something
-typically requiring 4 byte alignment, and then an odd C<bool> later on.
-(C<bool>s are typically defined as C<char>s). So it would be good for someone
-to review the ordering of the variables, to see how much alignment padding can
-be removed.
-
-It's also worth checking that all variables are actually used. Perl 5.8.0
-shipped with C<PL_nrs> still defined in F<thrdvar.h>, despite it being unused
-since a change over a year earlier. Had this been spotted before release, it
-could have been removed, but now it has to remain in the 5.8.x releases to
-keep the structure the same size, to retain binary compatibility.
-
-It's probably worth checking if all need to be the types they are. For example
-
- PERLVAR(Ierror_count, I32) /* how many errors so far, max 10 */
-
-might work as well if stored in a signed (or unsigned) 8 bit value, if the
-comment is accurate. C<PL_multi_open> and C<PL_multi_close> can probably
-become C<char>s. Finding variables to downsize coupled with rearrangement
-could shrink the interpreter structure; a size saving which is multiplied by
-the number of threads running.
+=head2 Weed out needless PERL_UNUSED_ARG
+
+The C code uses the macro C<PERL_UNUSED_ARG> to stop compilers warning about
+unused arguments. Often the arguments can't be removed, as there is an
+external constraint that determines the prototype of the function, so this
+approach is valid. However, there are some cases where C<PERL_UNUSED_ARG>
+could be removed. Specifically
+
+=over 4
+
+=item *
+
+The prototypes of (nearly all) static functions can be changed
+
+=item *
+
+Unused arguments generated by short cut macros are wasteful - the short cut
+macro used can be changed.
+
+=back
+
+=head2 Modernize the order of directories in @INC
+
+The way @INC is laid out by default, one cannot upgrade core (dual-life)
+modules without overwriting files. This causes problems for binary
+package builders. One possible proposal is laid out in this
+message:
+L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2002-04/msg02380.html>.
+
+=head2 -Duse32bit*
+
+Natively 64-bit systems need neither -Duse64bitint nor -Duse64bitall.
+On these systems, it might be the default compilation mode, and there
+is currently no guarantee that passing no use64bitall option to the
+Configure process will build a 32bit perl. Implementing -Duse32bit*
+options would be nice for perl 5.12.
=head2 Profile Perl - am I hot or not?
want to determine what ops I<really> are the most commonly used. And in turn
suggest evictions and promotions to achieve a better F<pp_hot.c>.
-=head2 Shrink struct context
-
-In F<cop.h>, we have
-
- struct context {
- U32 cx_type; /* what kind of context this is */
- union {
- struct block cx_blk;
- struct subst cx_subst;
- } cx_u;
- };
-
-There are less than 256 values for C<cx_type>, and the constituent parts
-C<struct block> and C<struct subst> both contain some C<U8> and C<U16> fields,
-so it should be possible to move them to the first word, and share space with
-a C<U8> C<cx_type>, saving 1 word.
+One piece of Perl code that might make a good testbed is F<installman>.
=head2 Allocate OPs from arenas
the various OP structures from arenas. The SV arena code can probably be
re-used for this.
-=head2 Merge the win32 and wince codebases
+Note that Configuring perl with C<-Accflags=-DPL_OP_SLAB_ALLOC> will use
+Perl_Slab_alloc() to pack optrees into a contiguous block, which is
+probably superior to the use of OP arenas, esp. from a cache locality
+standpoint. See L<Profile Perl - am I hot or not?>.
+
+=head2 Improve win32/wince.c
+
+Currently, numerous functions look virtually, if not completely,
+identical in both C<win32/wince.c> and C<win32/win32.c> files, which can't
+be good.
+
+=head2 Use secure CRT functions when building with VC8 on Win32
+
+Visual C++ 2005 (VC++ 8.x) deprecated a number of CRT functions on the basis
+that they were "unsafe" and introduced differently named secure versions of
+them as replacements, e.g. instead of writing
+
+ FILE* f = fopen(__FILE__, "r");
+
+one should now write
+
+ FILE* f;
+ errno_t err = fopen_s(&f, __FILE__, "r");
+
+Currently, the warnings about these deprecations have been disabled by adding
+-D_CRT_SECURE_NO_DEPRECATE to the CFLAGS. It would be nice to remove that
+warning suppressant and actually make use of the new secure CRT functions.
+
+There is also a similar issue with POSIX CRT function names like fileno having
+been deprecated in favour of ISO C++ conformant names like _fileno. These
+warnings are also currently suppressed by adding -D_CRT_NONSTDC_NO_DEPRECATE. It
+might be nice to do as Microsoft suggest here too, although, unlike the secure
+functions issue, there is presumably little or no benefit in this case.
+
+=head2 Fix POSIX::access() and chdir() on Win32
+
+These functions currently take no account of DACLs and therefore do not behave
+correctly in situations where access is restricted by DACLs (as opposed to the
+read-only attribute).
+
+Furthermore, POSIX::access() behaves differently for directories having the
+read-only attribute set depending on what CRT library is being used. For
+example, the _access() function in the VC6 and VC7 CRTs (wrongly) claim that
+such directories are not writable, whereas in fact all directories are writable
+unless access is denied by DACLs. (In the case of directories, the read-only
+attribute actually only means that the directory cannot be deleted.) This CRT
+bug is fixed in the VC8 and VC9 CRTs (but, of course, the directory may still
+not actually be writable if access is indeed denied by DACLs).
+
+For the chdir() issue, see ActiveState bug #74552:
+http://bugs.activestate.com/show_bug.cgi?id=74552
+
+Therefore, DACLs should be checked both for consistency across CRTs and for
+the correct answer.
+
+(Note that perl's -w operator should not be modified to check DACLs. It has
+been written so that it reflects the state of the read-only attribute, even
+for directories (whatever CRT is being used), for symmetry with chmod().)
+
+=head2 strcat(), strcpy(), strncat(), strncpy(), sprintf(), vsprintf()
+
+Maybe create a utility that checks after each libperl.a creation that
+none of the above (nor sprintf(), vsprintf(), or *SHUDDER* gets())
+ever creep back to libperl.a.
+
+ nm libperl.a | ./miniperl -alne '$o = $F[0] if /:$/; print "$o $F[1]" if $F[0] eq "U" && $F[1] =~ /^(?:strn?c(?:at|py)|v?sprintf|gets)$/'
+
+Note, of course, that this will only tell whether B<your> platform
+is using those naughty interfaces.
-Currently, code used to build Perl on Win32 and WinCE are maintained
-separately, but use much of the same code. We currently have a very high
-probability of the code diverging when it shouldn't. Ideally, the code
-for the two systems should be merged so that common code between the two
-could be shared.
+=head2 -D_FORTIFY_SOURCE=2, -fstack-protector
+
+Recent glibcs support C<-D_FORTIFY_SOURCE=2> and recent gcc
+(4.1 onwards?) supports C<-fstack-protector>, both of which give
+protection against various kinds of buffer overflow problems.
+These should probably be used for compiling Perl whenever available,
+Configure and/or hints files should be adjusted to probe for the
+availability of these features and enable them as appropriate.
+
+=head2 Arenas for GPs? For MAGIC?
+
+C<struct gp> and C<struct magic> are both currently allocated by C<malloc>.
+It might be a speed or memory saving to change to using arenas. Or it might
+not. It would need some suitable benchmarking first. In particular, C<GP>s
+can probably be changed with minimal compatibility impact (probably nothing
+outside of the core, or even outside of F<gv.c> allocates them), but they
+probably aren't allocated/deallocated often enough for a speed saving. Whereas
+C<MAGIC> is allocated/deallocated more often, but in turn, is also something
+more externally visible, so changing the rules here may bite external code.
+
+=head2 Shared arenas
+
+Several SV body structs are now the same size, notably PVMG and PVGV, PVAV and
+PVHV, and PVCV and PVFM. It should be possible to allocate and return same
+sized bodies from the same actual arena, rather than maintaining one arena for
+each. This could save 4-6K per thread, of memory no longer tied up in the
+not-yet-allocated part of an arena.
=head1 Tasks that need a knowledge of XS
the perl API that comes from writing modules that use XS to interface to
C.
-=head2 shrink C<PVBM>s
+=head2 Write an XS cookbook
+
+Create pod/perlxscookbook.pod with short, task-focused 'recipes' in XS that
+demonstrate common tasks and good practices. (Some of these might be
+extracted from perlguts.) The target audience should be XS novices, who need
+more examples than perlguts but something less overwhelming than perlapi.
+Recipes should provide "one pretty good way to do it" instead of TIMTOWTDI.
+
+Rather than focusing on interfacing Perl to C libraries, such a cookbook
+should probably focus on how to optimize Perl routines by re-writing them
+in XS. This will likely be more motivating to those who mostly work in
+Perl but are looking to take the next step into XS.
+
+Deconstructing and explaining some simpler XS modules could be one way to
+bootstrap a cookbook. (List::Util? Class::XSAccessor? Tree::Ternary_XS?)
+Another option could be deconstructing the implementation of some simpler
+functions in op.c.
+
+=head2 Allow XSUBs to inline themselves as OPs
+
+For a simple XSUB, often the subroutine dispatch takes more time than the
+XSUB itself. The tokeniser already has the ability to inline constant
+subroutines - it would be good to provide a way to inline other subroutines.
+
+Specifically, simplest approach looks to be to allow an XSUB to provide an
+alternative implementation of itself as a custom OP. A new flag bit in
+C<CvFLAGS()> would signal to the peephole optimiser to take an optree
+such as this:
+
+ b <@> leave[1 ref] vKP/REFC ->(end)
+ 1 <0> enter ->2
+ 2 <;> nextstate(main 1 -e:1) v:{ ->3
+ a <2> sassign vKS/2 ->b
+ 8 <1> entersub[t2] sKS/TARG,1 ->9
+ - <1> ex-list sK ->8
+ 3 <0> pushmark s ->4
+ 4 <$> const(IV 1) sM ->5
+ 6 <1> rv2av[t1] lKM/1 ->7
+ 5 <$> gv(*a) s ->6
+ - <1> ex-rv2cv sK ->-
+ 7 <$> gv(*x) s/EARLYCV ->8
+ - <1> ex-rv2sv sKRM*/1 ->a
+ 9 <$> gvsv(*b) s ->a
+
+perform the symbol table lookup of C<rv2cv> and C<gv(*x)>, locate the
+pointer to the custom OP that provides the direct implementation, and re-
+write the optree something like:
+
+ b <@> leave[1 ref] vKP/REFC ->(end)
+ 1 <0> enter ->2
+ 2 <;> nextstate(main 1 -e:1) v:{ ->3
+ a <2> sassign vKS/2 ->b
+ 7 <1> custom_x -> 8
+ - <1> ex-list sK ->7
+ 3 <0> pushmark s ->4
+ 4 <$> const(IV 1) sM ->5
+ 6 <1> rv2av[t1] lKM/1 ->7
+ 5 <$> gv(*a) s ->6
+ - <1> ex-rv2cv sK ->-
+ - <$> ex-gv(*x) s/EARLYCV ->7
+ - <1> ex-rv2sv sKRM*/1 ->a
+ 8 <$> gvsv(*b) s ->a
+
+I<i.e.> the C<gv(*)> OP has been nulled and spliced out of the execution
+path, and the C<entersub> OP has been replaced by the custom op.
+
+This approach should provide a measurable speed up to simple XSUBs inside
+tight loops. Initially one would have to write the OP alternative
+implementation by hand, but it's likely that this should be reasonably
+straightforward for the type of XSUB that would benefit the most. Longer
+term, once the run-time implementation is proven, it should be possible to
+progressively update ExtUtils::ParseXS to generate OP implementations for
+some XSUBs.
+
+=head2 Remove the use of SVs as temporaries in dump.c
+
+F<dump.c> contains debugging routines to dump out the contains of perl data
+structures, such as C<SV>s, C<AV>s and C<HV>s. Currently, the dumping code
+B<uses> C<SV>s for its temporary buffers, which was a logical initial
+implementation choice, as they provide ready made memory handling.
+
+However, they also lead to a lot of confusion when it happens that what you're
+trying to debug is seen by the code in F<dump.c>, correctly or incorrectly, as
+a temporary scalar it can use for a temporary buffer. It's also not possible
+to dump scalars before the interpreter is properly set up, such as during
+ithreads cloning. It would be good to progressively replace the use of scalars
+as string accumulation buffers with something much simpler, directly allocated
+by C<malloc>. The F<dump.c> code is (or should be) only producing 7 bit
+US-ASCII, so output character sets are not an issue.
+
+Producing and proving an internal simple buffer allocation would make it easier
+to re-write the internals of the PerlIO subsystem to avoid using C<SV>s for
+B<its> buffers, use of which can cause problems similar to those of F<dump.c>,
+at similar times.
+
+=head2 safely supporting POSIX SA_SIGINFO
+
+Some years ago Jarkko supplied patches to provide support for the POSIX
+SA_SIGINFO feature in Perl, passing the extra data to the Perl signal handler.
+
+Unfortunately, it only works with "unsafe" signals, because under safe
+signals, by the time Perl gets to run the signal handler, the extra
+information has been lost. Moreover, it's not easy to store it somewhere,
+as you can't call mutexs, or do anything else fancy, from inside a signal
+handler.
+
+So it strikes me that we could provide safe SA_SIGINFO support
-By removing unused elements and careful re-ordering, the structures for C<AV>s,
-C<HV>s, C<CV>s and C<GV>s have recently been shrunk considerably. C<PVIO>s
-probably aren't worth it, as typical programs don't use more than 8, and
-(at least) C<Filter::Util::Call> uses C<SvPVX>/C<SvCUR>/C<SvLEN> on a C<PVIO>,
-so it would mean code changes to modules on CPAN. C<PVBM>s might have some
-savings to win.
+=over 4
-=head2 Implicit Latin 1 => Unicode translation
+=item 1
-Conversions from byte strings to UTF-8 currently map high bit characters
-to Unicode without translation (or, depending on how you look at it, by
-implicitly assuming that the byte strings are in Latin-1). As perl assumes
-the C locale by default, upgrading a string to UTF-8 may change the
-meaning of its contents regarding character classes, case mapping, etc.
-This should probably emit a warning (at least).
+Provide global variables for two file descriptors
-This task is incremental - even a little bit of work on it will help.
+=item 2
+
+When the first request is made via C<sigaction> for C<SA_SIGINFO>, create a
+pipe, store the reader in one, the writer in the other
+
+=item 3
+
+In the "safe" signal handler (C<Perl_csighandler()>/C<S_raise_signal()>), if
+the C<siginfo_t> pointer non-C<NULL>, and the writer file handle is open,
+
+=over 8
+
+=item 1
+
+serialise signal number, C<struct siginfo_t> (or at least the parts we care
+about) into a small auto char buff
+
+=item 2
+
+C<write()> that (non-blocking) to the writer fd
+
+=over 12
+
+=item 1
+
+if it writes 100%, flag the signal in a counter of "signals on the pipe" akin
+to the current per-signal-number counts
+
+=item 2
+
+if it writes 0%, assume the pipe is full. Flag the data as lost?
+
+=item 3
+
+if it writes partially, croak a panic, as your OS is broken.
+
+=back
+
+=back
+
+=item 4
+
+in the regular C<PERL_ASYNC_CHECK()> processing, if there are "signals on
+the pipe", read the data out, deserialise, build the Perl structures on
+the stack (code in C<Perl_sighandler()>, the "unsafe" handler), and call as
+usual.
+
+=back
+
+I think that this gets us decent C<SA_SIGINFO> support, without the current risk
+of running Perl code inside the signal handler context. (With all the dangers
+of things like C<malloc> corruption that that currently offers us)
+
+For more information see the thread starting with this message:
+http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-03/msg00305.html
=head2 autovivification
temporarily retired in 5.8.1, and the -C has been repurposed, see
L<perlrun>.)
+Most probably the right way to do this would be this:
+L</"Virtualize operating system access">.
+
=head2 Unicode in %ENV
Currently the %ENV entries are always byte strings.
+See L</"Virtualize operating system access">.
+
+=head2 Unicode and glob()
+
+Currently glob patterns and filenames returned from File::Glob::glob()
+are always byte strings. See L</"Virtualize operating system access">.
=head2 use less 'memory'
running a shell. readpipe() (the function behind qx//) could be similarly
extended.
+=head2 Audit the code for destruction ordering assumptions
+
+Change 25773 notes
+
+ /* Need to check SvMAGICAL, as during global destruction it may be that
+ AvARYLEN(av) has been freed before av, and hence the SvANY() pointer
+ is now part of the linked list of SV heads, rather than pointing to
+ the original body. */
+ /* FIXME - audit the code for other bugs like this one. */
+
+adding the C<SvMAGICAL> check to
+
+ if (AvARYLEN(av) && SvMAGICAL(AvARYLEN(av))) {
+ MAGIC *mg = mg_find (AvARYLEN(av), PERL_MAGIC_arylen);
+
+Go through the core and look for similar assumptions that SVs have particular
+types, as all bets are off during global destruction.
+
+=head2 Extend PerlIO and PerlIO::Scalar
+
+PerlIO::Scalar doesn't know how to truncate(). Implementing this
+would require extending the PerlIO vtable.
+
+Similarly the PerlIO vtable doesn't know about formats (write()), or
+about stat(), or chmod()/chown(), utime(), or flock().
+
+(For PerlIO::Scalar it's hard to see what e.g. mode bits or ownership
+would mean.)
+
+PerlIO doesn't do directories or symlinks, either: mkdir(), rmdir(),
+opendir(), closedir(), seekdir(), rewinddir(), glob(); symlink(),
+readlink().
+See also L</"Virtualize operating system access">.
+=head2 -C on the #! line
+It should be possible to make -C work correctly if found on the #! line,
+given that all perl command line options are strict ASCII, and -C changes
+only the interpretation of non-ASCII characters, and not for the script file
+handle. To make it work needs some investigation of the ordering of function
+calls during startup, and (by implication) a bit of tweaking of that order.
+
+=head2 Organize error messages
+
+Perl's diagnostics (error messages, see L<perldiag>) could use
+reorganizing and formalizing so that each error message has its
+stable-for-all-eternity unique id, categorized by severity, type, and
+subsystem. (The error messages would be listed in a datafile outside
+of the Perl source code, and the source code would only refer to the
+messages by the id.) This clean-up and regularizing should apply
+for all croak() messages.
+
+This would enable all sorts of things: easier translation/localization
+of the messages (though please do keep in mind the caveats of
+L<Locale::Maketext> about too straightforward approaches to
+translation), filtering by severity, and instead of grepping for a
+particular error message one could look for a stable error id. (Of
+course, changing the error messages by default would break all the
+existing software depending on some particular error message...)
+
+This kind of functionality is known as I<message catalogs>. Look for
+inspiration for example in the catgets() system, possibly even use it
+if available-- but B<only> if available, all platforms will B<not>
+have catgets().
+
+For the really pure at heart, consider extending this item to cover
+also the warning messages (see L<perllexwarn>, C<warnings.pl>).
=head1 Tasks that need a knowledge of the interpreter
These tasks would need C knowledge, and knowledge of how the interpreter works,
or a willingness to learn.
-=head2 lexical pragmas
+=head2 forbid labels with keyword names
+
+Currently C<goto keyword> "computes" the label value:
+
+ $ perl -e 'goto print'
+ Can't find label 1 at -e line 1.
+
+It is controversial if the right way to avoid the confusion is to forbid
+labels with keyword names, or if it would be better to always treat
+bareword expressions after a "goto" as a label and never as a keyword.
+
+=head2 truncate() prototype
+
+The prototype of truncate() is currently C<$$>. It should probably
+be C<*$> instead. (This is changed in F<opcode.pl>)
+
+=head2 decapsulation of smart match argument
+
+Currently C<$foo ~~ $object> will die with the message "Smart matching a
+non-overloaded object breaks encapsulation". It would be nice to allow
+to bypass this by using explictly the syntax C<$foo ~~ %$object> or
+C<$foo ~~ @$object>.
+
+=head2 error reporting of [$a ; $b]
+
+Using C<;> inside brackets is a syntax error, and we don't propose to change
+that by giving it any meaning. However, it's not reported very helpfully:
-Document the new support for lexical pragmas in 5.9.3 and how %^H works.
-Maybe C<re>, C<encoding>, maybe other pragmas could be made lexical.
+ $ perl -e '$a = [$b; $c];'
+ syntax error at -e line 1, near "$b;"
+ syntax error at -e line 1, near "$c]"
+ Execution of -e aborted due to compilation errors.
+
+It should be possible to hook into the tokeniser or the lexer, so that when a
+C<;> is parsed where it is not legal as a statement terminator (ie inside
+C<{}> used as a hashref, C<[]> or C<()>) it issues an error something like
+I<';' isn't legal inside an expression - if you need multiple statements use a
+do {...} block>. See the thread starting at
+http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2008-09/msg00573.html
+
+=head2 lexicals used only once
+
+This warns:
+
+ $ perl -we '$pie = 42'
+ Name "main::pie" used only once: possible typo at -e line 1.
+
+This does not:
+
+ $ perl -we 'my $pie = 42'
+
+Logically all lexicals used only once should warn, if the user asks for
+warnings. An unworked RT ticket (#5087) has been open for almost seven
+years for this discrepancy.
+
+=head2 UTF-8 revamp
+
+The handling of Unicode is unclean in many places. For example, the regexp
+engine matches in Unicode semantics whenever the string or the pattern is
+flagged as UTF-8, but that should not be dependent on an internal storage
+detail of the string.
+
+=head2 Properly Unicode safe tokeniser and pads.
+
+The tokeniser isn't actually very UTF-8 clean. C<use utf8;> is a hack -
+variable names are stored in stashes as raw bytes, without the utf-8 flag
+set. The pad API only takes a C<char *> pointer, so that's all bytes too. The
+tokeniser ignores the UTF-8-ness of C<PL_rsfp>, or any SVs returned from
+source filters. All this could be fixed.
+
+=head2 state variable initialization in list context
+
+Currently this is illegal:
+
+ state ($a, $b) = foo();
+
+In Perl 6, C<state ($a) = foo();> and C<(state $a) = foo();> have different
+semantics, which is tricky to implement in Perl 5 as currently they produce
+the same opcode trees. The Perl 6 design is firm, so it would be good to
+implement the necessary code in Perl 5. There are comments in
+C<Perl_newASSIGNOP()> that show the code paths taken by various assignment
+constructions involving state variables.
+
+=head2 Implement $value ~~ 0 .. $range
+
+It would be nice to extend the syntax of the C<~~> operator to also
+understand numeric (and maybe alphanumeric) ranges.
+
+=head2 A does() built-in
+
+Like ref(), only useful. It would call the C<DOES> method on objects; it
+would also tell whether something can be dereferenced as an
+array/hash/etc., or used as a regexp, etc.
+L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-03/msg00481.html>
+
+=head2 Tied filehandles and write() don't mix
+
+There is no method on tied filehandles to allow them to be called back by
+formats.
+
+=head2 Propagate compilation hints to the debugger
+
+Currently a debugger started with -dE on the command-line doesn't see the
+features enabled by -E. More generally hints (C<$^H> and C<%^H>) aren't
+propagated to the debugger. Probably it would be a good thing to propagate
+hints from the innermost non-C<DB::> scope: this would make code eval'ed
+in the debugger see the features (and strictures, etc.) currently in
+scope.
=head2 Attach/detach debugger from running program
The old perltodo notes that lvalue functions don't work for list or hash
slices. This would be good to fix.
-=head2 LVALUE functions in the debugger
+=head2 regexp optimiser optional
-The old perltodo notes that lvalue functions don't work in the debugger. This
-would be good to fix.
+The regexp optimiser is not optional. It should configurable to be, to allow
+its performance to be measured, and its bugs to be easily demonstrated.
-=head2 _ prototype character
+=head2 delete &function
-Study the possibility of adding a new prototype character, C<_>, meaning
-"this argument defaults to $_".
+Allow to delete functions. One can already undef them, but they're still
+in the stash.
-=head2 state variables
+=head2 C</w> regex modifier
-C<my $foo if 0;> is deprecated, and should be replaced with
-C<state $x = "initial value\n";> the syntax from Perl 6.
-Rafael has sent a first cut patch to perl5-porters.
+That flag would enable to match whole words, and also to interpolate
+arrays as alternations. With it, C</P/w> would be roughly equivalent to:
-=head2 regexp optimiser optional
+ do { local $"='|'; /\b(?:P)\b/ }
-The regexp optimiser is not optional. It should configurable to be, to allow
-its performance to be measured, and its bugs to be easily demonstrated.
-
-=head2 UNITCHECK
-
-Introduce a new special block, UNITCHECK, which is run at the end of a
-compilation unit (module, file, eval(STRING) block). This will correspond to
-the Perl 6 CHECK. Perl 5's CHECK cannot be changed or removed because the
-O.pm/B.pm backend framework depends on it.
+See L<http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-01/msg00400.html>
+for the discussion.
=head2 optional optimizer
perl and XS at run time, so investigate using 2 ops to enter subs (one for
XS, one for perl) and swap between if a sub is redefined.
-=head2 Self ties
+=head2 Self-ties
-self ties are currently illegal because they caused too many segfaults. Maybe
-the causes of these could be tracked down and self-ties on all types re-
-instated.
+Self-ties are currently illegal because they caused too many segfaults. Maybe
+the causes of these could be tracked down and self-ties on all types
+reinstated.
=head2 Optimize away @_
The old perltodo notes "Look at the "reification" code in C<av.c>".
-=head2 What hooks would assertions need?
-
-Assertions are in the core, and work. However, assertions needed to be added
-as a core patch, rather than an XS module in ext, or a CPAN module, because
-the core has no hooks in the necessary places. It would be useful to
-investigate what hooks would need to be added to make it possible to provide
-the full assertion support from a CPAN module, so that we aren't constraining
-the imagination of future CPAN authors.
-
-=head2 Properly Unicode safe tokeniser and pads.
-
-The tokeniser isn't actually very UTF-8 clean. C<use utf8;> is a hack -
-variable names are stored in stashes as raw bytes, without the utf-8 flag
-set. The pad API only takes a C<char *> pointer, so that's all bytes too. The
-tokeniser ignores the UTF-8-ness of C<PL_rsfp>, or any SVs returned from
-source filters. All this could be fixed.
-
-
-
-
+=head2 Virtualize operating system access
+
+Implement a set of "vtables" that virtualizes operating system access
+(open(), mkdir(), unlink(), readdir(), getenv(), etc.) At the very
+least these interfaces should take SVs as "name" arguments instead of
+bare char pointers; probably the most flexible and extensible way
+would be for the Perl-facing interfaces to accept HVs. The system
+needs to be per-operating-system and per-file-system
+hookable/filterable, preferably both from XS and Perl level
+(L<perlport/"Files and Filesystems"> is good reading at this point,
+in fact, all of L<perlport> is.)
+
+This has actually already been implemented (but only for Win32),
+take a look at F<iperlsys.h> and F<win32/perlhost.h>. While all Win32
+variants go through a set of "vtables" for operating system access,
+non-Win32 systems currently go straight for the POSIX/Unix-style
+system/library call. Similar system as for Win32 should be
+implemented for all platforms. The existing Win32 implementation
+probably does not need to survive alongside this proposed new
+implementation, the approaches could be merged.
+
+What would this give us? One often-asked-for feature this would
+enable is using Unicode for filenames, and other "names" like %ENV,
+usernames, hostnames, and so forth.
+(See L<perlunicode/"When Unicode Does Not Happen">.)
+
+But this kind of virtualization would also allow for things like
+virtual filesystems, virtual networks, and "sandboxes" (though as long
+as dynamic loading of random object code is allowed, not very safe
+sandboxes since external code of course know not of Perl's vtables).
+An example of a smaller "sandbox" is that this feature can be used to
+implement per-thread working directories: Win32 already does this.
+
+See also L</"Extend PerlIO and PerlIO::Scalar">.
+
+=head2 Investigate PADTMP hash pessimisation
+
+The peephole optimiser converts constants used for hash key lookups to shared
+hash key scalars. Under ithreads, something is undoing this work.
+See http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-09/msg00793.html
+
+=head2 Store the current pad in the OP slab allocator
+
+=for clarification
+I hope that I got that "current pad" part correct
+
+Currently we leak ops in various cases of parse failure. I suggested that we
+could solve this by always using the op slab allocator, and walking it to
+free ops. Dave comments that as some ops are already freed during optree
+creation one would have to mark which ops are freed, and not double free them
+when walking the slab. He notes that one problem with this is that for some ops
+you have to know which pad was current at the time of allocation, which does
+change. I suggested storing a pointer to the current pad in the memory allocated
+for the slab, and swapping to a new slab each time the pad changes. Dave thinks
+that this would work.
+
+=head2 repack the optree
+
+Repacking the optree after execution order is determined could allow
+removal of NULL ops, and optimal ordering of OPs with respect to cache-line
+filling. The slab allocator could be reused for this purpose. I think that
+the best way to do this is to make it an optional step just before the
+completed optree is attached to anything else, and to use the slab allocator
+unchanged, so that freeing ops is identical whether or not this step runs.
+Note that the slab allocator allocates ops downwards in memory, so one would
+have to actually "allocate" the ops in reverse-execution order to get them
+contiguous in memory in execution order.
+
+See http://www.nntp.perl.org/group/perl.perl5.porters/2007/12/msg131975.html
+
+Note that running this copy, and then freeing all the old location ops would
+cause their slabs to be freed, which would eliminate possible memory wastage if
+the previous suggestion is implemented, and we swap slabs more frequently.
+
+=head2 eliminate incorrect line numbers in warnings
+
+This code
+
+ use warnings;
+ my $undef;
+
+ if ($undef == 3) {
+ } elsif ($undef == 0) {
+ }
+
+used to produce this output:
+
+ Use of uninitialized value in numeric eq (==) at wrong.pl line 4.
+ Use of uninitialized value in numeric eq (==) at wrong.pl line 4.
+
+where the line of the second warning was misreported - it should be line 5.
+Rafael fixed this - the problem arose because there was no nextstate OP
+between the execution of the C<if> and the C<elsif>, hence C<PL_curcop> still
+reports that the currently executing line is line 4. The solution was to inject
+a nextstate OPs for each C<elsif>, although it turned out that the nextstate
+OP needed to be a nulled OP, rather than a live nextstate OP, else other line
+numbers became misreported. (Jenga!)
+
+The problem is more general than C<elsif> (although the C<elsif> case is the
+most common and the most confusing). Ideally this code
+
+ use warnings;
+ my $undef;
+
+ my $a = $undef + 1;
+ my $b
+ = $undef
+ + 1;
+
+would produce this output
+
+ Use of uninitialized value $undef in addition (+) at wrong.pl line 4.
+ Use of uninitialized value $undef in addition (+) at wrong.pl line 7.
+
+(rather than lines 4 and 5), but this would seem to require every OP to carry
+(at least) line number information.
+
+What might work is to have an optional line number in memory just before the
+BASEOP structure, with a flag bit in the op to say whether it's present.
+Initially during compile every OP would carry its line number. Then add a late
+pass to the optimiser (potentially combined with L</repack the optree>) which
+looks at the two ops on every edge of the graph of the execution path. If
+the line number changes, flags the destination OP with this information.
+Once all paths are traced, replace every op with the flag with a
+nextstate-light op (that just updates C<PL_curcop>), which in turn then passes
+control on to the true op. All ops would then be replaced by variants that
+do not store the line number. (Which, logically, why it would work best in
+conjunction with L</repack the optree>, as that is already copying/reallocating
+all the OPs)
+
+(Although I should note that we're not certain that doing this for the general
+case is worth it)
+
+=head2 optimize tail-calls
+
+Tail-calls present an opportunity for broadly applicable optimization;
+anywhere that C<< return foo(...) >> is called, the outer return can
+be replaced by a goto, and foo will return directly to the outer
+caller, saving (conservatively) 25% of perl's call&return cost, which
+is relatively higher than in C. The scheme language is known to do
+this heavily. B::Concise provides good insight into where this
+optimization is possible, ie anywhere entersub,leavesub op-sequence
+occurs.
+
+ perl -MO=Concise,-exec,a,b,-main -e 'sub a{ 1 }; sub b {a()}; b(2)'
+
+Bottom line on this is probably a new pp_tailcall function which
+combines the code in pp_entersub, pp_leavesub. This should probably
+be done 1st in XS, and using B::Generate to patch the new OP into the
+optrees.
=head1 Big projects
Tasks that will get your name mentioned in the description of the "Highlights
-of 5.10"
+of 5.12"
=head2 make ithreads more robust
One bit would be to write the missing code in sv.c:Perl_dirp_dup.
+Fix Perl_sv_dup, et al so that threads can return objects.
+
=head2 iCOW
Sarathy and Arthur have a proposal for an improved Copy On Write which
This will allow the use of a regex from inside (?{ }), (??{ }) and
(?(?{ })|) constructs.
+
+=head2 Add class set operations to regexp engine
+
+Apparently these are quite useful. Anyway, Jeffery Friedl wants them.
+
+demerphq has this on his todo list, but right at the bottom.
+
+
+=head1 Tasks for microperl
+
+
+[ Each and every one of these may be obsolete, but they were listed
+ in the old Todo.micro file]
+
+
+=head2 make creating uconfig.sh automatic
+
+=head2 make creating Makefile.micro automatic
+
+=head2 do away with fork/exec/wait?
+
+(system, popen should be enough?)
+
+=head2 some of the uconfig.sh really needs to be probed (using cc) in buildtime:
+
+(uConfigure? :-) native datatype widths and endianness come to mind
+
projects / p5sagit/p5-mst-13.2.git / blobdiff