Document what's still to be done on the regular expression

[p5sagit/p5-mst-13.2.git] / pod / perltodo.pod

=head1 NAME

perltodo - Perl TO-DO List

=head1 DESCRIPTION

This is a list of wishes for Perl.  Send updates to
I<perl5-porters@perl.org>.  If you want to work on any of these
projects, be sure to check the perl5-porters archives for past ideas,
flames, and propaganda.  This will save you time and also prevent you
from implementing something that Larry has already vetoed.  One set
of archives may be found at:

    http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/

=head1 To do during 5.6.x

=head2 Support for I/O disciplines

C<perlio> provides this, but the interface could be a lot more
straightforward.

=head2 Autoload bytes.pm

When the lexer sees, for instance, C<bytes::length>, it should
automatically load the C<bytes> pragma.

=head2 Make "\u{XXXX}" et al work

Danger, Will Robinson! Discussing the semantics of C<"\x{F00}">,
C<"\xF00"> and C<"\U{F00}"> on P5P I<will> lead to a long and boring
flamewar.

=head2 Create a char *sv_pvprintify(sv, STRLEN *lenp, UV flags)

For displaying PVs with control characters, embedded nulls, and Unicode.
This would be useful for printing warnings, or data and regex dumping,
not_a_number(), and so on.

Requirements: should handle both byte and UTF8 strings.  isPRINT()
characters printed as-is, character less than 256 as \xHH, Unicode
characters as \x{HHH}.  Don't assume ASCII-like, either, get somebody
on EBCDIC to test the output.

Possible options, controlled by the flags:
- whitespace (other than ' ' of isPRINT()) printed as-is
- use isPRINT_LC() instead of isPRINT()
- print control characters like this: "\cA"
- print control characters like this: "^A"
- non-PRINTables printed as '.' instead of \xHH
- use \OOO instead of \xHH
- use the C/Perl-metacharacters like \n, \t
- have a maximum length for the produced string (read it from *lenp)
- append a "..." to the produced string if the maximum length is exceeded
- really fancy: print unicode characters as \N{...}

=head2 Overloadable regex assertions

This may or may not be possible with the current regular expression
engine. The idea is that, for instance, C<\b> needs to be
algorithmically computed if you're dealing with Thai text. Hence, the
B<\b> assertion wants to be overloaded by a function.

=head2 Unicode

=over 4

=item *

    Case Mappings? http://www.unicode.org/unicode/reports/tr21/

=item *

They have some tricks Perl doesn't yet implement like character
class subtraction.

	http://www.unicode.org/unicode/reports/tr18/

=back

See L<perlunicode/UNICODE REGULAR EXPRESSION SUPPORT LEVEL> for what's
there and what's missing.

=head2 use Thread for iThreads

Artur Bergman's C<iThreads> module is a start on this, but needs to
be more mature.

=head2 make perl_clone optionally clone ops

So that pseudoforking, mod_perl, iThreads and nvi will work properly
(but not as efficiently) until the regex engine is fixed to be threadsafe.

=head2 Work out exit/die semantics for threads

=head2 Typed lexicals for compiler

=head2 Compiler workarounds for Win32

=head2 AUTOLOADing in the compiler

=head2 Fixing comppadlist when compiling

=head2 Cleaning up exported namespace

=head2 Complete signal handling

Add C<PERL_ASYNC_CHECK> to opcodes which loop; replace C<sigsetjmp> with
C<sigjmp>; check C<wait> for signal safety.

=head2 Out-of-source builds

This was done for 5.6.0, but needs reworking for 5.7.x

=head2 POSIX realtime support

POSIX 1003.1 1996 Edition support--realtime stuff: POSIX semaphores,
message queues, shared memory, realtime clocks, timers, signals (the
metaconfig units mostly already exist for these)

=head2 UNIX98 support

Reader-writer locks, realtime/asynchronous IO

=head2 IPv6 Support

There are non-core modules, such as C<Net::IPv6>, but these will need
integrating when IPv6 actually starts to really happen.  See RFC 2292
and RFC 2553.

=head2 Long double conversion

Floating point formatting is still causing some weird test failures.

=head2 Locales

Locales and Unicode interact with each other in unpleasant ways.
One possible solution would be to adopt/support ICU:

	http://oss.software.ibm.com/developerworks/opensource/icu/project/

=head2 Thread-safe regexes

The regular expression engine is currently non-threadsafe.

=head2 Arithmetic on non-Arabic numerals

C<[1234567890]> aren't the only numerals any more.

=head2 POSIX Unicode character classes

([=a=] for equivalance classes, [.ch.] for collation.)
These are dependent on Unicode normalization and collation.

=head2 Factoring out common suffices/prefices in regexps (trie optimization)

Currently, the user has to optimize C<foo|far> and C<foo|goo> into
C<f(?:oo|ar)> and C<[fg]oo> by hand; this could be done automatically.

=head2 Security audit shipped utilities

All the code we ship with Perl needs to be sensible about temporary file
handling, locking, input validation, and so on.

=head2 Custom opcodes

Have a way to introduce user-defined opcodes without the subroutine call
overhead of an XSUB; the user should be able to create PP code. Simon
Cozens has some ideas on this.

=head2 spawnvp() on Win32

Win32 has problems spawning processes, particularly when the arguments
to the child process contain spaces, quotes or tab characters.

=head2 DLL Versioning

Windows needs a way to know what version of a XS or C<libperl> DLL it's
loading.

=head2 Introduce @( and @)

C<$(> may return "foo bar baz". Unfortunately, since groups can
theoretically have spaces in their names, this could be one, two or
three groups.

=head2 Floating point handling

C<NaN> and C<inf> support is particularly troublesome.
(fp_classify(), fp_class(), fp_class_d(), class(), isinf(),
isfinite(), finite(), isnormal(), unordered(), <ieeefp.h>,
<fp_class.h> (there are metaconfig units for all these) (I think),
fp_setmask(), fp_getmask(), fp_setround(), fp_getround()
(no metaconfig units yet for these).  Don't forget finitel(), fp_classl(),
fp_class_l(), (yes, both do, unfortunately, exist), and unorderedl().)

As of Perl 5.6.1 is a Perl macro, Perl_isnan().

=head2 IV/UV preservation

Nicholas Clark has done a lot of work on this, but work is continuing.
C<+>, C<-> and C<*> work, but guards need to be in place for C<%>, C</>,
C<&>, C<oct>, C<hex> and C<pack>.

=head2 Replace pod2html with something using Pod::Parser

The CPAN module C<Malik::Pod::Html> may be a more suitable basis for a
C<pod2html> convertor; the current one duplicates the functionality
abstracted in C<Pod::Parser>, which makes updating the POD language
difficult.

=head2 Automate module testing on CPAN

When a new Perl is being beta tested, porters have to manually grab
their favourite CPAN modules and test them - this should be done
automatically.

=head2 sendmsg and recvmsg

We have all the other BSD socket functions but these. There are
metaconfig units for these functions which can be added. To avoid these
being new opcodes, a solution similar to the way C<sockatmark> was added
would be preferable. (Autoload the C<IO::whatever> module.)

=head2 Rewrite perlre documentation

The new-style patterns need full documentation, and the whole document
needs to be a lot clearer.

=head2 Convert example code to IO::Handle filehandles

=head2 Document Win32 choices

=head2 Check new modules

=head2 Make roffitall find pods and libs itself

Simon Cozens has done some work on this but it needs a rethink.

=head1 To do at some point

These are ideas that have been regularly tossed around, that most
people believe should be done maybe during 5.8.x

=head2 Remove regular expression recursion

Because the regular expression engine is recursive, badly designed
expressions can lead to lots of recursion filling up the stack. Ilya
claims that it is easy to convert the engine to being iterative, but
this has still not yet been done. There may be a regular expression
engine hit squad meeting at TPC5.

=head2 Memory leaks after failed eval

Perl will leak memory if you C<eval "hlagh hlagh hlagh hlagh">. This is
partially because it attempts to build up an op tree for that code and
doesn't properly free it. The same goes for non-syntactically-correct
regular expressions. Hugo looked into this, but decided it needed a
mark-and-sweep GC implementation. 

Alan notes that: The basic idea was to extend the parser token stack
(C<YYSTYPE>) to include a type field so we knew what sort of thing each
element of the stack was.  The F<<perly.c> code would then have to be
postprocessed to record the type of each entry on the stack as it was
created, and the parser patched so that it could unroll the stack
properly on error.

This is possible to do, but would be pretty messy to implement, as it
would rely on even more sed hackery in F<perly.fixer>.

=head2 pack "(stuff)*"

That's to say, C<pack "(sI)40"> would be the same as C<pack "sI"x40>

=head2 bitfields in pack

=head2 Cross compilation

Make Perl buildable with a cross-compiler. This will play havoc with
Configure, which needs to how how the target system will respond to
its tests; maybe C<microperl> will be a good starting point here.
(Indeed, Bart Schuller reports that he compiled up C<microperl> for
the Agenda PDA and it works fine.)  A really big spanner in the works
is the bootstrapping build process of Perl: if the filesystem the
target systems sees is not the same what the build host sees, various
input, output, and (Perl) library files need to be copied back and forth.

As of 5.8.0 Configure mostly works for cross-compilation
(used successfully for iPAQ Linux), miniperl gets built,
but then building DynaLoader (and other extensions) fails
since MakeMaker knows nothing of cross-compilation.
(See INSTALL/Cross-compilation for the state of things.)

=head2 Perl preprocessor / macros

Source filters help with this, but do not get us all the way. For
instance, it should be possible to implement the C<??> operator somehow;
source filters don't (quite) cut it.

=head2 Perl lexer in Perl

Damian Conway is planning to work on this, but it hasn't happened yet.

=head2 Using POSIX calls internally

When faced with a BSD vs. SySV -style interface to some library or
system function, perl's roots show in that it typically prefers the BSD
interface (but falls back to the SysV one).  One example is getpgrp().
Other examples include C<memcpy> vs. C<bcopy>.  There are others, mostly in
F<<pp_sys.c>.

Mostly, this item is a suggestion for which way to start a journey into
an C<#ifdef> forest.  It is not primarily a suggestion to eliminate any of
the C<#ifdef> forests.

POSIX calls are perhaps more likely to be portable to unexpected
architectures. They are also perhaps more likely to be actively
maintained by a current vendor.  They are also perhaps more likely to be
available in thread-safe versions, if appropriate.

=head2 -i rename file when changed

It's only necessary to rename a file when inplace editing when the file
has changed. Detecting a change is perhaps the difficult bit.

=head2 All ARGV input should act like E<lt>E<gt>

=head2 Support for rerunning debugger

There should be a way of restarting the debugger on demand.

=head2 Test Suite for the Debugger

The debugger is a complex piece of software and fixing something
here may inadvertently break something else over there.  To tame
this chaotic behaviour, a test suite is necessary. 

=head2 my sub foo { }

The basic principle is sound, but there are problems with the semantics
of self-referential and mutually referential lexical subs: how to
declare the subs?

=head2 One-pass global destruction

Sweeping away all the allocated memory in one go is a laudable goal, but
it's difficult and in most cases, it's easier to let the memory get
freed by exiting.

=head2 Rewrite regexp parser

There has been talk recently of rewriting the regular expression parser
to produce an optree instead of a chain of opcodes; it's unclear whether
or not this would be a win. 

=head2 Cache recently used regexps

This is to speed up 

    for my $re (@regexps) {
        $matched++ if /$re/
    }

C<qr//> already gives us a way of saving compiled regexps, but it should
be done automatically.

=head2 Re-entrant functions

Add configure probes for C<_r> forms of system calls and fit them to the
core. Unfortunately, calling conventions for these functions and not
standardised.

=head2 Cross-compilation support

Bart Schuller reports that using C<microperl> and a cross-compiler, he
got Perl working on the Agenda PDA. However, one cannot build a full
Perl because Configure needs to get the results for the target platform,
for the host.

=head2 Bit-shifting bitvectors

Given:

    vec($v, 1000, 1) = 1;

One should be able to do

    $v <<= 1;

and have the 999'th bit set.

Currently if you try with shift bitvectors you shift the NV/UV, instead
of the bits in the PV.  Not very logical.

=head2 debugger pragma

The debugger is implemented in Perl in F<perl5db.pl>; turning it into a
pragma should be easy, but making it work lexically might be more
difficult. Fiddling with C<$^P> would be necessary.

=head2 use less pragma

Identify areas where speed/memory tradeoffs can be made and have a hint
to switch between them.

=head2 switch structures

Although we have C<Switch.pm> in core, Larry points to the dormant
C<nswitch> and C<cswitch> ops in F<pp.c>; using these opcodes would be
much faster.

=head2 Cache eval tree

=head2 rcatmaybe

=head2 Shrink opcode tables

=head2 Optimize away @_

Look at the "reification" code in C<av.c>

=head2 Prototypes versus indirect objects

Currently, indirect object syntax bypasses prototype checks.

=head2 Install HMTL

HTML versions of the documentation need to be installed by default; a
call to C<installhtml> from C<installperl> may be all that's necessary.

=head2 Prototype method calls

=head2 Return context prototype declarations

=head2 magic_setisa

=head2 Garbage collection

There have been persistent mumblings about putting a mark-and-sweep
garbage detector into Perl; Alan Burlison has some ideas about this.

=head2 IO tutorial

Mark-Jason Dominus has the beginnings of one of these.

=head2 pack/unpack tutorial

Simon Cozens has the beginnings of one of these.

=head2 Rewrite perldoc

There are a few suggestions for what to do with C<perldoc>: maybe a
full-text search, an index function, locating pages on a particular
high-level subject, and so on.

=head2 Install .3p manpages

This is a bone of contention; we can create C<.3p> manpages for each
built-in function, but should we install them by default? Tcl does this,
and it clutters up C<apropos>.

=head2 Unicode tutorial

Simon Cozens promises to do this before he gets old.

=head2 Update POSIX.pm for 1003.1-2

=head2 Retargetable installation

Allow C<@INC> to be changed after Perl is built.

=head2 POSIX emulation on non-POSIX systems

Make C<POSIX.pm> behave as POSIXly as possible everywhere, meaning we
have to implement POSIX equivalents for some functions if necessary.

=head2 Rename Win32 headers

=head2 Finish off lvalue functions

They don't work in the debugger, and they don't work for list or hash
slices.

=head2 Update sprintf documentation

Hugo van der Sanden plans to look at this.

=head2 Use fchown/fchmod internally

This has been done in places, but needs a thorough code review.
Also fchdir is available in some platforms.

=head1 Vague ideas

Ideas which have been discussed, and which may or may not happen.

=head2 ref() in list context

It's unclear what this should do or how to do it without breaking old
code.

=head2 Make tr/// return histogram of characters in list context

There is a patch for this, but it may require Unicodification.

=head2 Compile to real threaded code

=head2 Structured types

=head2 Modifiable $1 et al.

    ($x = "elephant") =~ /e(ph)/;
    $1 = "g"; # $x = "elegant"

What happens if there are multiple (nested?) brackets? What if the
string changes between the match and the assignment?

=head2 Procedural interfaces for IO::*, etc.

Some core modules have been accused of being overly-OO. Adding
procedural interfaces could demystify them.

=head2 RPC modules

=head2 Attach/detach debugger from running program

With C<gdb>, you can attach the debugger to a running program if you
pass the process ID. It would be good to do this with the Perl debugger
on a running Perl program, although I'm not sure how it would be done.

=head2 Alternative RE syntax module

    use Regex::Newbie;
    $re = Regex::Newbie->new
                       ->start
                       ->match("foo")
                       ->repeat(Regex::Newbie->class("char"),3)
                       ->end;
    /$re/;

=head2 GUI::Native

A non-core module that would use "native" GUI to create graphical
applications.

=head2 foreach(reverse ...)

Currently

    foreach (reverse @_) { ... }

puts C<@_> on the stack, reverses it putting the reversed version on the
stack, then iterates forwards. Instead, it could be special-cased to put
C<@_> on the stack then iterate backwards.

=head2 Constant function cache

=head2 Approximate regular expression matching

=head1 Ongoing

These items B<always> need doing:

=head2 Update guts documentation

Simon Cozens tries to do this when possible, and contributions to the
C<perlapi> documentation is welcome.

=head2 Add more tests

Michael Schwern will donate $500 to Yet Another Society when all core
modules have tests.

=head2 Update auxiliary tools

The code we ship with Perl should look like good Perl 5.

=head1 Recently done things

These are things which have been on the todo lists in previous releases
but have recently been completed.

=head2 Safe signal handling

A new signal model went into 5.7.1 without much fanfare. Operations and
C<malloc>s are no longer interrupted by signals, which are handled
between opcodes. This means that C<PERL_ASYNC_CHECK> now actually does
something. However, there are still a few things that need to be done.

=head2 Tie Modules

Modules which implement arrays in terms of strings, substrings or files
can be found on the CPAN.

=head2 gettimeofday

C<Time::Hires> has been integrated into the core.

=head2 setitimer and getimiter

Adding C<Time::Hires> got us this too.

=head2 Testing __DIE__ hook

Tests have been added.

=head2 CPP equivalent in Perl

A C Yardley will probably have done this by the time you can read this.
This allows for a generalization of the C constant detection used in
building C<Errno.pm>.

=head2 Explicit switch statements

C<Switch.pm> has been integrated into the core to give you all manner of
C<switch...case> semantics.

=head2 autocroak

This is C<Fatal.pm>.

=head2 UTF/EBCDIC

Nick Ing-Simmons has made UTF-EBCDIC (UTR13) work with Perl.

    EBCDIC?        http://www.unicode.org/unicode/reports/tr16/

=head2 UTF Regexes

Although there are probably some small bugs to be rooted out, Jarkko
Hietaniemi has made regular expressions polymorphic between bytes and
characters.

=head2 perlcc to produce executable

C<perlcc> was recently rewritten, and can now produce standalone
executables.

=head2 END blocks saved in compiled output

=head2 Secure temporary file module

Tim Jenness' C<File::Temp> is now in core.

=head2 Integrate Time::HiRes

This module is now part of core.

=head2 Turn Cwd into XS

Benjamin Sugars has done this.

=head2 Mmap for input

Nick Ing-Simmons' C<perlio> supports an C<mmap> IO method.

=head2 Byte to/from UTF8 and UTF8 to/from local conversion

C<Encode> provides this.

=head2 Add sockatmark support

Added in 5.7.1

=head2 Mailing list archives

http://lists.perl.org/, http://archive.develooper.com/

=head2 Bug tracking

Richard Foley has written the bug tracking system at http://bugs.perl.org/

=head2 Integrate MacPerl

Chris Nandor and Matthias Neeracher have integrated the MacPerl changes
into 5.6.0.

=head2 Web "nerve center" for Perl

http://use.perl.org/ is what you're looking for.

=head2 Regular expression tutorial

C<perlretut>, provided by Mark Kvale.

=head2 Debugging Tutorial

C<perldebtut>, written by Richard Foley.

=head2 Integrate new modules

Jarkko has been integrating madly into 5.7.x

=head2 Integrate profiler

C<Devel::DProf> is now a core module.

=head2 Y2K error detection

There's a configure option to detect unsafe concatenation with "19", and
a CPAN module. (C<D'oh::Year>)

=head2 Regular expression debugger

While not part of core, Mark-Jason Dominus has written C<Rx> and has
also come up with a generalised strategy for regular expression
debugging.

=head2 POD checker

That's, uh, F<podchecker>

=head2 "Dynamic" lexicals

=head2 Cache precompiled modules

=head1 Deprecated Wishes

These are items which used to be in the todo file, but have been
deprecated for some reason.

=head2 Loop control on do{}

This would break old code; use C<do{{ }}> instead.

=head2 Lexically scoped typeglobs

Not needed now we have lexical IO handles.

=head2 format BOTTOM

=head2 report HANDLE

Damian Conway's text formatting modules seem to be the Way To Go.

=head2 Generalised want()/caller())

=head2 Named prototypes

These both seem to be delayed until Perl 6.

=head2 Built-in globbing

The C<File::Glob> module has been used to replace the C<glob> function.

=head2 Regression tests for suidperl

C<suidperl> is deprecated in favour of common sense.

=head2 Cached hash values

We have shared hash keys, which perform the same job.

=head2 Add compression modules

The compression modules are a little heavy; meanwhile, Nick Clark is
working on experimental pragmata to do transparent decompression on
input.

=head2 Reorganise documentation into tutorials/references

Could not get consensus on P5P about this.

=head2 Remove distinction between functions and operators

Caution: highly flammable.

=head2 Make XS easier to use

Use C<Inline> instead, or SWIG.

=head2 Make embedding easier to use

Use C<Inline::CPR>.

=head2 man for perl

See the Perl Power Tools. (http://language.perl.com/ppt/)

=head2 my $Package::variable

Use C<our> instead.

=head2 "or" tests defined, not truth

Suggesting this on P5P B<will> cause a boring and interminable flamewar.

=head2 "class"-based lexicals

Use flyweight objects, secure hashes or, dare I say it, pseudo-hashes instead.
(Or whatever will replace pseudohashes in 5.10.)

=head2 byteperl

C<ByteLoader> covers this.

=head2 Lazy evaluation / tail recursion removal

C<List::Util> gives first() (a short-circuiting grep); tail recursion
removal is done manually, with C<goto &whoami;>. (However, MJD has
found that C<goto &whoami> introduces a performance penalty, so maybe
there should be a way to do this after all: C<sub foo {START: ... goto
START;> is better.)

=head2 Make "use utf8" the default

Because of backward compatibility this is difficult: scripts could not
contain B<any legacy eight-bit data> (like Latin-1) anymore, even in
string literals or pod.  Also would introduce a measurable slowdown of
at least few percentages since all regular expression operations would
be done in full UTF-8.  But if you want to try this, add
-DUSE_UTF8_SCRIPTS to your compilation flags.

=head2 Unicode collation and normalization

The Unicode::Collate and Unicode::Normalize modules
by SADAHIRO Tomoyuki have been included since 5.8.0.

    Collation?     http://www.unicode.org/unicode/reports/tr10/
    Normalization? http://www.unicode.org/unicode/reports/tr15/

=cut