Re: [perl #45605] Regexp failure with utf8-flagged string and byte-flagged pattern

[p5sagit/p5-mst-13.2.git] / README.cygwin

If you read this file _as_is_, just ignore the funny characters you
see. It is written in the POD format (see pod/perlpod.pod) which is
specially designed to be readable as is.

=head1 NAME

README.cygwin - Perl for Cygwin

=head1 SYNOPSIS

This document will help you configure, make, test and install Perl
on Cygwin.  This document also describes features of Cygwin that will
affect how Perl behaves at runtime.

B<NOTE:> There are pre-built Perl packages available for Cygwin and a
version of Perl is provided in the normal Cygwin install.  If you do
not need to customize the configuration, consider using one of those
packages.


=head1 PREREQUISITES FOR COMPILING PERL ON CYGWIN

=head2 Cygwin = GNU+Cygnus+Windows (Don't leave UNIX without it)

The Cygwin tools are ports of the popular GNU development tools for Win32
platforms.  They run thanks to the Cygwin library which provides the UNIX
system calls and environment these programs expect.  More information
about this project can be found at:

  F<http://www.cygwin.com/>

A recent net or commercial release of Cygwin is required.

At the time this document was last updated, Cygwin 1.5.24 was current.


=head2 Cygwin Configuration

While building Perl some changes may be necessary to your Cygwin setup so
that Perl builds cleanly.  These changes are B<not> required for normal
Perl usage.

B<NOTE:> The binaries that are built will run on all Win32 versions.
They do not depend on your host system (Win9x/WinME, WinNT/Win2K)
or your Cygwin configuration (I<ntea>, I<ntsec>, binary/text mounts).
The only dependencies come from hard-coded pathnames like C</usr/local>.
However, your host system and Cygwin configuration will affect Perl's
runtime behavior (see L</"TEST">).

=over 4

=item * C<PATH>

Set the C<PATH> environment variable so that Configure finds the Cygwin
versions of programs.  Any Windows directories should be removed or
moved to the end of your C<PATH>.

=item * I<nroff>

If you do not have I<nroff> (which is part of the I<groff> package),
Configure will B<not> prompt you to install I<man> pages.

=item * Permissions

On WinNT with either the I<ntea> or I<ntsec> C<CYGWIN> settings, directory
and file permissions may not be set correctly.  Since the build process
creates directories and files, to be safe you may want to run a
C<chmod -R +w *> on the entire Perl source tree.

Also, it is a well known WinNT "feature" that files created by a login
that is a member of the I<Administrators> group will be owned by the
I<Administrators> group.  Depending on your umask, you may find that you
can not write to files that you just created (because you are no longer
the owner).  When using the I<ntsec> C<CYGWIN> setting, this is not an
issue because it "corrects" the ownership to what you would expect on
a UNIX system.

=back

=head1 CONFIGURE PERL ON CYGWIN

The default options gathered by Configure with the assistance of
F<hints/cygwin.sh> will build a Perl that supports dynamic loading
(which requires a shared F<libperl.dll>).

This will run Configure and keep a record:

  ./Configure 2>&1 | tee log.configure

If you are willing to accept all the defaults run Configure with B<-de>.
However, several useful customizations are available.

=head2 Stripping Perl Binaries on Cygwin

It is possible to strip the EXEs and DLLs created by the build process.
The resulting binaries will be significantly smaller.  If you want the
binaries to be stripped, you can either add a B<-s> option when Configure
prompts you,

  Any additional ld flags (NOT including libraries)? [none] -s
  Any special flags to pass to gcc to use dynamic linking? [none] -s
  Any special flags to pass to ld2 to create a dynamically loaded library?
  [none] -s

or you can edit F<hints/cygwin.sh> and uncomment the relevant variables
near the end of the file.

=head2 Optional Libraries for Perl on Cygwin

Several Perl functions and modules depend on the existence of
some optional libraries.  Configure will find them if they are
installed in one of the directories listed as being used for library
searches.  Pre-built packages for most of these are available from
the Cygwin installer.

=over 4

=item * C<-lcrypt>

The crypt package distributed with Cygwin is a Linux compatible 56-bit
DES crypt port by Corinna Vinschen.

Alternatively, the crypt libraries in GNU libc have been ported to Cygwin.

The DES based Ultra Fast Crypt port was done by Alexey Truhan:

  ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/porters/Okhapkin_Sergey/cw32crypt-dist-0.tgz

NOTE: There are various export restrictions on DES implementations,
see the glibc README for more details.

The MD5 port was done by Andy Piper:

  ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/porters/Okhapkin_Sergey/libcrypt.tgz

=item * C<-lgdbm> (C<use GDBM_File>)

GDBM is available for Cygwin.

NOTE: The GDBM library only works on NTFS partitions.

=item * C<-ldb> (C<use DB_File>)

BerkeleyDB is available for Cygwin.

NOTE: The BerkeleyDB library only completely works on NTFS partitions 
and db-4.3 is flawed.

=item * C<cygserver> (C<use IPC::SysV>)

A port of SysV IPC is available for Cygwin.

NOTE: This has B<not> been extensively tested.  In particular,
C<d_semctl_semun> is undefined because it fails a Configure test
and on Win9x the I<shm*()> functions seem to hang.  It also creates
a compile time dependency because F<perl.h> includes F<<sys/ipc.h>>
and F<<sys/sem.h>> (which will be required in the future when compiling
CPAN modules). CURRENTLY NOT SUPPORTED!

=item * C<-lutil>

Included with the standard Cygwin netrelease is the inetutils package 
which includes libutil.a.

=back

=head2 Configure-time Options for Perl on Cygwin

The F<INSTALL> document describes several Configure-time options.  Some of
these will work with Cygwin, others are not yet possible.  Also, some of
these are experimental.  You can either select an option when Configure
prompts you or you can define (undefine) symbols on the command line.

=over 4

=item * C<-Uusedl>

Undefining this symbol forces Perl to be compiled statically.

=item * C<-Uusemymalloc>

By default Perl uses the C<malloc()> included with the Perl source.  If you
want to force Perl to build with the system C<malloc()> undefine this symbol.

=item * C<-Uuseperlio>

Undefining this symbol disables the PerlIO abstraction.  PerlIO is now the
default; it is not recommended to disable PerlIO.

=item * C<-Dusemultiplicity>

Multiplicity is required when embedding Perl in a C program and using
more than one interpreter instance.  This works with the Cygwin port.

=item * C<-Duse64bitint>

By default Perl uses 32 bit integers.  If you want to use larger 64
bit integers, define this symbol.

=item * C<-Duselongdouble>

I<gcc> supports long doubles (12 bytes).  However, several additional
long double math functions are necessary to use them within Perl
(I<{atan2, cos, exp, floor, fmod, frexp, isnan, log, modf, pow, sin, sqrt}l, 
strtold>).
These are B<not> yet available with Cygwin.

=item * C<-Dusethreads>

POSIX threads are implemented in Cygwin, define this symbol if you want
a threaded perl.

=item * C<-Duselargefiles>

Cygwin uses 64-bit integers for internal size and position calculations, 
this will be correctly detected and defined by Configure.

=item * C<-Dmksymlinks>

Use this to build perl outside of the source tree.  This works with Cygwin.
Details can be found in the F<INSTALL> document.  This is the recommended 
way to build perl from sources.

=back

=head2 Suspicious Warnings on Cygwin

You may see some messages during Configure that seem suspicious.

=over 4

=item * Win9x and C<d_eofnblk>

Win9x does not correctly report C<EOF> with a non-blocking read on a
closed pipe.  You will see the following messages:

  But it also returns -1 to signal EOF, so be careful!
  WARNING: you can't distinguish between EOF and no data!

  *** WHOA THERE!!! ***
      The recommended value for $d_eofnblk on this machine was "define"!
      Keep the recommended value? [y]

At least for consistency with WinNT, you should keep the recommended
value.

=item * Compiler/Preprocessor defines

The following error occurs because of the Cygwin C<#define> of
C<_LONG_DOUBLE>:

  Guessing which symbols your C compiler and preprocessor define...
  try.c:<line#>: missing binary operator

This failure does not seem to cause any problems.  With older gcc
versions, "parse error" is reported instead of "missing binary
operator".

=back

=head1 MAKE ON CYGWIN

Simply run I<make> and wait:

  make 2>&1 | tee log.make

=head1 TEST ON CYGWIN

There are two steps to running the test suite:

  make test 2>&1 | tee log.make-test

  cd t; ./perl harness 2>&1 | tee ../log.harness

The same tests are run both times, but more information is provided when
running as C<./perl harness>.

Test results vary depending on your host system and your Cygwin
configuration.  If a test can pass in some Cygwin setup, it is always
attempted and explainable test failures are documented.  It is possible
for Perl to pass all the tests, but it is more likely that some tests
will fail for one of the reasons listed below.

=head2 File Permissions on Cygwin

UNIX file permissions are based on sets of mode bits for
{read,write,execute} for each {user,group,other}.  By default Cygwin
only tracks the Win32 read-only attribute represented as the UNIX file
user write bit (files are always readable, files are executable if they
have a F<.{com,bat,exe}> extension or begin with C<#!>, directories are
always readable and executable).  On WinNT with the I<ntea> C<CYGWIN>
setting, the additional mode bits are stored as extended file attributes.
On WinNT with the default I<ntsec> C<CYGWIN> setting, permissions use the 
standard WinNT security descriptors and access control lists. Without one of
these options, these tests will fail (listing not updated yet):

  Failed Test           List of failed
  ------------------------------------
  io/fs.t               5, 7, 9-10
  lib/anydbm.t          2
  lib/db-btree.t        20
  lib/db-hash.t         16
  lib/db-recno.t        18
  lib/gdbm.t            2
  lib/ndbm.t            2
  lib/odbm.t            2
  lib/sdbm.t            2
  op/stat.t             9, 20 (.tmp not an executable extension)

=head2 NDBM_File and ODBM_File do not work on FAT filesystems

Do not use NDBM_File or ODBM_File on FAT filesystem.  They can be
built on a FAT filesystem, but many tests will fail:

 ../ext/NDBM_File/ndbm.t       13  3328    71   59  83.10%  1-2 4 16-71
 ../ext/ODBM_File/odbm.t      255 65280    ??   ??       %  ??
 ../lib/AnyDBM_File.t           2   512    12    2  16.67%  1 4
 ../lib/Memoize/t/errors.t      0   139    11    5  45.45%  7-11
 ../lib/Memoize/t/tie_ndbm.t   13  3328     4    4 100.00%  1-4
 run/fresh_perl.t                          97    1   1.03%  91

If you intend to run only on FAT (or if using AnyDBM_File on FAT),
run Configure with the -Ui_ndbm and -Ui_dbm options to prevent
NDBM_File and ODBM_File being built.

With NTFS (and no CYGWIN=nontsec), there should be no problems even if
perl was built on FAT.

=head2 C<fork()> failures in io_* tests

A C<fork()> failure may result in the following tests failing:

  ext/IO/lib/IO/t/io_multihomed.t
  ext/IO/lib/IO/t/io_sock.t
  ext/IO/lib/IO/t/io_unix.t

See comment on fork in L<Miscellaneous> below.

=head1 Specific features of the Cygwin port

=head2 Script Portability on Cygwin

Cygwin does an outstanding job of providing UNIX-like semantics on top of
Win32 systems.  However, in addition to the items noted above, there are
some differences that you should know about.  This is a very brief guide
to portability, more information can be found in the Cygwin documentation.

=over 4

=item * Pathnames

Cygwin pathnames can be separated by forward (F</>) or backward (F<\\>)
slashes.  They may also begin with drive letters (F<C:>) or Universal
Naming Codes (F<//UNC>).  DOS device names (F<aux>, F<con>, F<prn>,
F<com*>, F<lpt?>, F<nul>) are invalid as base filenames.  However, they
can be used in extensions (e.g., F<hello.aux>).  Names may contain all
printable characters except these:

  : * ? " < > |

File names are case insensitive, but case preserving.  A pathname that
contains a backslash or drive letter is a Win32 pathname (and not subject
to the translations applied to POSIX style pathnames).

For conversion we have C<Cygwin::win_to_posix_path()> and 
C<Cygwin::posix_to_win_path()>.

Pathnames may not contain Unicode characters. C<Cygwin> still uses the
ANSI API calls and no Unicode calls because of newlib deficiencies.
There's an unofficial unicode patch for cygwin at 
F<http://www.okisoft.co.jp/esc/utf8-cygwin/>

=item * Text/Binary

When a file is opened it is in either text or binary mode.  In text mode
a file is subject to CR/LF/Ctrl-Z translations.  With Cygwin, the default
mode for an C<open()> is determined by the mode of the mount that underlies
the file. See C<Cygwin::is_binmount()>. Perl provides a C<binmode()> function
to set binary mode on files that otherwise would be treated as text.
C<sysopen()> with the C<O_TEXT> flag sets text mode on files that otherwise
would be treated as binary:

    sysopen(FOO, "bar", O_WRONLY|O_CREAT|O_TEXT)

C<lseek()>, C<tell()> and C<sysseek()> only work with files opened in binary
mode.

The text/binary issue is covered at length in the Cygwin documentation.

=item * PerlIO

PerlIO overrides the default Cygwin Text/Binary behaviour.  A file will 
always treated as binary, regardless which mode of the mount it lives on,
just like it is in UNIX.  So CR/LF translation needs to be requested in 
either the C<open()> call like this:

  open(FH, ">:crlf", "out.txt");

which will do conversion from LF to CR/LF on the output, or in the 
environment settings (add this to your .bashrc):

  export PERLIO=crlf

which will pull in the crlf PerlIO layer which does LF -> CRLF conversion 
on every output generated by perl.

=item * F<.exe>

The Cygwin C<stat()>, C<lstat()> and C<readlink()> functions make the F<.exe>
extension transparent by looking for F<foo.exe> when you ask for F<foo>
(unless a F<foo> also exists).  Cygwin does not require a F<.exe>
extension, but I<gcc> adds it automatically when building a program.
However, when accessing an executable as a normal file (e.g., I<cp>
in a makefile) the F<.exe> is not transparent.  The I<install> included
with Cygwin automatically appends a F<.exe> when necessary.

=item * Cygwin vs. Windows process ids

Cygwin processes have their own pid, which is different from the
underlying windows pid.  Most posix compliant Proc functions expect
the cygwin pid, but several Win32::Process functions expect the
winpid. E.g. C<$$> is the cygwin pid of F</usr/bin/perl>, which is not
the winpid.  Use C<Cygwin::winpid_to_pid()> and C<Cygwin::winpid_to_pid()> 
to translate between them.

=item * Cygwin vs. Windows errors

Under Cygwin, $^E is the same as $!.  When using L<Win32 API Functions|Win32>,
use C<Win32::GetLastError()> to get the last Windows error.

=item * C<chown()>

On WinNT C<chown()> can change a file's user and group IDs.  On Win9x C<chown()>
is a no-op, although this is appropriate since there is no security model.

=item * Miscellaneous

File locking using the C<F_GETLK> command to C<fcntl()> is a stub that
returns C<ENOSYS>.

Win9x can not C<rename()> an open file (although WinNT can).

The Cygwin C<chroot()> implementation has holes (it can not restrict file
access by native Win32 programs).

Inplace editing C<perl -i> of files doesn't work without doing a backup 
of the file being edited C<perl -i.bak> because of windowish restrictions,
therefore Perl adds the suffix C<.bak> automatically if you use C<perl -i> 
without specifying a backup extension.

Using C<fork()> after loading multiple dlls may fail with an internal cygwin
error like the following:

  C:\CYGWIN\BIN\PERL.EXE: *** couldn't allocate memory 0x10000(4128768) for 'C:\CYGWIN\LIB\PERL5\5.6.1\CYGWIN-MULTI\AUTO\SOCKET\SOCKET.DLL' alignment, Win32 error 8

    200 [main] perl 377147 sync_with_child: child -395691(0xB8) died before initialization with status code 0x1
   1370 [main] perl 377147 sync_with_child: *** child state child loading dlls

Use the rebase utility to resolve the conflicting dll addresses.  The
rebase package is included in the Cygwin netrelease.  Use setup.exe from
F<http://www.cygwin.com/setup.exe> to install it and run rebaseall.

=back

=head2 Prebuilt methods:

=over 4

=item C<Cwd::cwd>

Returns the current working directory.

=item C<Cygwin::pid_to_winpid>

Translates a cygwin pid to the corresponding Windows pid (which may or
may not be the same).

=item C<Cygwin::winpid_to_pid>

Translates a Windows pid to the corresponding cygwin pid (if any).

=item C<Cygwin::win_to_posix_path>

Translates a Windows path to the corresponding cygwin path respecting
the current mount points. With a second non-null argument returns an
absolute path. Double-byte characters will not be translated.

=item C<Cygwin::posix_to_win_path>

Translates a cygwin path to the corresponding cygwin path respecting
the current mount points. With a second non-null argument returns an
absolute path. Double-byte characters will not be translated.

=item C<Cygwin::mount_table()>

Returns an array of [mnt_dir, mnt_fsname, mnt_type, mnt_opts].

  perl -e 'for $i (Cygwin::mount_table) {print join(" ",@$i),"\n";}'
  /bin c:\cygwin\bin system binmode,cygexec
  /usr/bin c:\cygwin\bin system binmode
  /usr/lib c:\cygwin\lib system binmode
  / c:\cygwin system binmode
  /cygdrive/c c: system binmode,noumount
  /cygdrive/d d: system binmode,noumount
  /cygdrive/e e: system binmode,noumount 

=item C<Cygwin::mount_flags>

Returns the mount type and flags for a specified mount point.
A comma-seperated string of mntent->mnt_type (always
"system" or "user"), then the mntent->mnt_opts, where
the first is always "binmode" or "textmode".

  system|user,binmode|textmode,exec,cygexec,cygdrive,mixed,
  notexec,managed,nosuid,devfs,proc,noumount

If the argument is "/cygdrive", just the volume mount settings are returned. 

User mounts override system mounts.

  $ perl -e 'print Cygwin::mount_flags "/usr/bin"'
  system,binmode,cygexec
  $ perl -e 'print Cygwin::mount_flags "/cygdrive"'
  binmode,cygdrive 

=item C<Cygwin::is_binmount>

Returns true if the given cygwin path is binary mounted, false if the
path is mounted in textmode.

=back

=head1 INSTALL PERL ON CYGWIN

This will install Perl, including I<man> pages.

  make install 2>&1 | tee log.make-install

NOTE: If C<STDERR> is redirected C<make install> will B<not> prompt
you to install I<perl> into F</usr/bin>.

You may need to be I<Administrator> to run C<make install>.  If you
are not, you must have write access to the directories in question.

Information on installing the Perl documentation in HTML format can be
found in the F<INSTALL> document.

=head1 MANIFEST ON CYGWIN

These are the files in the Perl release that contain references to Cygwin.
These very brief notes attempt to explain the reason for all conditional
code.  Hopefully, keeping this up to date will allow the Cygwin port to
be kept as clean as possible (listing not updated yet).

=over 4

=item Documentation

  INSTALL README.cygwin README.win32 MANIFEST
  Changes Changes5.005 Changes5.004 Changes5.6
  pod/perl.pod pod/perlport.pod pod/perlfaq3.pod
  pod/perldelta.pod pod/perl5004delta.pod pod/perl56delta.pod
  pod/perlhist.pod pod/perlmodlib.pod perl/buildtoc pod/perltoc.pod

=item Build, Configure, Make, Install

  cygwin/Makefile.SHs
  cygwin/ld2.in
  cygwin/perlld.in
  ext/IPC/SysV/hints/cygwin.pl
  ext/NDBM_File/hints/cygwin.pl
  ext/ODBM_File/hints/cygwin.pl
  hints/cygwin.sh
  Configure             - help finding hints from uname,
                          shared libperl required for dynamic loading
  Makefile.SH           - linklibperl
  Porting/patchls       - cygwin in port list
  installman            - man pages with :: translated to .
  installperl           - install dll/ld2/perlld, install to pods
  makedepend.SH         - uwinfix

=item Tests

  t/io/tell.t           - binmode
  t/lib/b.t             - ignore Cwd from os_extras
  t/lib/glob-basic.t    - Win32 directory list access differs from read mode
  t/op/magic.t          - $^X/symlink WORKAROUND, s/.exe//
  t/op/stat.t           - no /dev, skip Win32 ftCreationTime quirk
                          (cache manager sometimes preserves ctime of file
                          previously created and deleted), no -u (setuid)
  t/lib/cygwin.t	- builtin cygwin function tests

=item Compiled Perl Source

  EXTERN.h              - __declspec(dllimport)
  XSUB.h                - __declspec(dllexport)
  cygwin/cygwin.c       - os_extras (getcwd, spawn, and several Cygwin:: functions)
  perl.c                - os_extras
  perl.h                - binmode
  doio.c                - win9x can not rename a file when it is open
  pp_sys.c              - do not define h_errno, pp_system with spawn
  util.c                - use setenv

=item Compiled Module Source

  ext/POSIX/POSIX.xs    - tzname defined externally
  ext/SDBM_File/sdbm/pair.c
                        - EXTCONST needs to be redefined from EXTERN.h
  ext/SDBM_File/sdbm/sdbm.c
                        - binary open

=item Perl Modules/Scripts

  lib/Cwd.pm            - hook to internal Cwd::cwd
  lib/ExtUtils/MakeMaker.pm
                        - require MM_Cygwin.pm
  lib/ExtUtils/MM_Cygwin.pm
                        - canonpath, cflags, manifypods, perl_archive
  lib/File/Find.pm      - on remote drives stat() always sets st_nlink to 1
  lib/File/Spec/Cygwin.pm - case_tolerant
  lib/File/Spec/Unix.pm - preserve //unc
  lib/File/Temp.pm      - no directory sticky bit
  lib/perl5db.pl        - use stdin not /dev/tty
  utils/perldoc.PL      - version comment

=back

=head1 BUGS ON CYGWIN

Support for swapping real and effective user and group IDs is incomplete.
On WinNT Cygwin provides C<setuid()>, C<seteuid()>, C<setgid()> and C<setegid()>.
However, additional Cygwin calls for manipulating WinNT access tokens
and security contexts are required.

=head1 AUTHORS

Charles Wilson <cwilson@ece.gatech.edu>,
Eric Fifer <egf7@columbia.edu>,
alexander smishlajev <als@turnhere.com>,
Steven Morlock <newspost@morlock.net>,
Sebastien Barre <Sebastien.Barre@utc.fr>,
Teun Burgers <burgers@ecn.nl>,
Gerrit P. Haase <gp@familiehaase.de>,
Reini Urban <rurban@cpan.org>,
Jan Dubois <jand@activestate.com>.

=head1 HISTORY

Last updated: 2007-08-12