package Encode::Encoding;
# Base class for classes which implement encodings
use strict;
-our $VERSION = do { my @r = (q$Revision: 0.94 $ =~ /\d+/g); sprintf "%d."."%02d" x $#r, @r };
+our $VERSION = do { my @r = (q$Revision: 1.28 $ =~ /\d+/g); sprintf "%d."."%02d" x $#r, @r };
sub Define
{
my $canonical = shift;
$obj = bless { Name => $canonical },$obj unless ref $obj;
# warn "$canonical => $obj\n";
- Encode::define_encoding($obj, $canonical, @_);
+ Encode::define_encoding($obj, $canonical, @_);
}
sub name { shift->{'Name'} }
sub new_sequence { return $_[0] }
+sub perlio_ok { 0 }
+
+sub needs_lines { 0 }
+
sub DESTROY {}
1;
__PACKAGE__->Define(qw(myCanonical myAlias));
-=head 1 DESCRIPTION
+=head1 DESCRIPTION
As mentioned in L<Encode>, encodings are (in the current
implementation at least) defined by objects. The mapping of encoding
not actually loaded the encoding in question. This is because the
current "loading" process is all Perl and a bit slow.
-Once an encoding is loaded then value of the hash is object which
+Once an encoding is loaded, the value of the hash is the object which
implements the encoding. The object should provide the following
interface:
=item -E<gt>name
-Should return the string representing the canonical name of the encoding.
+MUST return the string representing the canonical name of the encoding.
=item -E<gt>new_sequence
This is a placeholder for encodings with state. It should return an
-object which implements this interface, all current implementations
+object which implements this interface. All current implementations
return the original object.
=item -E<gt>encode($string,$check)
-Should return the octet sequence representing I<$string>. If I<$check>
-is true it should modify I<$string> in place to remove the converted
-part (i.e. the whole string unless there is an error). If an error
-occurs it should return the octet sequence for the fragment of string
-that has been converted, and modify $string in-place to remove the
-converted part leaving it starting with the problem fragment.
+MUST return the octet sequence representing I<$string>.
+
+=over 2
+
+=item *
+
+If I<$check> is true, it SHOULD modify I<$string> in place to remove
+the converted part (i.e. the whole string unless there is an error).
+If perlio_ok() is true, SHOULD becomes MUST.
-If check is is false then C<encode> should make a "best effort" to
-convert the string - for example by using a replacement character.
+=item *
+
+If an error occurs, it SHOULD return the octet sequence for the
+fragment of string that has been converted and modify $string in-place
+to remove the converted part leaving it starting with the problem
+fragment. If perlio_ok() is true, SHOULD becomes MUST.
+
+=item *
+
+If I<$check> is is false then C<encode> MUST make a "best effort" to
+convert the string - for example, by using a replacement character.
+
+=back
=item -E<gt>decode($octets,$check)
-Should return the string that I<$octets> represents. If I<$check> is
-true it should modify I<$octets> in place to remove the converted part
-(i.e. the whole sequence unless there is an error). If an error
-occurs it should return the fragment of string that has been
-converted, and modify $octets in-place to remove the converted part
-leaving it starting with the problem fragment.
+MUST return the string that I<$octets> represents.
-If check is is false then C<decode> should make a "best effort" to
+=over 2
+
+=item *
+
+If I<$check> is true, it SHOULD modify I<$octets> in place to remove
+the converted part (i.e. the whole sequence unless there is an
+error). If perlio_ok() is true, SHOULD becomes MUST.
+
+=item *
+
+If an error occurs, it SHOULD return the fragment of string that has
+been converted and modify $octets in-place to remove the converted
+part leaving it starting with the problem fragment. If perlio_ok() is
+true, SHOULD becomes MUST.
+
+=item *
+
+If I<$check> is false then C<decode> should make a "best effort" to
convert the string - for example by using Unicode's "\x{FFFD}" as a
replacement character.
=back
-It should be noted that the check behaviour is different from the
+=item -E<gt>perlio_ok()
+
+If you want your encoding to work with PerlIO, you MUST define this
+method so that it returns 1 when PerlIO is enabled. Here is an
+example;
+
+ sub perlio_ok {
+ eval { require PerlIO::encoding };
+ if ($@){
+ return 0;
+ }else{
+ return 1;
+ }
+ }
+
+
+By default, this method is defined as follows;
+
+ sub perlio_ok { 0 }
+
+=item -E<gt>needs_lines()
+
+If your encoding can work with PerlIO but needs line buffering, you
+MUST define this method so it returns true. 7bit ISO-2022 encodings
+are one example that needs this. When this method is missing, false
+is assumed.
+
+=back
+
+It should be noted that the I<$check> behaviour is different from the
outer public API. The logic is that the "unchecked" case is useful
-when encoding is part of a stream which may be reporting errors
-(e.g. STDERR). In such cases it is desirable to get everything
+when the encoding is part of a stream which may be reporting errors
+(e.g. STDERR). In such cases, it is desirable to get everything
through somehow without causing additional errors which obscure the
-original one. Also the encoding is best placed to know what the
+original one. Also, the encoding is best placed to know what the
correct replacement character is, so if that is the desired behaviour
then letting low level code do it is the most efficient.
-In contrast if check is true, the scheme above allows the encoding to
-do as much as it can and tell layer above how much that was. What is
-lacking at present is a mechanism to report what went wrong. The most
-likely interface will be an additional method call to the object, or
-perhaps (to avoid forcing per-stream objects on otherwise stateless
-encodings) and additional parameter.
+By contrast, if I<$check> is true, the scheme above allows the
+encoding to do as much as it can and tell the layer above how much
+that was. What is lacking at present is a mechanism to report what
+went wrong. The most likely interface will be an additional method
+call to the object, or perhaps (to avoid forcing per-stream objects
+on otherwise stateless encodings) an additional parameter.
It is also highly desirable that encoding classes inherit from
C<Encode::Encoding> as a base class. This allows that class to define
-additional behaviour for all encoding objects. For example built in
-Unicode, UCS-2 and UTF-8 classes use :
+additional behaviour for all encoding objects. For example, built-in
+Unicode, UCS-2, and UTF-8 classes use
package Encode::MyEncoding;
use base qw(Encode::Encoding);
__PACKAGE__->Define(qw(myCanonical myAlias));
-To create an object with bless {Name => ...},$class, and call
+to create an object with C<< bless {Name => ...}, $class >>, and call
define_encoding. They inherit their C<name> method from
C<Encode::Encoding>.
=head2 Compiled Encodings
-F<Encode.xs> provides a class C<Encode::XS> which provides the
-interface described above. It calls a generic octet-sequence to
-octet-sequence "engine" that is driven by tables (defined in
-F<encengine.c>). The same engine is used for both encode and
-decode. C<Encode:XS>'s C<encode> forces Perl's characters to their
-UTF-8 form and then treats them as just another multibyte
-encoding. C<Encode:XS>'s C<decode> transforms the sequence and then
-turns the UTF-8-ness flag as that is the form that the tables are
-defined to produce. For details of the engine see the comments in
-F<encengine.c>.
-
-The tables are produced by the Perl script F<compile> (the name needs
-to change so we can eventually install it somewhere). F<compile> can
-currently read two formats:
-
-=over 4
-
-=item *.enc
-
-This is a coined format used by Tcl. It is documented in
-Encode/EncodeFormat.pod.
+For the sake of speed and efficiency, most of the encodings are now
+supported via a I<compiled form>: XS modules generated from UCM
+files. Encode provides the enc2xs tool to achieve that. Please see
+L<enc2xs> for more details.
-=item *.ucm
+=head1 SEE ALSO
-This is the semi-standard format used by IBM's ICU package.
+L<perlmod>, L<enc2xs>
-=back
-
-F<compile> can write the following forms:
+=begin future
=over 4
-=item *.ucm
-
-See above - the F<Encode/*.ucm> files provided with the distribution have
-been created from the original Tcl .enc files using this approach.
+=item Scheme 1
-=item *.c
+The fixup routine gets passed the remaining fragment of string being
+processed. It modifies it in place to remove bytes/characters it can
+understand and returns a string used to represent them. For example:
-Produces tables as C data structures - this is used to build in encodings
-into F<Encode.so>/F<Encode.dll>.
+ sub fixup {
+ my $ch = substr($_[0],0,1,'');
+ return sprintf("\x{%02X}",ord($ch);
+ }
-=item *.xs
+This scheme is close to how the underlying C code for Encode works,
+but gives the fixup routine very little context.
-In theory this allows encodings to be stand-alone loadable Perl
-extensions. The process has not yet been tested. The plan is to use
-this approach for large East Asian encodings.
+=item Scheme 2
-=back
-
-The set of encodings built-in to F<Encode.so>/F<Encode.dll> is
-determined by F<Makefile.PL>. The current set is as follows:
-
-=over 4
+The fixup routine gets passed the original string, an index into
+it of the problem area, and the output string so far. It appends
+what it wants to the output string and returns a new index into the
+original string. For example:
-=item ascii and iso-8859-*
+ sub fixup {
+ # my ($s,$i,$d) = @_;
+ my $ch = substr($_[0],$_[1],1);
+ $_[2] .= sprintf("\x{%02X}",ord($ch);
+ return $_[1]+1;
+ }
-That is all the common 8-bit "western" encodings.
+This scheme gives maximal control to the fixup routine but is more
+complicated to code, and may require that the internals of Encode be tweaked to
+keep the original string intact.
-=item IBM-1047 and two other variants of EBCDIC.
+=item Other Schemes
-These are the same variants that are supported by EBCDIC Perl as
-"native" encodings. They are included to prove "reversibility" of
-some constructs in EBCDIC Perl.
+Hybrids of the above.
-=item symbol and dingbats as used by Tk on X11.
+Multiple return values rather than in-place modifications.
-(The reason Encode got started was to support Perl/Tk.)
+Index into the string could be C<pos($str)> allowing C<s/\G...//>.
=back
-That set is rather ad hoc and has been driven by the needs of the
-tests rather than the needs of typical applications. It is likely
-to be rationalized.
+=end future
=cut
projects / p5sagit/p5-mst-13.2.git / blobdiff