From: Nicholas Clark Date: Thu, 29 Nov 2001 22:47:32 +0000 (+0000) Subject: Re: DRAFT perlpacktut.pod v0.0 X-Git-Url: http://git.shadowcat.co.uk/gitweb/gitweb.cgi?a=commitdiff_plain;h=47b6252e7f5b579fed84c3881f7cf1d3f6c0f2a4;p=p5sagit%2Fp5-mst-13.2.git Re: DRAFT perlpacktut.pod v0.0 Message-ID: <20011129224732.W37621@plum.flirble.org> p4raw-id: //depot/perl@13373 --- diff --git a/pod/perlpacktut.pod b/pod/perlpacktut.pod index b102543..28e585d 100644 --- a/pod/perlpacktut.pod +++ b/pod/perlpacktut.pod @@ -6,7 +6,7 @@ perlpacktut - tutorial on C and C C and C are two functions for transforming data according to a user-defined template, between the guarded way Perl stores values -and some well-defined representation as might be required in the +and some well-defined representation as might be required in the environment of a Perl program. Unfortunately, they're also two of the most misunderstood and most often overlooked functions that Perl provides. This tutorial will demystify them for you. @@ -42,7 +42,7 @@ of these two functions. To see how (un)packing works, we'll start with a simple template code where the conversion is in low gear: between the contents of a byte sequence and a string of hexadecimal digits. Let's use C, since -this is likely to remind you of a dump program, or some desparate last +this is likely to remind you of a dump program, or some desperate last message unfortunate programs are wont to throw at you before they expire into the wild blue yonder. Assuming that the variable C<$mem> holds a sequence of bytes that we'd like to inspect without assuming anything @@ -56,7 +56,7 @@ corresponding to a byte: 41204d414e204120504c414e20412043414e414c2050414e414d41 -What was in this chunk of memory? Numbers, charactes, or a mixture of +What was in this chunk of memory? Numbers, characters, or a mixture of both? Assuming that we're on a computer where ASCII (or some similar) encoding is used: hexadecimal values in the range C<0x40> - C<0x5A> indicate an uppercase letter, and <0x20> encodes a space. So we might @@ -141,7 +141,7 @@ the headers, and a handy ruler so we can keep track of where we are. 01/28/2001 Flea spray 24.99 01/29/2001 Camel rides to tourists 235.00 ->From this, we can see that the date column stretches from column 1 to +From this, we can see that the date column stretches from column 1 to column 10 - ten characters wide. The C-ese for "character" is C, and ten of them are C. So if we just wanted to extract the dates, we could say this: @@ -340,7 +340,7 @@ Unpacking C<$ps> with the same template returns the original integer value: my( $s ) = unpack( 's', $ps ); This is true for all numeric template codes. But don't expect miracles: -if the packed value exceeds the alotted byte capacity, high order bits +if the packed value exceeds the allotted byte capacity, high order bits are silently discarded, and unpack certainly won't be able to pull them back out of some magic hat. And, when you pack using a signed template code such as C, an excess value may result in the sign bit @@ -587,14 +587,14 @@ characters. Unicode 3.1 specifies 94,140 characters: The Basic Latin characters are assigned to the numbers 0 - 127. The Latin-1 Supplement with characters that are used in several European languages is in the next range, up to 255. After some more Latin extensions we find the character -sets from languages using non-roman alphabets, interspersed with a +sets from languages using non-Roman alphabets, interspersed with a variety of symbol sets such as currency symbols, Zapf Dingbats or Braille. (You might want to visit L for a look at some of them - my personal favourites are Telugu and Kannada.) The Unicode character sets associates characters with integers. Encoding these numbers in an equal number of bytes would more than double the -requirements for storing texts written in latin alphabets. +requirements for storing texts written in Latin alphabets. The UTF-8 encoding avoids this by storing the most common (from a western point of view) characters in a single byte while encoding the rarer ones in three or more bytes. @@ -627,7 +627,7 @@ In the previous section we've seen a network message that was constructed by prefixing the binary message length to the actual message. You'll find that packing a length followed by so many bytes of data is a frequently used recipe since appending a null byte won't work -if a null byte may be part of the data. - Here is an example where both +if a null byte may be part of the data. Here is an example where both techniques are used: after two null terminated strings with source and destination address, a Short Message (to a mobile phone) is sent after a length byte: @@ -638,7 +638,7 @@ Unpacking this message can be done with the same template: ( $src, $dst, $len, $sm ) = unpack( 'Z*Z*CA*', $msg ); -There's a subtle trap lurking in the offings: Adding another field after +There's a subtle trap lurking in the offing: Adding another field after the Short Message (in variable C<$sm>) is all right when packing, but this cannot be unpacked naively: @@ -678,6 +678,19 @@ C or C: # unpack $buf: '13 Humpty-Dumpty' my $txt = unpack( 'A4/A*', $buf ); +C is not implemented in Perls before 5.6, so if your code is required to +work on older Perls you'll need to C to get the length, +then use it to make a new unpack string. For example + + # pack a message: ASCIIZ, ASCIIZ, length, string, byte (5.005 compatible) + my $msg = pack( 'Z* Z* C A* C', $src, $dst, length $sm, $sm, $prio ); + + # unpack + ( undef, undef, $len) = unpack( 'Z* Z* C', $msg ); + ($src, $dst, $sm, $prio) = unpack ( "Z* Z* x A$len C", $msg ); + +But that second C is rushing ahead. It isn't using a simple literal +string for the template. So maybe we should introduce... =head2 Dynamic Templates @@ -695,12 +708,11 @@ additional delimiting null byte. Here's how: For the reverse operation, we'll have to determine the number of items in the buffer before we can let C rip it apart: - my $n = ( $env =~ s/\0/\0/g - 1 ); + my $n = $env =~ tr/\0// - 1; my %env = map { split( '=', $_ ) } unpack( 'Z*' x $n, $env ); -The substitution counts the null bytes. The C call returns a -list of name-value pairs each of which is taken apart in the C -block. +The C counts the null bytes. The C call returns a list of +name-value pairs each of which is taken apart in the C block. =head2 Another Portable Binary Encoding @@ -738,7 +750,9 @@ memory, or to or from a CPU register, if it is aligned at an even or multiple-of-four or even at a multiple-of eight address, a C compiler will give you this speed benefit by stuffing extra bytes into structures. If you don't cross the C shoreline this is not likely to cause you any -grief (although you should care when you design large data structures). +grief (although you should care when you design large data structures, +or you want your code to be portable between architectures (you do want +that, don't you?)). To see how this affects C and C, we'll compare these two C structures: @@ -791,7 +805,10 @@ from the list: my $gappy = pack( 'cxs cxxx l!', $c1, $s, $c2, $l ); Note the C after C: We want to make sure that we pack a long -integer as it is compiled by our C compiler. +integer as it is compiled by our C compiler. And even now, it will only +work for the platforms where the compiler aligns things as above. +And somebody somewhere has a platform where it doesn't. +[Probably a Cray, where Cs, Cs and Cs are all 8 bytes. :-)] Counting bytes and watching alignments in lengthy structures is bound to be a drag. Isn't there a way we can create the template with a simple @@ -859,7 +876,18 @@ of each component as well. Thus the correct template is: pack( 's!ax' x @buffer, map{ ( $_->{count}, $_->{glyph} ) } @buffer ); +=head2 Alignment, Take 3 + +And even if you take all the above into account, ANSI still lets this: + + typedef struct { + char foo[2]; + } foo_t; +vary in size. The alignment constraint of the structure can be greater than +any of its elements. [And if you think that this doesn't affect anything +common, dismember the next cellphone that you see. Many have ARM cores, and +the ARM structure rules make C == 4] =head2 Pointers for How to Use Them