4 IO::Compress::FAQ -- Frequently Asked Questions about IO::Compress
8 Common questions answered.
10 =head2 Compatibility with Unix compress/uncompress.
12 Although C<Compress::Zlib> has a pair of functions called C<compress> and
13 C<uncompress>, they are I<not> related to the Unix programs of the same
14 name. The C<Compress::Zlib> module is not compatible with Unix
17 If you have the C<uncompress> program available, you can use this to read
20 open F, "uncompress -c $filename |";
25 Alternatively, if you have the C<gunzip> program available, you can use
26 this to read compressed files
28 open F, "gunzip -c $filename |";
33 and this to write compress files, if you have the C<compress> program
36 open F, "| compress -c $filename ";
41 =head2 Accessing .tar.Z files
43 The C<Archive::Tar> module can optionally use C<Compress::Zlib> (via the
44 C<IO::Zlib> module) to access tar files that have been compressed with
45 C<gzip>. Unfortunately tar files compressed with the Unix C<compress>
46 utility cannot be read by C<Compress::Zlib> and so cannot be directly
47 accessed by C<Archive::Tar>.
49 If the C<uncompress> or C<gunzip> programs are available, you can use one
50 of these workarounds to read C<.tar.Z> files from C<Archive::Tar>
52 Firstly with C<uncompress>
58 open F, "uncompress -c $filename |";
59 my $tar = Archive::Tar->new(*F);
62 and this with C<gunzip>
68 open F, "gunzip -c $filename |";
69 my $tar = Archive::Tar->new(*F);
72 Similarly, if the C<compress> program is available, you can use this to
73 write a C<.tar.Z> file
80 my $fh = new IO::File "| compress -c >$filename";
81 my $tar = Archive::Tar->new();
86 =head2 Accessing Zip Files
88 This module provides support for reading/writing zip files using the
89 C<IO::Compress::Zip> and C<IO::Uncompress::Unzip> modules.
91 The primary focus of the C<IO::Compress::Zip> and C<IO::Uncompress::Unzip>
92 modules is to provide an C<IO::File> compatible streaming read/write
93 interface to zip files/buffers. They are not fully flegged archivers. If
94 you are looking for an archiver check out the C<Archive::Zip> module. You
95 can find it on CPAN at
97 http://www.cpan.org/modules/by-module/Archive/Archive-Zip-*.tar.gz
99 =head2 Compressed files and Net::FTP
101 The C<Net::FTP> module provides two low-level methods called C<stor> and
102 C<retr> that both return filehandles. These filehandles can used with the
103 C<IO::Compress/Uncompress> modules to compress or uncompress files read
104 from or written to an FTP Server on the fly, without having to create a
107 Firstly, here is code that uses C<retr> to uncompressed a file as it is
108 read from the FTP Server.
111 use IO::Uncompress::Gunzip qw(:all);
113 my $ftp = new Net::FTP ...
115 my $retr_fh = $ftp->retr($compressed_filename);
116 gunzip $retr_fh => $outFilename, AutoClose => 1
117 or die "Cannot uncompress '$compressed_file': $GunzipError\n";
119 and this to compress a file as it is written to the FTP Server
122 use IO::Compress::Gzip qw(:all);
124 my $stor_fh = $ftp->stor($filename);
125 gzip "filename" => $stor_fh, AutoClose => 1
126 or die "Cannot compress '$filename': $GzipError\n";
128 =head2 How do I recompress using a different compression?
130 This is easier that you might expect if you realise that all the
131 C<IO::Compress::*> objects are derived from C<IO::File> and that all the
132 C<IO::Uncompress::*> modules can read from an C<IO::File> filehandle.
134 So, for example, say you have a file compressed with gzip that you want to
135 recompress with bzip2. Here is all that is needed to carry out the
138 use IO::Uncompress::Gunzip ':all';
139 use IO::Compress::Bzip2 ':all';
141 my $gzipFile = "somefile.gz";
142 my $bzipFile = "somefile.bz2";
144 my $gunzip = new IO::Uncompress::Gunzip $gzipFile
145 or die "Cannot gunzip $gzipFile: $GunzipError\n" ;
147 bzip2 $gunzip => $bzipFile
148 or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ;
150 Note, there is a limitation of this technique. Some compression file
151 formats store extra information along with the compressed data payload. For
152 example, gzip can optionally store the original filename and Zip stores a
153 lot of information about the original file. If the original compressed file
154 contains any of this extra information, it will not be transferred to the
155 new compressed file usign the technique above.
157 =head2 Apache::GZip Revisited
159 Below is a mod_perl Apache compression module, called C<Apache::GZip>,
161 F<http://perl.apache.org/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.html#On_the_Fly_Compression>
163 package Apache::GZip;
164 #File: Apache::GZip.pm
167 use Apache::Constants ':common';
170 use constant GZIP_MAGIC => 0x1f8b;
171 use constant OS_MAGIC => 0x03;
176 my $file = $r->filename;
177 return DECLINED unless $fh=IO::File->new($file);
178 $r->header_out('Content-Encoding'=>'gzip');
179 $r->send_http_header;
180 return OK if $r->header_only;
182 tie *STDOUT,'Apache::GZip',$r;
183 print($_) while <$fh>;
190 # initialize a deflation stream
191 my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef;
193 # gzip header -- don't ask how I found out
194 $r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC));
196 return bless { r => $r,
207 my $data = $self->{d}->deflate($_);
208 $self->{r}->print($data);
209 # keep track of its length and crc
210 $self->{l} += length($_);
211 $self->{crc} = crc32($_,$self->{crc});
218 # flush the output buffers
219 my $data = $self->{d}->flush;
220 $self->{r}->print($data);
222 # print the CRC and the total length (uncompressed)
223 $self->{r}->print(pack("LL",@{$self}{qw/crc l/}));
228 Here's the Apache configuration entry you'll need to make use of it. Once
229 set it will result in everything in the /compressed directory will be
230 compressed automagically.
232 <Location /compressed>
233 SetHandler perl-script
234 PerlHandler Apache::GZip
237 Although at first sight there seems to be quite a lot going on in
238 C<Apache::GZip>, you could sum up what the code was doing as follows --
239 read the contents of the file in C<< $r->filename >>, compress it and write
240 the compressed data to standard output. That's all.
242 This code has to jump through a few hoops to achieve this because
248 The gzip support in C<Compress::Zlib> version 1.x can only work with a real
249 filesystem filehandle. The filehandles used by Apache modules are not
250 associated with the filesystem.
254 That means all the gzip support has to be done by hand - in this case by
255 creating a tied filehandle to deal with creating the gzip header and
260 C<IO::Compress::Gzip> doesn't have that filehandle limitation (this was one
261 of the reasons for writing it in the first place). So if
262 C<IO::Compress::Gzip> is used instead of C<Compress::Zlib> the whole tied
263 filehandle code can be removed. Here is the rewritten code.
265 package Apache::GZip;
268 use Apache::Constants ':common';
269 use IO::Compress::Gzip;
275 my $file = $r->filename;
276 return DECLINED unless $fh=IO::File->new($file);
277 $r->header_out('Content-Encoding'=>'gzip');
278 $r->send_http_header;
279 return OK if $r->header_only;
281 my $gz = new IO::Compress::Gzip '-', Minimal => 1
284 print $gz $_ while <$fh>;
289 or even more succinctly, like this, using a one-shot gzip
291 package Apache::GZip;
294 use Apache::Constants ':common';
295 use IO::Compress::Gzip qw(gzip);
299 $r->header_out('Content-Encoding'=>'gzip');
300 $r->send_http_header;
301 return OK if $r->header_only;
303 gzip $r->filename => '-', Minimal => 1
311 The use of one-shot C<gzip> above just reads from C<< $r->filename >> and
312 writes the compressed data to standard output.
314 Note the use of the C<Minimal> option in the code above. When using gzip
315 for Content-Encoding you should I<always> use this option. In the example
316 above it will prevent the filename being included in the gzip header and
317 make the size of the gzip data stream a slight bit smaller.
319 =head2 Using C<InputLength> to uncompress data embedded in a larger file/buffer.
321 A fairly common use-case is where compressed data is embedded in a larger
322 file/buffer and you want to read both.
324 As an example consider the structure of a zip file. This is a well-defined
325 file format that mixes both compressed and uncompressed sections of data in
328 For the purposes of this discussion you can think of a zip file as sequence
329 of compressed data streams, each of which is prefixed by an uncompressed
330 local header. The local header contains information about the compressed
331 data stream, including the name of the compressed file and, in particular,
332 the length of the compressed data stream.
334 To illustrate how to use C<InputLength> here is a script that walks a zip
335 file and prints out how many lines are in each compressed file (if you
336 intend write code to walking through a zip file for real see
337 L<IO::Uncompress::Unzip/"Walking through a zip file"> )
343 use IO::Uncompress::RawInflate qw(:all);
345 use constant ZIP_LOCAL_HDR_SIG => 0x04034b50;
346 use constant ZIP_LOCAL_HDR_LENGTH => 30;
348 my $file = $ARGV[0] ;
350 my $fh = new IO::File "<$file"
351 or die "Cannot open '$file': $!\n";
359 ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH
360 or die "Truncated file: $!\n";
362 my $signature = unpack ("V", substr($buffer, 0, 4));
364 last unless $signature == ZIP_LOCAL_HDR_SIG;
367 my $gpFlag = unpack ("v", substr($buffer, 6, 2));
368 my $compressedMethod = unpack ("v", substr($buffer, 8, 2));
369 my $compressedLength = unpack ("V", substr($buffer, 18, 4));
370 my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
371 my $filename_length = unpack ("v", substr($buffer, 26, 2));
372 my $extra_length = unpack ("v", substr($buffer, 28, 2));
375 $fh->read($filename, $filename_length) == $filename_length
376 or die "Truncated file\n";
378 $fh->read($buffer, $extra_length) == $extra_length
379 or die "Truncated file\n";
381 if ($compressedMethod != 8 && $compressedMethod != 0)
383 warn "Skipping file '$filename' - not deflated $compressedMethod\n";
384 $fh->read($buffer, $compressedLength) == $compressedLength
385 or die "Truncated file\n";
389 if ($compressedMethod == 0 && $gpFlag & 8 == 8)
391 die "Streamed Stored not supported for '$filename'\n";
394 next if $compressedLength == 0;
396 # Done reading the Local Header
398 my $inf = new IO::Uncompress::RawInflate $fh,
400 InputLength => $compressedLength
401 or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
410 print "$filename: $line_count\n";
413 The majority of the code above is concerned with reading the zip local
414 header data. The code that I want to focus on is at the bottom.
418 # read local zip header data
420 # get $compressedLength
422 my $inf = new IO::Uncompress::RawInflate $fh,
424 InputLength => $compressedLength
425 or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
434 print "$filename: $line_count\n";
437 The call to C<IO::Uncompress::RawInflate> creates a new filehandle C<$inf>
438 that can be used to read from the parent filehandle C<$fh>, uncompressing
439 it as it goes. The use of the C<InputLength> option will guarantee that
440 I<at most> C<$compressedLength> bytes of compressed data will be read from
441 the C<$fh> filehandle (The only exception is for an error case like a
442 truncated file or a corrupt data stream).
444 This means that once RawInflate is finished C<$fh> will be left at the
445 byte directly after the compressed data stream.
447 Now consider what the code looks like without C<InputLength>
451 # read local zip header data
453 # get $compressedLength
455 # read all the compressed data into $data
456 read($fh, $data, $compressedLength);
458 my $inf = new IO::Uncompress::RawInflate \$data,
460 or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
469 print "$filename: $line_count\n";
472 The difference here is the addition of the temporary variable C<$data>.
473 This is used to store a copy of the compressed data while it is being
476 If you know that C<$compressedLength> isn't that big then using temporary
477 storage won't be a problem. But if C<$compressedLength> is very large or
478 you are writing an application that other people will use, and so have no
479 idea how big C<$compressedLength> will be, it could be an issue.
481 Using C<InputLength> avoids the use of temporary storage and means the
482 application can cope with large compressed data streams.
484 One final point -- obviously C<InputLength> can only be used whenever you
485 know the length of the compressed data beforehand, like here with a zip
490 L<Compress::Zlib>, L<IO::Compress::Gzip>, L<IO::Uncompress::Gunzip>, L<IO::Compress::Deflate>, L<IO::Uncompress::Inflate>, L<IO::Compress::RawDeflate>, L<IO::Uncompress::RawInflate>, L<IO::Compress::Bzip2>, L<IO::Uncompress::Bunzip2>, L<IO::Compress::Lzop>, L<IO::Uncompress::UnLzop>, L<IO::Compress::Lzf>, L<IO::Uncompress::UnLzf>, L<IO::Uncompress::AnyInflate>, L<IO::Uncompress::AnyUncompress>
492 L<Compress::Zlib::FAQ|Compress::Zlib::FAQ>
494 L<File::GlobMapper|File::GlobMapper>, L<Archive::Zip|Archive::Zip>,
495 L<Archive::Tar|Archive::Tar>,
500 This module was written by Paul Marquess, F<pmqs@cpan.org>.
502 =head1 MODIFICATION HISTORY
504 See the Changes file.
506 =head1 COPYRIGHT AND LICENSE
508 Copyright (c) 2005-2009 Paul Marquess. All rights reserved.
510 This program is free software; you can redistribute it and/or
511 modify it under the same terms as Perl itself.