4 perliol - C API for Perl's implementation of IO in Layers.
8 /* Defining a layer ... */
14 This document describes the behavior and implementation of the PerlIO
15 abstraction described in L<perlapio> when C<USE_PERLIO> is defined (and
18 =head2 History and Background
20 The PerlIO abstraction was introduced in perl5.003_02 but languished as
21 just an abstraction until perl5.7.0. However during that time a number
22 of perl extentions switched to using it, so the API is mostly fixed to
23 maintain (source) compatibility.
25 The aim of the implementation is to provide the PerlIO API in a flexible
26 and platform neutral manner. It is also a trial of an "Object Oriented
27 C, with vtables" approach which may be applied to perl6.
29 =head2 Layers vs Disciplines
31 Initial discussion of the ability to modify IO streams behaviour used
32 the term "discipline" for the entities which were added. This came (I
33 believe) from the use of the term in "sfio", which in turn borrowed it
34 from "line disciplines" on Unix terminals. However, this document (and
35 the C code) uses the term "layer".
37 This is, I hope, a natural term given the implementation, and should avoid
38 connotations that are inherent in earlier uses of "discipline" for things
39 which are rather different.
41 =head2 Data Structures
43 The basic data structure is a PerlIOl:
45 typedef struct _PerlIO PerlIOl;
46 typedef struct _PerlIO_funcs PerlIO_funcs;
47 typedef PerlIOl *PerlIO;
51 PerlIOl * next; /* Lower layer */
52 PerlIO_funcs * tab; /* Functions for this layer */
53 IV flags; /* Various flags for state */
56 A C<PerlIOl *> is a pointer to to the struct, and the I<application> level
57 C<PerlIO *> is a pointer to a C<PerlIOl *> - i.e. a pointer to a pointer to
58 the struct. This allows the application level C<PerlIO *> to remain
59 constant while the actual C<PerlIOl *> underneath changes. (Compare perl's
60 C<SV *> which remains constant while its C<sv_any> field changes as the
61 scalar's type changes.) An IO stream is then in general represented as a
62 pointer to this linked-list of "layers".
64 It should be noted that because of the double indirection in a C<PerlIO *>,
65 a C<< &(perlio-E<gt>next) >> "is" a C<PerlIO *>, and so to some degree
66 at least one layer can use the "standard" API on the next layer down.
68 A "layer" is composed of two parts:
72 =item 1. The functions and attributes of the "layer class".
74 =item 2. The per-instance data for a particular handle.
78 =head2 Functions and Attributes
80 The functions and attributes are accessed via the "tab" (for table)
81 member of C<PerlIOl>. The functions (methods of the layer "class") are
82 fixed, and are defined by the C<PerlIO_funcs> type. They are broadly the
83 same as the public C<PerlIO_xxxxx> functions:
90 IV (*Pushed)(PerlIO *f,const char *mode,SV *arg);
91 IV (*Popped)(PerlIO *f);
92 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
95 int fd, int imode, int perm,
98 SV * (*Getarg)(PerlIO *f);
99 IV (*Fileno)(PerlIO *f);
100 /* Unix-like functions - cf sfio line disciplines */
101 SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
102 SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
103 SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
104 IV (*Seek)(PerlIO *f, Off_t offset, int whence);
105 Off_t (*Tell)(PerlIO *f);
106 IV (*Close)(PerlIO *f);
107 /* Stdio-like buffered IO functions */
108 IV (*Flush)(PerlIO *f);
109 IV (*Fill)(PerlIO *f);
110 IV (*Eof)(PerlIO *f);
111 IV (*Error)(PerlIO *f);
112 void (*Clearerr)(PerlIO *f);
113 void (*Setlinebuf)(PerlIO *f);
114 /* Perl's snooping functions */
115 STDCHAR * (*Get_base)(PerlIO *f);
116 Size_t (*Get_bufsiz)(PerlIO *f);
117 STDCHAR * (*Get_ptr)(PerlIO *f);
118 SSize_t (*Get_cnt)(PerlIO *f);
119 void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
124 The first few members of the struct give a "name" for the layer, the
125 size to C<malloc> for the per-instance data, and some flags which are
126 attributes of the class as whole (such as whether it is a buffering
127 layer), then follow the functions which fall into four basic groups:
133 Opening and setup functions
141 Stdio class buffering options.
145 Functions to support Perl's traditional "fast" access to the buffer.
149 A layer does not have to implement all the functions, but the whole table has
150 to be present. Unimplemented slots can be NULL (which will result in an error
151 when called) or can be filled in with stubs to "inherit" behaviour from
152 a "base class". This "inheritance" is fixed for all instances of the layer,
153 but as the layer chooses which stubs to populate the table, limited
154 "multiple inheritance" is possible.
156 =head2 Per-instance Data
158 The per-instance data are held in memory beyond the basic PerlIOl struct,
159 by making a PerlIOl the first member of the layer's struct thus:
163 struct _PerlIO base; /* Base "class" info */
164 STDCHAR * buf; /* Start of buffer */
165 STDCHAR * end; /* End of valid part of buffer */
166 STDCHAR * ptr; /* Current position in buffer */
167 Off_t posn; /* Offset of buf into the file */
168 Size_t bufsiz; /* Real size of buffer */
169 IV oneword; /* Emergency buffer */
172 In this way (as for perl's scalars) a pointer to a PerlIOBuf can be treated
173 as a pointer to a PerlIOl.
175 =head2 Layers in action.
179 +-----------+ +----------+ +--------+
180 PerlIO ->| |--->| next |--->| NULL |
181 +-----------+ +----------+ +--------+
182 | | | buffer | | fd |
183 +-----------+ | | +--------+
187 The above attempts to show how the layer scheme works in a simple case.
188 The application's C<PerlIO *> points to an entry in the table(s)
189 representing open (allocated) handles. For example the first three slots
190 in the table correspond to C<stdin>,C<stdout> and C<stderr>. The table
191 in turn points to the current "top" layer for the handle - in this case
192 an instance of the generic buffering layer "perlio". That layer in turn
193 points to the next layer down - in this case the lowlevel "unix" layer.
195 The above is roughly equivalent to a "stdio" buffered stream, but with
196 much more flexibility:
202 If Unix level C<read>/C<write>/C<lseek> is not appropriate for (say)
203 sockets then the "unix" layer can be replaced (at open time or even
204 dynamically) with a "socket" layer.
208 Different handles can have different buffering schemes. The "top" layer
209 could be the "mmap" layer if reading disk files was quicker using C<mmap>
210 than C<read>. An "unbuffered" stream can be implemented simply by
211 not having a buffer layer.
215 Extra layers can be inserted to process the data as it flows through.
216 This was the driving need for including the scheme in perl 5.7.0+ - we
217 needed a mechanism to allow data to be translated bewteen perl's
218 internal encoding (conceptually at least Unicode as UTF-8), and the
219 "native" format used by the system. This is provided by the
220 ":encoding(xxxx)" layer which typically sits above the buffering layer.
224 A layer can be added that does "\n" to CRLF translation. This layer can be used
225 on any platform, not just those that normally do such things.
229 =head2 Per-instance flag bits
231 The generic flag bits are a hybrid of C<O_XXXXX> style flags deduced from
232 the mode string passed to C<PerlIO_open()>, and state bits for typical buffer
241 =item PERLIO_F_CANWRITE
243 Writes are permitted, i.e. opened as "w" or "r+" or "a", etc.
245 =item PERLIO_F_CANREAD
247 Reads are permitted i.e. opened "r" or "w+" (or even "a+" - ick).
251 An error has occured (for C<PerlIO_error()>)
253 =item PERLIO_F_TRUNCATE
255 Truncate file suggested by open mode.
257 =item PERLIO_F_APPEND
259 All writes should be appends.
263 Layer is performing Win32-like "\n" mapped to CR,LF for output and CR,LF
264 mapped to "\n" for input. Normally the provided "crlf" layer is the only
265 layer that need bother about this. C<PerlIO_binmode()> will mess with this
266 flag rather than add/remove layers if the C<PERLIO_K_CANCRLF> bit is set
267 for the layers class.
271 Data written to this layer should be UTF-8 encoded; data provided
272 by this layer should be considered UTF-8 encoded. Can be set on any layer
273 by ":utf8" dummy layer. Also set on ":encoding" layer.
277 Layer is unbuffered - i.e. write to next layer down should occur for
278 each write to this layer.
282 The buffer for this layer currently holds data written to it but not sent
287 The buffer for this layer currently holds unconsumed data read from
290 =item PERLIO_F_LINEBUF
292 Layer is line buffered. Write data should be passed to next layer down
293 whenever a "\n" is seen. Any data beyond the "\n" should then be
298 File has been C<unlink()>ed, or should be deleted on C<close()>.
304 =item PERLIO_F_FASTGETS
306 This instance of this layer supports the "fast C<gets>" interface.
307 Normally set based on C<PERLIO_K_FASTGETS> for the class and by the
308 existance of the function(s) in the table. However a class that
309 normally provides that interface may need to avoid it on a
310 particular instance. The "pending" layer needs to do this when
311 it is pushed above an layer which does not support the interface.
312 (Perl's C<sv_gets()> does not expect the streams fast C<gets> behaviour
313 to change during one "get".)
317 =head2 Methods in Detail
321 =item IV (*Pushed)(PerlIO *f,const char *mode, SV *arg);
323 The only absoultely mandatory method. Called when the layer is pushed onto the stack.
324 The C<mode> argument may be NULL if this occurs post-open. The C<arg> will be non-C<NULL>
325 if an argument string was passed. In most cases this should call
326 C<PerlIOBase_pushed()> to convert C<mode> into the appropriate
327 C<PERLIO_F_XXXXX> flags in addition to any actions the layer itself takes.
328 If a layer is not expecting an argument it need neither save the one passed to it, nor
329 provide C<Getarg()> (it could perhaps C<Perl_warn> that the argument was un-expected).
331 =item IV (*Popped)(PerlIO *f);
333 Called when the layer is popped from the stack. A layer will normally be
334 popped after C<Close()> is called. But a layer can be popped without being
335 closed if the program is dynamically managing layers on the stream. In
336 such cases C<Popped()> should free any resources (buffers, translation
337 tables, ...) not held directly in the layer's struct.
338 It should also C<Unread()> any unconsumed data that has been read and buffered
339 from the layer below back to that layer, so that it can be re-provided to what
342 =item PerlIO * (*Open)(...);
344 The C<Open()> method has lots of arguments because it combines the functions
345 of perl's C<open>, C<PerlIO_open>, perl's C<sysopen>, C<PerlIO_fdopen> and C<PerlIO_reopen>.
346 The full prototype is as follows:
348 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
351 int fd, int imode, int perm,
353 int narg, SV **args);
355 Open should (perhaps indirectly) call C<PerlIO_allocate()> to allocate a slot in the table and
356 associate it with the layers information for the opened file, by calling C<PerlIO_push>.
357 The I<layers> AV is an array of all the layers destined for the C<PerlIO *>,
358 and any arguments passed to them, I<n> is the index into that array of the
359 layer being called. The macro C<PerlIOArg> will return a (possibly C<NULL>) SV *
360 for the argument passed to the layer.
362 The I<mode> string is an "C<fopen()>-like" string which would match the regular
363 expression C</^[I#]?[rwa]\+?[bt]?$/>.
365 The C<'I'> prefix is used during creation of C<stdin>..C<stderr> via special
366 C<PerlIO_fdopen> calls; the C<'#'> prefix means that this is C<sysopen> and that I<imode> and
367 I<perm> should be passed to C<PerlLIO_open3>; C<'r'> means B<r>ead, C<'w'> means B<w>rite
368 and C<'a'> means B<a>ppend. The C<'+'> suffix means that both reading and writing/appending
369 are permited. The C<'b'> suffix means file should be binary, and C<'t'> means it
370 is text. (Binary/Text should be ignored by almost all layers and binary IO done,
371 with PerlIO. The C<:crlf> layer should be pushed to handle the distinction.)
373 If I<old> is not C<NULL> then this is a C<PerlIO_reopen>. Perl iteself does not use
374 this (yet?) and semantics are a little vague.
376 If I<fd> not negative then it is the numeric file descriptor I<fd>, which will
377 be open in an manner compatible with the supplied mode string, the call is
378 thus equivalent to C<PerlIO_fdopen>. In this case I<nargs> will be zero.
380 If I<nargs> is greater than zero then it gives the number of arguments passed
381 to C<open>, otherwise it will be 1 if for example C<PerlIO_open> was called.
382 In simple cases SvPV(*args) is the pathname to open.
384 Having said all that translation-only layers do not need to provide C<Open()> at all,
385 but rather leave the opening to a lower level layer and wait to be "pushed".
386 If a layer does provide C<Open()> it should normaly call the C<Open()> method
387 of next layer down (if any) and then push itself on top if that succeeds.
389 =item SV * (*Getarg)(PerlIO *f);
391 Optional. If present should return an SV * representing the string argument
392 passed to the layer when it was pushed. e.g. ":encoding(ascii)" would
393 return an SvPV with value "ascii".
395 =item IV (*Fileno)(PerlIO *f);
397 Returns the Unix/Posix numeric file decriptor for the handle. Normally
398 C<PerlIOBase_fileno()> (which just asks next layer down) will suffice
401 =item SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
403 Basic read operation. Returns actual bytes read, or -1 on an error.
404 Typically will call Fill and manipulate pointers (possibly via the API).
405 C<PerlIOBuf_read()> may be suitable for derived classes which provide
408 =item SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
410 A superset of stdio's C<ungetc()>. Should arrange for future reads to
411 see the bytes in C<vbuf>. If there is no obviously better implementation
412 then C<PerlIOBase_unread()> provides the function by pushing a "fake"
413 "pending" layer above the calling layer.
415 =item SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
417 Basic write operation. Returns bytes written or -1 on an error.
419 =item IV (*Seek)(PerlIO *f, Off_t offset, int whence);
421 Position the file pointer. Should normally call its own C<Flush> method and
422 then the C<Seek> method of next layer down.
424 =item Off_t (*Tell)(PerlIO *f);
426 Return the file pointer. May be based on layers cached concept of
427 position to avoid overhead.
429 =item IV (*Close)(PerlIO *f);
431 Close the stream. Should normally call C<PerlIOBase_close()> to flush
432 itself and close layers below, and then deallocate any data structures
433 (buffers, translation tables, ...) not held directly in the data
436 =item IV (*Flush)(PerlIO *f);
438 Should make stream's state consistent with layers below. That is, any
439 buffered write data should be written, and file position of lower layers
440 adjusted for data read fron below but not actually consumed.
441 (Should perhaps C<Unread()> such data to the lower layer.)
443 =item IV (*Fill)(PerlIO *f);
445 The buffer for this layer should be filled (for read) from layer below.
447 =item IV (*Eof)(PerlIO *f);
449 Return end-of-file indicator. C<PerlIOBase_eof()> is normally sufficient.
451 =item IV (*Error)(PerlIO *f);
453 Return error indicator. C<PerlIOBase_error()> is normally sufficient.
455 =item void (*Clearerr)(PerlIO *f);
457 Clear end-of-file and error indicators. Should call C<PerlIOBase_clearerr()>
458 to set the C<PERLIO_F_XXXXX> flags, which may suffice.
460 =item void (*Setlinebuf)(PerlIO *f);
462 Mark the stream as line buffered. C<PerlIOBase_setlinebuf()> sets the
463 PERLIO_F_LINEBUF flag and is normally sufficient.
465 =item STDCHAR * (*Get_base)(PerlIO *f);
467 Allocate (if not already done so) the read buffer for this layer and
468 return pointer to it.
470 =item Size_t (*Get_bufsiz)(PerlIO *f);
472 Return the number of bytes that last C<Fill()> put in the buffer.
474 =item STDCHAR * (*Get_ptr)(PerlIO *f);
476 Return the current read pointer relative to this layer's buffer.
478 =item SSize_t (*Get_cnt)(PerlIO *f);
480 Return the number of bytes left to be read in the current buffer.
482 =item void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
484 Adjust the read pointer and count of bytes to match C<ptr> and/or C<cnt>.
485 The application (or layer above) must ensure they are consistent.
486 (Checking is allowed by the paranoid.)
493 The file C<perlio.c> provides the following layers:
499 A basic non-buffered layer which calls Unix/POSIX C<read()>, C<write()>,
500 C<lseek()>, C<close()>. No buffering. Even on platforms that distinguish
501 between O_TEXT and O_BINARY this layer is always O_BINARY.
505 A very complete generic buffering layer which provides the whole of
506 PerlIO API. It is also intended to be used as a "base class" for other
507 layers. (For example its C<Read()> method is implemented in terms of the
508 C<Get_cnt()>/C<Get_ptr()>/C<Set_ptrcnt()> methods).
510 "perlio" over "unix" provides a complete replacement for stdio as seen
511 via PerlIO API. This is the default for USE_PERLIO when system's stdio
512 does not permit perl's "fast gets" access, and which do not distinguish
513 between C<O_TEXT> and C<O_BINARY>.
517 A layer which provides the PerlIO API via the layer scheme, but
518 implements it by calling system's stdio. This is (currently) the default
519 if system's stdio provides sufficient access to allow perl's "fast gets"
520 access and which do not distinguish between C<O_TEXT> and C<O_BINARY>.
524 A layer derived using "perlio" as a base class. It provides Win32-like
525 "\n" to CR,LF translation. Can either be applied above "perlio" or serve
526 as the buffer layer itself. "crlf" over "unix" is the default if system
527 distinguishes between C<O_TEXT> and C<O_BINARY> opens. (At some point
528 "unix" will be replaced by a "native" Win32 IO layer on that platform,
529 as Win32's read/write layer has various drawbacks.) The "crlf" layer is
530 a reasonable model for a layer which transforms data in some way.
534 If Configure detects C<mmap()> functions this layer is provided (with
535 "perlio" as a "base") which does "read" operations by mmap()ing the
536 file. Performance improvement is marginal on modern systems, so it is
537 mainly there as a proof of concept. It is likely to be unbundled from
538 the core at some point. The "mmap" layer is a reasonable model for a
539 minimalist "derived" layer.
543 An "internal" derivative of "perlio" which can be used to provide
544 Unread() function for layers which have no buffer or cannot be bothered.
545 (Basically this layer's C<Fill()> pops itself off the stack and so resumes
546 reading from layer below.)
550 A dummy layer which never exists on the layer stack. Instead when
551 "pushed" it actually pops the stack(!), removing itself, and any other
552 layers until it reaches a layer with the class C<PERLIO_K_RAW> bit set.
556 Another dummy layer. When pushed it pops itself and sets the
557 C<PERLIO_F_UTF8> flag on the layer which was (and now is once more) the top
562 In addition F<perlio.c> also provides a number of C<PerlIOBase_xxxx()>
563 functions which are intended to be used in the table slots of classes
564 which do not need to do anything special for a particular method.
566 =head2 Extension Layers
568 Layers can made available by extension modules. When an unknown layer is encountered
569 the PerlIO code will perform the equivalent of :
573 Where I<layer> is the unknown layer. F<PerlIO.pm> will then attempt to :
575 require PerlIO::layer;
577 If after that process the layer is still not defined then the C<open> will fail.
579 The following extension layers are bundled with perl:
587 makes this layer available, although F<PerlIO.pm> "knows" where to find it.
588 It is an example of a layer which takes an argument as it is called thus:
590 open($fh,"<:encoding(iso-8859-7)",$pathname)
596 open($fh,"...",\$scalar)
598 When a handle is so opened, then reads get bytes from the string value of I<$scalar>,
599 and writes change the value. In both cases the position in I<$scalar> starts as zero
600 but can be altered via C<seek>, and determined via C<tell>.
602 =item ":Object" or ":Perl"
604 May be provided to allow layers to be implemented as perl code - implementation
605 is being investigated.