4 perliol - C API for Perl's implementation of IO in Layers.
8 /* Defining a layer ... */
14 This document describes the behavior and implementation of the PerlIO
15 abstraction described in L<perlapio> when C<USE_PERLIO> is defined (and
18 =head2 History and Background
20 The PerlIO abstraction was introduced in perl5.003_02 but languished as
21 just an abstraction until perl5.7.0. However during that time a number
22 of perl extentions switched to using it, so the API is mostly fixed to
23 maintain (source) compatibility.
25 The aim of the implementation is to provide the PerlIO API in a flexible
26 and platform neutral manner. It is also a trial of an "Object Oriented
27 C, with vtables" approach which may be applied to perl6.
29 =head2 Layers vs Disciplines
31 Initial discussion of the ability to modify IO streams behaviour used
32 the term "discipline" for the entities which were added. This came (I
33 believe) from the use of the term in "sfio", which in turn borrowed it
34 from "line disciplines" on Unix terminals. However, this document (and
35 the C code) uses the term "layer".
37 This is, I hope, a natural term given the implementation, and should avoid
38 connotations that are inherent in earlier uses of "discipline" for things
39 which are rather different.
41 =head2 Data Structures
43 The basic data structure is a PerlIOl:
45 typedef struct _PerlIO PerlIOl;
46 typedef struct _PerlIO_funcs PerlIO_funcs;
47 typedef PerlIOl *PerlIO;
51 PerlIOl * next; /* Lower layer */
52 PerlIO_funcs * tab; /* Functions for this layer */
53 IV flags; /* Various flags for state */
56 A C<PerlIOl *> is a pointer to to the struct, and the I<application> level
57 C<PerlIO *> is a pointer to a C<PerlIOl *> - i.e. a pointer to a pointer to
58 the struct. This allows the application level C<PerlIO *> to remain
59 constant while the actual C<PerlIOl *> underneath changes. (Compare perl's
60 C<SV *> which remains constant while its C<sv_any> field changes as the
61 scalar's type changes.) An IO stream is then in general represented as a
62 pointer to this linked-list of "layers".
64 It should be noted that because of the double indirection in a C<PerlIO *>,
65 a C<< &(perlio-E<gt>next) >> "is" a C<PerlIO *>, and so to some degree
66 at least one layer can use the "standard" API on the next layer down.
68 A "layer" is composed of two parts:
72 =item 1. The functions and attributes of the "layer class".
74 =item 2. The per-instance data for a particular handle.
78 =head2 Functions and Attributes
80 The functions and attributes are accessed via the "tab" (for table)
81 member of C<PerlIOl>. The functions (methods of the layer "class") are
82 fixed, and are defined by the C<PerlIO_funcs> type. They are broadly the
83 same as the public C<PerlIO_xxxxx> functions:
90 IV (*Pushed)(PerlIO *f,const char *mode,SV *arg);
91 IV (*Popped)(PerlIO *f);
92 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
95 int fd, int imode, int perm,
98 SV * (*Getarg)(PerlIO *f);
99 IV (*Fileno)(PerlIO *f);
100 /* Unix-like functions - cf sfio line disciplines */
101 SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
102 SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
103 SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
104 IV (*Seek)(PerlIO *f, Off_t offset, int whence);
105 Off_t (*Tell)(PerlIO *f);
106 IV (*Close)(PerlIO *f);
107 /* Stdio-like buffered IO functions */
108 IV (*Flush)(PerlIO *f);
109 IV (*Fill)(PerlIO *f);
110 IV (*Eof)(PerlIO *f);
111 IV (*Error)(PerlIO *f);
112 void (*Clearerr)(PerlIO *f);
113 void (*Setlinebuf)(PerlIO *f);
114 /* Perl's snooping functions */
115 STDCHAR * (*Get_base)(PerlIO *f);
116 Size_t (*Get_bufsiz)(PerlIO *f);
117 STDCHAR * (*Get_ptr)(PerlIO *f);
118 SSize_t (*Get_cnt)(PerlIO *f);
119 void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
124 The first few members of the struct give a "name" for the layer, the
125 size to C<malloc> for the per-instance data, and some flags which are
126 attributes of the class as whole (such as whether it is a buffering
127 layer), then follow the functions which fall into four basic groups:
131 =item 1. Opening and setup functions
133 =item 2. Basic IO operations
135 =item 3. Stdio class buffering options.
137 =item 4. Functions to support Perl's traditional "fast" access to the buffer.
141 A layer does not have to implement all the functions, but the whole table has
142 to be present. Unimplemented slots can be NULL (which will will result in an error
143 when called) or can be filled in with stubs to "inherit" behaviour from
144 a "base class". This "inheritance" is fixed for all instances of the layer,
145 but as the layer chooses which stubs to populate the table, limited
146 "multiple inheritance" is possible.
148 =head2 Per-instance Data
150 The per-instance data are held in memory beyond the basic PerlIOl struct,
151 by making a PerlIOl the first member of the layer's struct thus:
155 struct _PerlIO base; /* Base "class" info */
156 STDCHAR * buf; /* Start of buffer */
157 STDCHAR * end; /* End of valid part of buffer */
158 STDCHAR * ptr; /* Current position in buffer */
159 Off_t posn; /* Offset of buf into the file */
160 Size_t bufsiz; /* Real size of buffer */
161 IV oneword; /* Emergency buffer */
164 In this way (as for perl's scalars) a pointer to a PerlIOBuf can be treated
165 as a pointer to a PerlIOl.
167 =head2 Layers in action.
171 +-----------+ +----------+ +--------+
172 PerlIO ->| |--->| next |--->| NULL |
173 +-----------+ +----------+ +--------+
174 | | | buffer | | fd |
175 +-----------+ | | +--------+
179 The above attempts to show how the layer scheme works in a simple case.
180 The application's C<PerlIO *> points to an entry in the table(s)
181 representing open (allocated) handles. For example the first three slots
182 in the table correspond to C<stdin>,C<stdout> and C<stderr>. The table
183 in turn points to the current "top" layer for the handle - in this case
184 an instance of the generic buffering layer "perlio". That layer in turn
185 points to the next layer down - in this case the lowlevel "unix" layer.
187 The above is roughly equivalent to a "stdio" buffered stream, but with
188 much more flexibility:
194 If Unix level C<read>/C<write>/C<lseek> is not appropriate for (say)
195 sockets then the "unix" layer can be replaced (at open time or even
196 dynamically) with a "socket" layer.
200 Different handles can have different buffering schemes. The "top" layer
201 could be the "mmap" layer if reading disk files was quicker using C<mmap>
202 than C<read>. An "unbuffered" stream can be implemented simply by
203 not having a buffer layer.
207 Extra layers can be inserted to process the data as it flows through.
208 This was the driving need for including the scheme in perl 5.7.0+ - we
209 needed a mechanism to allow data to be translated bewteen perl's
210 internal encoding (conceptually at least Unicode as UTF-8), and the
211 "native" format used by the system. This is provided by the
212 ":encoding(xxxx)" layer which typically sits above the buffering layer.
216 A layer can be added that does "\n" to CRLF translation. This layer can be used
217 on any platform, not just those that normally do such things.
221 =head2 Per-instance flag bits
223 The generic flag bits are a hybrid of C<O_XXXXX> style flags deduced from
224 the mode string passed to C<PerlIO_open()>, and state bits for typical buffer
233 =item PERLIO_F_CANWRITE
235 Writes are permitted, i.e. opened as "w" or "r+" or "a", etc.
237 =item PERLIO_F_CANREAD
239 Reads are permitted i.e. opened "r" or "w+" (or even "a+" - ick).
243 An error has occured (for C<PerlIO_error()>)
245 =item PERLIO_F_TRUNCATE
247 Truncate file suggested by open mode.
249 =item PERLIO_F_APPEND
251 All writes should be appends.
255 Layer is performing Win32-like "\n" mapped to CR,LF for output and CR,LF
256 mapped to "\n" for input. Normally the provided "crlf" layer is the only
257 layer that need bother about this. C<PerlIO_binmode()> will mess with this
258 flag rather than add/remove layers if the C<PERLIO_K_CANCRLF> bit is set
259 for the layers class.
263 Data written to this layer should be UTF-8 encoded; data provided
264 by this layer should be considered UTF-8 encoded. Can be set on any layer
265 by ":utf8" dummy layer. Also set on ":encoding" layer.
269 Layer is unbuffered - i.e. write to next layer down should occur for
270 each write to this layer.
274 The buffer for this layer currently holds data written to it but not sent
279 The buffer for this layer currently holds unconsumed data read from
282 =item PERLIO_F_LINEBUF
284 Layer is line buffered. Write data should be passed to next layer down
285 whenever a "\n" is seen. Any data beyond the "\n" should then be
290 File has been C<unlink()>ed, or should be deleted on C<close()>.
296 =item PERLIO_F_FASTGETS
298 This instance of this layer supports the "fast C<gets>" interface.
299 Normally set based on C<PERLIO_K_FASTGETS> for the class and by the
300 existance of the function(s) in the table. However a class that
301 normally provides that interface may need to avoid it on a
302 particular instance. The "pending" layer needs to do this when
303 it is pushed above an layer which does not support the interface.
304 (Perl's C<sv_gets()> does not expect the streams fast C<gets> behaviour
305 to change during one "get".)
309 =head2 Methods in Detail
313 =item IV (*Pushed)(PerlIO *f,const char *mode, SV *arg);
315 The only absoultely mandatory method. Called when the layer is pushed onto the stack.
316 The C<mode> argument may be NULL if this occurs post-open. The C<arg> will be non-C<NULL>
317 if an argument string was passed. In most cases this should call
318 C<PerlIOBase_pushed()> to convert C<mode> into the appropriate
319 C<PERLIO_F_XXXXX> flags in addition to any actions the layer itself takes.
320 If a layer is not expecting an argument it need neither save the one passed to it, nor
321 provide C<Getarg()> (it could perhaps C<Perl_warn> that the argument was un-expected).
323 =item IV (*Popped)(PerlIO *f);
325 Called when the layer is popped from the stack. A layer will normally be
326 popped after C<Close()> is called. But a layer can be popped without being
327 closed if the program is dynamically managing layers on the stream. In
328 such cases C<Popped()> should free any resources (buffers, translation
329 tables, ...) not held directly in the layer's struct.
330 It should also C<Unread()> any unconsumed data that has been read and buffered
331 from the layer below back to that layer, so that it can be re-provided to what
334 =item PerlIO * (*Open)(...);
336 The C<Open()> method has lots of arguments because it combines the functions
337 of perl's C<open>, C<PerlIO_open>, perl's C<sysopen>, C<PerlIO_fdopen> and C<PerlIO_reopen>.
338 The full prototype is as follows:
340 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
343 int fd, int imode, int perm,
345 int narg, SV **args);
347 Open should (perhaps indirectly) call C<PerlIO_allocate()> to allocate a slot in the table and
348 associate it with the layers information for the opened file, by calling C<PerlIO_push>.
349 The I<layers> AV is an array of all the layers destined for the C<PerlIO *>,
350 and any arguments passed to them, I<n> is the index into that array of the
351 layer being called. The macro C<PerlIOArg> will return a (possibly C<NULL>) SV *
352 for the argument passed to the layer.
354 The I<mode> string is an "C<fopen()>-like" string which would match the regular
355 expression C</^[I#]?[rwa]\+?[bt]?$/>.
357 The C<'I'> prefix is used during creation of C<stdin>..C<stderr> via special
358 C<PerlIO_fdopen> calls; the C<'#'> prefix means that this is C<sysopen> and that I<imode> and
359 I<perm> should be passed to C<PerlLIO_open3>; C<'r'> means B<r>ead, C<'w'> means B<w>rite
360 and C<'a'> means B<a>ppend. The C<'+'> suffix means that both reading and writing/appending
361 are permited. The C<'b'> suffix means file should be binary, and C<'t'> means it
362 is text. (Binary/Text should be ignored by almost all layers and binary IO done,
363 with PerlIO. The C<:crlf> layer should be pushed to handle the distinction.)
365 If I<old> is not C<NULL> then this is a C<PerlIO_reopen>. Perl iteself does not use
366 this (yet?) and semantics are a little vague.
368 If I<fd> not negative then it is the numeric file descriptor I<fd>, which will
369 be open in an manner compatible with the supplied mode string, the call is
370 thus equivalent to C<PerlIO_fdopen>. In this case I<nargs> will be zero.
372 If I<nargs> is greater than zero then it gives the number of arguments passed
373 to C<open>, otherwise it will be 1 if for example C<PerlIO_open> was called.
374 In simple cases SvPV(*args) is the pathname to open.
376 Having said all that translation-only layers do not need to provide C<Open()> at all,
377 but rather leave the opening to a lower level layer and wait to be "pushed".
378 If a layer does provide C<Open()> it should normaly call the C<Open()> method
379 of next layer down (if any) and then push itself on top if that succeeds.
381 =item SV * (*Getarg)(PerlIO *f);
383 Optional. If present should return an SV * representing the string argument
384 passed to the layer when it was pushed. e.g. ":encoding(ascii)" would
385 return an SvPV with value "ascii".
387 =item IV (*Fileno)(PerlIO *f);
389 Returns the Unix/Posix numeric file decriptor for the handle. Normally
390 C<PerlIOBase_fileno()> (which just asks next layer down) will suffice
393 =item SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
395 Basic read operation. Returns actual bytes read, or -1 on an error.
396 Typically will call Fill and manipulate pointers (possibly via the API).
397 C<PerlIOBuf_read()> may be suitable for derived classes which provide
400 =item SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
402 A superset of stdio's C<ungetc()>. Should arrange for future reads to
403 see the bytes in C<vbuf>. If there is no obviously better implementation
404 then C<PerlIOBase_unread()> provides the function by pushing a "fake"
405 "pending" layer above the calling layer.
407 =item SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
409 Basic write operation. Returns bytes written or -1 on an error.
411 =item IV (*Seek)(PerlIO *f, Off_t offset, int whence);
413 Position the file pointer. Should normally call its own C<Flush> method and
414 then the C<Seek> method of next layer down.
416 =item Off_t (*Tell)(PerlIO *f);
418 Return the file pointer. May be based on layers cached concept of
419 position to avoid overhead.
421 =item IV (*Close)(PerlIO *f);
423 Close the stream. Should normally call C<PerlIOBase_close()> to flush
424 itself and close layers below, and then deallocate any data structures
425 (buffers, translation tables, ...) not held directly in the data
428 =item IV (*Flush)(PerlIO *f);
430 Should make stream's state consistent with layers below. That is, any
431 buffered write data should be written, and file position of lower layers
432 adjusted for data read fron below but not actually consumed.
433 (Should perhaps C<Unread()> such data to the lower layer.)
435 =item IV (*Fill)(PerlIO *f);
437 The buffer for this layer should be filled (for read) from layer below.
439 =item IV (*Eof)(PerlIO *f);
441 Return end-of-file indicator. C<PerlIOBase_eof()> is normally sufficient.
443 =item IV (*Error)(PerlIO *f);
445 Return error indicator. C<PerlIOBase_error()> is normally sufficient.
447 =item void (*Clearerr)(PerlIO *f);
449 Clear end-of-file and error indicators. Should call C<PerlIOBase_clearerr()>
450 to set the C<PERLIO_F_XXXXX> flags, which may suffice.
452 =item void (*Setlinebuf)(PerlIO *f);
454 Mark the stream as line buffered. C<PerlIOBase_setlinebuf()> sets the
455 PERLIO_F_LINEBUF flag and is normally sufficient.
457 =item STDCHAR * (*Get_base)(PerlIO *f);
459 Allocate (if not already done so) the read buffer for this layer and
460 return pointer to it.
462 =item Size_t (*Get_bufsiz)(PerlIO *f);
464 Return the number of bytes that last C<Fill()> put in the buffer.
466 =item STDCHAR * (*Get_ptr)(PerlIO *f);
468 Return the current read pointer relative to this layer's buffer.
470 =item SSize_t (*Get_cnt)(PerlIO *f);
472 Return the number of bytes left to be read in the current buffer.
474 =item void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
476 Adjust the read pointer and count of bytes to match C<ptr> and/or C<cnt>.
477 The application (or layer above) must ensure they are consistent.
478 (Checking is allowed by the paranoid.)
485 The file C<perlio.c> provides the following layers:
491 A basic non-buffered layer which calls Unix/POSIX C<read()>, C<write()>,
492 C<lseek()>, C<close()>. No buffering. Even on platforms that distinguish
493 between O_TEXT and O_BINARY this layer is always O_BINARY.
497 A very complete generic buffering layer which provides the whole of
498 PerlIO API. It is also intended to be used as a "base class" for other
499 layers. (For example its C<Read()> method is implemented in terms of the
500 C<Get_cnt()>/C<Get_ptr()>/C<Set_ptrcnt()> methods).
502 "perlio" over "unix" provides a complete replacement for stdio as seen
503 via PerlIO API. This is the default for USE_PERLIO when system's stdio
504 does not permit perl's "fast gets" access, and which do not distinguish
505 between C<O_TEXT> and C<O_BINARY>.
509 A layer which provides the PerlIO API via the layer scheme, but
510 implements it by calling system's stdio. This is (currently) the default
511 if system's stdio provides sufficient access to allow perl's "fast gets"
512 access and which do not distinguish between C<O_TEXT> and C<O_BINARY>.
516 A layer derived using "perlio" as a base class. It provides Win32-like
517 "\n" to CR,LF translation. Can either be applied above "perlio" or serve
518 as the buffer layer itself. "crlf" over "unix" is the default if system
519 distinguishes between C<O_TEXT> and C<O_BINARY> opens. (At some point
520 "unix" will be replaced by a "native" Win32 IO layer on that platform,
521 as Win32's read/write layer has various drawbacks.) The "crlf" layer is
522 a reasonable model for a layer which transforms data in some way.
526 If Configure detects C<mmap()> functions this layer is provided (with
527 "perlio" as a "base") which does "read" operations by mmap()ing the
528 file. Performance improvement is marginal on modern systems, so it is
529 mainly there as a proof of concept. It is likely to be unbundled from
530 the core at some point. The "mmap" layer is a reasonable model for a
531 minimalist "derived" layer.
535 An "internal" derivative of "perlio" which can be used to provide
536 Unread() function for layers which have no buffer or cannot be bothered.
537 (Basically this layer's C<Fill()> pops itself off the stack and so resumes
538 reading from layer below.)
542 A dummy layer which never exists on the layer stack. Instead when
543 "pushed" it actually pops the stack(!), removing itself, and any other
544 layers until it reaches a layer with the class C<PERLIO_K_RAW> bit set.
548 Another dummy layer. When pushed it pops itself and sets the
549 C<PERLIO_F_UTF8> flag on the layer which was (and now is once more) the top
554 In addition F<perlio.c> also provides a number of C<PerlIOBase_xxxx()>
555 functions which are intended to be used in the table slots of classes
556 which do not need to do anything special for a particular method.
558 =head2 Extension Layers
560 Layers can made available by extension modules. When an unknown layer is encountered
561 the PerlIO code will perform the equivalent of :
565 Where I<layer> is the unknown layer. F<PerlIO.pm> will then attempt to :
567 require PerlIO::layer;
569 If after that process the layer is still not defined then the C<open> will fail.
571 The following extension layers are bundled with perl:
579 makes this layer available, although F<PerlIO.pm> "knows" where to find it.
580 It is an example of a layer which takes an argument as it is called thus:
582 open($fh,"<:encoding(iso-8859-7)",$pathname)
588 open($fh,"...",\$scalar)
590 When a handle is so opened, then reads get bytes from the string value of I<$scalar>,
591 and writes change the value. In both cases the position in I<$scalar> starts as zero
592 but can be altered via C<seek>, and determined via C<tell>.
594 =item ":Object" or ":Perl"
596 May be provided to allow layers to be implemented as perl code - implementation
597 is being investigated.