4 perliol - C API for Perl's implementation of IO in Layers.
8 /* Defining a layer ... */
14 This document describes the behavior and implementation of the PerlIO
15 abstraction described in L<perlapio> when C<USE_PERLIO> is defined (and
18 =head2 History and Background
20 The PerlIO abstraction was introduced in perl5.003_02 but languished as
21 just an abstraction until perl5.7.0. However during that time a number
22 of perl extensions switched to using it, so the API is mostly fixed to
23 maintain (source) compatibility.
25 The aim of the implementation is to provide the PerlIO API in a flexible
26 and platform neutral manner. It is also a trial of an "Object Oriented
27 C, with vtables" approach which may be applied to perl6.
29 =head2 Layers vs Disciplines
31 Initial discussion of the ability to modify IO streams behaviour used
32 the term "discipline" for the entities which were added. This came (I
33 believe) from the use of the term in "sfio", which in turn borrowed it
34 from "line disciplines" on Unix terminals. However, this document (and
35 the C code) uses the term "layer".
37 This is, I hope, a natural term given the implementation, and should avoid
38 connotations that are inherent in earlier uses of "discipline" for things
39 which are rather different.
41 =head2 Data Structures
43 The basic data structure is a PerlIOl:
45 typedef struct _PerlIO PerlIOl;
46 typedef struct _PerlIO_funcs PerlIO_funcs;
47 typedef PerlIOl *PerlIO;
51 PerlIOl * next; /* Lower layer */
52 PerlIO_funcs * tab; /* Functions for this layer */
53 IV flags; /* Various flags for state */
56 A C<PerlIOl *> is a pointer to the struct, and the I<application> level
57 C<PerlIO *> is a pointer to a C<PerlIOl *> - i.e. a pointer to a pointer to
58 the struct. This allows the application level C<PerlIO *> to remain
59 constant while the actual C<PerlIOl *> underneath changes. (Compare perl's
60 C<SV *> which remains constant while its C<sv_any> field changes as the
61 scalar's type changes.) An IO stream is then in general represented as a
62 pointer to this linked-list of "layers".
64 It should be noted that because of the double indirection in a C<PerlIO *>,
65 a C<< &(perlio-E<gt>next) >> "is" a C<PerlIO *>, and so to some degree
66 at least one layer can use the "standard" API on the next layer down.
68 A "layer" is composed of two parts:
74 The functions and attributes of the "layer class".
78 The per-instance data for a particular handle.
82 =head2 Functions and Attributes
84 The functions and attributes are accessed via the "tab" (for table)
85 member of C<PerlIOl>. The functions (methods of the layer "class") are
86 fixed, and are defined by the C<PerlIO_funcs> type. They are broadly the
87 same as the public C<PerlIO_xxxxx> functions:
94 IV (*Pushed)(PerlIO *f,const char *mode,SV *arg);
95 IV (*Popped)(PerlIO *f);
96 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
99 int fd, int imode, int perm,
101 int narg, SV **args);
102 SV * (*Getarg)(PerlIO *f);
103 IV (*Fileno)(PerlIO *f);
104 /* Unix-like functions - cf sfio line disciplines */
105 SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
106 SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
107 SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
108 IV (*Seek)(PerlIO *f, Off_t offset, int whence);
109 Off_t (*Tell)(PerlIO *f);
110 IV (*Close)(PerlIO *f);
111 /* Stdio-like buffered IO functions */
112 IV (*Flush)(PerlIO *f);
113 IV (*Fill)(PerlIO *f);
114 IV (*Eof)(PerlIO *f);
115 IV (*Error)(PerlIO *f);
116 void (*Clearerr)(PerlIO *f);
117 void (*Setlinebuf)(PerlIO *f);
118 /* Perl's snooping functions */
119 STDCHAR * (*Get_base)(PerlIO *f);
120 Size_t (*Get_bufsiz)(PerlIO *f);
121 STDCHAR * (*Get_ptr)(PerlIO *f);
122 SSize_t (*Get_cnt)(PerlIO *f);
123 void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
128 The first few members of the struct give a "name" for the layer, the
129 size to C<malloc> for the per-instance data, and some flags which are
130 attributes of the class as whole (such as whether it is a buffering
131 layer), then follow the functions which fall into four basic groups:
137 Opening and setup functions
145 Stdio class buffering options.
149 Functions to support Perl's traditional "fast" access to the buffer.
153 A layer does not have to implement all the functions, but the whole table has
154 to be present. Unimplemented slots can be NULL (which will result in an error
155 when called) or can be filled in with stubs to "inherit" behaviour from
156 a "base class". This "inheritance" is fixed for all instances of the layer,
157 but as the layer chooses which stubs to populate the table, limited
158 "multiple inheritance" is possible.
160 =head2 Per-instance Data
162 The per-instance data are held in memory beyond the basic PerlIOl struct,
163 by making a PerlIOl the first member of the layer's struct thus:
167 struct _PerlIO base; /* Base "class" info */
168 STDCHAR * buf; /* Start of buffer */
169 STDCHAR * end; /* End of valid part of buffer */
170 STDCHAR * ptr; /* Current position in buffer */
171 Off_t posn; /* Offset of buf into the file */
172 Size_t bufsiz; /* Real size of buffer */
173 IV oneword; /* Emergency buffer */
176 In this way (as for perl's scalars) a pointer to a PerlIOBuf can be treated
177 as a pointer to a PerlIOl.
179 =head2 Layers in action.
183 +-----------+ +----------+ +--------+
184 PerlIO ->| |--->| next |--->| NULL |
185 +-----------+ +----------+ +--------+
186 | | | buffer | | fd |
187 +-----------+ | | +--------+
191 The above attempts to show how the layer scheme works in a simple case.
192 The application's C<PerlIO *> points to an entry in the table(s)
193 representing open (allocated) handles. For example the first three slots
194 in the table correspond to C<stdin>,C<stdout> and C<stderr>. The table
195 in turn points to the current "top" layer for the handle - in this case
196 an instance of the generic buffering layer "perlio". That layer in turn
197 points to the next layer down - in this case the lowlevel "unix" layer.
199 The above is roughly equivalent to a "stdio" buffered stream, but with
200 much more flexibility:
206 If Unix level C<read>/C<write>/C<lseek> is not appropriate for (say)
207 sockets then the "unix" layer can be replaced (at open time or even
208 dynamically) with a "socket" layer.
212 Different handles can have different buffering schemes. The "top" layer
213 could be the "mmap" layer if reading disk files was quicker using C<mmap>
214 than C<read>. An "unbuffered" stream can be implemented simply by
215 not having a buffer layer.
219 Extra layers can be inserted to process the data as it flows through.
220 This was the driving need for including the scheme in perl 5.7.0+ - we
221 needed a mechanism to allow data to be translated between perl's
222 internal encoding (conceptually at least Unicode as UTF-8), and the
223 "native" format used by the system. This is provided by the
224 ":encoding(xxxx)" layer which typically sits above the buffering layer.
228 A layer can be added that does "\n" to CRLF translation. This layer can be used
229 on any platform, not just those that normally do such things.
233 =head2 Per-instance flag bits
235 The generic flag bits are a hybrid of C<O_XXXXX> style flags deduced from
236 the mode string passed to C<PerlIO_open()>, and state bits for typical buffer
245 =item PERLIO_F_CANWRITE
247 Writes are permitted, i.e. opened as "w" or "r+" or "a", etc.
249 =item PERLIO_F_CANREAD
251 Reads are permitted i.e. opened "r" or "w+" (or even "a+" - ick).
255 An error has occurred (for C<PerlIO_error()>)
257 =item PERLIO_F_TRUNCATE
259 Truncate file suggested by open mode.
261 =item PERLIO_F_APPEND
263 All writes should be appends.
267 Layer is performing Win32-like "\n" mapped to CR,LF for output and CR,LF
268 mapped to "\n" for input. Normally the provided "crlf" layer is the only
269 layer that need bother about this. C<PerlIO_binmode()> will mess with this
270 flag rather than add/remove layers if the C<PERLIO_K_CANCRLF> bit is set
271 for the layers class.
275 Data written to this layer should be UTF-8 encoded; data provided
276 by this layer should be considered UTF-8 encoded. Can be set on any layer
277 by ":utf8" dummy layer. Also set on ":encoding" layer.
281 Layer is unbuffered - i.e. write to next layer down should occur for
282 each write to this layer.
286 The buffer for this layer currently holds data written to it but not sent
291 The buffer for this layer currently holds unconsumed data read from
294 =item PERLIO_F_LINEBUF
296 Layer is line buffered. Write data should be passed to next layer down
297 whenever a "\n" is seen. Any data beyond the "\n" should then be
302 File has been C<unlink()>ed, or should be deleted on C<close()>.
308 =item PERLIO_F_FASTGETS
310 This instance of this layer supports the "fast C<gets>" interface.
311 Normally set based on C<PERLIO_K_FASTGETS> for the class and by the
312 existence of the function(s) in the table. However a class that
313 normally provides that interface may need to avoid it on a
314 particular instance. The "pending" layer needs to do this when
315 it is pushed above a layer which does not support the interface.
316 (Perl's C<sv_gets()> does not expect the streams fast C<gets> behaviour
317 to change during one "get".)
321 =head2 Methods in Detail
325 =item IV (*Pushed)(PerlIO *f,const char *mode, SV *arg);
327 The only absolutely mandatory method. Called when the layer is pushed onto the stack.
328 The C<mode> argument may be NULL if this occurs post-open. The C<arg> will be non-C<NULL>
329 if an argument string was passed. In most cases this should call
330 C<PerlIOBase_pushed()> to convert C<mode> into the appropriate
331 C<PERLIO_F_XXXXX> flags in addition to any actions the layer itself takes.
332 If a layer is not expecting an argument it need neither save the one passed to it, nor
333 provide C<Getarg()> (it could perhaps C<Perl_warn> that the argument was un-expected).
335 =item IV (*Popped)(PerlIO *f);
337 Called when the layer is popped from the stack. A layer will normally be
338 popped after C<Close()> is called. But a layer can be popped without being
339 closed if the program is dynamically managing layers on the stream. In
340 such cases C<Popped()> should free any resources (buffers, translation
341 tables, ...) not held directly in the layer's struct.
342 It should also C<Unread()> any unconsumed data that has been read and buffered
343 from the layer below back to that layer, so that it can be re-provided to what
346 =item PerlIO * (*Open)(...);
348 The C<Open()> method has lots of arguments because it combines the functions
349 of perl's C<open>, C<PerlIO_open>, perl's C<sysopen>, C<PerlIO_fdopen> and C<PerlIO_reopen>.
350 The full prototype is as follows:
352 PerlIO * (*Open)(pTHX_ PerlIO_funcs *tab,
355 int fd, int imode, int perm,
357 int narg, SV **args);
359 Open should (perhaps indirectly) call C<PerlIO_allocate()> to allocate a slot in the table and
360 associate it with the layers information for the opened file, by calling C<PerlIO_push>.
361 The I<layers> AV is an array of all the layers destined for the C<PerlIO *>,
362 and any arguments passed to them, I<n> is the index into that array of the
363 layer being called. The macro C<PerlIOArg> will return a (possibly C<NULL>) SV *
364 for the argument passed to the layer.
366 The I<mode> string is an "C<fopen()>-like" string which would match the regular
367 expression C</^[I#]?[rwa]\+?[bt]?$/>.
369 The C<'I'> prefix is used during creation of C<stdin>..C<stderr> via special
370 C<PerlIO_fdopen> calls; the C<'#'> prefix means that this is C<sysopen> and that I<imode> and
371 I<perm> should be passed to C<PerlLIO_open3>; C<'r'> means B<r>ead, C<'w'> means B<w>rite
372 and C<'a'> means B<a>ppend. The C<'+'> suffix means that both reading and writing/appending
373 are permitted. The C<'b'> suffix means file should be binary, and C<'t'> means it
374 is text. (Binary/Text should be ignored by almost all layers and binary IO done,
375 with PerlIO. The C<:crlf> layer should be pushed to handle the distinction.)
377 If I<old> is not C<NULL> then this is a C<PerlIO_reopen>. Perl itself does not use
378 this (yet?) and semantics are a little vague.
380 If I<fd> not negative then it is the numeric file descriptor I<fd>, which will
381 be open in a manner compatible with the supplied mode string, the call is
382 thus equivalent to C<PerlIO_fdopen>. In this case I<nargs> will be zero.
384 If I<nargs> is greater than zero then it gives the number of arguments passed
385 to C<open>, otherwise it will be 1 if for example C<PerlIO_open> was called.
386 In simple cases SvPV(*args) is the pathname to open.
388 Having said all that translation-only layers do not need to provide C<Open()> at all,
389 but rather leave the opening to a lower level layer and wait to be "pushed".
390 If a layer does provide C<Open()> it should normally call the C<Open()> method
391 of next layer down (if any) and then push itself on top if that succeeds.
393 =item SV * (*Getarg)(PerlIO *f);
395 Optional. If present should return an SV * representing the string argument
396 passed to the layer when it was pushed. e.g. ":encoding(ascii)" would
397 return an SvPV with value "ascii".
399 =item IV (*Fileno)(PerlIO *f);
401 Returns the Unix/Posix numeric file descriptor for the handle. Normally
402 C<PerlIOBase_fileno()> (which just asks next layer down) will suffice
405 =item SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
407 Basic read operation. Returns actual bytes read, or -1 on an error.
408 Typically will call Fill and manipulate pointers (possibly via the API).
409 C<PerlIOBuf_read()> may be suitable for derived classes which provide
412 =item SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
414 A superset of stdio's C<ungetc()>. Should arrange for future reads to
415 see the bytes in C<vbuf>. If there is no obviously better implementation
416 then C<PerlIOBase_unread()> provides the function by pushing a "fake"
417 "pending" layer above the calling layer.
419 =item SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
421 Basic write operation. Returns bytes written or -1 on an error.
423 =item IV (*Seek)(PerlIO *f, Off_t offset, int whence);
425 Position the file pointer. Should normally call its own C<Flush> method and
426 then the C<Seek> method of next layer down.
428 =item Off_t (*Tell)(PerlIO *f);
430 Return the file pointer. May be based on layers cached concept of
431 position to avoid overhead.
433 =item IV (*Close)(PerlIO *f);
435 Close the stream. Should normally call C<PerlIOBase_close()> to flush
436 itself and close layers below, and then deallocate any data structures
437 (buffers, translation tables, ...) not held directly in the data
440 =item IV (*Flush)(PerlIO *f);
442 Should make stream's state consistent with layers below. That is, any
443 buffered write data should be written, and file position of lower layers
444 adjusted for data read from below but not actually consumed.
445 (Should perhaps C<Unread()> such data to the lower layer.)
447 =item IV (*Fill)(PerlIO *f);
449 The buffer for this layer should be filled (for read) from layer below.
451 =item IV (*Eof)(PerlIO *f);
453 Return end-of-file indicator. C<PerlIOBase_eof()> is normally sufficient.
455 =item IV (*Error)(PerlIO *f);
457 Return error indicator. C<PerlIOBase_error()> is normally sufficient.
459 =item void (*Clearerr)(PerlIO *f);
461 Clear end-of-file and error indicators. Should call C<PerlIOBase_clearerr()>
462 to set the C<PERLIO_F_XXXXX> flags, which may suffice.
464 =item void (*Setlinebuf)(PerlIO *f);
466 Mark the stream as line buffered. C<PerlIOBase_setlinebuf()> sets the
467 PERLIO_F_LINEBUF flag and is normally sufficient.
469 =item STDCHAR * (*Get_base)(PerlIO *f);
471 Allocate (if not already done so) the read buffer for this layer and
472 return pointer to it.
474 =item Size_t (*Get_bufsiz)(PerlIO *f);
476 Return the number of bytes that last C<Fill()> put in the buffer.
478 =item STDCHAR * (*Get_ptr)(PerlIO *f);
480 Return the current read pointer relative to this layer's buffer.
482 =item SSize_t (*Get_cnt)(PerlIO *f);
484 Return the number of bytes left to be read in the current buffer.
486 =item void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
488 Adjust the read pointer and count of bytes to match C<ptr> and/or C<cnt>.
489 The application (or layer above) must ensure they are consistent.
490 (Checking is allowed by the paranoid.)
497 The file C<perlio.c> provides the following layers:
503 A basic non-buffered layer which calls Unix/POSIX C<read()>, C<write()>,
504 C<lseek()>, C<close()>. No buffering. Even on platforms that distinguish
505 between O_TEXT and O_BINARY this layer is always O_BINARY.
509 A very complete generic buffering layer which provides the whole of
510 PerlIO API. It is also intended to be used as a "base class" for other
511 layers. (For example its C<Read()> method is implemented in terms of the
512 C<Get_cnt()>/C<Get_ptr()>/C<Set_ptrcnt()> methods).
514 "perlio" over "unix" provides a complete replacement for stdio as seen
515 via PerlIO API. This is the default for USE_PERLIO when system's stdio
516 does not permit perl's "fast gets" access, and which do not distinguish
517 between C<O_TEXT> and C<O_BINARY>.
521 A layer which provides the PerlIO API via the layer scheme, but
522 implements it by calling system's stdio. This is (currently) the default
523 if system's stdio provides sufficient access to allow perl's "fast gets"
524 access and which do not distinguish between C<O_TEXT> and C<O_BINARY>.
528 A layer derived using "perlio" as a base class. It provides Win32-like
529 "\n" to CR,LF translation. Can either be applied above "perlio" or serve
530 as the buffer layer itself. "crlf" over "unix" is the default if system
531 distinguishes between C<O_TEXT> and C<O_BINARY> opens. (At some point
532 "unix" will be replaced by a "native" Win32 IO layer on that platform,
533 as Win32's read/write layer has various drawbacks.) The "crlf" layer is
534 a reasonable model for a layer which transforms data in some way.
538 If Configure detects C<mmap()> functions this layer is provided (with
539 "perlio" as a "base") which does "read" operations by mmap()ing the
540 file. Performance improvement is marginal on modern systems, so it is
541 mainly there as a proof of concept. It is likely to be unbundled from
542 the core at some point. The "mmap" layer is a reasonable model for a
543 minimalist "derived" layer.
547 An "internal" derivative of "perlio" which can be used to provide
548 Unread() function for layers which have no buffer or cannot be bothered.
549 (Basically this layer's C<Fill()> pops itself off the stack and so resumes
550 reading from layer below.)
554 A dummy layer which never exists on the layer stack. Instead when
555 "pushed" it actually pops the stack(!), removing itself, and any other
556 layers until it reaches a layer with the class C<PERLIO_K_RAW> bit set.
560 Another dummy layer. When pushed it pops itself and sets the
561 C<PERLIO_F_UTF8> flag on the layer which was (and now is once more) the top
566 In addition F<perlio.c> also provides a number of C<PerlIOBase_xxxx()>
567 functions which are intended to be used in the table slots of classes
568 which do not need to do anything special for a particular method.
570 =head2 Extension Layers
572 Layers can made available by extension modules. When an unknown layer is encountered
573 the PerlIO code will perform the equivalent of :
577 Where I<layer> is the unknown layer. F<PerlIO.pm> will then attempt to :
579 require PerlIO::layer;
581 If after that process the layer is still not defined then the C<open> will fail.
583 The following extension layers are bundled with perl:
591 makes this layer available, although F<PerlIO.pm> "knows" where to find it.
592 It is an example of a layer which takes an argument as it is called thus:
594 open($fh,"<:encoding(iso-8859-7)",$pathname)
600 open($fh,"...",\$scalar)
602 When a handle is so opened, then reads get bytes from the string value of I<$scalar>,
603 and writes change the value. In both cases the position in I<$scalar> starts as zero
604 but can be altered via C<seek>, and determined via C<tell>.
606 =item ":Object" or ":Perl"
608 May be provided to allow layers to be implemented as perl code - implementation
609 is being investigated.