[urisagit/Perl-Docs.git] / extras / new_text

Somewhere along the line, I learned about a way to slurp files faster
than by setting $/ to undef. The method is very simple, you do a single
read call with the size of the file (which the -s operator provides).
This bypasses the I/O loop inside perl that checks for EOF and does all
sorts of processing. I then decided to experiment and found that
sysread is even faster as you would expect. sysread bypasses all of
Perl's stdio and reads the file from the kernel buffers directly into a
Perl scalar. This is why the slurp code in File::Slurp uses
sysopen/sysread/syswrite. All the rest of the code is just to support
the various options and data passing techniques.


Benchmarks can be enlightening, informative, frustrating and
deceiving. It would make no sense to create a new and more complex slurp
module unless it also gained signifigantly in speed. So I created a
benchmark script which compares various slurp methods with differing
file sizes and calling contexts. This script can be run from the main
directory of the tarball like this:

	perl -Ilib extras/slurp_bench.pl

If you pass in an argument on the command line, it will be passed to
timethese() and it will control the duration. It defaults to -2 which
makes each benchmark run to at least 2 seconds of cpu time.

The following numbers are from a run I did on my 300Mhz sparc. You will
most likely get much faster counts on your boxes but the relative speeds
shouldn't change by much. If you see major differences on your
benchmarks, please send me the results and your Perl and OS
versions. Also you can play with the benchmark script and add more slurp
variations or data files.

The rest of this section will be discussing the results of the
benchmarks. You can refer to extras/slurp_bench.pl to see the code for
the individual benchmarks. If the benchmark name starts with cpan_, it
is either from Slurp.pm or File::Slurp.pm. Those starting with new_ are
from the new File::Slurp.pm. Those that start with file_contents_ are
from a client's code base. The rest are variations I created to
highlight certain aspects of the benchmarks.

The short and long file data is made like this:

	my @lines = ( 'abc' x 30 . "\n")  x 100 ;
	my $text = join( '', @lines ) ;

	@lines = ( 'abc' x 40 . "\n")  x 1000 ;
	$text = join( '', @lines ) ;

So the short file is 9,100 bytes and the long file is 121,000
bytes. 

=head3 	Scalar Slurp of Short File

	file_contents        651/s
	file_contents_no_OO  828/s
	cpan_read_file      1866/s
	cpan_slurp          1934/s
	read_file           2079/s
	new                 2270/s
	new_buf_ref         2403/s
	new_scalar_ref      2415/s
	sysread_file        2572/s

=head3 	Scalar Slurp of Long File

	file_contents_no_OO 82.9/s
	file_contents       85.4/s
	cpan_read_file       250/s
	cpan_slurp           257/s
	read_file            323/s
	new                  468/s
	sysread_file         489/s
	new_scalar_ref       766/s
	new_buf_ref          767/s

The primary inference you get from looking at the mumbers above is that
when slurping a file into a scalar, the longer the file, the more time
you save by returning the result via a scalar reference. The time for
the extra buffer copy can add up. The new module came out on top overall
except for the very simple sysread_file entry which was added to
highlight the overhead of the more flexible new module which isn't that
much. The file_contents entries are always the worst since they do a
list slurp and then a join, which is a classic newbie and cargo culted
style which is extremely slow. Also the OO code in file_contents slows
it down even more (I added the file_contents_no_OO entry to show this).
The two CPAN modules are decent with small files but they are laggards
compared to the new module when the file gets much larger.

=head3 	List Slurp of Short File

	cpan_read_file          589/s
	cpan_slurp_to_array     620/s
	read_file               824/s
	new_array_ref           824/s
	sysread_file            828/s
	new                     829/s
	new_in_anon_array       833/s
	cpan_slurp_to_array_ref 836/s

=head3 	List Slurp of Long File

	cpan_read_file          62.4/s
	cpan_slurp_to_array     62.7/s
	read_file               92.9/s
	sysread_file            94.8/s
	new_array_ref           95.5/s
	new                     96.2/s
	cpan_slurp_to_array_ref 96.3/s
	new_in_anon_array       97.2/s


=head3 	Scalar Spew of Short File

	cpan_write_file 1035/s
	print_file      1055/s
	syswrite_file   1135/s
	new             1519/s
	print_join_file 1766/s
	new_ref         1900/s
	syswrite_file2  2138/s

=head3 	Scalar Spew of Long File

	cpan_write_file 164/s	20
	print_file      211/s	26
	syswrite_file   236/s	25
	print_join_file 277/s	2
	new             295/s	2
	syswrite_file2  428/s	25
	new_ref         608/s	2


=head3 List Spew of Short File

	cpan_write_file  794/s
	syswrite_file   1000/s
	print_file      1013/s
	new             1399/s
	print_join_file 1557/s

=head3 	List Spew of Long File

	cpan_write_file 112/s	12
	print_file      179/s	21
	syswrite_file   181/s	19
	print_join_file 205/s	2
	new             228/s	2
Commit	Line	Data
635c7876	1	Somewhere along the line, I learned about a way to slurp files faster
	2	than by setting $/ to undef. The method is very simple, you do a single
	3	read call with the size of the file (which the -s operator provides).
	4	This bypasses the I/O loop inside perl that checks for EOF and does all
	5	sorts of processing. I then decided to experiment and found that
	6	sysread is even faster as you would expect. sysread bypasses all of
	7	Perl's stdio and reads the file from the kernel buffers directly into a
	8	Perl scalar. This is why the slurp code in File::Slurp uses
	9	sysopen/sysread/syswrite. All the rest of the code is just to support
	10	the various options and data passing techniques.
	11
	12
	13	Benchmarks can be enlightening, informative, frustrating and
	14	deceiving. It would make no sense to create a new and more complex slurp
	15	module unless it also gained signifigantly in speed. So I created a
	16	benchmark script which compares various slurp methods with differing
	17	file sizes and calling contexts. This script can be run from the main
	18	directory of the tarball like this:
	19
	20	perl -Ilib extras/slurp_bench.pl
	21
	22	If you pass in an argument on the command line, it will be passed to
	23	timethese() and it will control the duration. It defaults to -2 which
	24	makes each benchmark run to at least 2 seconds of cpu time.
	25
	26	The following numbers are from a run I did on my 300Mhz sparc. You will
	27	most likely get much faster counts on your boxes but the relative speeds
	28	shouldn't change by much. If you see major differences on your
	29	benchmarks, please send me the results and your Perl and OS
	30	versions. Also you can play with the benchmark script and add more slurp
	31	variations or data files.
	32
	33	The rest of this section will be discussing the results of the
	34	benchmarks. You can refer to extras/slurp_bench.pl to see the code for
	35	the individual benchmarks. If the benchmark name starts with cpan_, it
	36	is either from Slurp.pm or File::Slurp.pm. Those starting with new_ are
	37	from the new File::Slurp.pm. Those that start with file_contents_ are
	38	from a client's code base. The rest are variations I created to
	39	highlight certain aspects of the benchmarks.
	40
	41	The short and long file data is made like this:
	42
	43	my @lines = ( 'abc' x 30 . "\n") x 100 ;
	44	my $text = join( '', @lines ) ;
	45
	46	@lines = ( 'abc' x 40 . "\n") x 1000 ;
	47	$text = join( '', @lines ) ;
	48
	49	So the short file is 9,100 bytes and the long file is 121,000
	50	bytes.
	51
	52	=head3 Scalar Slurp of Short File
	53
	54	file_contents 651/s
	55	file_contents_no_OO 828/s
	56	cpan_read_file 1866/s
	57	cpan_slurp 1934/s
	58	read_file 2079/s
	59	new 2270/s
	60	new_buf_ref 2403/s
	61	new_scalar_ref 2415/s
	62	sysread_file 2572/s
	63
	64	=head3 Scalar Slurp of Long File
65
66	file_contents_no_OO 82.9/s
67	file_contents 85.4/s
68	cpan_read_file 250/s
69	cpan_slurp 257/s
70	read_file 323/s
71	new 468/s
72	sysread_file 489/s
73	new_scalar_ref 766/s
74	new_buf_ref 767/s
75
76	The primary inference you get from looking at the mumbers above is that
77	when slurping a file into a scalar, the longer the file, the more time
78	you save by returning the result via a scalar reference. The time for
79	the extra buffer copy can add up. The new module came out on top overall
80	except for the very simple sysread_file entry which was added to
81	highlight the overhead of the more flexible new module which isn't that
82	much. The file_contents entries are always the worst since they do a
83	list slurp and then a join, which is a classic newbie and cargo culted
84	style which is extremely slow. Also the OO code in file_contents slows
85	it down even more (I added the file_contents_no_OO entry to show this).
86	The two CPAN modules are decent with small files but they are laggards
87	compared to the new module when the file gets much larger.
88
89	=head3 List Slurp of Short File
90
91	cpan_read_file 589/s
92	cpan_slurp_to_array 620/s
93	read_file 824/s
94	new_array_ref 824/s
95	sysread_file 828/s
96	new 829/s
97	new_in_anon_array 833/s
98	cpan_slurp_to_array_ref 836/s
99
100	=head3 List Slurp of Long File
101
102	cpan_read_file 62.4/s
103	cpan_slurp_to_array 62.7/s
104	read_file 92.9/s
105	sysread_file 94.8/s
106	new_array_ref 95.5/s
107	new 96.2/s
108	cpan_slurp_to_array_ref 96.3/s
109	new_in_anon_array 97.2/s
110
111
112	=head3 Scalar Spew of Short File
113
114	cpan_write_file 1035/s
115	print_file 1055/s
116	syswrite_file 1135/s
117	new 1519/s
118	print_join_file 1766/s
119	new_ref 1900/s
120	syswrite_file2 2138/s
121
122	=head3 Scalar Spew of Long File
123
124	cpan_write_file 164/s 20
125	print_file 211/s 26
126	syswrite_file 236/s 25
127	print_join_file 277/s 2
128	new 295/s 2
129	syswrite_file2 428/s 25
130	new_ref 608/s 2
131
132
133	=head3 List Spew of Short File
134
135	cpan_write_file 794/s
136	syswrite_file 1000/s
137	print_file 1013/s
138	new 1399/s
139	print_join_file 1557/s
140
141	=head3 List Spew of Long File
142
143	cpan_write_file 112/s 12
144	print_file 179/s 21
145	syswrite_file 181/s 19
146	print_join_file 205/s 2
147	new 228/s 2
148