Commit | Line | Data |
9dc10e63 |
1 | =head1 NAME |
2 | |
3 | Test::Tutorial - A tutorial about writing really basic tests |
4 | |
5 | =head1 DESCRIPTION |
6 | |
7 | AHHHHHHH!!!! NOT B<TESTING>! Anything but testing! |
8 | Beat me, whip me, send me to I<Detroit>, but don't make |
9 | me write tests! |
10 | |
11 | *sob* |
12 | |
13 | Besides, I don't know how to write the damned things. |
14 | |
15 | Is this you? Is writing tests right up there with writing |
16 | documentation and having your fingernails pulled out? |
17 | |
18 | |
19 | =head2 Nuts and bolts of testing. |
20 | |
21 | Here's the most basic test program. |
22 | |
23 | #!/usr/bin/perl -w |
24 | |
25 | print "1..1\n"; |
26 | |
27 | print 1 + 1 == 2 ? "ok 1\n" : "not ok 1\n"; |
28 | |
29 | since 1 + 1 is 2, it prints: |
30 | |
31 | 1..1 |
32 | ok 1 |
33 | |
34 | What this says is: C<1..1> "I'm going to run one test." [1] C<ok 1> |
35 | "The first test passed". And that's about all magic there is to |
36 | testing. Your basic unit of testing is the 'ok'. For each thing you |
37 | test, an 'ok' is printed. Simple. Test::Harness interprets your test |
38 | results to determine if you succeeded or failed (more on that later). |
39 | |
40 | Writing all these print statements rapidly gets tedious. Fortunately, |
41 | there's Test::Simple. It has one function, ok(). |
42 | |
43 | #!/usr/bin/perl -w |
44 | |
45 | use Test::Simple tests => 1; |
46 | |
47 | ok( 1 + 1 == 2 ); |
48 | |
49 | and that does the same thing as the code above. ok() is the backbone |
50 | of Perl testing, and we'll be using it instead of roll-your-own from |
51 | here on. If ok() gets a true value, the test passes. False, it |
52 | fails. |
53 | |
54 | #!/usr/bin/perl -w |
55 | |
56 | use Test::Simple tests => 2; |
57 | ok( 1 + 1 == 2 ); |
58 | ok( 2 + 2 == 5 ); |
59 | |
60 | from that comes |
61 | |
62 | 1..2 |
63 | ok 1 |
64 | not ok 2 |
65 | # Failed test (test.pl at line 5) |
66 | # Looks like you failed 1 tests of 2. |
67 | |
68 | C<1..2> "I'm going to run two tests." This number is used to ensure |
69 | your test program ran all the way through and didn't die or skip some |
70 | tests. C<ok 1> "The first test passed." C<not ok 2> "The second test |
71 | failed". Test::Simple helpfuly prints out some extra commentary about |
72 | your tests. |
73 | |
74 | It's not scary. Come, hold my hand. We're going to give an example |
75 | of testing a module. For our example, we'll be testing a date |
76 | library, Date::ICal. It's on CPAN, so download a copy and follow |
77 | along. [2] |
78 | |
79 | |
80 | =head2 Where to start? |
81 | |
82 | This is the hardest part of testing, where do you start? People often |
83 | get overwhelmed at the apparent enormity of the task of testing a |
84 | whole module. Best place to start is at the beginning. Date::ICal is |
85 | an object-oriented module, and that means you start by making an |
86 | object. So we test new(). |
87 | |
88 | #!/usr/bin/perl -w |
89 | |
90 | use Test::Simple tests => 2; |
91 | |
92 | use Date::ICal; |
93 | |
94 | my $ical = Date::ICal->new; # create an object |
95 | ok( defined $ical ); # check that we got something |
96 | ok( $ical->isa('Date::ICal') ); # and it's the right class |
97 | |
98 | run that and you should get: |
99 | |
100 | 1..2 |
101 | ok 1 |
102 | ok 2 |
103 | |
104 | congratulations, you've written your first useful test. |
105 | |
106 | |
107 | =head2 Names |
108 | |
109 | That output isn't terribly descriptive, is it? When you have two |
110 | tests you can figure out which one is #2, but what if you have 102? |
111 | |
112 | Each test can be given a little descriptive name as the second |
113 | argument to ok(). |
114 | |
115 | use Test::Simple tests => 2; |
116 | |
117 | ok( defined $ical, 'new() returned something' ); |
118 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); |
119 | |
120 | So now you'd see... |
121 | |
122 | 1..2 |
123 | ok 1 - new() returned something |
124 | ok 2 - and it's the right class |
125 | |
126 | |
127 | =head2 Test the manual |
128 | |
129 | Simplest way to build up a decent testing suite is to just test what |
130 | the manual says it does. [3] Let's pull something out of of the |
131 | Date::ICal SYNOPSIS and test that all it's bits work. |
132 | |
133 | #!/usr/bin/perl -w |
134 | |
135 | use Test::Simple tests => 8; |
136 | |
137 | use Date::ICal; |
138 | |
139 | $ical = Date::ICal->new( year => 1964, month => 10, day => 16, |
140 | hour => 16, min => 12, sec => 47, |
141 | tz => '0530' ); |
142 | |
143 | ok( defined $ical, 'new() returned something' ); |
144 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); |
145 | ok( $ical->sec == 47, ' sec()' ); |
146 | ok( $ical->min == 12, ' min()' ); |
147 | ok( $ical->hour == 16, ' hour()' ); |
148 | ok( $ical->day == 17, ' day()' ); |
149 | ok( $ical->month == 10, ' month()' ); |
150 | ok( $ical->year == 1964, ' year()' ); |
151 | |
152 | run that and you get: |
153 | |
154 | 1..8 |
155 | ok 1 - new() returned something |
156 | ok 2 - and it's the right class |
157 | ok 3 - sec() |
158 | ok 4 - min() |
159 | ok 5 - hour() |
160 | not ok 6 - day() |
161 | # Failed test (- at line 16) |
162 | ok 7 - month() |
163 | ok 8 - year() |
164 | # Looks like you failed 1 tests of 8. |
165 | |
166 | Whoops, a failure! [4] Test::Simple helpfully lets us know on what line |
167 | the failure occured, but not much else. We were supposed to get 17, |
168 | but we didn't. What did we get?? Dunno. We'll have to re-run the |
169 | test in the debugger or throw in some print statements to find out. |
170 | |
171 | Instead, we'll switch from Test::Simple to Test::More. Test::More |
172 | does everything Test::Simple does, and more! In fact, Test::More does |
173 | things I<exactly> the way Test::Simple does. You can literally swap |
174 | Test::Simple out and put Test::More in its place. That's just what |
175 | we're going to do. |
176 | |
177 | Test::More provides more informative ways to say 'ok'. ok() is nice |
178 | and generic, you can write almost any test with it, but it can't tell |
179 | you what went wrong. For that, we use the is() function. |
180 | |
181 | #!/usr/bin/perl -w |
182 | |
183 | use Test::More tests => 8; |
184 | |
185 | use Date::ICal; |
186 | |
187 | $ical = Date::ICal->new( year => 1964, month => 10, day => 16, |
188 | hour => 16, min => 12, sec => 47, |
189 | tz => '0530' ); |
190 | |
191 | ok( defined $ical, 'new() returned something' ); |
192 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); |
193 | is( $ical->sec, 47, ' sec()' ); |
194 | is( $ical->min, 12, ' min()' ); |
195 | is( $ical->hour, 16, ' hour()' ); |
196 | is( $ical->day, 17, ' day()' ); |
197 | is( $ical->month, 10, ' month()' ); |
198 | is( $ical->year, 1964, ' year()' ); |
199 | |
200 | "Is C<$ical->sec> 47?" "Is C<$ical->min> 12?" With is() in place, |
201 | you get some more information |
202 | |
203 | 1..8 |
204 | ok 1 - new() returned something |
205 | ok 2 - and it's the right class |
206 | ok 3 - sec() |
207 | ok 4 - min() |
208 | ok 5 - hour() |
209 | not ok 6 - day() |
210 | # Failed test (- at line 16) |
211 | # got: '16' |
212 | # expected: '17' |
213 | ok 7 - month() |
214 | ok 8 - year() |
215 | # Looks like you failed 1 tests of 8. |
216 | |
217 | letting us know that $ical->day returned 16, but we expected 17. A |
218 | quick check shows that the code is working fine, we made a mistake |
219 | when writing up the tests. Just change it to: |
220 | |
221 | is( $ical->day, 16, ' day()' ); |
222 | |
223 | and everything works. |
224 | |
225 | So any time you're doing a "this equals that" sort of test, use is(). |
226 | It even works on arrays. The test is always in scalar context, so you |
227 | can test how many elements are in a list this way. [5] |
228 | |
229 | is( @foo, 5, 'foo has 5 elements' ); |
230 | |
231 | |
232 | =head2 Sometimes the tests are wrong |
233 | |
234 | Which brings us to a very important lesson. Code has bugs. Tests are |
235 | code. Ergo, tests have bugs. A failing test could mean a bug in the |
236 | code, but don't discount the possibility that the test is wrong. |
237 | |
238 | On the flip side, don't be tempted to prematurely declare a test |
239 | incorrect just because you're having trouble finding the bug. |
240 | Invalidating a test isn't something to be taken lightly, and don't use |
241 | it as a cop out to avoid work. |
242 | |
243 | |
244 | =head2 Testing lots of values |
245 | |
246 | We're going to be wanting to test a lot of dates here, trying to trick |
247 | the code with lots of different edge cases. Does it work before 1970? |
248 | After 2038? Before 1904? Do years after 10,000 give it trouble? |
249 | Does it get leap years right? We could keep repeating the code above, |
250 | or we could set up a little try/expect loop. |
251 | |
252 | use Test::More tests => 32; |
253 | use Date::ICal; |
254 | |
255 | my %ICal_Dates = ( |
256 | # An ICal string And the year, month, date |
257 | # hour, minute and second we expect. |
258 | '19971024T120000' => # from the docs. |
259 | [ 1997, 10, 24, 12, 0, 0 ], |
260 | '20390123T232832' => # after the Unix epoch |
261 | [ 2039, 1, 23, 23, 28, 32 ], |
262 | '19671225T000000' => # before the Unix epoch |
263 | [ 1967, 12, 25, 0, 0, 0 ], |
264 | '18990505T232323' => # before the MacOS epoch |
265 | [ 1899, 5, 5, 23, 23, 23 ], |
266 | ); |
267 | |
268 | |
269 | while( my($ical_str, $expect) = each %ICal_Dates ) { |
270 | my $ical = Date::ICal->new( ical => $ical_str ); |
271 | |
272 | ok( defined $ical, "new(ical => '$ical_str')" ); |
273 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); |
274 | |
275 | is( $ical->year, $expect->[0], ' year()' ); |
276 | is( $ical->month, $expect->[1], ' month()' ); |
277 | is( $ical->day, $expect->[2], ' day()' ); |
278 | is( $ical->hour, $expect->[3], ' hour()' ); |
279 | is( $ical->min, $expect->[4], ' min()' ); |
280 | is( $ical->sec, $expect->[5], ' sec()' ); |
281 | } |
282 | |
283 | So now we can test bunches of dates by just adding them to |
284 | %ICal_Dates. Now that it's less work to test with more dates, you'll |
285 | be inclined to just throw more in as you think of them. |
286 | Only problem is, every time we add to that we have to keep adjusting |
287 | the C<use Test::More tests => ##> line. That can rapidly get |
288 | annoying. Instead we use 'no_plan'. This means we're just running |
289 | some tests, don't know how many. [6] |
290 | |
291 | use Test::More 'no_plan'; # instead of tests => 32 |
292 | |
293 | now we can just add tests and not have to do all sorts of math to |
294 | figure out how many we're running. |
295 | |
296 | |
297 | =head2 Informative names |
298 | |
299 | Take a look at this line here |
300 | |
301 | ok( defined $ical, "new(ical => '$ical_str')" ); |
302 | |
303 | we've added more detail about what we're testing and the ICal string |
304 | itself we're trying out to the name. So you get results like: |
305 | |
306 | ok 25 - new(ical => '19971024T120000') |
307 | ok 26 - and it's the right class |
308 | ok 27 - year() |
309 | ok 28 - month() |
310 | ok 29 - day() |
311 | ok 30 - hour() |
312 | ok 31 - min() |
313 | ok 32 - sec() |
314 | |
315 | if something in there fails, you'll know which one it was and that |
316 | will make tracking down the problem easier. So try to put a bit of |
317 | debugging information into the test names. |
318 | |
319 | |
320 | =head2 Skipping tests |
321 | |
322 | Poking around in the existing Date::ICal tests, I found this in |
323 | t/01sanity.t [7] |
324 | |
325 | #!/usr/bin/perl -w |
326 | |
327 | use Test::More tests => 7; |
328 | use Date::ICal; |
329 | |
330 | # Make sure epoch time is being handled sanely. |
331 | my $t1 = Date::ICal->new( epoch => 0 ); |
332 | is( $t1->epoch, 0, "Epoch time of 0" ); |
333 | |
334 | # XXX This will only work on unix systems. |
335 | is( $t1->ical, '19700101Z', " epoch to ical" ); |
336 | |
337 | is( $t1->year, 1970, " year()" ); |
338 | is( $t1->month, 1, " month()" ); |
339 | is( $t1->day, 1, " day()" ); |
340 | |
341 | # like the tests above, but starting with ical instead of epoch |
342 | my $t2 = Date::ICal->new( ical => '19700101Z' ); |
343 | is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" ); |
344 | |
345 | is( $t2->epoch, 0, " and back to ICal" ); |
346 | |
347 | The beginning of the epoch is different on most non-Unix operating |
348 | systems [8]. Even though Perl smooths out the differences for the most |
349 | part, certain ports do it differently. MacPerl is one off the top of |
350 | my head. [9] We I<know> this will never work on MacOS. So rather than |
351 | just putting a comment in the test, we can explicitly say it's never |
352 | going to work and skip the test. |
353 | |
354 | use Test::More tests => 7; |
355 | use Date::ICal; |
356 | |
357 | # Make sure epoch time is being handled sanely. |
358 | my $t1 = Date::ICal->new( epoch => 0 ); |
359 | is( $t1->epoch, 0, "Epoch time of 0" ); |
360 | |
361 | SKIP: { |
362 | skip('epoch to ICal not working on MacOS', 6) |
363 | if $^O eq 'MacOS'; |
364 | |
365 | is( $t1->ical, '19700101Z', " epoch to ical" ); |
366 | |
367 | is( $t1->year, 1970, " year()" ); |
368 | is( $t1->month, 1, " month()" ); |
369 | is( $t1->day, 1, " day()" ); |
370 | |
371 | # like the tests above, but starting with ical instead of epoch |
372 | my $t2 = Date::ICal->new( ical => '19700101Z' ); |
373 | is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" ); |
374 | |
375 | is( $t2->epoch, 0, " and back to ICal" ); |
376 | } |
377 | |
378 | A little bit of magic happens here. When running on anything but |
379 | MacOS, all the tests run normally. But when on MacOS, skip() causes |
380 | the entire contents of the SKIP block to be jumped over. It's never |
381 | run. Instead, it prints special output that tells Test::Harness that |
382 | the tests have been skipped. |
383 | |
384 | 1..7 |
385 | ok 1 - Epoch time of 0 |
386 | ok 2 # skip epoch to ICal not working on MacOS |
387 | ok 3 # skip epoch to ICal not working on MacOS |
388 | ok 4 # skip epoch to ICal not working on MacOS |
389 | ok 5 # skip epoch to ICal not working on MacOS |
390 | ok 6 # skip epoch to ICal not working on MacOS |
391 | ok 7 # skip epoch to ICal not working on MacOS |
392 | |
393 | This means your tests won't fail on MacOS. This means less emails |
394 | from MacPerl users telling you about failing tests that you know will |
395 | never work. You've got to be careful with skip tests. These are for |
396 | tests which don't work and B<never will>. It is not for skipping |
397 | genuine bugs (we'll get to that in a moment). |
398 | |
399 | The tests are wholely and completely skipped. [10] This will work. |
400 | |
401 | SKIP: { |
402 | skip("I don't wanna die!"); |
403 | |
404 | die, die, die, die, die; |
405 | } |
406 | |
407 | |
408 | =head2 Todo tests |
409 | |
410 | Thumbing through the Date::ICal man page, I came across this: |
411 | |
412 | ical |
413 | |
414 | $ical_string = $ical->ical; |
415 | |
416 | Retrieves, or sets, the date on the object, using any |
417 | valid ICal date/time string. |
418 | |
419 | "Retrieves or sets". Hmmm, didn't see a test for using ical() to set |
420 | the date in the Date::ICal test suite. So I'll write one. |
421 | |
422 | use Test::More tests => 1; |
423 | |
424 | my $ical = Date::ICal->new; |
425 | $ical->ical('20201231Z'); |
426 | is( $ical->ical, '20201231Z', 'Setting via ical()' ); |
427 | |
428 | run that and I get |
429 | |
430 | 1..1 |
431 | not ok 1 - Setting via ical() |
432 | # Failed test (- at line 6) |
433 | # got: '20010814T233649Z' |
434 | # expected: '20201231Z' |
435 | # Looks like you failed 1 tests of 1. |
436 | |
437 | Whoops! Looks like it's unimplemented. Let's assume we don't have |
438 | the time to fix this. [11] Normally, you'd just comment out the test |
439 | and put a note in a todo list somewhere. Instead, we're going to |
440 | explicitly state "this test will fail" by wraping it in a TODO block. |
441 | |
442 | use Test::More tests => 1; |
443 | |
444 | TODO: { |
445 | local $TODO = 'ical($ical) not yet implemented'; |
446 | |
447 | my $ical = Date::ICal->new; |
448 | $ical->ical('20201231Z'); |
449 | |
450 | is( $ical->ical, '20201231Z', 'Setting via ical()' ); |
451 | } |
452 | |
453 | Now when you run, it's a little different: |
454 | |
455 | 1..1 |
456 | not ok 1 - Setting via ical() # TODO ical($ical) not yet implemented |
457 | # got: '20010822T201551Z' |
458 | # expected: '20201231Z' |
459 | |
460 | Test::More doesn't say "Looks like you failed 1 tests of 1". That '# |
461 | TODO' tells Test::Harness "this is supposed to fail" and it treats a |
462 | failure as a successful test. So you can write tests even before |
463 | you've fixed the underlying code. |
464 | |
465 | If a TODO test passes, Test::Harness will report it "UNEXPECTEDLY |
466 | SUCCEEDED". When that happens, you simply remove the TODO block and |
467 | C<local $TODO> and turn it into a real test. |
468 | |
469 | |
470 | =head2 Testing with taint mode. |
471 | |
472 | Taint mode is a funny thing. It's the globalest of all global |
473 | features. Once you turn it on it effects B<all> code in your program |
474 | and B<all> modules used (and all the modules they use). If a single |
475 | piece of code isn't taint clean, the whole thing explodes. With that |
476 | in mind, it's very important to ensure your module works under taint |
477 | mode. |
478 | |
479 | It's very simple to have your tests run under taint mode. Just throw |
480 | a -T into the #! line. Test::Harness will read the switches in #! and |
481 | use them to run your tests. |
482 | |
483 | #!/usr/bin/perl -Tw |
484 | |
485 | use Test::More 'no_plan'; |
486 | |
487 | ...test normally here... |
488 | |
489 | So when you say "make test" it will be run with taint mode and |
490 | warnings on. |
491 | |
492 | |
493 | =head1 FOOTNOTES |
494 | |
495 | =over 4 |
496 | |
497 | =item 1 |
498 | |
499 | The first number doesn't really mean anything, but it has to be 1. |
500 | It's the second number that's important. |
501 | |
502 | =item 2 |
503 | |
504 | For those following along at home, I'm using version 1.31. It has |
505 | some bugs, which is good -- we'll uncover them with our tests. |
506 | |
507 | =item 3 |
508 | |
509 | You can actually take this one step further and test the manual |
510 | itself. Have a look at Pod::Tests (soon to be Test::Inline). |
511 | |
512 | =item 4 |
513 | |
514 | Yes, there's a mistake in the test suite. What! Me, contrived? |
515 | |
516 | =item 5 |
517 | |
518 | We'll get to testing the contents of lists later. |
519 | |
520 | =item 6 |
521 | |
522 | But what happens if your test program dies halfway through?! Since we |
523 | didn't say how many tests we're going to run, how can we know it |
524 | failed? No problem, Test::More employs some magic to catch that death |
525 | and turn the test into a failure, even if every test passed up to that |
526 | point. |
527 | |
528 | =item 7 |
529 | |
530 | I cleaned it up a little. |
531 | |
532 | =item 8 |
533 | |
534 | Most Operating Systems record time as the number of seconds since a |
535 | certain date. This date is the beginning of the epoch. Unix's starts |
536 | at midnight January 1st, 1970 GMT. |
537 | |
538 | =item 9 |
539 | |
540 | MacOS's epoch is midnight January 1st, 1904. VMS's is midnight, |
541 | November 17th, 1858, but vmsperl emulates the Unix epoch so it's not a |
542 | problem. |
543 | |
544 | =item 10 |
545 | |
546 | As long as the code inside the SKIP block at least compiles. Please |
547 | don't ask how. No, it's not a filter. |
548 | |
549 | =item 11 |
550 | |
551 | Do NOT be tempted to use TODO tests as a way to avoid fixing simple |
552 | bugs! |
553 | |
554 | =back |