3 Test::Harness::TAP - Documentation for the TAP format
7 TAP, the Test Anything Protocol, is Perl's simple text-based interface
8 between testing modules such as Test::More and the test harness
13 Exit code of the process.
17 TAP's general format is:
20 ok 1 Description # Directive
27 For example, a test file's output might look like:
30 ok 1 - Input file opened
31 not ok 2 - First line of the input valid
32 ok 3 - Read the rest of the file
33 not ok 4 - Summarized correctly # TODO Not written yet
35 =head1 HARNESS BEHAVIOR
37 In this document, the "harness" is any program analyzing TAP output.
38 Typically this will be Perl's I<prove> program, or the underlying
39 C<Test::Harness::runtests> subroutine.
41 A harness must only read TAP output from standard output and not
42 from standard error. Lines written to standard output matching
43 C</^(not )?ok\b/> must be interpreted as test lines. All other
44 lines must not be considered test output.
46 =head1 TESTS LINES AND THE PLAN
50 The plan tells how many tests will be run, or how many tests have
51 run. It's a check that the test file hasn't stopped prematurely.
52 It must appear once, whether at the beginning or end of the output.
54 The plan is usually the first line of TAP output and it specifies how
55 many test points are to follow. For example,
59 means you plan on running 10 tests. This is a safeguard in case your test
60 file dies silently in the middle of its run. The plan is optional but if
61 there is a plan before the test points it must be the first non-diagnostic
62 line output by the test file.
64 In certain instances a test file may not know how many test points
65 it will ultimately be running. In this case the plan can be the last
66 non-diagnostic line in the output.
68 The plan cannot appear in the middle of the output, nor can it appear more
73 The core of TAP is the test line. A test file prints one test line test
74 point executed. There must be at least one test line in TAP output. Each
75 test line comprises the following elements:
79 =item * C<ok> or C<not ok>
81 This tells whether the test point passed or failed. It must be
82 at the beginning of the line. C</^not ok/> indicates a failed test
83 point. C</^ok/> is a successful test point. This is the only mandatory
86 Note that unlike the Directives below, C<ok> and C<not ok> are
91 TAP expects the C<ok> or C<not ok> to be followed by a test point
92 number. If there is no number the harness must maintain
93 its own counter until the script supplies test numbers again. So
94 the following test output
103 has five tests. The sixth is missing. Test::Harness will generate
106 Failed 3/6 tests, 50.00% okay
110 Any text after the test number but before a C<#> is the description of
113 ok 42 this is the description of the test
115 Descriptions should not begin with a digit so that they are not confused
116 with the test point number.
118 The harness may do whatever it wants with the description.
122 The test point may include a directive, following a hash on the
123 test line. There are currently two directives allowed: C<TODO> and
124 C<SKIP>. These are discussed below.
132 =item * ok/not ok (required)
134 =item * Test number (recommended)
136 =item * Description (recommended)
138 =item * Directive (only when necessary)
144 Directives are special notes that follow a C<#> on the test line.
145 Only two are currently defined: C<TODO> and C<SKIP>. Note that
146 these two keywords are not case-sensitive.
150 If the directive starts with C<# TODO>, the test is counted as a
151 todo test, and the text after C<TODO> is the the explanation.
153 not ok 13 # TODO bend space and time
155 Note that if the TODO has an explanation it must be separated from
158 These tests represent a feature to be implemented or a bug to be fixed
159 and act as something of an executable "things to do" list. They are
160 B<not> expected to succeed. Should a todo test point begin succeeding,
161 the harness should report it as a bonus. This indicates that whatever
162 you were supposed to do has been done and you should promote this to a
165 =head2 Skipping tests
167 If the directive starts with C<# SKIP>, the test is counted as having
168 been skipped. If the whole test file succeeds, the count of skipped
169 tests is included in the generated output. The harness should report
170 the text after C< # SKIP\S*\s+> as a reason for skipping.
172 ok 23 # skip Insufficient flogiston pressure.
174 Similarly, one can include an explanation in a plan line,
175 emitted if the test file is skipped completely:
177 1..0 # Skipped: WWW::Mechanize not installed
183 As an emergency measure a test script can decide that further tests
184 are useless (e.g. missing dependencies) and testing should stop
185 immediately. In that case the test script prints the magic words
189 to standard output. Any message after these words must be displayed
190 by the interpreter as the reason why testing must be stopped, as
193 Bail out! MySQL is not running.
197 Additional information may be put into the testing output on separate
198 lines. Diagnostic lines should begin with a C<#>, which the harness must
199 ignore, at least as far as analyzing the test results. The harness is
200 free, however, to display the diagnostics. Typically diagnostics are
201 used to provide information about the environment in which test file is
202 running, or to delineate a group of tests.
204 ok 18 - Closed database connection
205 # End of database section.
206 # This starts the network part of the test.
207 # Daemon started on port 2112
208 ok 19 - Opened socket
210 ok 47 - Closed socket
211 # End of network tests
215 Any output line that is not a plan, a test line or a diagnostic is
216 incorrect. How a harness handles the incorrect line is undefined.
217 Test::Harness silently ignores incorrect lines, but will become more
218 stringent in the future.
222 All names, places, and events depicted in any example are wholly
223 fictitious and bear no resemblance to, connection with, or relation to any
224 real entity. Any such similarity is purely coincidental, unintentional,
227 =head2 Common with explanation
229 The following TAP listing declares that six tests follow as well as
230 provides handy feedback as to what the test is about to do. All six
235 # Create a new Board and Tile, then place
236 # the Tile onto the board.
238 ok 1 - The object isa Board
239 ok 2 - Board size is zero
240 ok 3 - The object isa Tile
241 ok 4 - Get possible places to put the Tile
242 ok 5 - Placing the tile produces no error
243 ok 6 - Board size is 1
245 =head2 Unknown amount and failures
247 This hypothetical test program ensures that a handful of servers are
248 online and network-accessible. Because it retrieves the hypothetical
249 servers from a database, it doesn't know exactly how many servers it
250 will need to ping. Thus, the test count is declared at the bottom after
251 all the test points have run. Also, two of the tests fail.
253 ok 1 - retrieving servers from the database
254 # need to ping 6 servers
255 ok 2 - pinged diamond
257 not ok 4 - pinged saphire
259 not ok 6 - pinged quartz
265 This listing reports that a pile of tests are going to be run. However,
266 the first test fails, reportedly because a connection to the database
267 could not be established. The program decided that continuing was
268 pointless and exited.
271 not ok 1 - database handle
272 Bail out! Couldn't connect to database.
274 =head2 Skipping a few
276 The following listing plans on running 5 tests. However, our program
277 decided to not run tests 2 thru 5 at all. To properly report this,
278 the tests are marked as being skipped.
281 ok 1 - approved operating system
283 ok 2 - # SKIP no /sys directory
284 ok 3 - # SKIP no /sys directory
285 ok 4 - # SKIP no /sys directory
286 ok 5 - # SKIP no /sys directory
288 =head2 Skipping everything
290 This listing shows that the entire listing is a skip. No tests were run.
292 1..0 # skip because English-to-French translator isn't installed
294 =head2 Got spare tuits?
296 The following example reports that four tests are run and the last two
297 tests failed. However, becauses the failing tests are marked as things
298 to do later, they are considered successes. Thus, a harness should report
299 this entire listing as a success.
302 ok 1 - Creating test program
303 ok 2 - Test program runs, no error
304 not ok 3 - infinite loop # TODO halting problem unsolved
305 not ok 4 - infinite loop 2 # TODO halting problem unsolved
307 =head2 Creative liberties
309 This listing shows an alternate output where the test numbers aren't
310 provided. The test also reports the state of a ficticious board game in
311 diagnostic form. Finally, the test count is reported at the end.
321 # +------+------+------+------+
323 # | |G N C | |C C G |
325 # +------+------+------+------+
327 # |R N G |G A G | |C C C |
329 # +------+------+------+------+
331 # | |G A G |G N R |R N R |
333 # +------+------+------+------+
334 ok - board has 7 tiles + starter tile
339 Andy Lester, based on the original Test::Harness documentation by Michael Schwern.
341 =head1 ACKNOWLEDGEMENTS
348 for help and contributions on this document.
350 The basis for the TAP format was created by Larry Wall in the
351 original test script for Perl 1. Tim Bunce and Andreas Koenig
352 developed it further with their modifications to Test::Harness.
356 Copyright 2003-2005 by
357 Michael G Schwern C<< <schwern@pobox.com> >>,
358 Andy Lester C<< <andy@petdance.com> >>.
360 This program is free software; you can redistribute it and/or
361 modify it under the same terms as Perl itself.
363 See L<http://www.perl.com/perl/misc/Artistic.html>.