diff options
Diffstat (limited to 'upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl')
| -rw-r--r-- | upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl | 533 |
1 files changed, 533 insertions, 0 deletions
diff --git a/upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl b/upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl new file mode 100644 index 00000000..95daa83d --- /dev/null +++ b/upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl @@ -0,0 +1,533 @@ +.\" -*- mode: troff; coding: utf-8 -*- +.\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) +.\" +.\" Standard preamble: +.\" ======================================================================== +.de Sp \" Vertical space (when we can't use .PP) +.if t .sp .5v +.if n .sp +.. +.de Vb \" Begin verbatim text +.ft CW +.nf +.ne \\$1 +.. +.de Ve \" End verbatim text +.ft R +.fi +.. +.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. +.ie n \{\ +. ds C` "" +. ds C' "" +'br\} +.el\{\ +. ds C` +. ds C' +'br\} +.\" +.\" Escape single quotes in literal strings from groff's Unicode transform. +.ie \n(.g .ds Aq \(aq +.el .ds Aq ' +.\" +.\" If the F register is >0, we'll generate index entries on stderr for +.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index +.\" entries marked with X<> in POD. Of course, you'll have to process the +.\" output yourself in some meaningful fashion. +.\" +.\" Avoid warning from groff about undefined register 'F'. +.de IX +.. +.nr rF 0 +.if \n(.g .if rF .nr rF 1 +.if (\n(rF:(\n(.g==0)) \{\ +. if \nF \{\ +. de IX +. tm Index:\\$1\t\\n%\t"\\$2" +.. +. if !\nF==2 \{\ +. nr % 0 +. nr F 2 +. \} +. \} +.\} +.rr rF +.\" ======================================================================== +.\" +.IX Title "TAP::Harness::Beyond 3perl" +.TH TAP::Harness::Beyond 3perl 2024-01-12 "perl v5.38.2" "Perl Programmers Reference Guide" +.\" For nroff, turn off justification. Always turn off hyphenation; it makes +.\" way too many mistakes in technical documents. +.if n .ad l +.nh +.SH NAME +Test::Harness::Beyond \- Beyond make test +.SH "Beyond make test" +.IX Header "Beyond make test" +Test::Harness is responsible for running test scripts, analysing +their output and reporting success or failure. When I type +\&\fImake test\fR (or \fI./Build test\fR) for a module, Test::Harness is usually +used to run the tests (not all modules use Test::Harness but the +majority do). +.PP +To start exploring some of the features of Test::Harness I need to +switch from \fImake test\fR to the \fIprove\fR command (which ships with +Test::Harness). For the following examples I'll also need a recent +version of Test::Harness installed; 3.14 is current as I write. +.PP +For the examples I'm going to assume that we're working with a +\&'normal' Perl module distribution. Specifically I'll assume that +typing \fImake\fR or \fI./Build\fR causes the built, ready-to-install module +code to be available below ./blib/lib and ./blib/arch and that +there's a directory called 't' that contains our tests. Test::Harness +isn't hardwired to that configuration but it saves me from explaining +which files live where for each example. +.PP +Back to \fIprove\fR; like \fImake test\fR it runs a test suite \- but it +provides far more control over which tests are executed, in what +order and how their results are reported. Typically \fImake test\fR +runs all the test scripts below the 't' directory. To do the same +thing with prove I type: +.PP +.Vb 1 +\& prove \-rb t +.Ve +.PP +The switches here are \-r to recurse into any directories below 't' +and \-b which adds ./blib/lib and ./blib/arch to Perl's include path +so that the tests can find the code they will be testing. If I'm +testing a module of which an earlier version is already installed +I need to be careful about the include path to make sure I'm not +running my tests against the installed version rather than the new +one that I'm working on. +.PP +Unlike \fImake test\fR, typing \fIprove\fR doesn't automatically rebuild +my module. If I forget to make before prove I will be testing against +older versions of those files \- which inevitably leads to confusion. +I either get into the habit of typing +.PP +.Vb 1 +\& make && prove \-rb t +.Ve +.PP +or \- if I have no XS code that needs to be built I use the modules +below \fIlib\fR instead +.PP +.Vb 1 +\& prove \-Ilib \-r t +.Ve +.PP +So far I've shown you nothing that \fImake test\fR doesn't do. Let's +fix that. +.SS "Saved State" +.IX Subsection "Saved State" +If I have failing tests in a test suite that consists of more than +a handful of scripts and takes more than a few seconds to run it +rapidly becomes tedious to run the whole test suite repeatedly as +I track down the problems. +.PP +I can tell prove just to run the tests that are failing like this: +.PP +.Vb 1 +\& prove \-b t/this_fails.t t/so_does_this.t +.Ve +.PP +That speeds things up but I have to make a note of which tests are +failing and make sure that I run those tests. Instead I can use +prove's \-\-state switch and have it keep track of failing tests for +me. First I do a complete run of the test suite and tell prove to +save the results: +.PP +.Vb 1 +\& prove \-rb \-\-state=save t +.Ve +.PP +That stores a machine readable summary of the test run in a file +called '.prove' in the current directory. If I have failures I can +then run just the failing scripts like this: +.PP +.Vb 1 +\& prove \-b \-\-state=failed +.Ve +.PP +I can also tell prove to save the results again so that it updates +its idea of which tests failed: +.PP +.Vb 1 +\& prove \-b \-\-state=failed,save +.Ve +.PP +As soon as one of my failing tests passes it will be removed from +the list of failed tests. Eventually I fix them all and prove can +find no failing tests to run: +.PP +.Vb 2 +\& Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU) +\& Result: NOTESTS +.Ve +.PP +As I work on a particular part of my module it's most likely that +the tests that cover that code will fail. I'd like to run the whole +test suite but have it prioritize these 'hot' tests. I can tell +prove to do this: +.PP +.Vb 1 +\& prove \-rb \-\-state=hot,save t +.Ve +.PP +All the tests will run but those that failed most recently will be +run first. If no tests have failed since I started saving state all +tests will run in their normal order. This combines full test +coverage with early notification of failures. +.PP +The \-\-state switch supports a number of options; for example to run +failed tests first followed by all remaining tests ordered by the +timestamps of the test scripts \- and save the results \- I can use +.PP +.Vb 1 +\& prove \-rb \-\-state=failed,new,save t +.Ve +.PP +See the prove documentation (type prove \-\-man) for the full list +of state options. +.PP +When I tell prove to save state it writes a file called '.prove' +('_prove' on Windows) in the current directory. It's a YAML document +so it's quite easy to write tools of your own that work on the saved +test state \- but the format isn't officially documented so it might +change without (much) warning in the future. +.SS "Parallel Testing" +.IX Subsection "Parallel Testing" +If my tests take too long to run I may be able to speed them up by +running multiple test scripts in parallel. This is particularly +effective if the tests are I/O bound or if I have multiple CPU +cores. I tell prove to run my tests in parallel like this: +.PP +.Vb 1 +\& prove \-rb \-j 9 t +.Ve +.PP +The \-j switch enables parallel testing; the number that follows it +is the maximum number of tests to run in parallel. Sometimes tests +that pass when run sequentially will fail when run in parallel. For +example if two different test scripts use the same temporary file +or attempt to listen on the same socket I'll have problems running +them in parallel. If I see unexpected failures I need to check my +tests to work out which of them are trampling on the same resource +and rename temporary files or add locks as appropriate. +.PP +To get the most performance benefit I want to have the test scripts +that take the longest to run start first \- otherwise I'll be waiting +for the one test that takes nearly a minute to complete after all +the others are done. I can use the \-\-state switch to run the tests +in slowest to fastest order: +.PP +.Vb 1 +\& prove \-rb \-j 9 \-\-state=slow,save t +.Ve +.SS "Non-Perl Tests" +.IX Subsection "Non-Perl Tests" +The Test Anything Protocol (http://testanything.org/) isn't just +for Perl. Just about any language can be used to write tests that +output TAP. There are TAP based testing libraries for C, C++, PHP, +Python and many others. If I can't find a TAP library for my language +of choice it's easy to generate valid TAP. It looks like this: +.PP +.Vb 4 +\& 1..3 +\& ok 1 \- init OK +\& ok 2 \- opened file +\& not ok 3 \- appended to file +.Ve +.PP +The first line is the plan \- it specifies the number of tests I'm +going to run so that it's easy to check that the test script didn't +exit before running all the expected tests. The following lines are +the test results \- 'ok' for pass, 'not ok' for fail. Each test has +a number and, optionally, a description. And that's it. Any language +that can produce output like that on STDOUT can be used to write +tests. +.PP +Recently I've been rekindling a two-decades-old interest in Forth. +Evidently I have a masochistic streak that even Perl can't satisfy. +I want to write tests in Forth and run them using prove (you can +find my gforth TAP experiments at +https://svn.hexten.net/andy/Forth/Testing/). I can use the \-\-exec +switch to tell prove to run the tests using gforth like this: +.PP +.Vb 1 +\& prove \-r \-\-exec gforth t +.Ve +.PP +Alternately, if the language used to write my tests allows a shebang +line I can use that to specify the interpreter. Here's a test written +in PHP: +.PP +.Vb 6 +\& #!/usr/bin/php +\& <?php +\& print "1..2\en"; +\& print "ok 1\en"; +\& print "not ok 2\en"; +\& ?> +.Ve +.PP +If I save that as t/phptest.t the shebang line will ensure that it +runs correctly along with all my other tests. +.SS "Mixing it up" +.IX Subsection "Mixing it up" +Subtle interdependencies between test programs can mask problems \- +for example an earlier test may neglect to remove a temporary file +that affects the behaviour of a later test. To find this kind of +problem I use the \-\-shuffle and \-\-reverse options to run my tests +in random or reversed order. +.SS "Rolling My Own" +.IX Subsection "Rolling My Own" +If I need a feature that prove doesn't provide I can easily write my own. +.PP +Typically you'll want to change how TAP gets \fIinput\fR into and \fIoutput\fR +from the parser. App::Prove supports arbitrary plugins, and TAP::Harness +supports custom \fIformatters\fR and \fIsource handlers\fR that you can load using +either prove or Module::Build; there are many examples to base mine on. +For more details see App::Prove, TAP::Parser::SourceHandler, and +TAP::Formatter::Base. +.PP +If writing a plugin is not enough, you can write your own test harness; one of +the motives for the 3.00 rewrite of Test::Harness was to make it easier to +subclass and extend. +.PP +The Test::Harness module is a compatibility wrapper around TAP::Harness. +For new applications I should use TAP::Harness directly. As we'll +see, prove uses TAP::Harness. +.PP +When I run prove it processes its arguments, figures out which test +scripts to run and then passes control to TAP::Harness to run the +tests, parse, analyse and present the results. By subclassing +TAP::Harness I can customise many aspects of the test run. +.PP +I want to log my test results in a database so I can track them +over time. To do this I override the summary method in TAP::Harness. +I start with a simple prototype that dumps the results as a YAML +document: +.PP +.Vb 1 +\& package My::TAP::Harness; +\& +\& use base \*(AqTAP::Harness\*(Aq; +\& use YAML; +\& +\& sub summary { +\& my ( $self, $aggregate ) = @_; +\& print Dump( $aggregate ); +\& $self\->SUPER::summary( $aggregate ); +\& } +\& +\& 1; +.Ve +.PP +I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness +is on Perl's \f(CW@INC\fR include path I can +.PP +.Vb 1 +\& prove \-\-harness=My::TAP::Harness \-rb t +.Ve +.PP +If I don't have My::TAP::Harness installed on \f(CW@INC\fR I need to provide +the correct path to perl when I run prove: +.PP +.Vb 1 +\& perl \-Ilib \`which prove\` \-\-harness=My::TAP::Harness \-rb t +.Ve +.PP +I can incorporate these options into my own version of prove. It's +pretty simple. Most of the work of prove is handled by App::Prove. +The important code in prove is just: +.PP +.Vb 1 +\& use App::Prove; +\& +\& my $app = App::Prove\->new; +\& $app\->process_args(@ARGV); +\& exit( $app\->run ? 0 : 1 ); +.Ve +.PP +If I write a subclass of App::Prove I can customise any aspect of +the test runner while inheriting all of prove's behaviour. Here's +myprove: +.PP +.Vb 2 +\& #!/usr/bin/env perl use lib qw( lib ); # Add ./lib to @INC +\& use App::Prove; +\& +\& my $app = App::Prove\->new; +\& +\& # Use custom TAP::Harness subclass +\& $app\->harness( \*(AqMy::TAP::Harness\*(Aq ); +\& +\& $app\->process_args( @ARGV ); exit( $app\->run ? 0 : 1 ); +.Ve +.PP +Now I can run my tests like this +.PP +.Vb 1 +\& ./myprove \-rb t +.Ve +.SS "Deeper Customisation" +.IX Subsection "Deeper Customisation" +Now that I know how to subclass and replace TAP::Harness I can +replace any other part of the harness. To do that I need to know +which classes are responsible for which functionality. Here's a +brief guided tour; the default class for each component is shown +in parentheses. Normally any replacements I write will be subclasses +of these default classes. +.PP +When I run my tests TAP::Harness creates a scheduler +(TAP::Parser::Scheduler) to work out the running order for the +tests, an aggregator (TAP::Parser::Aggregator) to collect and analyse +the test results and a formatter (TAP::Formatter::Console) to display +those results. +.PP +If I'm running my tests in parallel there may also be a multiplexer +(TAP::Parser::Multiplexer) \- the component that allows multiple +tests to run simultaneously. +.PP +Once it has created those helpers TAP::Harness starts running the +tests. For each test it creates a new parser (TAP::Parser) which +is responsible for running the test script and parsing its output. +.PP +To replace any of these components I call one of these harness +methods with the name of the replacement class: +.PP +.Vb 5 +\& aggregator_class +\& formatter_class +\& multiplexer_class +\& parser_class +\& scheduler_class +.Ve +.PP +For example, to replace the aggregator I would +.PP +.Vb 1 +\& $harness\->aggregator_class( \*(AqMy::Aggregator\*(Aq ); +.Ve +.PP +Alternately I can supply the names of my substitute classes to the +TAP::Harness constructor: +.PP +.Vb 3 +\& my $harness = TAP::Harness\->new( +\& { aggregator_class => \*(AqMy::Aggregator\*(Aq } +\& ); +.Ve +.PP +If I need to reach even deeper into the internals of the harness I +can replace the classes that TAP::Parser uses to execute test scripts +and tokenise their output. Before running a test script TAP::Parser +creates a grammar (TAP::Parser::Grammar) to decode the raw TAP into +tokens, a result factory (TAP::Parser::ResultFactory) to turn the +decoded TAP results into objects and, depending on whether it's +running a test script or reading TAP from a file, scalar or array +a source or an iterator (TAP::Parser::IteratorFactory). +.PP +Each of these objects may be replaced by calling one of these parser +methods: +.PP +.Vb 5 +\& source_class +\& perl_source_class +\& grammar_class +\& iterator_factory_class +\& result_factory_class +.Ve +.SS Callbacks +.IX Subsection "Callbacks" +As an alternative to subclassing the components I need to change I +can attach callbacks to the default classes. TAP::Harness exposes +these callbacks: +.PP +.Vb 5 +\& parser_args Tweak the parameters used to create the parser +\& made_parser Just made a new parser +\& before_runtests About to run tests +\& after_runtests Have run all tests +\& after_test Have run an individual test script +.Ve +.PP +TAP::Parser also supports callbacks; bailout, comment, plan, test, +unknown, version and yaml are called for the corresponding TAP +result types, ALL is called for all results, ELSE is called for all +results for which a named callback is not installed and EOF is +called once at the end of each TAP stream. +.PP +To install a callback I pass the name of the callback and a subroutine +reference to TAP::Harness or TAP::Parser's callback method: +.PP +.Vb 3 +\& $harness\->callback( after_test => sub { +\& my ( $script, $desc, $parser ) = @_; +\& } ); +.Ve +.PP +I can also pass callbacks to the constructor: +.PP +.Vb 8 +\& my $harness = TAP::Harness\->new({ +\& callbacks => { +\& after_test => sub { +\& my ( $script, $desc, $parser ) = @_; +\& # Do something interesting here +\& } +\& } +\& }); +.Ve +.PP +When it comes to altering the behaviour of the test harness there's +more than one way to do it. Which way is best depends on my +requirements. In general if I only want to observe test execution +without changing the harness' behaviour (for example to log test +results to a database) I choose callbacks. If I want to make the +harness behave differently subclassing gives me more control. +.SS "Parsing TAP" +.IX Subsection "Parsing TAP" +Perhaps I don't need a complete test harness. If I already have a +TAP test log that I need to parse all I need is TAP::Parser and the +various classes it depends upon. Here's the code I need to run a +test and parse its TAP output +.PP +.Vb 1 +\& use TAP::Parser; +\& +\& my $parser = TAP::Parser\->new( { source => \*(Aqt/simple.t\*(Aq } ); +\& while ( my $result = $parser\->next ) { +\& print $result\->as_string, "\en"; +\& } +.Ve +.PP +Alternately I can pass an open filehandle as source and have the +parser read from that rather than attempting to run a test script: +.PP +.Vb 6 +\& open my $tap, \*(Aq<\*(Aq, \*(Aqtests.tap\*(Aq +\& or die "Can\*(Aqt read TAP transcript ($!)\en"; +\& my $parser = TAP::Parser\->new( { source => $tap } ); +\& while ( my $result = $parser\->next ) { +\& print $result\->as_string, "\en"; +\& } +.Ve +.PP +This approach is useful if I need to convert my TAP based test +results into some other representation. See TAP::Convert::TET +(http://search.cpan.org/dist/TAP\-Convert\-TET/) for an example of +this approach. +.SS "Getting Support" +.IX Subsection "Getting Support" +The Test::Harness developers hang out on the tapx-dev mailing +list[1]. For discussion of general, language independent TAP issues +there's the tap\-l[2] list. Finally there's a wiki dedicated to the +Test Anything Protocol[3]. Contributions to the wiki, patches and +suggestions are all welcome. +.PP +[1] <http://www.hexten.net/mailman/listinfo/tapx\-dev> +[2] <http://testanything.org/mailman/listinfo/tap\-l> +[3] <http://testanything.org/> |