diff options
Diffstat (limited to 'upstream/mageia-cauldron/man3pm/TAP::Harness.3pm')
| -rw-r--r-- | upstream/mageia-cauldron/man3pm/TAP::Harness.3pm | 609 |
1 files changed, 609 insertions, 0 deletions
diff --git a/upstream/mageia-cauldron/man3pm/TAP::Harness.3pm b/upstream/mageia-cauldron/man3pm/TAP::Harness.3pm new file mode 100644 index 00000000..6838c4b4 --- /dev/null +++ b/upstream/mageia-cauldron/man3pm/TAP::Harness.3pm @@ -0,0 +1,609 @@ +.\" -*- mode: troff; coding: utf-8 -*- +.\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) +.\" +.\" Standard preamble: +.\" ======================================================================== +.de Sp \" Vertical space (when we can't use .PP) +.if t .sp .5v +.if n .sp +.. +.de Vb \" Begin verbatim text +.ft CW +.nf +.ne \\$1 +.. +.de Ve \" End verbatim text +.ft R +.fi +.. +.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. +.ie n \{\ +. ds C` "" +. ds C' "" +'br\} +.el\{\ +. ds C` +. ds C' +'br\} +.\" +.\" Escape single quotes in literal strings from groff's Unicode transform. +.ie \n(.g .ds Aq \(aq +.el .ds Aq ' +.\" +.\" If the F register is >0, we'll generate index entries on stderr for +.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index +.\" entries marked with X<> in POD. Of course, you'll have to process the +.\" output yourself in some meaningful fashion. +.\" +.\" Avoid warning from groff about undefined register 'F'. +.de IX +.. +.nr rF 0 +.if \n(.g .if rF .nr rF 1 +.if (\n(rF:(\n(.g==0)) \{\ +. if \nF \{\ +. de IX +. tm Index:\\$1\t\\n%\t"\\$2" +.. +. if !\nF==2 \{\ +. nr % 0 +. nr F 2 +. \} +. \} +.\} +.rr rF +.\" ======================================================================== +.\" +.IX Title "TAP::Harness 3pm" +.TH TAP::Harness 3pm 2023-11-28 "perl v5.38.2" "Perl Programmers Reference Guide" +.\" For nroff, turn off justification. Always turn off hyphenation; it makes +.\" way too many mistakes in technical documents. +.if n .ad l +.nh +.SH NAME +TAP::Harness \- Run test scripts with statistics +.SH VERSION +.IX Header "VERSION" +Version 3.44 +.SH DESCRIPTION +.IX Header "DESCRIPTION" +This is a simple test harness which allows tests to be run and results +automatically aggregated and output to STDOUT. +.SH SYNOPSIS +.IX Header "SYNOPSIS" +.Vb 3 +\& use TAP::Harness; +\& my $harness = TAP::Harness\->new( \e%args ); +\& $harness\->runtests(@tests); +.Ve +.SH METHODS +.IX Header "METHODS" +.SS "Class Methods" +.IX Subsection "Class Methods" +\fR\f(CI\*(C`new\*(C'\fR\fI\fR +.IX Subsection "new" +.PP +.Vb 5 +\& my %args = ( +\& verbosity => 1, +\& lib => [ \*(Aqlib\*(Aq, \*(Aqblib/lib\*(Aq, \*(Aqblib/arch\*(Aq ], +\& ) +\& my $harness = TAP::Harness\->new( \e%args ); +.Ve +.PP +The constructor returns a new \f(CW\*(C`TAP::Harness\*(C'\fR object. It accepts an +optional hashref whose allowed keys are: +.IP \(bu 4 +\&\f(CW\*(C`verbosity\*(C'\fR +.Sp +Set the verbosity level: +.Sp +.Vb 6 +\& 1 verbose Print individual test results to STDOUT. +\& 0 normal +\& \-1 quiet Suppress some test output (mostly failures +\& while tests are running). +\& \-2 really quiet Suppress everything but the tests summary. +\& \-3 silent Suppress everything. +.Ve +.IP \(bu 4 +\&\f(CW\*(C`timer\*(C'\fR +.Sp +Append run time for each test to output. Uses Time::HiRes if +available. +.IP \(bu 4 +\&\f(CW\*(C`failures\*(C'\fR +.Sp +Show test failures (this is a no-op if \f(CW\*(C`verbose\*(C'\fR is selected). +.IP \(bu 4 +\&\f(CW\*(C`comments\*(C'\fR +.Sp +Show test comments (this is a no-op if \f(CW\*(C`verbose\*(C'\fR is selected). +.IP \(bu 4 +\&\f(CW\*(C`show_count\*(C'\fR +.Sp +Update the running test count during testing. +.IP \(bu 4 +\&\f(CW\*(C`normalize\*(C'\fR +.Sp +Set to a true value to normalize the TAP that is emitted in verbose modes. +.IP \(bu 4 +\&\f(CW\*(C`lib\*(C'\fR +.Sp +Accepts a scalar value or array ref of scalar values indicating which +paths to allowed libraries should be included if Perl tests are +executed. Naturally, this only makes sense in the context of tests +written in Perl. +.IP \(bu 4 +\&\f(CW\*(C`switches\*(C'\fR +.Sp +Accepts a scalar value or array ref of scalar values indicating which +switches should be included if Perl tests are executed. Naturally, this +only makes sense in the context of tests written in Perl. +.IP \(bu 4 +\&\f(CW\*(C`test_args\*(C'\fR +.Sp +A reference to an \f(CW@INC\fR style array of arguments to be passed to each +test program. +.Sp +.Vb 1 +\& test_args => [\*(Aqfoo\*(Aq, \*(Aqbar\*(Aq], +.Ve +.Sp +if you want to pass different arguments to each test then you should +pass a hash of arrays, keyed by the alias for each test: +.Sp +.Vb 4 +\& test_args => { +\& my_test => [\*(Aqfoo\*(Aq, \*(Aqbar\*(Aq], +\& other_test => [\*(Aqbaz\*(Aq], +\& } +.Ve +.IP \(bu 4 +\&\f(CW\*(C`color\*(C'\fR +.Sp +Attempt to produce color output. +.IP \(bu 4 +\&\f(CW\*(C`exec\*(C'\fR +.Sp +Typically, Perl tests are run through this. However, anything which +spits out TAP is fine. You can use this argument to specify the name of +the program (and optional switches) to run your tests with: +.Sp +.Vb 1 +\& exec => [\*(Aq/usr/bin/ruby\*(Aq, \*(Aq\-w\*(Aq] +.Ve +.Sp +You can also pass a subroutine reference in order to determine and +return the proper program to run based on a given test script. The +subroutine reference should expect the TAP::Harness object itself as the +first argument, and the file name as the second argument. It should +return an array reference containing the command to be run and including +the test file name. It can also simply return \f(CW\*(C`undef\*(C'\fR, in which case +TAP::Harness will fall back on executing the test script in Perl: +.Sp +.Vb 2 +\& exec => sub { +\& my ( $harness, $test_file ) = @_; +\& +\& # Let Perl tests run. +\& return undef if $test_file =~ /[.]t$/; +\& return [ qw( /usr/bin/ruby \-w ), $test_file ] +\& if $test_file =~ /[.]rb$/; +\& } +.Ve +.Sp +If the subroutine returns a scalar with a newline or a filehandle, it +will be interpreted as raw TAP or as a TAP stream, respectively. +.IP \(bu 4 +\&\f(CW\*(C`merge\*(C'\fR +.Sp +If \f(CW\*(C`merge\*(C'\fR is true the harness will create parsers that merge STDOUT +and STDERR together for any processes they start. +.IP \(bu 4 +\&\f(CW\*(C`sources\*(C'\fR +.Sp +\&\fINEW to 3.18\fR. +.Sp +If set, \f(CW\*(C`sources\*(C'\fR must be a hashref containing the names of the +TAP::Parser::SourceHandlers to load and/or configure. The values are a +hash of configuration that will be accessible to the source handlers via +"config_for" in TAP::Parser::Source. +.Sp +For example: +.Sp +.Vb 5 +\& sources => { +\& Perl => { exec => \*(Aq/path/to/custom/perl\*(Aq }, +\& File => { extensions => [ \*(Aq.tap\*(Aq, \*(Aq.txt\*(Aq ] }, +\& MyCustom => { some => \*(Aqconfig\*(Aq }, +\& } +.Ve +.Sp +The \f(CW\*(C`sources\*(C'\fR parameter affects how \f(CW\*(C`source\*(C'\fR, \f(CW\*(C`tap\*(C'\fR and \f(CW\*(C`exec\*(C'\fR parameters +are handled. +.Sp +For more details, see the \f(CW\*(C`sources\*(C'\fR parameter in "new" in TAP::Parser, +TAP::Parser::Source, and TAP::Parser::IteratorFactory. +.IP \(bu 4 +\&\f(CW\*(C`aggregator_class\*(C'\fR +.Sp +The name of the class to use to aggregate test results. The default is +TAP::Parser::Aggregator. +.IP \(bu 4 +\&\f(CW\*(C`version\*(C'\fR +.Sp +\&\fINEW to 3.22\fR. +.Sp +Assume this TAP version for TAP::Parser instead of default TAP +version 12. +.IP \(bu 4 +\&\f(CW\*(C`formatter_class\*(C'\fR +.Sp +The name of the class to use to format output. The default is +TAP::Formatter::Console, or TAP::Formatter::File if the output +isn't a TTY. +.IP \(bu 4 +\&\f(CW\*(C`multiplexer_class\*(C'\fR +.Sp +The name of the class to use to multiplex tests during parallel testing. +The default is TAP::Parser::Multiplexer. +.IP \(bu 4 +\&\f(CW\*(C`parser_class\*(C'\fR +.Sp +The name of the class to use to parse TAP. The default is +TAP::Parser. +.IP \(bu 4 +\&\f(CW\*(C`scheduler_class\*(C'\fR +.Sp +The name of the class to use to schedule test execution. The default is +TAP::Parser::Scheduler. +.IP \(bu 4 +\&\f(CW\*(C`formatter\*(C'\fR +.Sp +If set \f(CW\*(C`formatter\*(C'\fR must be an object that is capable of formatting the +TAP output. See TAP::Formatter::Console for an example. +.IP \(bu 4 +\&\f(CW\*(C`errors\*(C'\fR +.Sp +If parse errors are found in the TAP output, a note of this will be +made in the summary report. To see all of the parse errors, set this +argument to true: +.Sp +.Vb 1 +\& errors => 1 +.Ve +.IP \(bu 4 +\&\f(CW\*(C`directives\*(C'\fR +.Sp +If set to a true value, only test results with directives will be +displayed. This overrides other settings such as \f(CW\*(C`verbose\*(C'\fR or +\&\f(CW\*(C`failures\*(C'\fR. +.IP \(bu 4 +\&\f(CW\*(C`ignore_exit\*(C'\fR +.Sp +If set to a true value instruct \f(CW\*(C`TAP::Parser\*(C'\fR to ignore exit and wait +status from test scripts. +.IP \(bu 4 +\&\f(CW\*(C`jobs\*(C'\fR +.Sp +The maximum number of parallel tests to run at any time. Which tests +can be run in parallel is controlled by \f(CW\*(C`rules\*(C'\fR. The default is to +run only one test at a time. +.IP \(bu 4 +\&\f(CW\*(C`rules\*(C'\fR +.Sp +A reference to a hash of rules that control which tests may be executed in +parallel. If no rules are declared and CPAN::Meta::YAML is available, +\&\f(CW\*(C`TAP::Harness\*(C'\fR attempts to load rules from a YAML file specified by the +\&\f(CW\*(C`rulesfile\*(C'\fR parameter. If no rules file exists, the default is for all +tests to be eligible to be run in parallel. +.Sp +Here some simple examples. For the full details of the data structure +and the related glob-style pattern matching, see +"Rules data structure" in TAP::Parser::Scheduler. +.Sp +.Vb 4 +\& # Run all tests in sequence, except those starting with "p" +\& $harness\->rules({ +\& par => \*(Aqt/p*.t\*(Aq +\& }); +\& +\& # Equivalent YAML file +\& \-\-\- +\& par: t/p*.t +\& +\& # Run all tests in parallel, except those starting with "p" +\& $harness\->rules({ +\& seq => [ +\& { seq => \*(Aqt/p*.t\*(Aq }, +\& { par => \*(Aq**\*(Aq }, +\& ], +\& }); +\& +\& # Equivalent YAML file +\& \-\-\- +\& seq: +\& \- seq: t/p*.t +\& \- par: ** +\& +\& # Run some startup tests in sequence, then some parallel tests than some +\& # teardown tests in sequence. +\& $harness\->rules({ +\& seq => [ +\& { seq => \*(Aqt/startup/*.t\*(Aq }, +\& { par => [\*(Aqt/a/*.t\*(Aq,\*(Aqt/b/*.t\*(Aq,\*(Aqt/c/*.t\*(Aq], } +\& { seq => \*(Aqt/shutdown/*.t\*(Aq }, +\& ], +\& +\& }); +\& +\& # Equivalent YAML file +\& \-\-\- +\& seq: +\& \- seq: t/startup/*.t +\& \- par: +\& \- t/a/*.t +\& \- t/b/*.t +\& \- t/c/*.t +\& \- seq: t/shutdown/*.t +.Ve +.Sp +This is an experimental feature and the interface may change. +.IP \(bu 4 +\&\f(CW\*(C`rulesfiles\*(C'\fR +.Sp +This specifies where to find a YAML file of test scheduling rules. If not +provided, it looks for a default file to use. It first checks for a file given +in the \f(CW\*(C`HARNESS_RULESFILE\*(C'\fR environment variable, then it checks for +\&\fItestrules.yml\fR and then \fIt/testrules.yml\fR. +.IP \(bu 4 +\&\f(CW\*(C`stdout\*(C'\fR +.Sp +A filehandle for catching standard output. +.IP \(bu 4 +\&\f(CW\*(C`trap\*(C'\fR +.Sp +Attempt to print summary information if run is interrupted by +SIGINT (Ctrl-C). +.PP +Any keys for which the value is \f(CW\*(C`undef\*(C'\fR will be ignored. +.SS "Instance Methods" +.IX Subsection "Instance Methods" +\fR\f(CI\*(C`runtests\*(C'\fR\fI\fR +.IX Subsection "runtests" +.PP +.Vb 1 +\& $harness\->runtests(@tests); +.Ve +.PP +Accepts an array of \f(CW@tests\fR to be run. This should generally be the +names of test files, but this is not required. Each element in \f(CW@tests\fR +will be passed to \f(CWTAP::Parser::new()\fR as a \f(CW\*(C`source\*(C'\fR. See +TAP::Parser for more information. +.PP +It is possible to provide aliases that will be displayed in place of the +test name by supplying the test as a reference to an array containing +\&\f(CW\*(C`[ $test, $alias ]\*(C'\fR: +.PP +.Vb 2 +\& $harness\->runtests( [ \*(Aqt/foo.t\*(Aq, \*(AqFoo Once\*(Aq ], +\& [ \*(Aqt/foo.t\*(Aq, \*(AqFoo Twice\*(Aq ] ); +.Ve +.PP +Normally it is an error to attempt to run the same test twice. Aliases +allow you to overcome this limitation by giving each run of the test a +unique name. +.PP +Tests will be run in the order found. +.PP +If the environment variable \f(CW\*(C`PERL_TEST_HARNESS_DUMP_TAP\*(C'\fR is defined it +should name a directory into which a copy of the raw TAP for each test +will be written. TAP is written to files named for each test. +Subdirectories will be created as needed. +.PP +Returns a TAP::Parser::Aggregator containing the test results. +.PP +\fR\f(CI\*(C`summary\*(C'\fR\fI\fR +.IX Subsection "summary" +.PP +.Vb 1 +\& $harness\->summary( $aggregator ); +.Ve +.PP +Output the summary for a TAP::Parser::Aggregator. +.PP +\fR\f(CI\*(C`aggregate_tests\*(C'\fR\fI\fR +.IX Subsection "aggregate_tests" +.PP +.Vb 1 +\& $harness\->aggregate_tests( $aggregate, @tests ); +.Ve +.PP +Run the named tests and display a summary of result. Tests will be run +in the order found. +.PP +Test results will be added to the supplied TAP::Parser::Aggregator. +\&\f(CW\*(C`aggregate_tests\*(C'\fR may be called multiple times to run several sets of +tests. Multiple \f(CW\*(C`Test::Harness\*(C'\fR instances may be used to pass results +to a single aggregator so that different parts of a complex test suite +may be run using different \f(CW\*(C`TAP::Harness\*(C'\fR settings. This is useful, for +example, in the case where some tests should run in parallel but others +are unsuitable for parallel execution. +.PP +.Vb 8 +\& my $formatter = TAP::Formatter::Console\->new; +\& my $ser_harness = TAP::Harness\->new( { formatter => $formatter } ); +\& my $par_harness = TAP::Harness\->new( +\& { formatter => $formatter, +\& jobs => 9 +\& } +\& ); +\& my $aggregator = TAP::Parser::Aggregator\->new; +\& +\& $aggregator\->start(); +\& $ser_harness\->aggregate_tests( $aggregator, @ser_tests ); +\& $par_harness\->aggregate_tests( $aggregator, @par_tests ); +\& $aggregator\->stop(); +\& $formatter\->summary($aggregator); +.Ve +.PP +Note that for simpler testing requirements it will often be possible to +replace the above code with a single call to \f(CW\*(C`runtests\*(C'\fR. +.PP +Each element of the \f(CW@tests\fR array is either: +.IP \(bu 4 +the source name of a test to run +.IP \(bu 4 +a reference to a [ source name, display name ] array +.PP +In the case of a perl test suite, typically \fIsource names\fR are simply the file +names of the test scripts to run. +.PP +When you supply a separate display name it becomes possible to run a +test more than once; the display name is effectively the alias by which +the test is known inside the harness. The harness doesn't care if it +runs the same test more than once when each invocation uses a +different name. +.PP +\fR\f(CI\*(C`make_scheduler\*(C'\fR\fI\fR +.IX Subsection "make_scheduler" +.PP +Called by the harness when it needs to create a +TAP::Parser::Scheduler. Override in a subclass to provide an +alternative scheduler. \f(CW\*(C`make_scheduler\*(C'\fR is passed the list of tests +that was passed to \f(CW\*(C`aggregate_tests\*(C'\fR. +.PP +\fR\f(CI\*(C`jobs\*(C'\fR\fI\fR +.IX Subsection "jobs" +.PP +Gets or sets the number of concurrent test runs the harness is +handling. By default, this value is 1 \-\- for parallel testing, this +should be set higher. +.PP +\fR\f(CI\*(C`make_parser\*(C'\fR\fI\fR +.IX Subsection "make_parser" +.PP +Make a new parser and display formatter session. Typically used and/or +overridden in subclasses. +.PP +.Vb 1 +\& my ( $parser, $session ) = $harness\->make_parser; +.Ve +.PP +\fR\f(CI\*(C`finish_parser\*(C'\fR\fI\fR +.IX Subsection "finish_parser" +.PP +Terminate use of a parser. Typically used and/or overridden in +subclasses. The parser isn't destroyed as a result of this. +.SH CONFIGURING +.IX Header "CONFIGURING" +\&\f(CW\*(C`TAP::Harness\*(C'\fR is designed to be easy to configure. +.SS Plugins +.IX Subsection "Plugins" +\&\f(CW\*(C`TAP::Parser\*(C'\fR plugins let you change the way TAP is \fIinput\fR to and \fIoutput\fR +from the parser. +.PP +TAP::Parser::SourceHandlers handle TAP \fIinput\fR. You can configure them +and load custom handlers using the \f(CW\*(C`sources\*(C'\fR parameter to "new". +.PP +TAP::Formatters handle TAP \fIoutput\fR. You can load custom formatters by +using the \f(CW\*(C`formatter_class\*(C'\fR parameter to "new". To configure a formatter, +you currently need to instantiate it outside of TAP::Harness and pass it in +with the \f(CW\*(C`formatter\*(C'\fR parameter to "new". This \fImay\fR be addressed by adding +a \fIformatters\fR parameter to "new" in the future. +.ie n .SS """Module::Build""" +.el .SS \f(CWModule::Build\fP +.IX Subsection "Module::Build" +Module::Build version \f(CW0.30\fR supports \f(CW\*(C`TAP::Harness\*(C'\fR. +.PP +To load \f(CW\*(C`TAP::Harness\*(C'\fR plugins, you'll need to use the \f(CW\*(C`tap_harness_args\*(C'\fR +parameter to \f(CW\*(C`new\*(C'\fR, typically from your \f(CW\*(C`Build.PL\*(C'\fR. For example: +.PP +.Vb 10 +\& Module::Build\->new( +\& module_name => \*(AqMyApp\*(Aq, +\& test_file_exts => [qw(.t .tap .txt)], +\& use_tap_harness => 1, +\& tap_harness_args => { +\& sources => { +\& MyCustom => {}, +\& File => { +\& extensions => [\*(Aq.tap\*(Aq, \*(Aq.txt\*(Aq], +\& }, +\& }, +\& formatter_class => \*(AqTAP::Formatter::HTML\*(Aq, +\& }, +\& build_requires => { +\& \*(AqModule::Build\*(Aq => \*(Aq0.30\*(Aq, +\& \*(AqTAP::Harness\*(Aq => \*(Aq3.18\*(Aq, +\& }, +\& )\->create_build_script; +.Ve +.PP +See "new" +.ie n .SS """ExtUtils::MakeMaker""" +.el .SS \f(CWExtUtils::MakeMaker\fP +.IX Subsection "ExtUtils::MakeMaker" +ExtUtils::MakeMaker does not support TAP::Harness out-of-the-box. +.ie n .SS """prove""" +.el .SS \f(CWprove\fP +.IX Subsection "prove" +prove supports \f(CW\*(C`TAP::Harness\*(C'\fR plugins, and has a plugin system of its +own. See "FORMATTERS" in prove, "SOURCE HANDLERS" in prove and App::Prove +for more details. +.SH "WRITING PLUGINS" +.IX Header "WRITING PLUGINS" +If you can't configure \f(CW\*(C`TAP::Harness\*(C'\fR to do what you want, and you can't find +an existing plugin, consider writing one. +.PP +The two primary use cases supported by TAP::Harness for plugins are \fIinput\fR +and \fIoutput\fR: +.IP "Customize how TAP gets into the parser" 2 +.IX Item "Customize how TAP gets into the parser" +To do this, you can either extend an existing TAP::Parser::SourceHandler, +or write your own. It's a pretty simple API, and they can be loaded and +configured using the \f(CW\*(C`sources\*(C'\fR parameter to "new". +.IP "Customize how TAP results are output from the parser" 2 +.IX Item "Customize how TAP results are output from the parser" +To do this, you can either extend an existing TAP::Formatter, or write your +own. Writing formatters are a bit more involved than writing a +\&\fISourceHandler\fR, as you'll need to understand the TAP::Parser API. A +good place to start is by understanding how "aggregate_tests" works. +.Sp +Custom formatters can be loaded configured using the \f(CW\*(C`formatter_class\*(C'\fR +parameter to "new". +.SH SUBCLASSING +.IX Header "SUBCLASSING" +If you can't configure \f(CW\*(C`TAP::Harness\*(C'\fR to do exactly what you want, and writing +a plugin isn't an option, consider extending it. It is designed to be (mostly) +easy to subclass, though the cases when sub-classing is necessary should be few +and far between. +.SS Methods +.IX Subsection "Methods" +The following methods are ones you may wish to override if you want to +subclass \f(CW\*(C`TAP::Harness\*(C'\fR. +.IP """new""" 4 +.IX Item """new""" +.PD 0 +.IP """runtests""" 4 +.IX Item """runtests""" +.IP """summary""" 4 +.IX Item """summary""" +.PD +.SH REPLACING +.IX Header "REPLACING" +If you like the \f(CW\*(C`prove\*(C'\fR utility and TAP::Parser but you want your +own harness, all you need to do is write one and provide \f(CW\*(C`new\*(C'\fR and +\&\f(CW\*(C`runtests\*(C'\fR methods. Then you can use the \f(CW\*(C`prove\*(C'\fR utility like so: +.PP +.Vb 1 +\& prove \-\-harness My::Test::Harness +.Ve +.PP +Note that while \f(CW\*(C`prove\*(C'\fR accepts a list of tests (or things to be +tested), \f(CW\*(C`new\*(C'\fR has a fairly rich set of arguments. You'll probably want +to read over this code carefully to see how all of them are being used. +.SH "SEE ALSO" +.IX Header "SEE ALSO" +Test::Harness |