197 lines
9.2 KiB
Groff
197 lines
9.2 KiB
Groff
.\" **************************************************************************
|
|
.\" * _ _ ____ _
|
|
.\" * Project ___| | | | _ \| |
|
|
.\" * / __| | | | |_) | |
|
|
.\" * | (__| |_| | _ <| |___
|
|
.\" * \___|\___/|_| \_\_____|
|
|
.\" *
|
|
.\" * Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
|
.\" *
|
|
.\" * This software is licensed as described in the file COPYING, which
|
|
.\" * you should have received as part of this distribution. The terms
|
|
.\" * are also available at https://curl.se/docs/copyright.html.
|
|
.\" *
|
|
.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell
|
|
.\" * copies of the Software, and permit persons to whom the Software is
|
|
.\" * furnished to do so, under the terms of the COPYING file.
|
|
.\" *
|
|
.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
|
|
.\" * KIND, either express or implied.
|
|
.\" *
|
|
.\" * SPDX-License-Identifier: curl
|
|
.\" *
|
|
.\" **************************************************************************
|
|
.\"
|
|
.TH runtests.pl 1 "06 Jun 2023" runtests runtests
|
|
.SH NAME
|
|
runtests.pl \- run one or more test cases
|
|
.SH SYNOPSIS
|
|
.B runtests.pl [options] [tests]
|
|
.SH DESCRIPTION
|
|
\fIruntests.pl\fP runs one, several or all the existing test cases in curl's
|
|
test suite. It is often called from the root Makefile of the curl package with
|
|
\&'make test'.
|
|
.SH "TESTS"
|
|
Specify which test(s) to run by specifying test numbers or keywords.
|
|
|
|
If no test number or keyword is given, all existing tests that the script can
|
|
find will be considered for running. You can specify single test cases to run
|
|
by specifying test numbers space-separated, like "1 3 5 7 11", and you can
|
|
specify a range of tests like "45 to 67".
|
|
|
|
Specify tests to not run with a leading exclamation point, like "!66", which
|
|
runs all available tests except number 66.
|
|
|
|
Prefix a test number with a tilde (~) to still run it, but ignore the results.
|
|
|
|
It is also possible to specify tests based on a keyword describing the test(s)
|
|
to run, like "FTPS". The keywords are strings used in the individual tests.
|
|
|
|
You can also specify keywords with a leading exclamation point and the keyword
|
|
or phrase, like "!HTTP NTLM auth" to run all tests \fBexcept\fP those using
|
|
this keyword. Remember that the exclamation marks and spaces will need to be
|
|
quoted somehow when entered at many command shells.
|
|
|
|
Prefix a keyword with a tilde (~) to still run it, but ignore the results.
|
|
.SH OPTIONS
|
|
.IP "-a"
|
|
Continue running the rest of the test cases even if one test fails. By
|
|
default, the test script stops as soon as an error is detected.
|
|
.IP "-ac <curl>"
|
|
Provide a path to a curl binary to talk to APIs (currently only CI test APIs).
|
|
.IP "-am"
|
|
Display test results in automake style output (PASS/FAIL: [number] [name]).
|
|
.IP "-c <curl>"
|
|
Provide a path to a custom curl binary to run the tests with. Default is the
|
|
curl executable in the build tree.
|
|
.IP "-d"
|
|
Enable protocol debug: have the servers display protocol output. If used in
|
|
conjunction with parallel testing, it will be difficult to associate the logs
|
|
with the test being run.
|
|
.IP "-E <exclude_file>"
|
|
Load the \fBexclude_file\fP with additional reasons why certain tests
|
|
should be skipped. Useful when testing with external HTTP proxies in
|
|
which case some of the tests aren't appropriate.
|
|
The file contains colon-delimited lines. The first field contains the
|
|
type of exclusion, the second field contains a pattern and the final
|
|
field contains the reason why matching tests should be skipped.
|
|
The exclusion types are \fkeyword\fP, \ftest\fP, and \ftool\fP.
|
|
.IP "-e"
|
|
Run the test event-based (if possible). This will make runtests invoke curl
|
|
with --test-event option. This option only works if both curl and libcurl were
|
|
built debug-enabled.
|
|
.IP "-f"
|
|
Force the test to run even if mentioned in DISABLED.
|
|
.IP "-g"
|
|
Run the given test(s) with gdb. This is best used on a single test case and
|
|
curl built --disable-shared. This then fires up gdb with command line set to
|
|
run the specified test case. Simply (set a break-point and) type 'run' to
|
|
start.
|
|
.IP "-gw"
|
|
Run the given test(s) with gdb as a windowed application.
|
|
.IP "-h, --help"
|
|
Displays a help text about this program's command line options.
|
|
.IP "-k"
|
|
Keep output and log files in log/ after a test run, even if no error was
|
|
detected. Useful for debugging.
|
|
.IP "-j[num]"
|
|
Spawn num processes to run tests. This defaults to 0 to run tests serially
|
|
within a single process. Using a number greater than one allows multiple tests
|
|
to run in parallel, speeding up a test run. The optimum number is dependent on
|
|
the system and set of tests to run, but 7*number of CPU cores is a good figure
|
|
to start with, or 1.3*number of CPU cores if Valgrind is in use. Enabling
|
|
parallel tests is not recommended in conjunction with the \-g option.
|
|
.IP "-L <file>"
|
|
Load and execute the specified file which should contain perl code.
|
|
This option allows one to change \fIruntests.pl\fP behaviour by overwriting
|
|
functions and variables and is useful when testing external proxies
|
|
using curl's regression test suite.
|
|
.IP "-l"
|
|
Lists all test case names.
|
|
.IP "-n"
|
|
Disable the check for and use of valgrind.
|
|
.IP "--no-debuginfod"
|
|
Delete the DEBUGINFOD_URLS variable if that is defined. Makes valgrind, gdb
|
|
etc not able to use this functionality.
|
|
.IP "-o <variablename=value>"
|
|
Overwrite the specified internal \fBvariable\fP with \fBvalue\fP.
|
|
Useful to change variables that didn't get a dedicated flag to change them.
|
|
Check the source to see which variables are available.
|
|
.IP "-P <proxy>"
|
|
Use the specified HTTP proxy when executing tests, even if the tests
|
|
themselves don't specify a proxy. This option allows one to test external
|
|
proxies using curl's regression test suite.
|
|
.IP "-p"
|
|
Prints out all files in "log/" to stdout when a test case fails. Very
|
|
practical when used in the automated and distributed tests since then the
|
|
people checking the failures and the reasons for them might not have physical
|
|
access to the machine and logs.
|
|
.IP "-R"
|
|
Run the tests in a scrambled, or randomized, order instead of sequentially.
|
|
|
|
The random seed initially set for this is fixed per month and can be set with
|
|
\fI--seed\fP.
|
|
.IP "-r"
|
|
Display run time statistics. (Requires Perl Time::HiRes module)
|
|
.IP "-rf"
|
|
Display full run time statistics. (Requires Perl Time::HiRes module)
|
|
.IP "-rm"
|
|
Force removal of files by killing locking processes. (Windows only,
|
|
requires Sysinternals handle[64].exe to be on PATH)
|
|
.IP "--repeat=[num]"
|
|
This will repeat the given set of test numbers this many times. If no test
|
|
numbers are given, it will repeat ALL tests this many times. It iteratively
|
|
adds the new sequence at the end of the initially given one.
|
|
|
|
If \fB-R\fP is also used, the scrambling is done after the repeats have
|
|
extended the test sequence.
|
|
.IP "-s"
|
|
Shorter output. Speaks less than default.
|
|
.IP "--seed=[num]"
|
|
When using \fI--shallow\fP or \fI-R\fP that randomize certain aspects of the
|
|
behavior, this option can set the initial seed. If not set, the random seed
|
|
will be set based on the currently set local year and month and the first line
|
|
of the "curl -V" output.
|
|
.IP "--shallow=[num]"
|
|
Used together with \fB-t\fP. This limits the number of tests to fail in
|
|
torture mode to no more than 'num' per test case. If this reduces the amount,
|
|
the script will randomly discard entries to fail until the amount is 'num'.
|
|
|
|
The random seed initially set for this is fixed per month and can be set with
|
|
\fI--seed\fP.
|
|
.IP "-t[num]"
|
|
Selects a \fBtorture\fP test for the given tests. This makes runtests.pl first
|
|
run the tests once and count the number of memory allocations made. It then
|
|
reruns the test that number of times, each time forcing one of the allocations
|
|
to fail until all allocs have been tested. By setting \fInum\fP you can force
|
|
the allocation with that number to be set to fail at once instead of looping
|
|
through everyone, which is very handy when debugging and then often in
|
|
combination with \fI-g\fP.
|
|
.IP "-u"
|
|
Error instead of warning on server unexpectedly alive.
|
|
.IP "-v"
|
|
Enable verbose output. Speaks more than by default. If used in conjunction with
|
|
parallel testing, it will be difficult to associate the logs with the test
|
|
being run.
|
|
.IP "-vc <curl>"
|
|
Provide a path to a custom curl binary to run when verifying that the servers
|
|
running are indeed our test servers. Default is the curl executable in the
|
|
build tree.
|
|
.SH "RUNNING TESTS"
|
|
Many tests have conditions that must be met before the test case can run
|
|
fine. They could depend on built-in features in libcurl or features present in
|
|
the operating system or even in third-party libraries that curl may or may not
|
|
use.
|
|
.P
|
|
The test script checks most of these by itself to determine when it is
|
|
safe to attempt to run each test. Those which cannot be run due to
|
|
failed requirements will simply be skipped and listed at the completion
|
|
of all test cases. In some unusual configurations, the test script
|
|
cannot make the correct determination for all tests. In these cases,
|
|
the problematic tests can be skipped using the "!keyword" skip feature
|
|
documented earlier.
|
|
.SH "WRITING TESTS"
|
|
The simplest way to write test cases is to start with a similar existing test,
|
|
save it with a new number and then adjust it to fit. There's an attempt to
|
|
document the test case file format in the tests/FILEFORMAT.md.
|