testcode.py¶
testcode.py - a command-line interface to testcode.
Synopsis¶
testcode.py [options] [action1 [action2...]]
Description¶
Run a set of actions on a set of tests.
Requires two configuration files, jobconfig and userconfig. See testcode documentation for further details.
testcode.py provides a command-line interface to testcode, a simple framework for comparing output from (principally numeric) programs to previous output to reveal regression errors or miscompilation.
Actions¶
‘’run’’ is th default action.
- compare
- compare set of test outputs from a previous testcode run against the benchmark outputs.
- compare
- diff set of test outputs from a previous testcode run against the benchmark outputs.
- make-benchmarks
- create a new set of benchmarks and update the userconfig file with the new benchmark id. Also runs the ‘run’ action unless the ‘compare’ action is also given.
- run
- run a set of tests and compare against the benchmark outputs.
- tidy
- Remove files from previous testcode runs from the test directories.
Options¶
| -h, --help | show this help message and exit |
| -b BENCHMARK, --benchmark=BENCHMARK | |
| Set the file ID of the benchmark files. If BENCHMARK is in the format t:ID, then the test files with the corresponding ID are used. This allows two sets of tests to be compared. Default: specified in the [user] section of the userconfig file. | |
| -c CATEGORY, --category=CATEGORY | |
| Select the category/group of tests. Can be specified multiple times. Default: use the _default_ category if run is an action unless make-benchmarks is an action. All other cases use the _all_ category by default. The _default_ category contains all tests unless otherwise set in the jobconfig file. | |
| -e EXECUTABLE, --executable=EXECUTABLE | |
| Set the executable(s) to be used to run the tests. Can be a path or name of an option in the userconfig file, in which case all test programs are set to use that value, or in the format program_name=value, which affects only the specified program. Only relevant to the run action. Default: exe variable set for each program listed in the userconfig file. | |
| --jobconfig=JOBCONFIG | |
| Set path to the job configuration file. Default: jobconfig. | |
| --job-option=JOB_OPTION | |
| Override/add setting to jobconfig. Takes three arguments. Format: section_name option_name value. Default: none. | |
| -n NTHREADS, --nthreads=NTHREADS | |
| Set the number of tests to run concurrently. Only relevant to the run action. Default: 1. | |
| --older-than=OLDER_THAN | |
| Set the age (in days) of files to remove. Only relevant to the tidy action. Default: 14 days. | |
| -p NPROCS, --processors=NPROCS | |
| Set the number of processors to run each test on. Only relevant to the run action. Default: run tests as serial jobs. | |
| -q, --quiet | Print only minimal output. Default: False. |
| -s QUEUE_SYSTEM, --submit=QUEUE_SYSTEM | |
| Submit tests to a queueing system of the specified type. Only PBS system is currently implemented. Only relevant to the run action. Default: none. | |
| -t TEST_ID, --test-id=TEST_ID | |
| Set the file ID of the test outputs. If TEST_ID is in the format b:ID, then the benchmark files with the corresponding ID are used. This allows two sets of benchmarks to be compared. Default: unique filename based upon date if running tests and most recent test_id if comparing tests. | |
| --userconfig=USERCONFIG | |
| Set path to the user configuration file. Default: userconfig. | |
| --user-option=USER_OPTION | |
| Override/add setting to userconfig. Takes three arguments. Format: section_name option_name value. Default: none. | |
License¶
Modified BSD License. See LICENSE in the source code for more details.
Bugs¶
Contact James Spencer (j.spencer@imperial.ac.uk) regarding bug reports, suggestions for improvements or code contributions.