1
0
mirror of https://github.com/ARMmbed/mbedtls.git synced 2025-05-15 03:07:16 +08:00

Document test case descriptions

This commit is contained in:
Gilles Peskine 2019-09-27 20:22:41 +02:00
parent 717cd76e8a
commit e94bc87ebe

View File

@ -4,6 +4,36 @@ This document is an overview of the Mbed TLS test framework and test tools.
This document is incomplete. You can help by expanding it. This document is incomplete. You can help by expanding it.
## Unit tests
See https://tls.mbed.org/kb/development/test_suites
### Unit test descriptions
Each test case has a description which succinctly describes for a human audience what the test does. The first non-comment line of each paragraph in a `.data` file is the test description. The following rules and guidelines apply:
* Test descriptions may not contain semicolons, line breaks and other control characters, or non-ASCII characters. <br>
Rationale: keep the tools that process test descriptions (`generate_test_code.py`, [outcome file](#outcome-file) tools) simple.
* Test descriptions must be unique within a `.data` file. If you can't think of a better description, the convention is to append `#1`, `#2`, etc. <br>
Rationale: make it easy to relate a failure log to the test data. Avoid confusion between cases in the [outcome file](#outcome-file).
* Test descriptions should be a maximum of **66 characters**. <br>
Rationale: 66 characters is what our various tools assume (leaving room for 14 more characters on an 80-column line). Longer descriptions may be truncated or may break a visual alignment. <br>
We have a lot of test cases with longer descriptions, but they should be avoided. At least please make sure that the first 66 characters describe the test uniquely.
* Make the description descriptive. “foo: x=2, y=4” is more descriptive than “foo #2”. “foo: 0<x<y, both even is even better if these inequalities and parities are why this particular test data was chosen.
* Avoid changing the description of an existing test case without a good reason. This breaks the tracking of failures across CI runs, since this tracking is based on the descriptions.
`tests/scripts/check-test-cases.py` enforces some rules and warns if some guidelines are violated.
## TLS tests
### SSL extension tests
#### SSL test case descriptions
Each test case in `ssl-opt.sh` has a description which succinctly describes for a human audience what the test does. The test description is the first parameter to `run_tests`.
The same rules and guidelines apply as for [unit test descriptions](#unit-test-descriptions). In addition, the description must be written on the same line as `run_test`, in double quotes, for the sake of `check-test-cases.py`.
## Running tests ## Running tests
### Outcome file ### Outcome file