48 Commits

Author SHA1 Message Date
Chris Johns
3368e6b1aa test, rtemstoolkit: Fix regx escape errors on python 3.12 2024-04-22 16:42:08 +10:00
Chris Johns
a22b396a32 tester/tftp: Add a session timeout
- Fix listener done state

- Finish open with the state as finished

Closes #4959
2023-11-08 08:34:14 +11:00
Sebastian Huber
c05c4a9cbf tester: Use shorter SHA512 encoding
Encode the SHA512 digest in base64urlsafe instead of a hex string to
reduce the report size.
2023-08-11 07:51:49 +02:00
Sebastian Huber
1cd476e155 tester: Sort test reports for JSON/YAML
This makes the reports more git friendly.

Update 4671.
2022-08-08 06:42:34 +02:00
Sebastian Huber
ef4b8c53f9 tester: Normalize JSON and YAML reports
Report the same data in JSON and YAML reports.  Do not report redundant
information.

Update 4671.
2022-07-26 14:31:32 +02:00
Chris Johns
9d880d78b2 tester: Count tester header errors
- Remove the hard tester error and count the header errors.
2021-09-09 18:33:54 +10:00
Cláudio Maia
2251c5a878 tester: Add yaml format to the supported report formats 2020-12-16 07:43:29 +01:00
Kinsey Moore
79a9a46a20 tester: Prevent crash with json output
This fixes an issue with json report output that causes a crash due to
the incorrect object member being accessed for test-too-long. This also
changes treatment of fatal errors to match how they are displayed in the
tester log output and changes "fatal error" to "fatal-error" to better
match the naming of all the other failure types.
2020-10-05 18:58:40 -05:00
Chris Johns
87262b429d Move all python commands to use env python
- If you host does not provide a python command consult the User
  manual for ways you can run the python commands.

- Full package paths are being used to avoid namespace pollution and
  crosstalk.
2020-09-26 10:49:27 +10:00
Chris Johns
cd8fa2fe74 tester: Add support for test-too-long
- A test that loops generating output did not timeout. Monitor the
  the session time and set a maximum test period.
2020-09-08 16:15:43 +10:00
Sebastian Huber
0d66fd5264 tester: Use prefix for rtems_tools by default
Commit 72c684eff2cd932b4948e21902680a93473340c3 removed the default
value of rtems_tools.  If the --rtems--tools option was omitted the
rtems-test command printed lots of

	error: run.cfg:61: macro '%{rtems_tools}' not found

error messages.
2020-02-26 07:29:53 +01:00
Hesham Almatary
c02459ee65 tester: Add basic support to generate JUnit test reports 2019-11-22 09:05:23 +00:00
Kinsey Moore
e9f48e9fac Fix exception when no output format is specified
The JSON log generation patch introduced a bug when the report output
generation was not configured due to attempting to iterate over 'None'.
2019-09-24 08:32:52 +10:00
Chris Johns
72c684eff2 tester: Minor fix to prefix handling. 2019-09-05 09:47:01 +10:00
Kinsey Moore
5fcaf843bd Add JSON log generation
Add log formatter hooks and JSON log formatter to the test infrastructure
for consumption by automated processes or report generators.
2019-08-22 10:41:48 +10:00
Vijay Kumar Banerjee
a1498ab899 Coverage : install covoar in tester/bin 2018-11-27 16:55:26 +11:00
Chris Johns
f632bd8b60 tester: Minor formatting change. 2018-11-26 09:56:13 +11:00
Chris Johns
e058db0281 python: Provide support to select a valid python version.
- Update imports after wrapping the code.
- Fix python3 issues.
- Fix config path issues for in repo and install runs.

Closes #3537
2018-11-08 18:13:45 +11:00
Chris Johns
3bd8def210 config: Consolidate the version information into a single configuration file 2018-10-03 11:38:09 +10:00
Chris Johns
e341a65347 tester: Make the path to covoar absolute to ignore the env PATH.
Using the environment's path to find covoar allow invalid versions
to be used which may vary in subtle ways. Find and use the covoar
that is build with the version of 'rtems-test'.

This patch means you do not need to install the tools before
running improving the development experience.

Closes #3458
2018-06-18 12:26:16 +10:00
Vijay Kumar Banerjee
b762312fae tester: Add script to generate html coverage report from covoar output
Add support in tester to run covoar and generate an html report to display
the summary of the coverage reports generated from covoar.

Co-authored-by : Cillian O'Donnell <cpodonnell8@gmail.com>
2018-06-05 18:09:19 -05:00
Chris Johns
b7c4753de0 teater: Filter out an *.norun.* executables. 2018-04-10 14:00:09 +10:00
Chris Johns
30218f5ca5 tester: Add reporting the RTEMS version, build and tools.
Update #3217.
2017-11-11 16:21:52 +11:00
Chris Johns
24d51f7e4b tester: Change the mail subject so it list by BSP in the archives.
Change the subject to place the arch/bsp first and then a short
score card for the tests so the archive can be listed by subject and
you can see regressions.
2017-11-04 11:22:08 +11:00
Chris Johns
42e39d2644 tester: Fix the output so the report end analysis works. 2017-11-03 18:20:31 +11:00
Chris Johns
c68beb8181 tester: Add the rtems-run command. 2017-11-03 17:59:53 +11:00
Chris Johns
bb218b62b0 tester: Minor fix to the test results mail. 2017-10-25 10:21:22 +11:00
Chris Johns
c73e03040f tester: Check mail parameters early so users can fix. 2017-10-25 10:13:28 +11:00
Chris Johns
41d0c343ee tester: Add a BSP field to each BSP INI configuration section.
The INI section in a BSP configuration is the name of the INI file so
each BSP configuration section needs a BSP.

Updates #3204.
2017-10-25 09:03:43 +11:00
Chris Johns
bf58911519 tester: Refactor to use INI format files for BSP configurations.
- Add support for user condfigurations files with the --user-config.
- Add support for a $HOME/.rtemstesterrc for a user configuration.

Closes #3204.
2017-10-24 22:21:35 +11:00
Chris Johns
cec5878ad3 tester: Remove pass status messages from the email log posts. 2017-10-21 16:20:04 +11:00
Chris Johns
89e8c2abb6 tester: Add support to email results. 2017-10-12 06:13:47 +11:00
Chris Johns
662e1f7d9a tester: Add a tester off to turn the target off when finished.
Turning the target off stops it sitting asking for a download if you have
more than one target in a test set up. Any target can jump in and
nick the download.
2017-10-10 13:30:31 +11:00
Chris Johns
3a867a4919 Add TFTP as a back end option for testing. Add telnet as a console option.
TFTP runs a local TFTP server on port 69 or another specified port and
serves each test for any requested file.

Telnet is now a console option.
2017-09-21 18:26:20 +10:00
Chris Johns
15a3e06518 tester: Add support for expected-fail, user-input, and benchmarks.
Count the errors for the states listed and add running totals.

Kill the test process if the test state is user-input or benchmark.
2017-04-04 13:32:03 +10:00
Chris Johns
dfa2468af6 tester: Fix reraising an exception with Python2 and Python3.
Updates #2619.
2016-03-16 21:39:39 +11:00
Chris Johns
6f8b07c3b6 rtemstoolkit: Fix execute's writer thread to not eval() the input.
The conversion to Python3 added an eval() call which is wrong.

Fix the spelling in execute.

Fix labels in the tester gdb locking.

Check the debug-trace arguments.

Close #2642.
2016-03-14 14:59:11 +11:00
Chris Johns
c9ed2483c6 Remove exception test code left in.
Updates #2619.
2016-03-09 14:34:26 +11:00
Chris Johns
2de37f3432 Python 2 and python 3 refactor fixes.
Updates #2619.
2016-03-09 14:27:42 +11:00
Chris Johns
b0fa2ae998 Update rtems-tool to support Python 2 and 3.
Add solaris and netbsd.

Close #2619.
2016-03-03 16:53:39 +11:00
Chris Johns
efc4f099b4 Add release versioning support.
Support a top level VERSION file that defines an RTEMS release.

Fix the install of the python modules including thertems-test.

Update the git python module to the RSB version. Fix the options to
not call clean and to call dirty.

Update the version python module.

Fix the rtld C++ support to the VERSION file and the top level waf
script.
2015-12-09 20:08:19 +11:00
Sebastian Huber
04a52040ae Python 3 compatibility 2015-11-12 11:15:23 +01:00
Chris Johns
9600c39638 Fixes to the run the waf built tests when installed. 2015-02-08 18:14:16 +11:00
Chris Johns
b7d48ef5a4 Install the rtems-test command.
This installs the Python RTEMS Toolkit.

The copmiler has been switched from forcing gcc to allowing waf
to detect the host's tool chain.
2015-02-08 17:12:04 +11:00
Ben Gras
e4014f05e8 typo 2014-11-02 08:55:51 +11:00
Chris Johns
c04a84917a tester: Correctly handle contro-c.
Add support to kill running tests if the user presses control-c.
2014-05-31 20:03:05 +10:00
Ben Gras
803b63d76e add --filter to specify the set of selected tests.
. parameter is a glob
  . default is *.exe as it was
  . it is a convenient way to select a particular test to run over and
    over again while debugging it, without laboriously manipulating the
    .exe's that are present (i.e. moving/deleting them)
2014-05-09 21:39:01 +10:00
Chris Johns
50fdf12244 rt: Add the rtems-tester. 2014-02-15 06:30:06 +11:00