154 Commits

Author SHA1 Message Date
Chris Johns
3368e6b1aa test, rtemstoolkit: Fix regx escape errors on python 3.12 2024-04-22 16:42:08 +10:00
Chris Johns
384acf1bcb tester/console: Fix restart regx check
The fix to the TFTP timeout has a simple bug.

Updates #4959
2023-12-14 17:07:04 +11:00
Chris Johns
a22b396a32 tester/tftp: Add a session timeout
- Fix listener done state

- Finish open with the state as finished

Closes #4959
2023-11-08 08:34:14 +11:00
Sam Price
7dc34d2cab rtems-tools: stty rename / spelling bug.
This patch fixes a spelling/rename mistake in rtems-tools.
2023-09-11 14:11:48 -05:00
Sebastian Huber
c05c4a9cbf tester: Use shorter SHA512 encoding
Encode the SHA512 digest in base64urlsafe instead of a hex string to
reduce the report size.
2023-08-11 07:51:49 +02:00
Muhammad Sulthan Mazaya
bd2d3d5523 rtemstoolkit: Fix shlex.split to use posix mode and add unit test for pipe operation
Turns out subprocess.Popen operates on posix mode. Also, there is an
issue with previous implementation of pipe command that is fixed by
Chris. Now, it can also accepts command in form of a string. The unit
test for that is added via this patch.
2023-06-21 14:30:48 +10:00
Muhammad Sulthan Mazaya
60e793a17c tester/rt: use shlex.split to split command args
The regular split-by-space function used to split command arguments
creates compatibility issues with many shell command syntaxes. A
specific example is the handling of string arguments, as shown below:

%define renode_args -e start_opts -e "s %{bsp_resc_script}"

Thus, it is changed to use shlex.split instead. It splits
the command arguments using shell-like syntax. More about shlex
module here: https://docs.python.org/3/library/shlex.html
2023-06-21 09:09:06 +10:00
Sebastian Huber
203077030d tester: Check for begin/end of test
Check for "BEGIN OF TEST" and "END OF TEST" to not use other information blocks
such as "END OF GCOV" to determine the test status.
2022-10-07 06:26:31 +02:00
Sebastian Huber
1cd476e155 tester: Sort test reports for JSON/YAML
This makes the reports more git friendly.

Update 4671.
2022-08-08 06:42:34 +02:00
Sebastian Huber
ef4b8c53f9 tester: Normalize JSON and YAML reports
Report the same data in JSON and YAML reports.  Do not report redundant
information.

Update 4671.
2022-07-26 14:31:32 +02:00
Sebastian Huber
e3a5912249 tester: Fix string representation 2022-07-26 13:43:37 +02:00
Karel Gardas
43843b85fe tester/gdb: allow kill on the test end
Sponsored-By:	Precidata
2022-05-29 19:47:05 +02:00
Sebastian Huber
f73627e364 tester/exe: Fix adjust timeouts by the step size
The bug was introduced by cfd5aa41e847752cd98cde65515df7ce45ff9665.

The code monitors the test and generates a timeout test failure.  There are two
layers of timeout, one for output and the other for a test running too long.

The test needs to generate some output within the first timeout period and the
second timeout detects if the test has run too long. The first timeout detects
a target has locked up. The second is for a test looping generating output.

With exe type tests the output is buffered in a separate thread.
2022-01-10 13:55:23 +01:00
Alex White
ba4648bf68 rtems-bsp-builder: Fix mail support
This fixes a problem with mailer options support that occurred because
check.py uses argparse.ArgumentParser instead of tester.rt.options.
2021-10-29 16:44:02 -05:00
Chris Johns
59880c0933 tester/wait: Add a wait directive to the tester
- Lets you test with stand alone TFTP or other services
2021-09-21 17:41:15 +10:00
Chris Johns
cfd5aa41e8 tester/exe: Adjust timeouts by the step size 2021-09-21 17:41:02 +10:00
Chris Johns
dac7ef233a tester/tftp: Fix recovery of timed out TFTP sessions
- Add support to retry the tftp session if the target has not
  started

- Add target handlers for the test directives to allow recovery
  on error
2021-09-19 08:31:58 +10:00
Chris Johns
ad7b862cda tester: Simplify the console data handling 2021-09-19 08:31:58 +10:00
Ryan Long
c6870de792 check.py: Fix incorrect use of os.linesep
Replaced a couple of calls to os.linesep() with os.linesep because
os.linesep() does not exist.
2021-09-15 10:53:54 -05:00
Chris Johns
e730a2d781 tester/tftp: Fix prptocol timeout handling
- Change some of the notification messages
2021-09-09 18:33:54 +10:00
Chris Johns
9d880d78b2 tester: Count tester header errors
- Remove the hard tester error and count the header errors.
2021-09-09 18:33:54 +10:00
Chris Johns
c73fa1691c tester/telnet: Only reopen once a second
- Limit the reopen to once a second

- Make sure the socket is closed when reopening
2021-09-09 18:33:54 +10:00
Ryan Long
527848d5cc rtems-bsp-builder: Change to waf build system
Closes #4124
2021-05-25 12:41:13 -05:00
Kinsey Moore
a918844889 tester: Avoid TFTP race condition
The u-boot TFTP client will fail a fully completed file retrieval if the
final ACK gets an ICMP response that the port is unavailable which can
occur if the port is closed before the ACK arrives from the client. This
change causes the TFTP server to wait for the final ACK before closing
the socket and prevents transfers from failing due to the race
condition.
2021-05-18 13:58:34 -05:00
Alex White
b3fcd10685 coverage.py: Call covoar once
Updates #4374
2021-04-06 14:18:55 -05:00
Alex White
57e7b22faf tester: Limit branch coverage percentage precision 2021-03-30 13:15:59 -05:00
Alex White
8ce2cfc917 coverage: Fix option processing on FreeBSD
Covoar uses getopt() to process the command line options. If getopt() is
POSIX-compliant, it will return -1 when it encounters the first
non-option command line argument. It appears that it behaves this way on
FreeBSD, but on Linux getopt() continues to process arguments while
skipping any non-options. This changes the order of arguments passed to
covoar by coverage.py to group all options at the beginning. This allows
hosts with POSIX-compliant getopt() implementations to correctly process
all command line options.
2021-03-30 13:15:59 -05:00
Alex White
bfee5507b9 coverage: Give coverage bars red background 2021-03-30 13:15:58 -05:00
Alex White
4d21ff8d60 coverage/reports: Share common JS and CSS in reports
This moves all of the javascript and CSS files that are shared by the
symbol set HTML reports to the shared parent directory. It also includes
the javascript and CSS in the top-level index file.
2021-03-30 13:15:58 -05:00
Alex White
04597495e5 coverage/reports: Improve formatting and clarity
The coverage reports contain places where they display incorrect or
vague information particularly when some statistic is unavailable. This
has been fixed. The formatting and wording of various things has been
improved as well.
2021-03-30 13:15:58 -05:00
Alex White
4aabd9bbf7 covoar/reports: Add new statistics to summary
The following new statistics have been added to the summary report:
number of unreferenced symbols, total branch paths found, number of
branch paths not executed, and percentage of branch paths covered.
2021-03-30 13:15:58 -05:00
Alex White
f78ec7ceeb tester: Remove target from BSP coverage configs 2021-03-30 13:15:57 -05:00
Cláudio Maia
2251c5a878 tester: Add yaml format to the supported report formats 2020-12-16 07:43:29 +01:00
Kinsey Moore
2bb7d3a1bb tester: Report fatal-error test states as failures
Tests resulting in the fatal-error state are counted as failures, but
are not currently reported in the text summary. This has already been
fixed for JSON output.
2020-11-25 16:18:03 -06:00
Kinsey Moore
79a9a46a20 tester: Prevent crash with json output
This fixes an issue with json report output that causes a crash due to
the incorrect object member being accessed for test-too-long. This also
changes treatment of fatal errors to match how they are displayed in the
tester log output and changes "fatal error" to "fatal-error" to better
match the naming of all the other failure types.
2020-10-05 18:58:40 -05:00
Chris Johns
87262b429d Move all python commands to use env python
- If you host does not provide a python command consult the User
  manual for ways you can run the python commands.

- Full package paths are being used to avoid namespace pollution and
  crosstalk.
2020-09-26 10:49:27 +10:00
Chris Johns
cd8fa2fe74 tester: Add support for test-too-long
- A test that loops generating output did not timeout. Monitor the
  the session time and set a maximum test period.
2020-09-08 16:15:43 +10:00
Chris Johns
7a29b47a0f tester: Fix TFTP server Python2 issues.
- Add a --show-backtrace option to make it easier for users to
  get an exception backtrace if something goes wrong.

- Fix the --packet-trace option so it actually decodes the packets
2020-09-07 10:07:11 +10:00
Chris Johns
eb3608133b tester: Change to a simpler TFTP server
- Add a simpler TFTP to allow parallel test hardware

- Remove the imported tftpy server

Closes #4063
2020-09-01 09:18:04 +10:00
Chris Johns
4f5485eb51 rtems-test: target_exe_filter fails when there is no filter
Closes #4006
2020-06-17 08:43:00 +10:00
Chris Johns
753eb949b3 tester/gdb: Add lock timing and remote fetching registers.
- Add timing for the locks to aid performance profiling
- Remove fetching registers as the MI parser is slow on pyton2

Updates #3864
2020-03-30 09:52:57 +11:00
Sebastian Huber
274981f77d Improve Python 3 compatibility 2020-03-17 11:36:05 +01:00
Sebastian Huber
0d66fd5264 tester: Use prefix for rtems_tools by default
Commit 72c684eff2cd932b4948e21902680a93473340c3 removed the default
value of rtems_tools.  If the --rtems--tools option was omitted the
rtems-test command printed lots of

	error: run.cfg:61: macro '%{rtems_tools}' not found

error messages.
2020-02-26 07:29:53 +01:00
Sebastian Huber
29ad0ec7a8 tester: Prefer '_' as test state separator char 2019-12-19 10:09:22 +01:00
Hesham Almatary
c02459ee65 tester: Add basic support to generate JUnit test reports 2019-11-22 09:05:23 +00:00
Kinsey Moore
e9f48e9fac Fix exception when no output format is specified
The JSON log generation patch introduced a bug when the report output
generation was not configured due to attempting to iterate over 'None'.
2019-09-24 08:32:52 +10:00
Chris Johns
128ee7b7ec bsp-builder: Do not clean the build space when starting with --no-clean 2019-09-05 09:47:01 +10:00
Chris Johns
72c684eff2 tester: Minor fix to prefix handling. 2019-09-05 09:47:01 +10:00
Chris Johns
89442b7c0c tester/tftp: Handle ctl-C when active. 2019-09-05 09:47:01 +10:00
Kinsey Moore
5fcaf843bd Add JSON log generation
Add log formatter hooks and JSON log formatter to the test infrastructure
for consumption by automated processes or report generators.
2019-08-22 10:41:48 +10:00