[tools 2/2] testbot/reporttest: Match the terminology of the rest of the TestBot code.
Francois Gouget
fgouget at codeweavers.com
Fri Feb 5 09:07:24 CST 2021
Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
---
testbot/src/reporttest/report.template | 94 +++++++++++++-------------
testbot/src/reporttest/reporttest.c | 8 +--
2 files changed, 51 insertions(+), 51 deletions(-)
diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index eb9999e3e..b541e41d8 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -211,7 +211,7 @@ Test output:
stub advapi32:cred
advapi32:crypt start dlls/advapi32/tests/crypt.c -
------ A unit test with one failed test
+----- A test unit with one failed test
----- Expected assessement: 1 failure
This also has a valid source file name which one can link to.
crypt.c:1: Test failed: Something wrong
@@ -219,7 +219,7 @@ crypt.c:1: Test failed: Something wrong
advapi32:crypt:031c done (1) in 0s
advapi32:crypt_lmhash start dlls/advapi32/tests/crypt_lmhash.c -
------ A unit test with two failed tests
+----- A test unit with two failed tests
----- Expected assessement: 2 failures
crypt_lmhash.c:10: Test failed: Something wrong
crypt_lmhash.c:20: Test failed: Second failure here
@@ -227,14 +227,14 @@ crypt_lmhash.c:20: Test failed: Second failure here
advapi32:crypt_lmhash:07c0 done (1) in 0s
advapi32:crypt_md4 start dlls/advapi32/tests/crypt_md4.c -
------ A unit test with two test summary lines
+----- A test unit with two test summary lines
----- Expected assessement: Success
0123:crypt_md4: 11 tests executed (0 marked as todo, 0 failures), 0 skipped.
0ca0:crypt_md4: 11 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:crypt_md4:0ca0 done (0) in 0s
advapi32:crypt_md5 start dlls/advapi32/tests/crypt_md5.c -
------ A unit test with a failed subprocess test repeated in the parent
+----- A test unit with a failed subprocess test repeated in the parent
----- Expected assessement: 1 failure (also acceptable: 2 failures)
crypt_md5.c:20: Test failed: A failure here, presumably in the subprocess
0123:crypt_md5: 11 tests executed (0 marked as todo, 1 failure), 0 skipped.
@@ -246,35 +246,35 @@ Note that this also applies to the todo and skip counts.
advapi32:crypt_md5:087c done (3) in 0s
advapi32:crypt_sha start dlls/advapi32/tests/crypt_sha.c -
------ A unit test with a failed subprocess test ignored by the parent
+----- A test unit with a failed subprocess test ignored by the parent
----- Expected assessement: 1 failure
crypt_sha.c:20: Test failed: A failure here, presumably in the subprocess
0456:crypt_sha: 1 tests executed (0 marked as todo, 1 failure), 0 skipped.
-The unit test counts on the framework to pick up the failure reported by the
+The test unit counts on the framework to pick up the failure reported by the
subprocess.
08f8:crypt_sha: 1 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:crypt_sha:08f8 done (3) in 0s
advapi32:eventlog start dlls/advapi32/tests/eventlog.c -
------ A unit test with an unreported failed test
+----- A test unit with an unreported failed test
----- Expected assessement: 1 failure, "Unreported failure" error
eventlog.c:20: Test failed: Something wrong
088c:eventlog: 408 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:eventlog:088c done (3) in 4s
advapi32:lsa start dlls/advapi32/tests/lsa.c -
------ A unit test with no test summary line
+----- A test unit with no test summary line
----- Expected assessement: "Missing test summary" error
advapi32:lsa:083c done (0) in 0s
advapi32:registry start dlls/advapi32/tests/registry.c -
------ A unit test with no main process test summary line
+----- A test unit with no main process test summary line
----- Expected assessement: "Missing test summary" error
1234:registry: 765 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:registry:0804 done (0) in 0s
advapi32:security start dlls/advapi32/tests/security.c -
------ A unit test with a misplaced but successful subprocess test summary line
+----- A test unit with a misplaced but successful subprocess test summary line
----- Expected assessement: Success
The subprocess may be a server process that exits automatically once the test
is complete. That's ok as long as it's not meant to report failures.
@@ -283,24 +283,24 @@ advapi32:security:0148 done (0) in 9s
0954:security: 18 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:service start dlls/advapi32/tests/service.c -
------ A unit test with a misplaced and failed subprocess test summary line
+----- A test unit with a misplaced and failed subprocess test summary line
----- Expected assessement: "Misplaced test summary line" error
Same scenario as above but the subprocess can actually report failures so the
main process must wait for it. Ideally the error should be reported against
-this unit test but reporting against the next one is also acceptable.
+this test unit but reporting against the next one is also acceptable.
037c:service: 1930 tests executed (0 marked as todo, 0 failures), 0 skipped.
advapi32:service:037c done (0) in 11s
service.c:20: Test failed: A failure in the subprocess
4567:service: 10 tests executed (0 marked as todo, 1 failure), 0 skipped.
advpack:advpack start dlls/advpack/tests/advpack.c -
------ A standard successful unit test
+----- A standard successful test unit
----- Expected assessement: Success, unless the previous test interferes
0da0:advpack: 99 tests executed (0 marked as todo, 0 failures), 0 skipped.
advpack:advpack:0da0 done (0) in 0s
advpack:files start dlls/advpack/tests/files.c -
------ A unit test with a failed test in the main process before a subprocess
+----- A test unit with a failed test in the main process before a subprocess
----- Expected assessement: 1 failure
files.c:20: Test failed: A failure in the main process
4321:files: 99 tests executed (0 marked as todo, 0 failures), 0 skipped.
@@ -308,7 +308,7 @@ files.c:20: Test failed: A failure in the main process
advpack:files:0880 done (3) in 0s
advpack:install start dlls/advpack/tests/install.c -
------ A unit test with a crash
+----- A test unit with a crash
----- Expected assessement: Crash
07b0:install: unhandled exception c0000005 at 0040167C
advpack:install:07b0 done (-1073741819) in 0s
@@ -320,7 +320,7 @@ amstream: 152 tests executed (0 marked as todo, 0 failures), 0 skipped.
amstream:amstream done (0) in 0s
apphelp:apphelp start dlls/apphelp/tests/apphelp.c -
------ A unit test with no execution time
+----- A test unit with no execution time
----- Expected assessement: Success
0860:apphelp: 15 tests executed (0 marked as todo, 0 failures), 0 skipped.
apphelp:apphelp:0860 done (0)
@@ -338,7 +338,7 @@ module: 544 tests executed (0 marked as todo, 0 failures), 0 skipped.
atl:module:0920 done (0) in 0s
atl:registrar start dlls/atl/tests/registrar.c -
------ A unit test with a foreign test summary line
+----- A test unit with a foreign test summary line
----- Expected assessement: "Misplaced test summary line" error
9876:alien: 10 tests executed (0 marked as todo, 1 failures), 0 skipped.
0922:registrar: 17 tests executed (0 marked as todo, 0 failures), 0 skipped.
@@ -350,47 +350,47 @@ avifil32 is missing and thus no test will be run for it and no entry will be
found in the test report.
browseui:autocomplete start dlls/browseui/tests/autocomplete.c -
------ A unit test with a skipped test
+----- A test unit with a skipped test
----- Expected assessement: Success, 1 skipped
autocomplete.c:42: Tests skipped: Another time
052c:autocomplete: 66 tests executed (0 marked as todo, 0 failures), 1 skipped.
browseui:autocomplete:052c done (0) in 0s
browseui:progressdlg start dlls/browseui/tests/progressdlg.c -
------ A unit test with an unreported skipped test
+----- A test unit with an unreported skipped test
----- Expected assessement: 1 skipped, "Unreported skip" error
progressdlg.c:42: Tests skipped: Another time
0680:progressdlg: 4 tests executed (0 marked as todo, 0 failures), 0 skipped.
browseui:progressdlg:0680 done (0) in 0s
cabinet:extract start dlls/cabinet/tests/extract.c -
------ A unit test with a missing skipped test line
+----- A test unit with a missing skipped test line
----- Expected assessement: "Missing tests skipped line" error
0fd4:extract: 347 tests executed (0 marked as todo, 0 failures), 1 skipped.
cabinet:extract:0fd4 done (0) in 0s
cabinet:fdi start dlls/cabinet/tests/fdi.c -
------ A unit test with a todo test
+----- A test unit with a todo test
----- Expected assessement: Success, 1 todo
fdi.c:42: Test marked todo: Hello Wine!
0690:fdi: 265 tests executed (1 marked as todo, 0 failures), 0 skipped.
cabinet:fdi:0690 done (0) in 0s
cmd.exe:batch start programs/cmd/tests/batch.c -
------ A unit test with an unreported todo test
+----- A test unit with an unreported todo test
----- Expected assessement: "Unreported todo line" error
batch.c:42: Test marked todo: Hello Wine!
0ddc:batch: 1701 tests executed (0 marked as todo, 0 failures), 0 skipped.
cmd.exe:batch:0ddc done (0) in 3s
comcat:comcat start dlls/comcat/tests/comcat.c -
------ A unit test with a missing todo test line
+----- A test unit with a missing todo test line
----- Expected assessement: "Missing todo line" error
0948:comcat: 9 tests executed (1 marked as todo, 0 failures), 0 skipped.
comcat:comcat:0948 done (0) in 0s
comctl32:animate start dlls/comctl32/tests/animate.c -
------ A unit test with a timeout
+----- A test unit with a timeout
----- Expected assessement: Timeout
comctl32:animate:0b14 done (258) in 42s
@@ -399,7 +399,7 @@ comctl32:button skipped dlls/comctl32/tests/button.c -
----- Expected assessement: Skipped by user
comctl32:comboex start dlls/comctl32/tests/comboex.c -
------ A unit test with failed, todo and skipped tests in the main process
+----- A test unit with failed, todo and skipped tests in the main process
----- Expected assessement: 2 failures, 1 todo, 1 skip
comboex.c:40: Test failed: A failure in the main process
comboex.c:41: Test succeeded inside todo block: Success in the main process
@@ -411,20 +411,20 @@ This line says nothing is wrong because it's from the subprocess
comctl32:comboex:0c9c done (2) in 0s
comctl32:datetime start dlls/comctl32/tests/datetime.c -
------ A unit test reporting a failure but with no test failed line
+----- A test unit reporting a failure but with no test failed line
----- Expected assessement: "Missing test failed line" error
0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped.
comctl32:datetime:0998 done (1) in 0s
comctl32:dpa start dlls/comctl32/tests/dpa.c -
------ A unit test with a non-zero exit code
+----- A test unit with a non-zero exit code
----- Expected assessement: "Non-zero exit code" error
-This would cause things like 'make test' to consider the unit test failed.
+This would cause things like 'make test' to consider the test unit failed.
0998:dpa: 16 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:dpa:0998 done (1) in 0s
comctl32:header start dlls/comctl32/tests/header.c -
------ A unit test with a zero exit code despite failures
+----- A test unit with a zero exit code despite failures
----- Expected assessement: 1 failure, "Zero exit code" error
A non-zero exit code is necessary for 'make test'.
header.c:1: Test failed: Something wrong
@@ -432,27 +432,27 @@ header.c:1: Test failed: Something wrong
comctl32:header:0998 done (0) in 0s
comctl32:imagelist start - -
------ A unit test with no source filename
+----- A test unit with no source filename
----- Expected assessement: Success
TestLauncher does not set the source filename field.
0998:imagelist: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
comctl32:imagelist:0998 done (0) in 0s
comctl32:ipaddress start dlls/comctl32/tests/ipaddress.c -
------ A unit test with a missing done line
+----- A test unit with a missing done line
----- Expected assessement: "Missing done line" error
Issued upon seeing the next start line.
0998:ipaddress: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
stub comctl32:listview
------ A unit test with a missing start line
+----- A test unit with a missing start line
----- Expected assessement: "Missing start line" error
0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
comctl32:misc:0998 done (0) in 0s
comctl32:monthcal start dlls/comctl32/tests/monthcal.c -
------ A unit test with a misplaced todo subprocess test summary line
+----- A test unit with a misplaced todo subprocess test summary line
----- Expected assessement: 1 todo, "Misplaced summary line" error
See the previous misplaced test summary line scenarios.
0148:monthcal: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
@@ -461,13 +461,13 @@ security.c:41: Test marked todo: Hello Wine!
0955:security: 18 tests executed (1 marked as todo, 0 failures), 0 skipped.
comctl32:mru start dlls/comctl32/tests/mru.c -
------ A standard successful unit test
+----- A standard successful test unit
----- Expected assessement: Success, unless the previous test interferes
0148:mru: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:mru:0148 done (0) in 9s
comctl32:pager start dlls/comctl32/tests/pager.c -
------ A unit test with a misplaced subprocess skip test summary line
+----- A test unit with a misplaced subprocess skip test summary line
----- Expected assessement: Success
See the previous misplaced test summary line scenarios.
0148:pager: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
@@ -476,18 +476,18 @@ security.c:42: Tests skipped: Another time
0956:security: 18 tests executed (0 marked as todo, 0 failures), 1 skipped.
comctl32:progress start dlls/comctl32/tests/progress.c -
------ A standard successful unit test
+----- A standard successful test unit
----- Expected assessement: Success
0148:progress: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:progress:0148 done (0) in 9s
comctl32:propsheet start dlls/comctl32/tests/propsheet.c -
------ A unit test with no main process test summary line and no pid anywhere
+----- A test unit with no main process test summary line and no pid anywhere
----- Expected assessement: "Missing test summary" error
comctl32:propsheet done (0) in 9s
comctl32:rebar start dlls/comctl32/tests/rebar.c -
------ A unit test with an unreported crash in a subprocess
+----- A test unit with an unreported crash in a subprocess
----- Expected assessement: "Zero exit code" error
Even in subprocesses crashes are equivalent to 'Test failed' errors and as
such the main process cannot report success, i.e. it must have a non-zero exit
@@ -500,7 +500,7 @@ subprocess crashes either. This is why the error is "Zero exit code" and not
comctl32:rebar:07b0 done (0) in 0s
comctl32:status start dlls/comctl32/tests/status.c -
------ A unit test with a subprocess crash but no pid
+----- A test unit with a subprocess crash but no pid
----- Expected assessement: "Zero exit code" error
See the comctl32:rebar comment.
status: unhandled exception c0000005 at 0040167C
@@ -508,7 +508,7 @@ status: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:status:07b0 done (0) in 0s
comctl32:subclass start dlls/comctl32/tests/subclass.c -
------ A unit test with a subprocess crash, no pid and a non-zero exit code
+----- A test unit with a subprocess crash, no pid and a non-zero exit code
----- Expected assessement: 1 failure
See the comctl32:rebar comment. Then note that here the main process does not
claim success. Also note that the exit code does not correspond to that of a
@@ -518,7 +518,7 @@ subclass: 2251 tests executed (0 marked as todo, 0 failures), 0 skipped.
comctl32:subclass:07b0 done (1) in 0s
comctl32:syslink start dlls/comctl32/tests/syslink.c -
------ A unit test with garbled test lines
+----- A test unit with garbled test lines
----- Expected assessement: 2 failures, 1 todo, 1 skip
Multi-threaded and multi-process tests should really try to avoid races in
the report. But analysis tools should also try to be resilient.
@@ -535,30 +535,30 @@ me off
comctl32:syslink:07b0 done (2) in 0s
comctl32:tab start dlls/comctl32/tests/tab.c -
------ A unit test crash with no pid
+----- A test unit crash with no pid
----- Expected assessement: crashed
tab: unhandled exception c0000005 at 0040167C
comctl32:tab done (-1073741819) in 0s
comctl32:taskdialog start dlls/comctl32/tests/taskdialog.c -
------ A unit test crash with unsigned exit code (TestLauncher)
+----- A test unit crash with unsigned exit code (TestLauncher)
----- Expected assessement: crashed
taskdialog: unhandled exception c0000005 at 0040167C
comctl32:taskdialog done (3221225477) in 0s
comctl32:toolbar start dlls/comctl32/tests/toolbar.c -
------ A unit test crash with no unhandled exception line
+----- A test unit crash with no unhandled exception line
----- Expected assessement: crashed
comctl32:toolbar done (-1073741819) in 0s
comctl32:tooltips start dlls/comctl32/tests/toolbar.c -
------ A unit test crash with no unhandled exception line
+----- A test unit crash with no unhandled exception line
----- and an unsigned exit code.
----- Expected assessement: crashed
comctl32:tooltips done (3221225477) in 0s
comctl32:trackbar start dlls/comctl32/tests/trackbar.c -
------ A unit test with a subtest
+----- A test unit with a subtest
trackbar.c:496: Subtest driver
driver.c:40: Test failed: Something wrong
driver.c:41: Test succeeded inside todo block: Something right!
@@ -1051,7 +1051,7 @@ stub xcopy.exe:xcopy
stub xmllite:reader
xmllite:writer start dlls/xmllite/tests/writer.c -
------ A unit test with a misplaced test failed line
+----- A test unit with a misplaced test failed line
----- Expected assessement: "Misplaced test failed line" error
See the similar scenario before but note the lack of the extra test summary line
here. Also beware of the end-of-file.
diff --git a/testbot/src/reporttest/reporttest.c b/testbot/src/reporttest/reporttest.c
index b2e7a78bf..f6ad9b315 100644
--- a/testbot/src/reporttest/reporttest.c
+++ b/testbot/src/reporttest/reporttest.c
@@ -75,8 +75,8 @@ int main(int argc, char** argv)
{
fprintf(stderr, "error: Outputting the lines of a specific test (%s) is not supported.\n", argv[i]);
/* This would require adjusting the report template to use the real
- * process pid. Plus some tests imply mixing the output of unit
- * tests which would not make sense.
+ * process pid. Plus some tests imply mixing the output of test
+ * units which would not make sense.
*/
return 2;
}
@@ -232,7 +232,7 @@ int main(int argc, char** argv)
char* unit = strchr(line, ':');
if (!unit)
{
- fprintf(stderr, "error: The line below does not have a unit!\n");
+ fprintf(stderr, "error: The line below does not have a test unit!\n");
fputs(line, stdout);
return 1;
}
@@ -240,7 +240,7 @@ int main(int argc, char** argv)
unit++;
unit[strlen(unit)-1] = '\0';
fprintf(logfile, "%s:%s start fake/%s/%s.c -\n", line+5, unit, line+5, unit);
- fprintf(logfile, "----- A standard successful unit test\n");
+ fprintf(logfile, "----- A standard successful test unit\n");
fprintf(logfile, "----- Expected assessment: Success\n");
fprintf(logfile, "1234:%s: 2 tests executed (0 marked as todo, 0 failures), 0 skipped.\n", unit);
fprintf(logfile, "%s:%s:1234 done (0) in 0s\n", line+5, unit);
--
2.20.1
More information about the wine-devel
mailing list