Francois Gouget : testbot/reporttest: Match the tests to the wtbsuite.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: a26e789fad7cc0486b6e967ad97b4113082aa24c
URL:    https://source.winehq.org/git/tools.git/?a=commit;h=a26e789fad7cc0486b6e967ad97b4113082aa24c

Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Mon Feb  8 15:21:56 2021 +0100

testbot/reporttest: Match the tests to the wtbsuite.

Where a test mirrors a wtbsuite test, add the latter's commit message as
a comment to make it simpler to compare the two.
Also tweak some comments to better describe the tests.

Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard at winehq.org>

---

 testbot/src/reporttest/report.template | 30 ++++++++++++++++++++++++------
 1 file changed, 24 insertions(+), 6 deletions(-)

diff --git a/testbot/src/reporttest/report.template b/testbot/src/reporttest/report.template
index c08b4d4..8da3d82 100644
--- a/testbot/src/reporttest/report.template
+++ b/testbot/src/reporttest/report.template
@@ -209,8 +209,11 @@ Dll info:
     xmllite=1.0.1018.0
 Test output:
 stub advapi32:cred
+----- WTBS Simple test patch
+----- Expected assessement: Success
 
 advapi32:crypt start dlls/advapi32/tests/crypt.c -
+----- WTBS Simple test failure
 ----- A test unit with one failed test
 ----- Expected assessement: 1 failure
 This also has a valid source file name which one can link to.
@@ -219,10 +222,11 @@ crypt.c:1: Test failed: Something wrong
 advapi32:crypt:031c done (1) in 0s
 
 advapi32:crypt_lmhash start dlls/advapi32/tests/crypt_lmhash.c -
+----- WTBS Simple diacritic failure
 ----- A test unit with two failed tests
------ Expected assessement: 2 failures
+----- Expected assessement: 2 failures despite exit code
 crypt_lmhash.c:10: Test failed: Something wrong
-crypt_lmhash.c:20: Test failed: Second failure here
+crypt_lmhash.c:20: Test failed: Second failure with added diacritics: ??????????
 07c0:crypt_lmhash: 95 tests executed (0 marked as todo, 2 failures), 0 skipped.
 advapi32:crypt_lmhash:07c0 done (1) in 0s
 
@@ -246,7 +250,8 @@ Note that this also applies to the todo and skip counts.
 advapi32:crypt_md5:087c done (3) in 0s
 
 advapi32:crypt_sha start dlls/advapi32/tests/crypt_sha.c -
------ A test unit with a failed subprocess test ignored by the parent
+----- A test unit with a failed subprocess test ignored by the parent test
+----- summary but accounted for in the exit code.
 ----- Expected assessement: 1 failure
 crypt_sha.c:20: Test failed: A failure here, presumably in the subprocess
 0456:crypt_sha: 1 tests executed (0 marked as todo, 1 failure), 0 skipped.
@@ -256,6 +261,7 @@ subprocess.
 advapi32:crypt_sha:08f8 done (3) in 0s
 
 advapi32:eventlog start dlls/advapi32/tests/eventlog.c -
+----- WTBS Unreported test failures
 ----- A test unit with an unreported failed test
 ----- Expected assessement: 1 failure, "Unreported failure" error
 eventlog.c:20: Test failed: Something wrong
@@ -263,6 +269,7 @@ eventlog.c:20: Test failed: Something wrong
 advapi32:eventlog:088c done (3) in 4s
 
 advapi32:lsa start dlls/advapi32/tests/lsa.c -
+----- WTBS Missing test summary line
 ----- A test unit with no test summary line
 ----- Expected assessement: "Missing test summary" error
 advapi32:lsa:083c done (0) in 0s
@@ -308,6 +315,7 @@ files.c:20: Test failed: A failure in the main process
 advpack:files:0880 done (3) in 0s
 
 advpack:install start dlls/advpack/tests/install.c -
+----- WTBS Simple test crash
 ----- A test unit with a crash
 ----- Expected assessement: Crash
 07b0:install: unhandled exception c0000005 at 0040167C
@@ -371,6 +379,8 @@ comcat:comcat start dlls/comcat/tests/comcat.c -
 comcat:comcat:0948 done (0) in 0s
 
 comctl32:animate start dlls/comctl32/tests/animate.c -
+----- WTBS Timeout in a test
+----- WTBS Timeout in a test - barely
 ----- A test unit with a timeout
 ----- Expected assessement: Timeout
 comctl32:animate:0b14 done (258) in 42s
@@ -380,7 +390,10 @@ comctl32:button skipped dlls/comctl32/tests/button.c -
 ----- Expected assessement: Skipped by user
 
 comctl32:comboex start dlls/comctl32/tests/comboex.c -
------ A test unit with failed, todo and skipped tests in the main process
+----- WTBS Simple success in a todo test
+----- WTBS Simple trace, ok, todo, skip sampler
+----- A test unit with a subprocess and failed, todo and skipped tests in the
+----- main process.
 ----- Expected assessement: 2 failures, 1 todo, 1 skip
 comboex.c:40: Test failed: A failure in the main process
 comboex.c:41: Test succeeded inside todo block: Success in the main process
@@ -394,6 +407,7 @@ comctl32:comboex:0c9c done (2) in 0s
 comctl32:datetime start dlls/comctl32/tests/datetime.c -
 ----- A test unit reporting a failure but with no test failed line
 ----- Expected assessement: "Missing test failed line" error
+----- + TestFailures=2 (1 for the exit code + 1 for the missing failure line)
 0998:datetime: 651 tests executed (0 marked as todo, 1 failures), 0 skipped.
 comctl32:datetime:0998 done (1) in 0s
 
@@ -425,8 +439,10 @@ comctl32:ipaddress start dlls/comctl32/tests/ipaddress.c -
 Issued upon seeing the next start line.
 0998:ipaddress: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
 
+----- Keep this stub in case the previous test is mishandled
 stub comctl32:listview
 
+----- WTBS A test unit with no start line
 ----- A test unit with a missing start line
 ----- Expected assessement: "Missing start line" error
 0998:misc: 16 tests executed (0 marked as todo, 0 failure), 0 skipped.
@@ -490,6 +506,7 @@ crashed process which makes sense since it did not crash.
 comctl32:subclass:07b0 done (1) in 0s
 
 comctl32:syslink start dlls/comctl32/tests/syslink.c -
+----- WTBS Garbled trace, ok, todo, skip sampler
 ----- A test unit with garbled test lines
 ----- Expected assessement: 2 failures, 1 todo, 1 skip
 Multi-threaded and multi-process tests should really try to avoid races in
@@ -518,13 +535,14 @@ comctl32:toolbar done (-1073741819) in 0s
 stub comctl32:tooltips
 
 comctl32:trackbar start dlls/comctl32/tests/trackbar.c -
+----- WTBS Test failures, todos and skips in a subtest
 ----- A test unit with a subtest
+----- Expected assessement: 2 failures, 1 todo, 1 skip
 trackbar.c:496: Subtest driver
 driver.c:40: Test failed: Something wrong
 driver.c:41: Test succeeded inside todo block: Something right!
 driver.c:42: Test marked todo: Hello Wine!
 driver.c:43: Tests skipped: Another time
------ Expected assessement: 2 failures, 1 todo, 1 skip
 07b1:trackbar: 3 tests executed (1 marked as todo, 2 failures), 1 skipped.
 comctl32:trackbar:07b1 done (2) in 0s
 
@@ -1011,7 +1029,7 @@ stub xcopy.exe:xcopy
 stub xmllite:reader
 
 xmllite:writer start dlls/xmllite/tests/writer.c -
------ A test unit with a misplaced test failed line
+----- A test unit with a misplaced (late) test failed line
 ----- Expected assessement: "Misplaced test failed line" error
 See the similar scenario before but note the lack of the extra test summary line
 here. Also beware of the end-of-file.