The first thing to test for is the documented behavior of APIs
and such as CreateFile. For instance one can create a file using a
long pathname, check that the behavior is correct when the file
already exists, try to open the file using the corresponding short
pathname, convert the filename to Unicode and try to open it using
CreateFileW, and all other things which are documented and that
applications rely on.
While the testing framework is not specifically geared towards this
type of tests, it is also possible to test the behavior of Windows
messages. To do so, create a window, preferably a hidden one so that
it does not steal the focus when running the tests, and send messages
to that window or to controls in that window. Then, in the message
procedure, check that you receive the expected messages and with the
correct parameters.
For instance you could create an edit control and use WM_SETTEXT to
set its contents, possibly check length restrictions, and verify the
results using WM_GETTEXT. Similarly one could create a listbox and
check the effect of LB_DELETESTRING on the list's number of items,
selected items list, highlighted item, etc. For concrete examples,
see dlls/user/tests/win.c and the related tests.
However, undocumented behavior should not be tested for unless there
is an application that relies on this behavior, and in that case the
test should mention that application, or unless one can strongly
expect applications to rely on this behavior, typically APIs that
return the required buffer size when the buffer pointer is NULL.