The first thing to test for is the documented behavior of APIs
and such as
CreateFile. For instance one can create a file using a
long pathname, check that the behavior is correct when the file
already exists, try to open the file using the corresponding short
pathname, convert the filename to Unicode and try to open it using
CreateFileW, and all other things which are documented and that
applications rely on.
While the testing framework is not specifically geared towards this type of tests, it is also possible to test the behavior of Windows messages. To do so, create a window, preferably a hidden one so that it does not steal the focus when running the tests, and send messages to that window or to controls in that window. Then, in the message procedure, check that you receive the expected messages and with the correct parameters.
For instance you could create an edit control and use WM_SETTEXT to set its contents, possibly check length restrictions, and verify the results using WM_GETTEXT. Similarly one could create a listbox and check the effect of LB_DELETESTRING on the number of items of the list, selected items list, highlighted item, etc. For concrete examples, see dlls/user32/tests/win.c and the related tests.
However, undocumented behavior should not be tested for unless there is an application that relies on this behavior, and in that case the test should mention that application, or unless one can strongly expect applications to rely on this behavior, typically APIs that return the required buffer size when the buffer pointer is NULL.