Modifying Subsystem in Winelib executables

[Damjan Jovanovic]

On Tue, Sep 29, 2009 at 3:01 PM, Greg Geldorp <ggeldorp at vmware.com> wrote:
>> From: Paul Vriens [mailto:paul.vriens.wine at gmail.com]
>>
>> Can't we just compile shell32 tests as a GUI app?
>
> I don't think that will work, but I'll look into it. Wine tests expect
> to have access to stdout. At the very least you won't see output when
> running the test manually on Windows. Not sure if it would affect
> running from winetest.exe.
>
>> What other changes are needed to fix the errors on Vista+? (I know that
>> we have popups now)
>
> Yeah, there are basically two problems with shell32:shlexec on Vista+.
> First is that there appears to be no way to suppress the popups that you
> get when there's no registered handler for a filename extension. I've got
> that one more or less under control, by detecting this condition and
> compensating for it when it first occurs and skipping similar tests
> afterwards. I can submit a patch for that right now if it helps.
>
> The second problem is when using DDE. My guess is the ShellExecuteEx() code
> for that looks something like this (at least, that's how I would write
> it :-)):
>
> try_establish_dde_connection();
> if (no_connection)
> {
>  CreateProcess();
>  WaitForInputIdle();
>  retry_establish_dde_connection();
> }
>
> The child app is supposed to do something like this:
>
> WinMain()
> {
>  setup_dde_server();
>  while (GetMessage())
>  {
>    ...
>  }
> }
>
> The WaitForInputIdle() in the parent should wait for the child to setup its
> DDE
> server, so when retry_establish_dde_conection() is called the DDE server is
> up
> and running. Except that this doesn't work for console apps. A console app
> can
> of course still have a message loop, but WaitForInputIdle() returns
> immediately
> for console apps. That means if the standard DDE setup is used you'll have a
> race condition when the child app is a console app, there's a good chance
> the
> childs setup_dde_server() won't have finished by the time the parent calls
> its
> retry_establish_dde_connection() function. For the shlexec test, the child
> launched can really only be shell32_test.exe, which hits this problem
> because
> that is a console app.
>
> There's a pretty cool hack in shlexec.c to work around this, I wish I would
> have thought of it myself :-). The idea is that the parent sets up the DDE
> server. That DDE server will decline the first attempt to establish a DDE
> connection. That will cause ShellExecuteEx() to launch the child process, a
> second copy of shell32_test.exe that will only check for correct parameters
> but
> will not try to do any DDE related stuff. In the mean time, we still have
> the
> DDE server in the parent running and when the parent calls
> retry_establish_dde_connection() its own DDE server responds and accepts the
> connection. The DDE server in the parent then checks the DDE conversation.
> Problem solved, except it doesn't work anymore on Vista+...
>
> At first I thought maybe ShellExecuteEx() will refuse to talk to a DDE
> server
> in its own process, but now I don't believe that's the problem. I do see the
> DDE server being probed for the initial connection which it refuses. It's
> just
> that no second connect request is received by the DDE server. Spying on
> messages sent by the parent makes me think that in
> retry_establish_dde_connection() it will only send WM_DDE_INITIATE messages
> to
> new windows (windows which were not present during the initial
> try_establish_dde_connection() call). That kind of makes sense, if a window
> didn't want to establish a conversation on a specific app/topic before it
> probably doesn't want to have that conversation at all, so there's no use in
> asking again. It breaks our hack though.
>
> I'm still trying to figure out a solution for this. Would be nice if the
> child
> could simply be a GUI app so WaitForInputIdle() would work as intended, but
> I don't see a way to make that happen. Another idea is to hook
> WaitForInputIdle
> and don't return until the child has set an event, indicating its DDE server
> is
> up and running. And obviously I'm open to other ideas.
>

Can you not change the application type to GUI at runtime using
MSVCRT's __set_app_type() function? Or does that do something
different?

> Ge.
>
>
>
>

Damjan Jovanovic