[...]
Since this is anti-cheat it is tricky, and things might not be what they seem.
When you say "it messes something up", are you sure the hook is working
correctly to begin with? If not, that needs to be fixed rather than handling
the fallout exception.
I don't know if it's working correctly, but that's a separate issue,
I'm looking into
both things.
Do you know if the recursive invocation happens on
Windows too? You mention
that the process is killed, so it doesn't sound like it is supposed to happen.
Yeah it's definitely not supposed to happen but Wine still could
handle it better
(hence this email).
Here is small example that will deadlock Wine in exactly same way
https://gist.github.com/davispuh/6c0ae7e5a10500a5b7a759b817e66214
In Windows such program will be killed and you'll get dialog with
button to attach
debugger.
Anti-cheat / Anti-debugger / DRM systems usually
don't tell you nicely if they
think something is wrong. When they think you are trying to mess with them
they usually pretend to proceed for a while, do something else and then at a
much later point deliberately crash the process in a weird way. So your
exception handler recursion might be the Anti-cheat's underhanded attempt to
kill the process.
True, but in this case it wasn't intended way, fixing LoadLibraryEx
non-NULL hFile
prevented stack overflow and deadlock
https://gitlab.winehq.org/wine/wine/-/merge_requests/4587
^ But I don't know if that was anti-debug/reversing trick or just
working around some
bigger issue.
Game still doesn't work but now behaviour is way better because it
doesn't deadlock
and 100% of time presents game's internal crash dialog (it's used for
all purposes,
including if debugger is detected).