What to do when an application misuses an API ...
Michael Casadevall
mcasadevall at ubuntu.com
Wed Jul 24 22:33:25 CDT 2013
On Wed, Jul 24, 2013 at 1:22 AM, Damjan Jovanovic <damjan.jov at gmail.com
<mailto:damjan.jov at gmail.com>> wrote:
On Wed, Jul 24, 2013 at 4:17 AM, Michael Casadevall
<mcasadevall at ubuntu.com <mailto:mcasadevall at ubuntu.com>> wrote:
> Hey all,
> I've been working on trying to get Sid Meier's Civilization V
(Civ5) up to a
> point where it works relatively flawlessly on WINE. The largest
issue is
> that installing the latest expansion pack has completely hosed
the game
> under WINE unless some rather ugly workarounds have been taken.
> Specifically, the main issue is that Civ5 loads its data files in
the order
> returned by Find{First,Next}File(), and breaks spectacularly if
that order
> is disrupted. I documented my findings here:
> http://bugs.winehq.org/show_bug.cgi?id=34122
>
> As I sat to look at a possible patch, I'm not sure what I can
actually do.
> Its clearly stated on the MSDN page that the order returned is
dependent on
> the underlying filesystem, and Googling suggests that while NTFS
seems to
> always return in a roughly alphabetical order, FAT32 will return
by creation
> date, and with network drives all bets are off. I was even able
to reproduce
> similar bugs in Civ5 by launching it from a network drive (which
I had
> already tested in advance to make sure it returned an odd file
order).
> FindFirstFile essentially calls down to NtQueryDirectoryFile,
which in turn
> either calls a system specific syscall or readdir to get a list
of files in
> a directory.
>
> My first thought is to extend the underlying read_directory_*
functions in
> ntdll/directory.c so that they return an alphabetized list
(essentially
> taking the results from getdents/getdirentries/readdir, sort
them, then
> returning in the sort order whichever entry is next in line).
This would
> produce a behavior similar (but not identical) to what is seen when
> operating under an NTFS filesystem.
>
> This approach seems wrong to me, and is further compounded by the
fact this
> is a bug in an application, and not in WINE specifically; Under
the correct
> circumstances, Windows will return a "non-sorted" list of files
and in those
> cases Civilization V breaks. The flipside is that in the most
common case,
> it works correctly, and furthermore, given how easy it is to
assume that
> Find*File does return a sorted list, I won't be surprised if this
issue has
> cropped up elsewhere (figuring out the underlying cause was
exceedingly
> annoying). Any insight in to what the most correct course of
action would be
> most appreciated
>
> I'm perfectly willing to put the legwork in fixing this bug, but
I want to
> make sure I'm fixing it in a way that will be accepted :-).
>
> Regards,
> Michael
>
The other possibility to consider is a special FUSE-based filesystem
that orders filenames just like Windows does. This filesystem could
cache names and perform much better than reading all the directory
contents on each FindFirstFile, just like the
http://www.brain-dump.org/projects/ciopfs/ provides a performance
benefit for case-insensitive filename matching.
Regards
Damjan
(warning, long reply with ASCII art)
First, thanks you all for your replies.
After these emails, and a long discussion in #winehackers, I grabbed
ciopfs's source, thumbed through FUSE's documentation, and tweaked it to
return files in the order, and then ensuring that ciopfs passed its test
suite and was more or less stable. This was then followed by making a
new prefix, installing Steam, and attempting to install Civilization V
over it, just to be stied by the fact that ciopfs introduces subtle bugs
to running applications. As a test, I built and ran coreutils in a
normal ciopfs overlayed directory, and found it failed a large chunk of
its test suite (the same
exact code worked properly when copied into a normal directory).
Furthermore, after sitting down and playing with FUSE for awhile, I'm
convinced that its absolutely the wrong way to handle this sort of
problem. By definition, anything running through FUSE is going to have a
massive performance penalty due to the additional calls from
kernelspace-userpsace.
For instance, from a syscall perspective, here's what a normal
FindFirstFile() looks like
+----------------+
| FindFirstFile()|
+-------^--------+
|
+----------v-----------+
|NtQueryDirectoryFile()|
+----------^-----------+
|
|
|<------------------+Crossing into kernelspace
|
+--------v-----+
|getdirents64()|
+--------^-----+
|
+--------v-------+
|Kernel VFS Layer|
+--------^-------+
|
+-----v-----+
|ext4 driver|
+-----------+
(in case this gets screwed up by your mail client:
http://paste.ubuntu.com/5909570/)
Meanwhile, here's what a simplified call with a FUSE filesystem looks like:
+----------------+
| FindFirstFile()|
+-------^--------+
|
+----------v-----------+ +--------------+
|NtQueryDirectoryFile()| | ciopfs |
+----------^-----------+ +--------------+
| ^ ^
| | |
|<---Crossing-into-kernelspace+---------+------>|
| | |
+--------v-----+ | |
|getdirents64()| | |
+--------+-----+ | |
| | |
+--------v-------+ | +-v-+
|Kernel VFS Layer| | |VFS|
+--------^-------+ | +-^-+
| | |
| | +--v-+
+-----v-----+ | |ext4|
|FUSE Module|<--------------------------------+ +----+
+-----------+
(http://paste.ubuntu.com/5909572/)
Any sort of FUSE module that acts as an overlay for an existing
filesystem is looking at a minimum of six roundtrips across
kernel/userspace boundaries vs. just the two of a normal filesystem
call. Furthermore, from a user perspective, this makes life difficult,
as they either must setup FUSE environment, place their prefix on it,
make sure FUSE is running when trying to launch a WINE app, etc.
Alternatively, WINE would need to grow code to handle this, and now
another large chunk of code where things can (and likely will) go wrong,
with no obvious answer on why. The more I look at it, the more I see the
path to madness. While I don't doubt its possible to ciopfs (or some
other FUSE filesystem) to work 100%, the performance penalties are
extremely high.
The flipside of the coin is I can also see the problem on why hitting
Find*Files() with a patch to reorder things is undesirable.
So, I decided to step back and look at this problem from a new perspective.
I think we're mostly agreed here that the app here is just getting
lucky, and is in this case fundamentally broken. I was thumbing through
my copy of "The New Old Thing" (an excellent read), and thinking what
would be done In the Windows world, assuming that some years down the
line, Windows changed something under the hood that broke app compatibility.
In the WIndows world, there are three things that can generally be done
when app compatilbity is broken: change the core, write a shim, or patch
the application.
As it exists in WINE today, we can only change our DLLs; patching
applications on the fly is a legal quagmire and is fragile. Thus when
this ugly case of when an application is broken, we have no good way to
correct it. Thus I propose extending WINE
During last nights discussion, some work has been done on implementing
apphelp.dll which contains the shim database functionality. Said author
had no intend on implementing actual shimming in WINE, but as time goes
on, I suspect there will be more and more cases where shims would be a
more appropriate fix than fixing WINE's core libraries.
While I do have an interest in further exploring the possibilities of
implementing shimming in WINE, it will likely be a few weeks before I
could dedicate the proper time to do so. As such, in the near future,
I'll cook up a patch with what I learned hacking on FUSE to reorder the
Find*File() results, and post it to the bug, with the hope that it will
become the basis of a shim in the near future.
Michael
(and as an aside, I will see if I can reproduce the bug on a FAT32
install of Windows XP and if so, see if I can get the game developer to
fix it)
More information about the wine-devel
mailing list