[PATCH] mscoree: Implement VTable fixup for x86_64 architecture.

[Paul Gofman]

On 01/21/2016 12:13 AM, Vincent Povirk wrote:
>> I have progressed a bit in attempts to load mixed code DLL into 64-bit
>> process. To do this I built wine-mono pure assembly dlls (mscorlib and
>> others) with -platform:x64 flag which provides x86_64 pure assembly dlls
>> and thus allows to workaround DLL load issue.
> Interesting. Do you have a patch to Mono that does this?
>
I just hot fixed it in a makefile after standard configuration & first
time build.
File is mono/mcs/build/library.make, line 283 (recipe for $(build_lib):
$(LIBRARY_COMPILE) $(LIBRARY_FLAGS) $(LIB_MCS_FLAGS)
$(MCS_FLAGS_RESOURCE_STRINGS) -target:library -platform:x64 -debug+
-out:$@ $(BUILT_SOURCES_cmdline) @$(response)
I inserted -platform:x64 to the original line.
I also commented out next line '$(Q) $(SN) -R $@ $(LIBRARY_SNK)' as
assembly signing did not work for me for x64 arch dlls. I tested the
unsigned DLLs with win32 setup, it worked OK so I suppose this is not a
problem for testing.

After that 'make clean; make' in mono/mcs/class does the job without
rebuilding the whole thing. Then I just copied the dlls to mono dir in
wineprefix.


>> In this state ReallyFixupVTable is called from x86_64 thunk, where
>> mono_marshal_get_vtfixup_ftnptr() starts executing 64-bit mixed code
>> DLL's assembly initialization, but currently fails with my DLL on
>> "invalid IL instruction" somewhere around System.IntPtr.ctor call (the
>> same class works on i686), which is probably some mono issue. So I
>> cannot come up with a fully working test case for now, though I did not
>> give up yet.
> I can try to build a 64-bit managed C++ dll in visual studio and call
> it from a C program.
I can send a simple test program which I am using now if it helps. It is
a bit weird though as using MFC in the main app. And I still have a
problem running it in wine64 ('System.InvalidProgramException: Invalid
IL code in <Module>:gcroot<System::String ^>.= (gcroot<System::String
^>*,string): IL_0004: call      0x0a000012' on calling
IntPtr.ctor(void*)) which I am trying to fix. So maybe a simpler one
could help. Unfortunately I don't have MSVC and cannot build mixed
assemblies myself.

> I guess it's not important then. If you're touching those lines
> anyway, I guess it makes sense to fix it. But more importantly, I
> don't think we should commit this until we can test it somehow. 
Yes, sure