HLSL Compiler and d3dcompiler_xx.dll

[Roderick Colenbrander]

On Wed, Apr 14, 2010 at 5:19 PM, Matijn Woudt <tijnema at gmail.com> wrote:
> On Wed, Apr 14, 2010 at 4:51 PM, Roderick Colenbrander
> <thunderbird2k at gmail.com> wrote:
>> On Wed, Apr 14, 2010 at 4:38 PM, Matijn Woudt <tijnema at gmail.com> wrote:
>>> On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger <stefandoesinger at gmx.at> wrote:
>>>>
>>>> Am 14.04.2010 um 15:44 schrieb Henri Verbeet:
>>>>
>>>>> On 14 April 2010 15:07, Stefan Dösinger <stefandoesinger at gmx.at> wrote:
>>>>>>> 3) Implement the compiler in d3dcompiler_xx.
>>>>>> I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them?
>>>>>>
>>>>> Quite frankly, I also believe that's where some of the issues in the
>>>>> early versions of those assembler patches came from. I don't know if
>>>>> that compiler has seen any work since 2008, but I'd be careful with
>>>>> taking it as too much of an example.
>>>> Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though.
>>>
>>> First thing would probably be cleaning up and getting something minimal past AJ.
>>>
>>>>
>>>> Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it.
>>>>
>>>
>>> Am I supposed to find your compiler code there? It seems like a git
>>> tree of ages ago (where d3dx9_36 only contains math and font code).
>>>
>>>
>>>
>>
>> It might also make sense to explore whether compilers like LLVM (used
>> a lot these days including for OpenCL implementations by Apple,
>> Nvidia; and don't forget Gallium3D) and Open64 (used in Cuda) and
>> others.
>>
>> Roderick
>>
>
> I have thought about using LLVM, but I don't like it because of:
> 1) Having another wine dependency (~40MB for me on ubuntu)

I agree that an extra dependency should be avoided if possible.
Remember that if we use some generic framework (if it makes sense)
they would do a lot of hard work for us (especially optimizations
later on). Further other projects might want to help as well (there is
some XNA project for mono which wants a HLSL but there are likely
others as well). The 40MB is not a big deal, disk space is really
cheap these days.

> 2) Code generated by LLVM will most likely not generated exactly the
> same bytecode as native compiler does, even though the shader has same
> effect on both. This makes it really hard to verify the HLSL compiler
> is working. My idea was to make wine's hlsl compiler produce exactly
> the same bytecode as native does. We might even find bugs in the
> native compiler, and either fix or duplicate these bugs.

I don't know much details about the HLSL compiler but I think this is
nearly impossible. I wouldn't be surprised if Microsoft has added more
and more tweaks to each new revision of the HLSL compiler, so creating
100% the same code is hard. I think we need very strict d3dx9 test
cases which don't necessarily check the bytecode (in simple cases you
can) but which check whether the shader when fed with data works
correctly.

> LLVM might be useful if we needed a really fast compiler, but native
> compiler isn't really fast AFAIK. Once games are calling it a few
> hundred times it might be worth the effort.

My main worry is that in the end if we use a compiler framework
(remember we don't have a lot of manpower for this) I think it would
reduce in more efficient shaders and it prevents some reinventing of
the wheel.

Roderick