fog with pixel shaders < 3.0

[Fabian Bieler]

I investigated the issue a bit further. Henri was right, the solution is much 
simpler than I initially thought. :)

On Monday 12 March 2007 12:56, you wrote:
> Am Montag 12 März 2007 10:35 schrieb Fabian Bieler:
>
> With pixel Shaders < SM 3.0, the fixed function fog calculation is still
> performed in Direct3D, but not in opengl? So we have to add a fog emulation
> to our pixel shaders?
That's how I see the problem.
>
> One thing to consider is that there are 2 different fog types in d3d: Fog
> Calculated per Vertex, and per Fragment fog(or something like this). In gl
> the difference is just the fog niceness hint(GL_NICEST, GL_FASTEST), but in
> d3d they seem to be handled very differently with vertex shaders and
> transformed vertices. And last but not least there is the No-Fog[1] fog
> which has both vertex and table fog set to NONE but is still quite foggy.
As I see it, the No-Fog fog is a vertex fog where the fog weighting parameter 
for every vertex is calculated by the application on the cpu rather than by 
geometry pipeline.

Anyhow from MSDN:
"When using a vertex shader, you must use vertex fog. This is accomplished by 
using the vertex shader to write the per-vertex fog intensity to the oFog 
register. After the pixel shader completes, the oFog data is used to linearly 
interpolate with the fog color. This intensity is not available in a pixel 
shader."

This is what we have to do, imo: Take the value from the oFog register, and 
store it in gl_FogFragCoord (which is currently already done). At the end of 
the fragment shader, we have to clamp gl_FogFragCoord to [0,1] and used it to 
mix the fragment color with the fog color, if fog is enabled.
An Idea would be to use shader constants as the clamping parameters and to 
clamp gl_FogFragCoord to [1,1] if fogging is disabled and [0,1] if fogging is 
enabled.

On a sidenote: MSDN also states:
"An application can implement fog with a vertex shader, and pixel fog 
simultaneously if desired."
However, I think this is rather a corner case. Which application would want to 
use two fogs simultaneously?

> > P.S: I noticed that fog-state management is not yet included in the state
> > table. Should a patch that fixes fog with pixel shaders wait until this
> > is done, or is this of no importance?
>
> No, it should be there :-) See STATE_RENDER(WINED3DRS_FOGENABLE), and
> states linked to it. Or did I miss anything? The apply handler should be
> state_fog.
>
sorry, I misunderstood a comment in the code.