wined3d: Use higher precision to convert floats into text.

[Michael Karcher]

Am Montag, den 26.05.2008, 19:00 -0600 schrieb Vitaliy Margolen:
> Chris Robinson wrote:
> > On Monday 26 May 2008 12:37:04 pm Vitaliy Margolen wrote:
> >> -            shader_addline(buffer, "PARAM srgb_mul_low = {%f, %f, %f,
> >> 1.0};\n",
> >> +            shader_addline(buffer, "PARAM srgb_mul_low = {%.12f, 
> >> %.12f, %.12f, 1.0};\n",
> > 
> > I don't know if ARB shaders would except the notation (GLSL does), but 
> > would %.8e or somesuch work better than %.12f at maintaining accuracy?
> I thought they were clamped? In either case .12 should be enough precision 
> to avoid problems.

There are two problems with %f that %e readily avoids. Only one of them
is irrelevant for clamped values (I take your word for that. I am not
experienced in 3D shader languages at all, but I recall that when I
skimmed over some documentation yesterday, it seemed to mention a range
of -63..+63.)

The first problem of %f is unpredictable large strings:
  char buffer[50];
  sprintf(buffer,"%f",1e100);
is a nice way of producing a buffer overflow. Now imagine the number
coming from a remote system. "%40f" does not help. It just sets a
minimum length, not a maximum length, so it will still overflow. Unlike
strings, "%.40f" won't help either, as it just causes 40 digits after
the decimal dot/comma to be printed, but does not limit what is printed
before. This problem is irrelevant for clamped values.

The second problem of %f is unpredicted loss of precision:
  printf("%.6f %.5f\n",1.467e-6,1.467e-6);
will print "0.000001 0.00000", so the first value is of by 46%, the
second one is lost completely. As I do not know whether such small
values are used in Direct3D (but having problems fixed by "%.12f"
strongly indicates it!), I am not sure whether this point applies here.

Now consider the two examples with "%.8e":
In the first case, the buffer will contain
  "1.00000000e+100"
which fits nicely into the 50 allocated characters. In the second case,
lets assume a format string of "%.6e %.2e", the output will be
 "1.467000e-6 1.46e-6"
In the second case, precision is still lost, but as one can be sure that
the error is smaller than half a percent (extreme case is 1.0049).

In this case (passing numbers as text between two programs, not intended
for the user), I would definitely plead for using "%.8e" which should
even enable round-trips for every 32 bit float pattern. On presenting
numbers two the user, %g might be worth considering. %f is IMHO
inappropriate if you are not very confident that the number you want to
print is in a sensible order of magnitude. (Using "%.3f" is OK, even if
the value might be very near to zero, if you just care whether it is
less then 0.0005, and smaller values are definitely rounding errors. No
one likes to see 2.4 - 10*0.24=1.6e-16 or something like that. On the
other hand 2.4e-16 - 9*0.24e-16 should usually not print as zero)

Regards,
  Michael Karcher