Conclusion

Regardless of how you interpret NVIDIA's opinion about 3D Mark 2003, what remains is a lingering aftertaste. The total score of 3D Mark was already controversial with version 2001 and earlier, but in the individual tests, you could always reach some conclusions about a graphics card. In version 2003, however, Futuremark has reduced these tests further. What's missing, for example, are the Polygon Count and Light tests. Even the CPU scores are questionable, because the results depend on the graphics card. The Image Quality tests are only restricted to screen shots, instead of showing special scenes that could demonstrate advantages and disadvantages of a card's specific FSAA implementation.

It will be interesting to see how Futuremark reacts to the criticisms, especially with regard to Pixel Shader 1.4. At the NVIDIA Developer Event "Dawn Till Dusk" in London, I met a game developer (who requested anonymity) who had some interesting things to say. He explained to me that it's no problem for a graphics driver to recognize a special and well-known shader code, and then to replace it with a special, optimized code while the game or benchmark is executed. The software would not notice this little trick. Fixed (and well-known) pixel and vertex shader tests as used by 3D Mark therefore lose their meaning entirely. As a result, the best way to gauge the performance of a card is still to use a "real" game. Now, NVIDIA is of the same opinion:

So, where do you find a true gamers' benchmark? How about running actual games? Most popular games include a benchmark mode for just this purpose. Doom3, Unreal Tournament 2003, and Serious Sam Second Encounter are all far better indicators of current and upcoming game performance. And, because the vendors of these games have licensed their game engines to other game developers, you can expect that the next generation of games will have these game engines at their core. Today's consumers no longer have to rely on artificial attempts at mimicking games. In most cases, consumer's favorite games already have a ready-made benchmark built right in.

NVIDIA's sudden change of mind is baffling - up till now, 3D Mark was always a well regarded test among the manufacturers, including NVIDIA. What initially looked like an attempt to hide the FX's weaker test performance cannot be confirmed with the appearance of the new driver. Perhaps NVIDIA feared a dwindling influence on the development of 3DMark. Especially with the new "The way it's meant to be Played " program, NVIDIA tries to increase its influence on game developers. Of course, gamers with NVIDIA cards benefit from NVIDIA's involvement, as they get better graphics and performance in real games. Owners of non-NVIDIA cards might have something else to say though. At the NVIDIA Developer Forum, it became clear the Pixel Shader (2.0) code doesn't run equally well on all hardware. Ultimately, it all boils down to the fact that developers need to use special code for different chips in order to ensure good performance on all possible platforms. This explains the attention that NVIDIA devoted to developers. Is the main reason for NVIDIA's criticism at 3D Mark 2003 due to Futuremark's refusal to use optimized code? In this case, NVIDIA is right in saying that 3D Mark is unrealistic - for one thing, because it was developed without NVIDIA's input, and for another thing, because it's difficult to imagine that actual "neutral" code would be useful for performance evaluation.

Please follow-up by reading 3DMark 2003: Talking Back to NVIDIA .