Well, looks like I kicked up a bit of a stink with my last column , which analyzed NVIDIA's criticism of Futuremark's new 3DMark 2003. I seem to have started a few discussions as well. Well, good - that was my intention!

Meanwhile, Futuremark, the developer of the software in question, has responded to NVIDIA's harsh attack on its benchmark. In a PDF document published on its website, Futuremark addresses NVIDIA's accusations and explains its own view of the situation in a kind of rebuttal. Lastly, ATi has also issued an official statement on the topic, clarifying its own position on 3DMark 2003.

But let us begin with Futuremark. The company is now (officially) responding to NVIDIA's accusations. Let me make one thing absolutely clear first, though: the following excerpts in italics are direct quotes from the respective companies. They are not necessarily indicative of my opinion on this matter.

Futuremark:

3DMark 03 is a synthetic benchmark that is designed for the sole purpose of enabling objective performance measurement of DirectX 9 compatible hardware today. There are no other significant DirectX 9 applications published yet, and the most awaited DirectX 9 game is most likely at least six months away. 3DMark 03 is a forward-looking tool that provides unique value to consumers in the form of impartial information to support their purchasing decisions today.

We continue to recommend to independent testers to complement their analysis by using published games in measuring performance. However, it has to be noted that those results are valid only for that game, whereas 3DMark 03 can provide a forward-looking overall view of the performance and features of the hardware.

Futuremark is positioning the new 3DMark as a test to evaluate the performance of DirectX 9 class video accelerators in games appearing in the next 6-12 months. It does not, however, test the performance of current video cards in current games - which was one of the points criticized by NVIDIA in its "Lab Report."

NVIDIA:

The primary goal of any benchmark is to arm the consumer with the right information to make the best possible purchase decision. As the gamers' benchmark, 3DMark 03 must emulate as closely as possible the kind of experience that the gaming enthusiast will expect on their machine. It must exercise graphics hardware in the same manner that consumer games will. The graphics features, rendering paths, and effects must all emulate games, or the consumer will be misinformed and their expectations misguided.

3DMark 03 combines custom artwork with a custom rendering engine that creates a set of demo scenes that, while pretty, have very little to do with actual games . It is much better termed a demo than a benchmark. The examples included in this report illustrate that 3DMark 03 does not represent games, can never be used as a stand-in for games, and should not be used as a gamers' benchmark.

The ultimate injury to the consumer of such a benchmark is three-fold. First, of course, the consumer is misguided. A purchase decision based on ineffectual data will lead consumers to wrong conclusions. Second, it causes graphics hardware manufacturers to focus attention and engineering resources on optimizing for artificially fabricated cases that are a-typical of games. Such optimizations generally do nothing to improve real game performance, and provide no benefit to the consumer. Finally, the extra engineering effort focused on such benchmarks reduces the effort available for activities beneficial to consumers - improving the actual gaming experience.