FCAT VR: GPU And CPU Performance in Virtual Reality

Proof of Concept: Hardware Versus Software Capture

First things first. Before we start pitting graphics architectures against each other or digging into quality settings, we need to establish the comparability of hardware- and software-based FCAT.

For this, we’ll use three different GPUs with varying performance under Gunfire Games’ Chronos. It’s one of the most taxing titles we’ve tested, and we deliberately select the Epic quality preset to push each board as hard as possible. Simultaneously, we record one run using both performance tools. They’re processed independently and aligned using Nvidia’s FCAT VR Analyzer to ensure they overlay as perfectly as possible.

We find it easiest to look at the fastest card first, and that’s Nvidia’s Titan X (Pascal).

The Analyzer included with Nvidia’s toolset generates three charts. Up top, we see frame time over time, with both the hardware- and software-based results plotted together. Below that are the two runs, graphed separately, illustrating frame type over time. The legend shows real frames, in green, synthesized frames (via asynchronous spacewarp) in yellow, and dropped frames in red.

Even at Chronos’ most demanding settings, a Titan X is fast enough to sustain 90 real frames per second throughout our test sequence. Both the hardware and software captures recognize four dropped frames, which frankly may correspond to scene changes as the RPG changes perspective. Regardless, you can see in the frame time chart up top where small spikes cause the hardware capture to flip from 11 to 22ms, and where the unconstrained frame time measured by software briefly blips. The red line staying far enough below 11ms is what facilitates an ideal result.

If it weren’t for a 90 Hz refresh, the Titan X would average 134 FPS in this workload.

The GeForce GTX 1070 chart looks significantly different, requiring some more information to properly analyze. Let’s start with the frame time over time chart up top.

Our software capture oscillates above and below 11ms. When it’s below, we get 90 newly-rendered frames per second. When the line spends too much time above 11ms, asynchronous spacewarp kicks in and we end up synthesizing alternating frames.

To the hardware-based video capture solution, it looks like we’re dropping every other frame, pushing the green line up to 22ms (or 45 FPS). That’s because the tool is looking at the overlay’s colored bars and determining if the regular pattern missed, and then how many colors it missed by. Based on that number, the scripts determine how long it took to display the last frame (one missed color equals 11 + 11ms, or 22ms, two missed colors equal 11 + 22ms, or 33ms, and so on).

There’s another interesting observation to address. In several instances, the video capture reflects 90 real frames rendered per second as the software tool shows frame times exceeding 11ms. How is that possible, when we’re taught work on a frame has to be completed in less than 11ms to appear during the subsequent scan? In all actuality, the envelope can expand/contract due to preemption/parallelization done by the VR runtime. Oculus' adaptive queue ahead feature is designed to facilitate this, so an 11ms cut-off is not absolute, though it’s generally true. In short, optimizations are not enough to keep you from dropping frames if you render at >11ms for an extended period of time.

The next two charts break down the composition of each 90 FPS run. In the last one (hardware capture), Nvidia’s scripts report 1964 dropped frames. In reality, asynchronous spacewarp is active when the 1070 isn’t able to render 90 real frames per second. But that’s not apparent in a recorded video, so we end up with lots of red. Data analyzed by the FCAT VR Capture tool can convey this information, though. As a result, the chart corresponding to software capture tells us only 12 frames were dropped, while 1952 were synthesized.

The introduction of asynchronous spacewarp bumped the Rift’s minimum graphics specification down to a GeForce GTX 960, but Chronos’ Epic detail preset demonstrates how taxing VR workloads need more than just a mainstream GPU.

Our hardware-based capture system reflects an almost-constant 22ms frame time. Were it not for warping, this would be a fairly abysmal VR experience. The stats tell us we generated 3616 new frames through the run and dropped 3626 frames. That slightly-higher-than 1:1 ratio of dropped to real frames is likely due to the four spikes up to 33ms and subsequent dips to 11ms.

Picking through the software capture tells us that the frames originally reported as dropped by a video sequence were in fact synthesized frames, generated by ASW. The stats spit back 3685 new frames and 3695 synthesized ones. One question remains unanswered for us here: if ASW generates extrapolated frames using previous real frames, how does the technology operate in situations where the frame rate drops below 45? Does it simply extrapolate based on an extrapolation? This isn’t the most extreme example we've seen of Nvidia’s tool reporting more synthesized than real frames. We'll address the question in more depth shortly, and present a potential answer.

Hardware Vs. Software: It Works

The real purpose of this first look at FCAT VR is to test the relationship between hardware-based capture and Nvidia’s FCAT VR Capture tool.

We now have the infrastructure in place to intercept and record a 2160x1200 signal at 90 Hz, analyze the output, and chart it just as we did during the days of Challenging FPS: Testing SLI And CrossFire Using Video Capture. Nvidia knows this methodology is expensive and time-consuming, though. So it developed software to make VR benchmarking more accessible.

By instead tapping into telemetry data exposed to developers by Oculus’ runtime and SteamVR, Nvidia’s FCAT VR Capture application facilitates Fraps-like recording of several different statistics, which can be analyzed and charted through a Python-based Analyzer GUI.

Anyone could have accessed this information previously. In fact, we have an unpublished story in our CMS from last year extensively measuring graphics performance on the Vive. But we uncovered so many issues that changed with each subsequent driver update that it was never possible to push the piece live before its findings were rendered obsolete. Eventually we shelved it. The point is that Nvidia isn’t presenting anything you couldn’t get at yourself, given the time and effort. By making the tool available publicly, the technical community has an opportunity to dissect it. And by facilitating a comparison to hardware-based capture, we can verify agreement between the two metrics.

MORE: Gaming At 3840x2160: Is Your PC Ready For A 4K Display

MORE: How Well Do Workstation Graphics Cards Play Games?

This thread is closed for comments
11 comments
    Your comment
  • ingtar33
    no ryzen? just an 8320?
  • AndrewJacksonZA
    A really nice article and a great introduction to FCAT VR. Thanks Chris!

    Obvious request that probably everyone will ask of you: please include Ryzen in your tests once you have a sane moment after the hectic launch period. :-)

    Now that that's out the way, I can get to my actual question: I would like to know if "Game A" is available on both the Rift and the Vive, would FCAT VR be able to tell you what, if anything, the performance difference is given the same computer hardware please?

    Please can you also include multi-GPU setups. They can definitely help, depending on the title, e.g.
    https://www.hardocp.com/article/2016/11/30/serious_sam_vr_mgpu_nvidia_gtx_pascal_follow_up/5
    https://www.hardocp.com/article/2016/10/24/serious_sam_vr_mgpu_amd_rx_480_follow_up/

    I found this statement to be very interesting: "it’s possible to illustrate that 11ms isn’t an absolute threshold for rendering new frames at 90 Hz."

    Thank you
    Andrew
  • cangelini
    Hi guys,

    Ryzen wasn't in my lab yet when this was written, but I do have plans there ;)

    Andrew, yes, FCAT VR should allow us to test the same title on two different HMDs and compare their performance.

    Multi-GPU is a plan as well, particularly once games begin incorporating better support for it (right now, that's a bit of a problem).
  • WhyAreYou
    Thanks, looks good!
  • playingwithplato
    Do FCAT VR's measurement algorithms tend to favor NVDIA chipsets? Curious, wonder if AMD will release a similar testing tool to measure buffer store/retrieval and render speed <11ms? Would like to see that applied to en environment with their GPUs.
  • ffrgtm
    Fantastic article! I would love to see results with a secondary GPU (non-SLi) dedicated to physics and compare that to the cpu swap tests in AZ Sunshine you've just shown us.
  • ffrgtm
    159646 said:
    Do FCAT VR's measurement algorithms tend to favor NVDIA chipsets? Curious, wonder if AMD will release a similar testing tool to measure buffer store/retrieval and render speed <11ms? Would like to see that applied to en environment with their GPUs.

    A quote from another article on FCAT:

    Quote:
    "While the FCAT VR tool is developed by Nvidia, the company insists it is headset and GPU agnostic, and meant only to capture data. The tool itself doesn’t contain a benchmark; according to the company, the tool logs information directly from the VR runtime."


    I'm inclined to believe Nvidia's claim of brand blindness right now... but if AMD ever manages to put forth some real competition then I don't think we could be blamed for becoming more skeptical. It's hard to forget just how far Nvidia and AMD have gone to skew results in the past.
  • AndrewJacksonZA
    134065 said:
    Andrew, yes, FCAT VR should allow us to test the same title on two different HMDs and compare their performance. Multi-GPU is a plan as well, particularly once games begin incorporating better support for it (right now, that's a bit of a problem).
    Thank you.


    One thing that I might've missed in the article: Has Nvidia open-sourced FCAT and FCAT VR so that everyone can see the code, check it for unbiasedness (is that even a word? :-) and help contribute to the program to make it even better?
  • cangelini
    Yup, check it out: http://www.geforce.com/whats-new/guides/fcat-vr-download-and-how-to-guide

    There's a download link in there. I'd be curious to hear from any TH readers who want to mess with it as well!
  • thinkspeak
    Any chance of doing this again but with stock and OC on the maxwell, pascal and AMD cards? The 980 ti really opens up and they generally overclock well, in some cases exceeding the 1070 which would be useful for those debating an upgrading for VR
  • mitch074
    I guess my o/c'ed 4670k and reference RX480 should be able to handle this correctly, then... Now, if headset prices could go down!