How We Test USB Wi-Fi Adapters

Tom's Hardware shows you how we test USB-attached 802.11ac wireless networking adapters.

Introduction

Choosing networking gear can be a daunting process. Buying the cheapest product in this market segment is often a lousy idea due to quality issues and poor support, but purchasing the most expensive adapter for no specific reason can be equally foolhardy. Too often, we simply choose a particular brand of networking gear at the onset based on a recommendation or a sale, and simply keep purchasing it.

Product testing of Wi-Fi gear has traditionally focused on only one piece of hardware--the wireless router. While that's the central hub of a home network, it is not the only piece of equipment that should be optimized for the best networking experience. Indeed, in our testing, choosing the best wireless AC USB adapter can be equally important to choosing the right router for your application.

The ideal would be for home users to have access to several brands of equipment and to pick the components that work best after testing them. But the practical reality is that most of us buy parts one at a time as we build our networks out, without access to multiple products simultaneously. This leads to the inevitable read-through of user "product reviews" on various retailer websites. While occasionally informative, and often entertaining, it is too often unclear how a particular product really performs when there are evaluations mostly based on a lack of real data, a majority subjective, leaving the average home user at a loss to choose the best gear in an objective fashion. For a review of the technology behind wireless AC USB adapters, check out our recent USB Wi-Fi 101 article.

MORE: All Networking Content
MORE: Networking in the Forums

Testing Methodology

With that background in mind, we set out to both objectively and subjectively test wireless AC USB adapters. In the scientific method (remember that from your high school science class?) this is known as quantitative and qualitative testing.

The quantitative method focuses on data gleaned from benchmarks. It needs to be generated in a reproducible fashion, under controlled conditions. To that end, Tom's Hardware has invested in setting up a new hardware testing laboratory to generate networking results. In the area of wireless networking, the important metrics include data throughput and signal strength on the two frequencies that are used: the original 2.4GHz that goes all the way back to the 802.11b standard, as well as the 5GHz band first introduced in 802.11a. The 5GHz frequency was mostly used commercially, while more commonly used in households as part of the 802.11n specification, and was pushed to the forefront by 802.11ac.

While not data-based, qualitative observations are also important in choosing the best gear. This subjective evaluation includes the overall design of the adapter, the internal contents of the device, the installation process and the included software. While not quantifiable like a speed metric or signal strength on a chart or graph, these more subjective points are also critical if the adapter comes with unstable software or protrudes too far from the USB port, for example. The affordability factor of these devices is also a consideration, as many users want good value for their money.

With a comprehensive methodology to evaluate wireless AC USB adapters, we selected several major manufacturers and asked them to submit some of their current models. Some of the products we received will be tested in upcoming articles.

Testing Setup

The router chosen for the setup is the Asus RT-AC66U. This is a current router that features:

  • Three external antennas
  • Broadcom 802.11ac controller
  • Gigabit Ethernet ports (1 x WAN, 4 x LAN)
  • 256MB of RAM
  • AC1750, 2.4GHz at 450 Mb/s and 5GHz at 1.3 Gb/s
  • AiRadar- beamforming technology

The goal was to utilize a router that can match or exceed the capabilities of the wireless AC USB adapters tested, and this Asus model has the go-fast top speed. The latest firmware was used to enhance stability and throughput.

We positioned the RT-AC66U on a cart, which remained in the same location throughout testing. Its antennas were positioned with the center one standing vertically and the two on the side pointing 45-degrees out laterally, as recommended in the owner's manual.

When testing wireless networking hardware, a mobile platform is ideal to facilitate variable distances. We picked Sony's SVS13112FXS laptop, which uses an Intel Core i5-3210M processor at 2.5GHz. It's based on the Ivy Bridge architecture and includes HD Graphics 4000. You also get a 13.3-inch display and plenty of battery life, along with 6GB of DDR3-1333. The operating system is Windows 7 64-bit restored to its factory settings.

During testing, the on-board wireless card was disabled in the BIOS, and all other software other than WirelessMon, IxChariot and TightVNC was disabled. This is done to ensure consistent results, as background processes can affect the results as they consume resources.

Perhaps most important, the Sony notebook has three USB ports, two of which transfer at 3.0 rates. It is important to have these AC-class adapters plugged into a USB 3.0 port capable of 5 Gb/s data rates. USB 2.0 with its theoretical maximum of 480 Mb/s would be a likely performance bottleneck, particularly on the 5GHz band.

For signal strength testing, the laptop was held at its measured distance for 20 to 30 seconds to acquire the signal reading in dB (decibels). In order to level the playing field, the wireless AC USB adapters were plugged directly into the USB 3.0 ports for benchmarking (in other words, the docks or USB extension cords that are provided with some models were not used, since they could give some, but not all, products a signal strength advantage. For those adapters that include extendable antennas, they were positioned upright in a vertical position, and not tuned any further. Our goal was to test the adapter, and not the usefulness of the USB extension cable.

Each of the wireless AC USB adapters was benchmarked with its latest software. While these devices ship with drivers on a CD, they're sometimes outdated. To rectify this, we download any bundled drivers or utilities from the manufacturer's website.

The Sony notebook connects to an ASRock Vision X 471D that plays the server role in our testing. This server features a mobile Haswell processor, Intel's Core i7-4712MQ. It was reviewed previously by Tom's Hardware last year. 

Quantitative Testing

Throughput is one of a Wi-Fi adapter's key specifications. Since these are all AC1200-capable models, you might not expect much variation between them.

The router and the overall network topography remained constant throughout testing. Throughput was measured at five, 25 and 60 feet. Sixty-foot testing included two intervening walls that were in the way due to the physical setup of the test lab. The distances were determined with a measuring tape, except for the 60-foot test, which we calculated using the Pythagorean theorem (finally, a use for trigonometry outside of math class!). Line of sight was lost on that one as we turned a corner to get far enough away from the router. Tests reaching 75 feet usually go out into an atrium just outside the suite's entrance. The setup is in the first floor of a commercial building, and the walls are constructed of plaster.

Above is a figure of the office test setup that shows the various distances, and the intervening wall structures at 56 feet. If 75-foot testing is done, the procedure puts the tested adapter outside of the office. The diagram is not to scale.

A labeled image shows the physical setup of the testing location. Note that the router on the left-hand side is standing at an upright angle.

When a router and adapter are too close together, the Wi-Fi signals can cause interference (as represented in the image above). This is why a router and an access point (AP) should not be set up in close proximity, and not on the same Wi-Fi channel.

In most cases, the speeds close to the router at the five-foot distance were slower than at 25 or 50 feet. This phenomenon can happen when the signals are too strong between the router and the adapter, creating interference as a result of being too close to each other. Simply put, this is one of those cases when a wired connection is preferred over wireless.

A screenshot of the type of data that the popular Web-based Speedtest (www.speedtest.net) generates. While it will provide an indication of the download and upload speed, it measures the Internet connection and not the wireless network speeds.

Measuring the speed of a network connection is a vital part of each adapter's quantitative evaluation. Simply running an Internet speed metric, such as Speedtest.net, will give you the speed of the online network connection, but it's not the right test to use to compare Wi-Fi speeds. Unless you have a gigabit connection (such as Google Fiber), the throughput of a home's internal network will be faster than the Internet, invalidating the use of a Web-based tool for these tests.

A screen capture from Windows 7 showing the network speed estimate over a 5GHz Wi-Fi network; in this case it was 325.0 Mb/s.

There are better ways to approximate wireless network speed. The first is Windows' Network Center, which can provide a rough estimate of the network adapter's connection. It's not an indicator of actual performance though, so it is not the best number to use since it does not get measured directly.

For home users with multiple devices on their network, by transferring a file between two computers within a Windows Home Group, knowing the size of the file and timing the transfer time, speed can be measured and Mb/s derived. One of the devices should be wireless and the other one wired, preferably on a 10/100/1000 Mb/s Ethernet port, to best measure the wireless connection's speed. The limitation of this method is that the transfer is manually timed, introducing an element of inaccuracy (the last time I did this I was using a stopwatch app for my smartphone, practicing starting the transfer and the stopwatch simultaneously, which is difficult to perfect).

This is the raw data that IxChariot generates. Note the throughput at the 50-foot distance on the 5GHz band fluctuated significantly from a minimum of 18.5 to a maximum of 231.8 Mb/s, which, while it averaged out to 116.8 Mb/s, hardly tells the whole story!

Given the limitations of the previously mentioned techniques, a dedicated software solution was chosen: IxChariot.

This software can measure network performance in a reliable and consistent fashion, including TCP throughput. It'll report minimum and maximum speeds, as well as calculate the average. TightVNCViewer, a remote desktop software solution, is used to remotely access our ASRock server in order to log into a desktop session and retrieve the IP address. Then, TightVNCViewer is terminated. When working with IxChariot, we designate the server's IP address as Endpoint 1 and the laptop's IP address as Endpoint 2. We use the High Performance Throughput test to determine speeds that are reported in Mb/s. Throughput tests are run to completion, which means, specifically, the test is run until 100 timing records are finished and the results recorded with screenshots.

A hypothetical bar graph of the speeds obtained (x-axis, expressed in Mb/s) from four wireless AC USB adapters (y-axis). Note that the maximum speeds are in black, the minimum speeds in red and the average speeds in blue. While one product may have lower minimum values, it may not necessarily have lower averages. This same observation also applies to peaks values.

PassMark's WirelessMon is a software package we use to measure signal strength in five-foot increments on the 2.4 and 5GHz bands. The WirelessMon software provides a signal strength reading between the device under test and the router for a given distance. The notebook is held in each spot for 20 to 30 seconds to get this reading. This data set looks at how good the antenna is on the adapter, or if a manufacturer's implementation of beamforming (a Wi-Fi technology designed for directional signal transmission) is working or not, as the distance increases compared to the competition. All of this data gets put into an Excel spreadsheet with relevant notations on how the test went specifically for that USB AC1200 Wi-Fi adapter. With the data collected on each product, all data points are put into a separate Excel spreadsheet for comparative analysis.

A screenshot of the software PassMark's WirelessMon.

A hypothetical plot of four wireless AC USB adapters with the distance from the router expressed in feet (x-axis), and the signal strength in decibels (dB, y-axis). While the red and black lines follow a linear progression, the blue and green plots illustrate how nonlinear signal strength may be.

Qualitative Testing

To further expand our performance data, the devices are set up independently on a Lenovo S400 in a residential home. In this case, the router we're using is a Netis AC1200 with a broadband cable connection. Each of the USB AC1200 Wi-Fi adapters is installed and used separately in a more typical home environment with the usual challenges that home users face: placing the router downstairs and not directly under the adapter, steel ductwork reflecting the signal in the basement and the multitude of competing wireless networks creating interference.

The point was not to redo the numbers from the Tom's Hardware lab testing, but rather to confirm the numbers generated in our office, as well as see how each wireless AC USB adapter performs under typical residential use. For example, does it stream HD video smoothly or with hiccups, does the adapter stay cool or get hot, and is the connection to the Wi-Fi network maintained or dropped? In addition, the stability and usability of the software could be assessed, as well as the out-of-box experience and setup process. Don't think this all goes smoothly in our expert hands; along the way, we lost an adapter for the initial article, as neither the lab nor this writer could get it up and running. Stay tuned, as this will be featured in a subsequent article.

Getting Inside

It is important to see what these wireless AC USB adapters are made of. While the adapters are enclosed in plastic black shells, upon opening them it becomes clear why the better-performing adapters have the higher speeds, and the answer seems dependent on the antenna solution. Getting inside of these adapters was not via some manufacturer-supplied images, but rather through old-fashioned grunt work, with an assortment of tools, including Torx screwdrivers, pliers (both lineman and needle-nose), flat-bladed screwdrivers of assorted sizes and even micro screwdrivers. Some cases came apart easier than others. The Netgear could be reconstructed, but the D-Link was essentially destroyed in the process. Be aware that this adventure can become a one-way trip to breaking the adapter!

Final Thoughts

While this test process was used specifically for wireless AC1200 USB adapters, it can be easily scaled to higher-speed wireless products like AC1750-class adapters as they become available. This will include anticipated wireless improvements like MU-MIMO, which stands for Multi-User Multiple-Input Multiple-Output. It's an update to the 802.11ac standard known as Wave 2, with maximum speeds anticipated of 6.93 Gb/s. MU-MIMO allows multiple users to simultaneously access the same wireless channel by providing spatial degrees of freedom, albeit only in the downstream direction. In addition, the testing suite will also be applicable to WiGig, a future wireless standard that promises high speeds over short distances on the unlicensed 60GHz band.

Helping users choose a wireless AC USB adapter starts with a comprehensive set of benchmarks, accurately measured across multiple distances and on both the 2.4GHz and 5GHz frequencies. This is supported by looking at the out-of-box experience, the setup process, the stability and the software. Taking the devices apart provides another level of analysis. This is all done so that when we recommend a wireless AC USB adapter to our readers, we have full confidence that that recommendation will be the best one to suit their needs, while steering them clear of hardware with issues.

MORE: All Networking Content
MORE: Networking in the Forums

Jonas DeMuro is an Associate Contributing Writer for Tom's Hardware.

Follow us on Twitter @tomshardware, on Facebook and on Google+.

This thread is closed for comments
8 comments
    Your comment
  • blackmagnum
    I am waiting for Apple to update their cheaper Airport Express base station to AC bandwidth. The current one gave me no proplem at all for various types and brands of wifi adaptors.
  • marraco
    I have a linksys router and a huawei. I also have some laptops with internal WiFi plaques, some USB WiFi dongles, and many phones and tablets..

    The huawei dongle works well with the huawei router, and so does SOME internal WIFi plaques.

    On the lynksys router, MOST internal WiFi plaques work better, but not all, and not all the USB devices.

    Worse, everything changes when many devices are added to the network, like a large number of laptops, phones, tablets which the family, friends and neighbors overlap, accessing my network or other networks I don't use.

    At this moment IssiDer shows 2 dozen devices at reach of my PC, and I don't have any control over them.

    So, I never trust Tomshardware reviews. They don't test WiFi equipment against a variety of different devices, and they don't check for interference from other networks, and under different channels.

    The focus tends to be over performance in isolated and controlled state, but that says little or nothing about real world behavior.

    When the air gets filled with other networks, and many different devices are connected to the router, robustness is far more valuable than pure speed.

    Also, it does not matter if a specific router performs best with the non disclosed device inside the laptop used by tomshardware for benchmarking, because I may have a different WiFi plaque, and different WiFi USB dongles.
  • Foxlife
    Please keep in mind and add to test methodology the fact that those usb dongles eat up cpu usage, in some case quite significantly vs integrated (within the laptops themselves) solutions.
  • tsnor
    Foxlife said "Please keep in mind and add to test methodology the fact that those usb dongles eat up cpu usage, in some case quite significantly vs integrated (within the laptops themselves) solutions."

    I'm surprised by this, given the internal wifi solutions are typically simply mini pci cards. Do you have any references ?

    ---

    I'm looking forward to the actual test results! The test procedures look good.
  • dthx
    Nice article. I do appreciate the transparency on your testing methods.
    It is just one way of testing and my experience with some brands that I'll avoid to buy at all cost often differs from your test results. For example, I always had bad luck with anything d-link makes.
    But nothing's perfect in this world, especially when playing with radio waves, So although your methodology will never be 100% accurate due to possible interference from the neighbourhood and maybe the microwave oven of your kitchen, I'd rather choose your methodology (close to the real life usage) over any tests done in specialized labs with all tests executed inside a Faraday cage.
  • getochkn
    Why does your picture not show the 2 end antenna's at 45 degree angles like you said you did?
  • willy481
    I think most people would rather not have to buy an external WiFi dongle for their laptop or NUC - double whamy as as pay twice to lose 1 port, so are you able to test the most popular mPCIe WiFi-AC cards that laptop makers typically include, in particular the Intel Dual Band Wireless-AC 7260 and its recent replacement the 7265 ?. I realise this not as simple as testing external dongle due the fact laptops dont all use identical antenna designs - so just test in a laptop that uses a typical antenna design -better than nothing; perhaps also test the popular D54250 NUC which uses the above 7260 - think all Intel NUCs use similar box & dual internal antenna
  • EdJulio
    Quote:
    Why does your picture not show the 2 end antenna's at 45 degree angles like you said you did?


    The cart is designed to be mobile so it got shifted around during testing. Plus we're constantly switching routers around. The pic caught the cart on an off-day. I swapped out the old pic with a new one. Thanks for the note!!!