Tom's Hardware shows you how we test USB-attached 802.11ac wireless networking adapters.
Choosing networking gear can be a daunting process. Buying the cheapest product in this market segment is often a lousy idea due to quality issues and poor support, but purchasing the most expensive adapter for no specific reason can be equally foolhardy. Too often, we simply choose a particular brand of networking gear at the onset based on a recommendation or a sale, and simply keep purchasing it.
Product testing of Wi-Fi gear has traditionally focused on only one piece of hardware--the wireless router. While that's the central hub of a home network, it is not the only piece of equipment that should be optimized for the best networking experience. Indeed, in our testing, choosing the best wireless AC USB adapter can be equally important to choosing the right router for your application.
The ideal would be for home users to have access to several brands of equipment and to pick the components that work best after testing them. But the practical reality is that most of us buy parts one at a time as we build our networks out, without access to multiple products simultaneously. This leads to the inevitable read-through of user "product reviews" on various retailer websites. While occasionally informative, and often entertaining, it is too often unclear how a particular product really performs when there are evaluations mostly based on a lack of real data, a majority subjective, leaving the average home user at a loss to choose the best gear in an objective fashion. For a review of the technology behind wireless AC USB adapters, check out our recent USB Wi-Fi 101 article.
With that background in mind, we set out to both objectively and subjectively test wireless AC USB adapters. In the scientific method (remember that from your high school science class?) this is known as quantitative and qualitative testing.
The quantitative method focuses on data gleaned from benchmarks. It needs to be generated in a reproducible fashion, under controlled conditions. To that end, Tom's Hardware has invested in setting up a new hardware testing laboratory to generate networking results. In the area of wireless networking, the important metrics include data throughput and signal strength on the two frequencies that are used: the original 2.4GHz that goes all the way back to the 802.11b standard, as well as the 5GHz band first introduced in 802.11a. The 5GHz frequency was mostly used commercially, while more commonly used in households as part of the 802.11n specification, and was pushed to the forefront by 802.11ac.
While not data-based, qualitative observations are also important in choosing the best gear. This subjective evaluation includes the overall design of the adapter, the internal contents of the device, the installation process and the included software. While not quantifiable like a speed metric or signal strength on a chart or graph, these more subjective points are also critical if the adapter comes with unstable software or protrudes too far from the USB port, for example. The affordability factor of these devices is also a consideration, as many users want good value for their money.
With a comprehensive methodology to evaluate wireless AC USB adapters, we selected several major manufacturers and asked them to submit some of their current models. Some of the products we received will be tested in upcoming articles.
The router chosen for the setup is the Asus RT-AC66U. This is a current router that features:
- Three external antennas
- Broadcom 802.11ac controller
- Gigabit Ethernet ports (1 x WAN, 4 x LAN)
- 256MB of RAM
- AC1750, 2.4GHz at 450 Mb/s and 5GHz at 1.3 Gb/s
- AiRadar- beamforming technology
The goal was to utilize a router that can match or exceed the capabilities of the wireless AC USB adapters tested, and this Asus model has the go-fast top speed. The latest firmware was used to enhance stability and throughput.
We positioned the RT-AC66U on a cart, which remained in the same location throughout testing. Its antennas were positioned with the center one standing vertically and the two on the side pointing 45-degrees out laterally, as recommended in the owner's manual.
When testing wireless networking hardware, a mobile platform is ideal to facilitate variable distances. We picked Sony's SVS13112FXS laptop, which uses an Intel Core i5-3210M processor at 2.5GHz. It's based on the Ivy Bridge architecture and includes HD Graphics 4000. You also get a 13.3-inch display and plenty of battery life, along with 6GB of DDR3-1333. The operating system is Windows 7 64-bit restored to its factory settings.
During testing, the on-board wireless card was disabled in the BIOS, and all other software other than WirelessMon, IxChariot and TightVNC was disabled. This is done to ensure consistent results, as background processes can affect the results as they consume resources.
Perhaps most important, the Sony notebook has three USB ports, two of which transfer at 3.0 rates. It is important to have these AC-class adapters plugged into a USB 3.0 port capable of 5 Gb/s data rates. USB 2.0 with its theoretical maximum of 480 Mb/s would be a likely performance bottleneck, particularly on the 5GHz band.
For signal strength testing, the laptop was held at its measured distance for 20 to 30 seconds to acquire the signal reading in dB (decibels). In order to level the playing field, the wireless AC USB adapters were plugged directly into the USB 3.0 ports for benchmarking (in other words, the docks or USB extension cords that are provided with some models were not used, since they could give some, but not all, products a signal strength advantage. For those adapters that include extendable antennas, they were positioned upright in a vertical position, and not tuned any further. Our goal was to test the adapter, and not the usefulness of the USB extension cable.
Each of the wireless AC USB adapters was benchmarked with its latest software. While these devices ship with drivers on a CD, they're sometimes outdated. To rectify this, we download any bundled drivers or utilities from the manufacturer's website.
The Sony notebook connects to an ASRock Vision X 471D that plays the server role in our testing. This server features a mobile Haswell processor, Intel's Core i7-4712MQ. It was reviewed previously by Tom's Hardware last year.
Throughput is one of a Wi-Fi adapter's key specifications. Since these are all AC1200-capable models, you might not expect much variation between them.
The router and the overall network topography remained constant throughout testing. Throughput was measured at five, 25 and 60 feet. Sixty-foot testing included two intervening walls that were in the way due to the physical setup of the test lab. The distances were determined with a measuring tape, except for the 60-foot test, which we calculated using the Pythagorean theorem (finally, a use for trigonometry outside of math class!). Line of sight was lost on that one as we turned a corner to get far enough away from the router. Tests reaching 75 feet usually go out into an atrium just outside the suite's entrance. The setup is in the first floor of a commercial building, and the walls are constructed of plaster.
Above is a figure of the office test setup that shows the various distances, and the intervening wall structures at 56 feet. If 75-foot testing is done, the procedure puts the tested adapter outside of the office. The diagram is not to scale.
A labeled image shows the physical setup of the testing location. Note that the router on the left-hand side is standing at an upright angle.
When a router and adapter are too close together, the Wi-Fi signals can cause interference (as represented in the image above). This is why a router and an access point (AP) should not be set up in close proximity, and not on the same Wi-Fi channel.
In most cases, the speeds close to the router at the five-foot distance were slower than at 25 or 50 feet. This phenomenon can happen when the signals are too strong between the router and the adapter, creating interference as a result of being too close to each other. Simply put, this is one of those cases when a wired connection is preferred over wireless.
A screenshot of the type of data that the popular Web-based Speedtest (www.speedtest.net) generates. While it will provide an indication of the download and upload speed, it measures the Internet connection and not the wireless network speeds.
Measuring the speed of a network connection is a vital part of each adapter's quantitative evaluation. Simply running an Internet speed metric, such as Speedtest.net, will give you the speed of the online network connection, but it's not the right test to use to compare Wi-Fi speeds. Unless you have a gigabit connection (such as Google Fiber), the throughput of a home's internal network will be faster than the Internet, invalidating the use of a Web-based tool for these tests.
A screen capture from Windows 7 showing the network speed estimate over a 5GHz Wi-Fi network; in this case it was 325.0 Mb/s.
There are better ways to approximate wireless network speed. The first is Windows' Network Center, which can provide a rough estimate of the network adapter's connection. It's not an indicator of actual performance though, so it is not the best number to use since it does not get measured directly.
For home users with multiple devices on their network, by transferring a file between two computers within a Windows Home Group, knowing the size of the file and timing the transfer time, speed can be measured and Mb/s derived. One of the devices should be wireless and the other one wired, preferably on a 10/100/1000 Mb/s Ethernet port, to best measure the wireless connection's speed. The limitation of this method is that the transfer is manually timed, introducing an element of inaccuracy (the last time I did this I was using a stopwatch app for my smartphone, practicing starting the transfer and the stopwatch simultaneously, which is difficult to perfect).
This is the raw data that IxChariot generates. Note the throughput at the 50-foot distance on the 5GHz band fluctuated significantly from a minimum of 18.5 to a maximum of 231.8 Mb/s, which, while it averaged out to 116.8 Mb/s, hardly tells the whole story!
Given the limitations of the previously mentioned techniques, a dedicated software solution was chosen: IxChariot.
This software can measure network performance in a reliable and consistent fashion, including TCP throughput. It'll report minimum and maximum speeds, as well as calculate the average. TightVNCViewer, a remote desktop software solution, is used to remotely access our ASRock server in order to log into a desktop session and retrieve the IP address. Then, TightVNCViewer is terminated. When working with IxChariot, we designate the server's IP address as Endpoint 1 and the laptop's IP address as Endpoint 2. We use the High Performance Throughput test to determine speeds that are reported in Mb/s. Throughput tests are run to completion, which means, specifically, the test is run until 100 timing records are finished and the results recorded with screenshots.
A hypothetical bar graph of the speeds obtained (x-axis, expressed in Mb/s) from four wireless AC USB adapters (y-axis). Note that the maximum speeds are in black, the minimum speeds in red and the average speeds in blue. While one product may have lower minimum values, it may not necessarily have lower averages. This same observation also applies to peaks values.
PassMark's WirelessMon is a software package we use to measure signal strength in five-foot increments on the 2.4 and 5GHz bands. The WirelessMon software provides a signal strength reading between the device under test and the router for a given distance. The notebook is held in each spot for 20 to 30 seconds to get this reading. This data set looks at how good the antenna is on the adapter, or if a manufacturer's implementation of beamforming (a Wi-Fi technology designed for directional signal transmission) is working or not, as the distance increases compared to the competition. All of this data gets put into an Excel spreadsheet with relevant notations on how the test went specifically for that USB AC1200 Wi-Fi adapter. With the data collected on each product, all data points are put into a separate Excel spreadsheet for comparative analysis.
A screenshot of the software PassMark's WirelessMon.
A hypothetical plot of four wireless AC USB adapters with the distance from the router expressed in feet (x-axis), and the signal strength in decibels (dB, y-axis). While the red and black lines follow a linear progression, the blue and green plots illustrate how nonlinear signal strength may be.
To further expand our performance data, the devices are set up independently on a Lenovo S400 in a residential home. In this case, the router we're using is a Netis AC1200 with a broadband cable connection. Each of the USB AC1200 Wi-Fi adapters is installed and used separately in a more typical home environment with the usual challenges that home users face: placing the router downstairs and not directly under the adapter, steel ductwork reflecting the signal in the basement and the multitude of competing wireless networks creating interference.
The point was not to redo the numbers from the Tom's Hardware lab testing, but rather to confirm the numbers generated in our office, as well as see how each wireless AC USB adapter performs under typical residential use. For example, does it stream HD video smoothly or with hiccups, does the adapter stay cool or get hot, and is the connection to the Wi-Fi network maintained or dropped? In addition, the stability and usability of the software could be assessed, as well as the out-of-box experience and setup process. Don't think this all goes smoothly in our expert hands; along the way, we lost an adapter for the initial article, as neither the lab nor this writer could get it up and running. Stay tuned, as this will be featured in a subsequent article.
It is important to see what these wireless AC USB adapters are made of. While the adapters are enclosed in plastic black shells, upon opening them it becomes clear why the better-performing adapters have the higher speeds, and the answer seems dependent on the antenna solution. Getting inside of these adapters was not via some manufacturer-supplied images, but rather through old-fashioned grunt work, with an assortment of tools, including Torx screwdrivers, pliers (both lineman and needle-nose), flat-bladed screwdrivers of assorted sizes and even micro screwdrivers. Some cases came apart easier than others. The Netgear could be reconstructed, but the D-Link was essentially destroyed in the process. Be aware that this adventure can become a one-way trip to breaking the adapter!
While this test process was used specifically for wireless AC1200 USB adapters, it can be easily scaled to higher-speed wireless products like AC1750-class adapters as they become available. This will include anticipated wireless improvements like MU-MIMO, which stands for Multi-User Multiple-Input Multiple-Output. It's an update to the 802.11ac standard known as Wave 2, with maximum speeds anticipated of 6.93 Gb/s. MU-MIMO allows multiple users to simultaneously access the same wireless channel by providing spatial degrees of freedom, albeit only in the downstream direction. In addition, the testing suite will also be applicable to WiGig, a future wireless standard that promises high speeds over short distances on the unlicensed 60GHz band.
Helping users choose a wireless AC USB adapter starts with a comprehensive set of benchmarks, accurately measured across multiple distances and on both the 2.4GHz and 5GHz frequencies. This is supported by looking at the out-of-box experience, the setup process, the stability and the software. Taking the devices apart provides another level of analysis. This is all done so that when we recommend a wireless AC USB adapter to our readers, we have full confidence that that recommendation will be the best one to suit their needs, while steering them clear of hardware with issues.
Jonas DeMuro is an Associate Contributing Writer for Tom's Hardware.