How We Test Headphones
See How We Test Other Products:
Here at Reviewed, we've a decade. Over time, we've refined our methods, working with several partners and inventing some of our own tests from scratch to produce the most accurate, data-driven reviews of headphones around.
While many other testers rely on ‘golden ears’ (those with good hearing who subjectively judge headphones), we use proven scientific test methods developed by ourselves as well as industry experts. This allows us to quantify headphone performance, giving us reliable, repeatable results.
All headphones are tested in our Cambridge, MA labs, in a consistent, climate-controlled room. In accordance with our ethics policy we also believe it's important to be transparent with both our methodology and the equipment we use. Though our scoring model is proprietary for the purposes of preserving a level playing field amongst all manufacturers, below you can find an outline of all of our testing procedures and equipment.
SoundCheck, from Listen, Inc. is an electroacoustic measurement and analysis package widely used for testing audio devices, both on the production line and in R&D applications. It is a PC—and sound card—based system, which communicates with both analog and digital equipment using standard, non-proprietary interfaces.
The Brüel & Kjær hardware (see below) is directly controlled through SoundCheck, with the test signals passing from the computer to a digital amplifier/modulator. Tests are fully customizable, enabling the sound signal, various methods of analysis, and result output format to be selected.
Head and Torso Simulator
The Head and Torso Simulator (HATS for short), is a semi-robotic human analog with built-in ear and mouth simulators that provide a realistic reproduction of the acoustic properties of an average adult human head and torso. In addition to sensitive recording equipment, HATS even has detachable pinna (the visible part of your ear that acts to gather sound), designed to mimic a real human's ear as closely as possible.
To test, we mount the headphones on HATS the same way you'd put headphones on your own head. However, we use the SoundCheck software to determine the perfect fit before testing. The sensitive microphones in HATS pick up the sound from the headphones and feed it back to the SoundCheck system through two high-end microphone amplifiers. The entire system is calibrated and tested on a regular basis to ensure accurate, consistent results.
The BQC-4148 is a module that allows us to interface with HATS and perform our tests wirelessly, over Bluetooth. Instead of passing the test audio signals through the amplifier to the headphones, it goes directly from the computer, to the BQC-4148, to the Bluetooth headphones.
Most smartphones and laptops will have a variety of different Bluetooth profiles and codecs, and this unit allows us to control for that. Working in conjunction with SoundCheck, we use the BQC-4148 to reliably test and analyze audio performance of headphones using Bluetooth wireless technology. While many headphones offer codecs such as APT-X, we currently test using standard Bluetooth 4.0 protocols that represent most people's setups.
Our objective audio tests are broken down into several categories, including: frequency response, isolation, distortion, tracking, and leakage. While not all tests are given equal weight, we specifically try to reward products that excel in all of these areas.
Frequency response describes how a set of headphones emphasizes different notes across the audible spectrum. To test this, we first put the headphones on HATS, and make sure they have a proper fit. We then use SoundCheck to send a frequency sweep through the headphones, which covers a range from 20 Hz to 20,000Hz. This frequency range covers all the frequencies the human ear can hear as a young child; the older you are, the less you can hear higher-frequency sound. We use 20,000 Hz as an upper limit to cover most potential listeners, though we only score up to 12.5kHz.
First, the sound is played through the headphones and into HATS's precision microphone ears. The microphones inside HATS record the playback and sends the data back to SoundCheck. We then compare the original sound file to what HATS recorded to determine how the headphones have altered the original sound file.
We plot the responses from each ear to create a graph such as the one below, which shows the frequency response of the headphones from low notes (at the left) to high notes (at the right). The response curve we feature includes a diffuse field correction and HRT, which corrects for the effects of the head, torso, pinna and ear canal on the response curve.
There is no single ideal response for headphones, but there are industry standards for different types of headphones. For consumer headphones like you'd use on the subway or out on the town, we compare the frequency response against the ISO 226:2003 "equal loudness" curve—the curve at which most human ears hear notes at the same loudness—with the least overall deviation. While many people prefer non-standard curves that derive a more customized sound, we have found over years of testing that headphones calibrated to the ISO 226:2003 standard work well across the widest variety of genres. The headphone companies seem to agree, since most headphones we test follow this guideline somewhat.
For studio headphones, we set a range of +/- 5 dB(SPL), and our testing system puts these limits against the response curve of the headphones. Our scores are then based on how much the headphones go outside these limits, so we are not scoring on the exact curve, but rather on the smoothness of the curves. Although headphones are almost always a matter of personal taste, a good pair of headphones should produce a good, clean frequency response that does not overly exaggerate or diminish nearby frequencies. In short, we're looking for the headphones to accurately reproduce the music you're putting through them.
Distortion is a measure of how accurately the headphones reproduce the sounds that make up the music you listen to; if they clip, compress, or otherwise mangle the signal, your music won't sound the way it should. Using the SoundCheck system, we measure the Total Harmonic Distortion (THD), which is a measure of not just the distortion at the fundamental level, but at every harmonic as well. A decent set of headphones should be able to play the music stored on your computer or music device without any noticeable buzzing or clipping.
We measure the THD of headphones using the same sample sound as the one in the frequency response test, and it produces a graph like the one below. Here, you can see the distortion for both channels averaged as the red line. Big peaks in this graph indicate that the headphones have a problem with the harmonics of that frequency; notes at that frequency may sound buzzy or rough, like a distorted guitar. THD is measured as a percentage, indicating how much the original sound is distorted. Lower is better here; the closer each line is to 0, the less distortion will impact your music.
Our distortion score is based on the total amount of distortion across all frequencies recorded over our tolerance cutoff line; the higher the level, the lower the score. In the complex assortment of frequencies that make up your favorite music, you are very unlikely to be able to detect low levels of distortion—especially in lower frequencies. This is why any levels recorded under the blue tolerance cutoff line are not penalized.
Headphones have two channels, one for each ear. Both of these channels should sound the same, but we test these channels, just to make sure. Our testing system analyzes the sound produced by both channels, and creates a graph to show any differences between the two as a percentage of the frequency response, measured from 20 Hz to 10,000 Hz.
If both channels were producing the same sound, the red line would have a value of zero. When the left channel is louder, the red line will rise above zero; when the right channel is louder, the red line will dip below zero. A difference of 1-2% isn't noticeable, and generally, we don't expect to see anything above 4-5%. Tracking differences larger than 4-5% could cause noticeable and unpleasant-sounding channel shifts.
Our score for this test is based on the total imbalance between the two channels above +/- 2dB; the more the line bounces outside the blue area on the graph, the lower the score.
In many situations—especially if you're looking for portable headphones—you'll want to block out the noise of the outside world. Public transit, air travel, traffic noise, or obnoxious roommates can be distracting and interrupt your music. Therefore, we test how much external noise the headphones can block out. We do this by placing a speaker next to the HATS and playing pink noise, which has equal power across the entire frequency range.
SoundCheck measures and records the noise levels. First, we aim the speakers playing the pink noise at HATS, who is initially not wearing headphones. Then, we next put the headphones on HATS and play the pink noise again. Lastly, we run the test again with active noise cancellation turned on (if applicable). Once we're done, SoundCheck compares the three curves: without headphones, with headphones, and with active noise cancellation on (if the headphones have this feature). This produces a graph that looks something like this:
This shows how much of the sound was blocked; the higher the line, the more sound was prevented from reaching the ears. In the above plot, Passive and Active Attenuation plot right on top of one another. In the case of headphones that include active noise cancellation (ANC), we are able to tease out exactly how much noise ANC can actually eliminate.
Our score is based on how well the headphones block the noise; the more they block, the better the score. With active noise canceling headphones, we use whichever test provides the best results as the basis for our score. The best results are almost invariably with the active cancellation enabled, but the feature often creates a lot of distortion in your music.
If you're trying to block out the world by listening to music, it's only polite to return the favor; your fellow commuters want to hear their own conversations, not your music. Leakage is also an important figure to know if you're planning on being in a quiet public place, like a library or museum.
We use a sound meter, placed six inches away from HATS, to measure the ambient noise of the room when everything is quiet. Next, we measure the sound level with 90 dB(SPL) of pink noise playing back through the headphones. We then calculate the difference between these two levels; the more sound the headphones leak, the lower the score.
In addition to our rigorous sound tests, we also use the headphones casually in our everyday lives; this way, we can tell our readers about some of the more intangible aspects that come with owning a pair of headphones. How comfortable are they to wear for 10 minutes? What about for an hour? If they’re wireless, what kind of codec do they have? Can the headphones collapse into a more portable container? We answer all these questions and more in an effort to give consumers an idea of the full experience, both musical and not, they are in for if they buy a pair of headphones.
Want to find out about the latest headphones we’ve reviewed? Be sure to check out our library of headphone reviews. If you want to read about our favorite headphones, head on over to the Best Right Now page, where you’ll find our top picks in many headphone categories, from Best Wireless Earbuds to Best Headphones for Kids, to the Best Gaming Headphones, and back again.
To develop our testing methodology we are constantly looking across the industry at the current trade standards, the methods of our colleagues, and the enterprising work done by other sites and testing outfits. Though we've never hesitated to develop our own methodologies and equipment where we had to, we're also grateful for the assistance of several partners who have helped us in developing one of the most complete, rigorous, and scientific testing models in the industry. We'd like to offer special thanks to the following groups who have been essential in helping us achieve our goal.
ListenInc, headquartered in Boston, MA was founded in 1995, and offers leading electroacoustic analysis solutions. The company's flagship product, SoundCheck, is used by manufacturers and reviewers alike to test headphones and other sound equipment. Listen, Inc is dedicated to research and development, often presenting white papers on new measurement techniques and standards at conferences.
Brüel & Kjær
Brüel & Kjær makes industry leading sound and vibration measurement equipment used by car manufacturers, cell carriers, and audio equipment manufacturers. In business since 1943, Brüel & Kjær is noted for its inventions, including the first charge accelerometer and the first Airport Noise Monitoring System. Brüel & Kjær is a subsidiary of UK-based Spectris plc.