How We Test Headphones
See How We Test Other Products:
Here at Reviewed.com, we've been testing headphones for nearly a decade. Over that time we've refined our methods, working with several partners and inventing some of our own tests from scratch to produce the most accurate, data-driven reviews of headphones around.
While many other testers rely on ‘golden ears’ (those with good hearing who subjectively judge headphones), we use proven scientific test methods developed by ourselves as well as industry experts. This allows us to quantify headphone performance, giving us reliable, repeatable results.
All headphones are tested in our Cambridge, MA labs, in a consistent, climate-controlled room. In accordance with our ethics policy we also believe it's important to be transparent with both our methodology and the equipment we use. Though our scoring model is proprietary for the purposes of preserving a level playing field amongst all manufacturers, below you can find an outline of all of our testing procedures and equipment.
SoundCheck, from Listen, Inc. is an electroacoustic measurement and analysis package widely used for testing audio devices, both on the production line and in R&D applications. It is a PC—and sound card—based system, which communicates with both analog and digital equipment using standard, non-proprietary interfaces.
The Brüel & Kjær hardware (see below) is directly controlled through SoundCheck, with the test signals passing from the soundcard to the high-end audio amplifier we use to drive the headphones (a Crown D45). Tests are fully programmable, enabling the sound signal, various methods of analysis and result output format to be selected.
Head and Torso Simulator
The Head and Torso Simulator (HATS for short), is a semi-robotic human analog with built-in ear and mouth simulators that provide a realistic reproduction of the acoustic properties of an average adult human head and torso. In addition to sensitive recording equipment, HATS even has detachable pinna (the visible part of your ear that acts to gather sound), designed to mimic a real human's ear as closely as possible.
To test, we mount the headphones on HATS the same way you'd put headphones on your own head. However, we use the SoundCheck software to determine the perfect fit before testing. The sensitive microphones in HATS pick up the sound from the headphones and feed it back to the SoundCheck system through two high-end microphone amplifiers. The entire system is calibrated and tested on a regular basis to ensure accurate, consistent results.
The BQC-4148 is a module that allows us to interface with HATS and perform our tests wirelessly, over Bluetooth. Instead of passing the test audio signals through the amplifier to the headphones, it goes directly from the computer, to the BQC-4148, to the Bluetooth headphones.
Most phones will have a variety of different Bluetooth profiles and codecs, and this unit allows us to control for that. Working in conjunction with SoundCheck, we use the BQC-4148 to reliably test and analyze audio performance of headphones using Bluetooth wireless technology. While many headphones offer codecs such as APT-X, we currently test using standard Bluetooth 4.0 protocols that represent most people's setups.
Our objective audio tests are broken down into several categories, including: frequency response, isolation, distortion, tracking, sound pressure level, and leakage. While not all tests are given equal weight, we specifically try to reward products that excel in all of our these areas.
Frequency response describes how a set of headphones emphasizes different notes across the audible spectrum. To test this, we first put the headphones on HATS, and make sure they have a proper fit. We then use SoundCheck to send a frequency sweep through the headphones, which covers a range from 20 to 20,000Hz. This frequency range covers all the frequencies the human ear can hear as a young child; the older you are, the less you can hear higher-frequency sound. We use 20kHz as an upper limit to cover most potential listeners, though we only score up to 12.5kHz.
First, the sound is played through the headphones and into HATS's precision microphone ears. HATS records the playback and sends the data back to SoundCheck. We then can compare the original sound file to what HATS recorded to determine how the headphones have altered the original sound file.
We plot the responses from each ear to create a graph such as the one below, which shows the frequency response of the headphones from low notes (at the left) to high notes (at the right). The response curve we feature includes a diffuse field correction and HRT, which corrects for the effects of the head, torso, pinna and ear canal on the response curve.
There is no single ideal response for headphones, but there are industry standards for different types of headphones. For consumer headphones like you'd use on the subway or out on the town, we compare the frequency response against the ISO 226:2003 "equal loudness" curve—the curve at which most human ears hear notes at the same loudness—with the least overall deviation. While many people prefer non-standard curves that derive a more customized sound, we have found over years of testing that headphones calibrated to the ISO 226:2003 standard work well across the widest variety of genres. The headphone companies seem to agree, since most headphones we test follow this guideline somewhat.
For studio headphones, we set a range of +/- 5 dB(SPL), and our testing system puts these limits against the response curve of the headphones. Our scores are then based on how much the headphones go outside these limits, so we are not scoring on the exact curve, but rather on the smoothness of the curves. Although headphones are almost always a matter of personal taste, a good pair of headphones should produce a good, clean frequency response that does not overly exaggerate or diminish nearby frequencies. In short, we're looking for the headphones to accurately reproduce the music you're putting through them.
Distortion is a measure of how accurately the headphones reproduce the sounds that make up the music you listen to; if they clip, compress, or otherwise mangle the signal, your music won't sound the way it should. Using the SoundCheck system, we measure the Total Harmonic Distortion (THD), which is a measure of not just the distortion at the fundamental level, but at every harmonic as well. A decent set of headphones should be able to accurately reproduce both the fundamental frequency and the harmonics at the same level as they are present in the original sound.
We measure the THD of headphones using the same recorded sample sound as the frequency response test, and it produces a graph like the one below. Here, you can see the distortion for both channels averaged as the red line. Big peaks in this graph indicate that the headphones are have a problem with the harmonics of that frequency; notes at that frequency may sound buzzy or rough, like a distorted guitar. THD is measured as a percentage, indicating how much the original sound is distorted. Lower is better here; the closer each line is to 0, the less distortion will impact your music.
Our distortion score is based on the total amount of distortion across all frequencies recorded over our tolerance cutoff line; the higher the level, the lower the score. In the complex assortment of frequencies that make up your favorite music, you are very unlikely to be able to detect low levels of distortion—especially in lower frequencies. This is why any levels recorded under the blue tolerance cutoff line are not penalized.
Headphones have two channels, one for each ear. Both of these channels should sound the same, so we test this to make sure. Our testing system analyzes the sound both channels produce and creates a graph to show any differences between the two as a percentage of the frequency response, measured from 20 to 10,000Hz.
If both channels were producing the same sound, the red line would stay along the zero line. When the left channel is louder, the red line will rise above zero; when the right channel is louder, the red line will dip below zero. A difference of a couple of percentage points here isn't noticeable, but we don't expect to see anything above 4-5%; that could make unpleasant-sounding channel shifts.
Our score for this test is based on the total imbalance between the two channels above +/- 2dB; the more the line bounces outside the blue area on the graph, the lower the score.
In this test, we examine how high we can increase the volume on the headphones before they reach a peak distortion (THD) of 3%. The SoundCheck test sequence gradually increases the level of the stimulus and analyzes the distortion until it reaches this threshold, or the level of the sound reaches 120 dB(SPL). We don't test any higher than 120 dBSPL, because anything higher could cause permanent hearing damage. We also don't recommend listening at 120dB, for the record.
Isolation / Attenuation
In many situations—especially if you're looking for portable headphones—you'll want to block out the noise of the outside world. Public transit, air travel, traffic noise, or obnoxious roommates can be distracting and interrupt your music. Therefore, we test how much external noise the headphones can block out. We do this by placing a speaker next to the HATS and playing pink noise, which has equal power across the entire frequency range.
SoundCheck measures and records the noise levels. We next put the headphones on HATS and blast both with noise again. Afterwards, we run the test again with active noise cancellation turned on (if applicable). Once we're done, SoundCheck compares the three curves: without headphones, with headphones, and with active noise cancellation on (if the headphones have this feature). This produces a graph that looks something like this:
This shows how much of the sound was blocked; the higher the line, the more sound was prevented from reaching the ears.
In the case of headphones that include active noise cancellation (ANC), we feature a graph with two lines: one for the active noise cancellation disabled (green) and one for the noise canceling enabled (blue).
Our score is based on how well the headphones block the noise; the more they block, the better the score. With active noise canceling headphones, we use whichever test provides the best results as the basis for our score. The best results are almost invariably with the active cancellation enabled, but the feature often creates a lot of distortion in your music.
If you're trying to block out the world by listening to music, it's only polite to return the favor; the person sitting next to you on the bus wants to hear their own conversation, not your music. Leakage is also an important figure to know if you're planning on being in a quiet public place, like a library or museum.
Our test uses a sensitive microphone, placed six inches away from HATS. We first take a measurement of the ambient noise of the room. Next, we measure the sound level with 90 dBSPL of pink noise playing back through the headphones. We then calculate the difference between these two levels; the more sound the headphones leak, the lower the score.
To develop our testing methodology we are constantly looking across the industry at the current trade standards, the methods of our colleagues, and the enterprising work done by other sites and testing outfits. Though we've never hesitated to develop our own methodologies and equipment where we had to, we're also grateful for the assistance of several partners who have helped us in developing one of the most complete, rigorous, and scientific testing models in the industry. We'd like to offer special thanks to the following groups who have been essential in helping us achieve our goal.
ListenInc, headquartered in Boston, MA was founded in 1995, and offers leading electroacoustic analysis solutions. The company's flagship product, SoundCheck, is used by manufacturers and reviewers alike to test headphones and other sound equipment. ListenInc is dedicated to research and development, often presenting white papers on new measurement techniques and standards at conferences.
Brüel & Kjær
Brüel & Kjær makes industry leading sound and vibration measurement equipment used by car manufacturers, cell carriers, and audio equipment manufacturers. In business since 1943, Brüel & Kjær is noted for its inventions, including the first charge accelerometer and the first Airport Noise Monitoring System. Brüel & Kjær is a subsidiary of UK-based Spectris plc.