MP3s don’t matter (until they do)

Standard

I’ve written before on some of the differences in MP3s vs WAVs, specifically how MP3s seem to invoke more negativity than WAVs in a blind test. I don’t know about you, but I thought those results were interesting and weird. So, I thought it made sense to kind of zoom out and try and get a bigger picture of this phenomenon.

A logical first step was to ask “Can people even hear the difference between WAVs and MP3s in their day-to-day life? If so, in what circumstances?” As the title implies, people generally can’t tell in most circumstances but once they do, it is a very pronounced shift.

The Experiment

I made an online experiment, asking people to listen to 16 different pairs of song segments and select the one they thought sounded better. There were 4 levels of MP3 compression: 320k, 192k, 128k, and 64k.

‘Why those levels of compression?’ you might be wondering. Amazon and Tidal deliver at 320k, Spotify premium does 192k, YouTube does 128k, and Pandora’s free streaming is 64k.

For each pair, one version of the segment was a WAV and the other was an MP3. (See below for more detail.) I also asked basic demographic information and how they usually listen to music and how they were listening to the experiment. For example, a lot of people use Spotify regularly for music listening on their phones, and a lot of people used their phones to do the experiment. Doing the experiment gave up a lot of control over how and where people listened, but the goal was to capture a realistic listening environment.

The Songs

I selected songs that are generally considered to be good recordings capable of offering a kind of audiophile experience. Also, I tried to choose “brighter” sounding recordings because they are particularly susceptible to MP3 artifacts. The thought behind this was to maximize the chance for identification of sonic differences, because I was doubtful there would be any difference until a very high level of compression.

I also split the songs into eras: Pre and Post MP3. I thought that maybe music production techniques might change to accommodate the MP3 medium, and maybe MP3s would be easier to detect in recordings that were not conceived for the medium.

The Song List by Era

Pre MP3 (pre 1993):

  1. David Bowie – Golden Years (1999 remaster)
  2. NIN – Terrible Lie
  3. Cowboy Junkies – Sweet Jane
  4. U2 – With Or Without You
  5. Lou Reed – Underneath the Bottle
  6. Lou Reed & John Cale – Style It Takes
  7. Yes – You and I
  8. Pink Floyd – Time

Post MP3:

  1. Buena Vista Social Club – Chan Chan
  2. Lou Reed – Future Farmers of America
  3. Air – Tropical Disease
  4. David Bowie – Battle for Britain
  5. Squarepusher – Ultravisitor
  6. The Flaming Lips – Race for the Prize
  7. Daft Punk – Giving Life Back to Music
  8. Nick Cave & The Bad Seeds – Jesus Alone

The Song List by Compression Level

320k

  1. Cowboy Junkies – Sweet Jane
  2. Lou Reed – Underneath the Bottle
  3. Squarepusher – Ultravisitor
  4. Daft Punk – Giving Life Back to Music

192k

  1. David Bowie – Golden Years (1999 remaster)
  2. NIN – Terrible Lie
  3. The Flaming Lips – Race for the Prize
  4. Air – Tropical Disease

128k

  1. U2 – With Or Without You
  2. Lou Reed & John Cale – Style It Takes
  3. Buena Vista Social Club – Chan Chan
  4. Nick Cave & The Bad Seeds – Jesus Alone

64k

  1. Pink Floyd – Time
  2. Bowie – Battle for Britain
  3. Lou Reed – Future Farmers of America
  4. Yes – You and I

The Participants

I had a total of 17 participants complete the experiment (and 1 more do part of the listening task) and a whole lot of bogus entries by bots…. sigh. Here’s some info on the real humans that did the experiment:

Pie Charts2.png

Note: options with 0 responses are not shown

Pie Charts3.png

Pie Charts4.png

“Which best describes your favorite way to listen to music that you have regular access to?” was the full question. I didn’t want everyone to think back to that one time they heard a really nice stereo!

Pie Charts5.png

Pie Charts6.png

Pie Charts7.png

“This includes informal or self-taught training. Examples of this include – but are not limited to – musicians, audio engineers, and audiophiles.”

 

Unfortunately, the sample size wasn’t big enough to do any interesting statistical analyses with this demographic info, but it’s still informative to help understand who created this data set.

The Results

Participants reliably (meaning, a statistically significant binomial test) selected WAVs as higher fidelity when the MP3s were 64k. Other than that, there was no statistical difference.

OUTPUT.png

OUTPUT1.png

OUTPUT2.png

OUTPUT3.png

11 to 57 in favor of WAV, p <0.001

When I first looked at the Pre/Post MP3 comparison, I was flummoxed. There is a statistical difference in the Post MP3 category… favoring WAVs.

866

That’s pretty counter-intuitive. That would be like finding that people preferred listening to the Beatles on CD instead of vinyl. It just doesn’t make sense. Why would recordings sound worse in the new hip medium that everyone’s using?

They don’t. My categorization was clumsy. So, yes, I selected 8 songs that were recorded after MP3s were invented, but what I didn’t consider is that the MP3 was not a cultural force until about a decade later, and not a force in the music industry until later than that even. So I went back and looked at just the Post MP3 category and split it again. Figuring out when the MP3 because a major force in the recording industry was a rabbit hole I didn’t want to go down, so I used a proxy: Jonathan Sterne, a scholar who looks at recording technology, published an article in 2006 discussing the MP3 as a cultural artifact. And luckily enough, using 2006 ended up being fruitful because of my 8 songs in the Post MP3 category, none were released on or even near 2006. I had 5 released before and 3 released after, and when I analyzed those groups, there was a strong preference for WAV in the older recordings but not in the newest recordings. This suggests that yes, recordings, after a certain date, are generally recorded to sound just as good as MP3s of a certain quality or WAVs. Here’s the analysis:

better-mp31

25 to 60 in favor of WAV, p < 0.001

 

better-mp3

So, to sum up: the debate between WAV and MP3 doesn’t matter in terms of identifying fidelity differences in real world situations for these participants UNTIL the compression levels are extreme. And, recordings designed for CDs and not MP3s sound better on CDs than MP3s, but it doesn’t matter for older recordings. If I had to guess it could be because some of the limitations of the vinyl medium are similar to MP3 (gasp! Heresy!) and so recordings designed for vinyl work kinda well as MP3s, too.

Beauty and the Beast: is there any difference between listening to MP3 vs CD quality?

Standard

TL;DR: yes. But come on! There’s a bunch of graphs and some lame jokes if you actually read the post.

Preface

As I sit here at my desk, I am surrounded by audio equipment and CDs. Spotify is open right now (streaming quality set to “Extreme,” thank you very much). My favorite pair of headphones are within arm’s reach. My studio monitors are effortlessly reproducing a lovely Terry Riley piece. Clearly, I am spoiled. But wait, let’s rewind a moment: I’ve got a stack of CD’s next to me, but I’m streaming compressed audio when I could be enjoying clean, uncompressed audio from my CDs? Why would I do that? (I also have a record player and a few choice vinyls, but it’s an obviously inferior format to CD so it’s not part of the comparison.)

I do it because it’s convenient. And there’s a massive amount of diversity on Spotify that simply isn’t legally accessible to me given my grad student budget. And I’m not alone: a whole heck of a lot of people in the US use streaming services. But all of them, save one, stream in what’s called lossy formats. In fact, other than listening to a CD or vinyl, the music you listen to is probably in a lossy format. It means the previously uncompressed and pristine digital audio of a CD is reduced not just in file size, but in information it contains.WAVs, by comparison, are lossless. It’s kind of bonkers to think, but MP3s and other lossy formats throw away a LOT of sound. That’s partially why they’re so small. The goal, of course, is to only throw away things you can’t hear.

It might sound kind of like science fiction (or the fantasy of scared parents of metal fans): unheard sounds in recordings? It’s true, though. In fact, our cognitive systems are really excellent at filtering out unwanted noise. It’s called the cocktail party effect. So why not automate the process and only save the parts that we hear anyway? It might not be that simple. I, along with a classmate and our advisor, decided to test if there was a difference in the subjective enjoyment of music listening between WAVs and MP3s.

The Experiment

We selected eight songs: four recorded before MP3s were even a glimmer in the Fraunhofer Institute’s eye, and four very recent songs. We did this because there’s an idea floating around in audio engineering and audiophile circles that, for example, the Beatles sound better on vinyl than CD because the albums were recorded for the idiosyncrasies of vinyl in mind. The easiest way to control for this was to have two “early” songs and two “recent” songs as MP3 and another set of two and two as WAVs.

The Song List

  • Aretha Franklin – RESPECT
  • Michael Jackson – Thriller *
  • The Eagles – Hotel California
  • The Beatles – Help! *
  • Carly Rae Jepsen – Call Me Maybe
  • Sia – Chandelier *
  • Rihanna – We Found Love
  • Daft Punk – Get Lucky *

* = MP3, 128k, LAME encoder

Note: the oldest available CD mastering was used for the pre-MP3 songs to eliminate / reduce the chance that some modern mastering techniques would be used to make it more MP3 friendly. For example, “Hotel California” was sourced from the original CD release in 1989.

We had people come in, put on headphones we provide them with, and listen to all 8 songs presented to each person in a random order. After each song, they would rate how positive it made them feel, how negative it made them feel, and how much they enjoyed it. The reason we asked positive and negative separately is because we conceptualize those feelings as representing activations of appetitive or aversive systems, respectively. They can activate separately or they can activate together.

Keep in mind, we told the participants nothing about the sound quality, MP3s or WAVs. As far as they knew, they just had to listen to 8 songs and respond to those 3 questions for each.

Results

I instigated this experiment because I didn’t think there would be a difference. We ended up hypothesizing that there would be a difference between the formats, such that people would like WAVs more. But to be honest I was skeptical, even if I had a theory-driven rationalization as to why I thought it would come out this way. (More on that later.) I thought people might even prefer MP3s since our participants are young and have probably been listening to MP3s their whole lives, give or take.

H1 figure.png

F(1, 17) = 2.162, p = 0.16

The graph above shows the mean positivity results by Format. It’s not statistically significant, but it is in the direction we predicted. Admittedly, this one result alone isn’t convincing. But wait — there’s more!

H2 figure.png

F(1, 17) = 5.224, p < 0.05

And this is a prime example of why we split out positivity and negativity into two measurements: the negative scores are significant, and support our hypothesis that people would like MP3s less.

H3 figure.png

F(1, 17) = 1.7, = 0.21

Again, not statistically significant findings here but the data are trending in the direction we predicted.

RQ1 figure.png

F(1,17) = 5.285, p < 0.05

And here’s the kicker: people rated early era songs as MP3s more negatively than anything else. And this finding is statistically significant.

Discussion

So what gives? Well, it could be as simple as our participants just hated “Thriller” and “Help!” as songs. But more than they hated The Eagles‘ “Hotel California?” I sincerely doubt it. But it is possible, I’ll admit that openly.

Here’s what I think went on, though: remember how I said that MP3s strip out a lot of information, most of which you can’t hear anyway? I bet that process is flawed. It clearly works very well, but I bet that it is imperfect and listening to MP3s is actually MORE work for your brain than uncompressed audio (like WAVs). Our minds are very lazy and, under most circumstances, seek the path of least resistance when hit with a task. If MP3s tax the cognitive systems more than WAVs because we need to actively fill in some of the missing gaps or work harder to do our usual filtering, then it seems logical that we would rate the experience more negatively.

Moving Forward

This study isn’t perfect. I would prefer to have run it with a counterbalanced design where some participants heard Song A as MP3 and others heard Song A as a WAV. That would help remove unwanted effects of the song itself. That, and while I have some ideas as to why these results came about, this experiment doesn’t prove or even directly support my ideas. I need more information before I can put that claim forward more strongly.

The good news is that we have a lot more research in the pipeline regarding audio compression and how it impacts the listening experience.