Bias in the Anime Music Tournament

The Anime Blog Anime Music Tournament to crown the best anime song of all time has been going on for a while over at http://animusictourney.wordpress.com/ and we’ve reached the Sweet Sixteen. (You should all check it out, and vote for the songs I like.) Over the course of the tournament, there has been grumbling in the comments to the effect of “people only voted for <song I didn’t like> over <song I like> because of nostalgia [or recency bias, or because they liked the show, or whatever]”.  Sometimes this made sense to me, such as when “Connect” beat “Grey Wednesday”, which I can only attribute to people voting for Madoka Magica over Penguindrum rather than on the merits of the songs themselves.  But it’s easy to fool yourself into thinking the songs you like lost because of bias while the songs you didn’t like lost because they just plain weren’t good songs.  So I decided to sit down with the data and get some hard numbers.

There were three possible sources of bias I decided to examine: voting for the more recent song, voting for the song that came from a better show, or voting for a song you were familiar with over one you weren’t familiar with.  Out of the 240 matches that have been completed in the Animusic Tourney so far:

*In 119 matches the newer song won, while in 119 matches the older song won. (In two matches there was no age difference.)
*In 135 matches the song whose show was rated higher on myanimelist.net won, while in 101 matches the lower-rated show’s song won. (In four matches they were rated the same.)
*In 132 matches the song whose show had more viewers on MAL won, while in 107 matches the song whose show had fewer viewers won. (In one match they had the same number. This was the “Kimi no Shiranai Monogatari” vs. “Staple Stable” matchup.)

So clearly there’s no question of this data showing any recency or nostalgia bias.  Forget no statistically significant difference, there’s no difference at all.  As far as the other two criteria are concerned it seems clear that people are to at least some extent voting for shows instead of songs.  We would only have a 3% chance of seeing results this extreme if the show’s rating were entirely unconnected to the song’s performance in the tournament.  The impact of familiarity is less clear. We would have a 12% chance of seeing results this extreme by pure chance, and to some extent familiarity is probably “piggybacking” off show quality, because people are more likely to watch a show if they hear it’s good.

Just because there is an effect doesn’t necessarily mean that there’s an evil bias at work.  There could legitimately be a connection between the quality of a show and the quality of the show’s music.  For example, one of the things that makes a show good is if it has good songs.  Would Suzumiya Haruhi no Yuuutsu have been as beloved without its incredible concert scene?  (The recent concert scene in White Album was unimpressive in part because its songs weren’t as amazing as “God Knows”.)  It’s nice at least to have some hard numbers to think about though.  And hopefully this will get people to stop whining about “recency bias” and “nostalgia bias”.

EDIT: Based on the comments I decided to test a few more hypotheses.  First, there was some indication that there could be a generation gap. I set the cutoff between the old and the new at 2006, since that was when HD anime started airing – it makes a pretty good break between “old” and “new” anime.  There were 53 cases of an “old” anime beating a “new” anime, and 60 cases of the reverse.  This suggests a possible bias toward the newer generation of anime, but a result like that could show up 57% of the time due to pure chance, so it’s hard to draw conclusions.

Second, there was a suggestion to look at only songs that were a certain distance apart in time, because a song from fall of 2005 beating a song from spring of 2006 doesn’t tell you much about nostalgia bias.  This makes sense to me.  Limiting the comparison to songs with at least 3 years difference has the newer song winning 85 matches compared to the older song winning 76 matches.  This is a bit in favor of the newer song, but again, the odds of such a result happening by pure chance are 42% so who knows.

Data is here if you want to play with it yourself. I corrected a couple errors but they don’t materially change the conclusions.