Skip to contents
Hot Pod Insider

The Challenge of Measuring — and Contextualizing — Performance

Downloads, Success, and Who Are We Supposed To Trust

First week back, and I am indeed a little rusty! One clarification from Tuesday’s newsletter: my bad on mis-remembering the supposed rollout of the new Apple in-episode analytics were. I had written down in my notes that it was pegged to the iOS11 update — turns out that there were no explicit mention of a date from the company, and that the timeline was broadly “later this year.” From what I’ve been hearing, the holding position is actually pegged to the question of iOS11 adoption, and that a critical mass of adopters needs to be achieved before a proper analytics story can be consistently and effectively produced.

We’re going to stick to this theme of analytics and a proper story for this week’s members newsletter. It’s based on something I’ve been thinking about for a very long time now, and it’s something that I’ve been very frustrated with for a very long time.

Non-narratives. Someone really high up in a podcast company, who I happen to think is very smart, once opined out loud to me, “The whole conversation around podcast measurement really frustrates me — has anybody even seen how bad radio metrics are?” Indeed, I share the frazzled sentiment, but I also think it’s besides the point. The quantitative, evaluative assessment of a piece of media (or anything else, really) is, in part, an act of collective social fiction; we take measurement numbers to be a tool that gets us closer to some approximate understanding about the nature or performance of a thing. Those measurement numbers will always be a subject of debate and iteration, and we’re seeing that play out in bits and pieces: with the incoming Apple analytics, with the recent waves of hosting and player analytics adjustments towards IAB standards, with the light push-backs and begrudging acceptances of the visibility of something like the Podtrac industry ranker. But the fact of the matter is, as much as it’s important for those debates and iterations take place, it’s also important that the industry continues to unfold its story. Which is to say, it’s also important to try, today, to create good and trustworthy stories about how podcasts are performing in the marketplace, and how podcasting as an industry is unfolding in the broader media ecosystem.

This week, we saw two chunky pieces by the journalist Ken Doctor on the success of the New York Times’ The Daily — here and here — which I won’t discuss at length here, because I’ve dedicated enough digital ink to that show in this here members’ newsletter. Rather, I’d like to draw your attention to this tweet reacting to those pieces, by the media reporter and critic Mathew Ingram (formerly of Fortune, now at the Columbia Journalism Review), that caught my eye:

Ingram’s observation here drew an instant reactionary “no duh” response from me internally, but I checked myself, because this is one of those situations where it behooves me to remember that not everybody — in fact, very few people outside of myself — actually spends the time to obsessively research, dig around, and think about the podcast industry and its many immature but evolving layers. And so I proceeded to find myself slumping into a light malaise: here we are again, confronted with yet another piece of evidence that the meta-narrative of the podcast is once again trapped within the confines of what is unambiguously defining trait, that of the inadequacies of its current means of identity and public evaluation, the download.

Still, I am nonetheless drawn to this notion of “judging the success of a podcast” — because, in many ways, you could argue that has nothing at all to do specifically with downloads. That, I’d argue, has everything to do with whether the industry is in a good enough position to tell a contiguous, coherent, and credible story about itself through various structures of accessible, reliable public information.

I’ve written a little about this before, but it remains, to this date, that there are very few reliable public sources of information and value indicators for the podcast industry with which we can use to attribute a sense of success to a show. Indeed, we have some data sources, of course, but they are all ridden with caveats, and they all don’t fit together very well. To wit, we have the Apple Podcast charts (caveats caveats caveats), the Podtrac industry ranker (caveats caveats caveats), miscellaneous Best Of lists and reviews and glowing recommendation write-ups (the primary caveat being that it’s all subjective), but the industry still lacks a centralized, easily communicable capacity to tell a singular story. (Someone once raised the importance of awards rituals for this to me— rituals that have sufficiently accumulate the clout to matter in some shape or form to they various constituencies key to the health and conduct of the industry. We don’t have that yet, I’d argue.) Instead, what we have is an environment where every individual story by an individual publisher or network is fighting with brute force to promote itself independent of the context of the larger ecosystem. Which is to say, it’s one thing to claim that your podcast has garnered, say, 1 million downloads across six episodes in a period of a month in a vacuum; it’s another thing to contextualize that number in relational terms. How many other shows have pulled off that achievement? Is that small, is that big? Is that perfectly normal, is that outsized? And perhaps most importantly, how many caveats and massaging do we need to deploy in order to even attempt a contextualization of so and so show against other shows within the genre? Is this something that we can do?

Very occasionally, we get a truly edge case that indisputably makes the argument for its complete dominance by virtue of it being an edge case. The obvious example here would be S-Town, which broke 40 million global downloads in under 30 days across six episodes. Its claim towards its significance is instantly clear, even though we can still hit that metric with a bunch of caveats: global vs. US. the fact that it was a binge-drop, etc. etc. But as much as that particular edge case is prominent, we need to be able to easily engage in the act of evaluating and contextualizing more mundane or ordinary releases. Only then can we start accumulating a good sense of data, and in doing so, lay down the foundations for a clear sense of a track record.

The point here is simple: it’s not enough to know how many downloads — or listens or impressions or whatever — a show garners. We need enough to know what it means. The industry needs a reliable paradigm to be accurately, consistently, and fairly judged. This problem, I’d argue, is an existential one for the industry, because without the production of these stories through reliable numbers and contextual meaning, we’re losing the opportunity to create a foundational body of history about the industry and ecosystem with every passing day.

There are certain processes taking place as I write this newsletter that are not-so-slowly and gradually prodding the ecosystem towards a unified system of comparable metrics: the aforementioned shifts towards the IAB standards, the expectation of the new analytics, the slow alignment between all the various parties, and so on. I’ll be writing about those with some depth in next week’s issue. That said, I can’t help but think that we need to do a whole lot more right now to start pumping out an immediate framework in which we can chalk up wins. So far, I’ve been running short items printing out download numbers  as reported to me directly by publishers themselves, but I’ve always felt squeamish doing that due to the very basic questions of accountability (E.g. can you really trust these numbers without a strong third party? Can I actually extract numbers from publishers if the shows didn’t end up being as strong as they were hoping? Those numbers need to be out there, even if underwhelming). I’ll still source and print those babies out, of course, because we still need to keep the ball rolling on even that kind of data. But I’m thinking: there must be a better way for right now. There has to be.

Anyway. That’s what’s on my mind these days. I hope you’re winding down well from the week.

Side note: Let me echo Ingram by saying that this isn’t to impugn The Daily or the New York Times’ publicizing the fact the podcast has beaten 100M downloads. Furthermore, this is not to dispute that achievement — The Daily, I think, is a remarkable step forward for the medium, and that it’s beaten 100M downloads in about 180~ editions is a testament to a news product well developed and well executed.