Skip to contents
Hot Pod Insider

How do we trust the download numbers they give us?

Plus: SiriusXM gets podcast exclusives and a neat show from Seattle

Measurement Credulity

You might have notice in this week’s issue that I’ve begun using new metric in appraising show performance: downloads per episode average, which I’ll be the first to admit is kind of a weird one. I’ve used it a bunch of times in the past, but I specifically used the measure twice on Tuesday:

  • The first when discussing WBUR’s Season Ticket, where I wrote “In its first two weeks, the show received approximately 200,000 downloads across its first 10 dispatches (a 20,000 per episode average).”
  • And the second when assessing The Daily’s 100M milestone, where I wrote “As of the article’s pub date, October 17, The Daily had delivered 186 editions, which means the show has a 530,000~ download per episode average.”

It’s not the cleanest measure, and of course it’s by no means the “one metric to rule them all” (it shouldn’t even be anybody’s so-called “north star metric”), but the primary reason I deploy it nowadays is simply that it accounts for the medium’s increasingly diversifying levels of production. We have weeklies, we have dailies, we have monthlies, we have short-runs, and we have shows that have been publishing for eternities, and so you can’t just stack them up against each other with broad download reads because high-volume shows have the structural advantage. You can say that a show has reached, say, a half million download milestone — which is a pretty common type of press release, or PR move, that hits my inbox — but given an endless time horizon, any continuously publishing show will at some point beat any milestone.

To control for that, then, it’s important to contextualize the overall download volume within its overall production volume, and as a result we can come with a rough reading of the value for an individual product unit within the context of its production at a given point in time. We’d need to apply more nuances to the measure, like an additional layer to control for shows that are newer versus shows that have been publishing with more history weighing them down, but I think it’s a pretty good start. (I’d love to know what you think, and if there’s a fundamental flaw that I’m not seeing here.)

Now, I’m laying out my process here to illustrate a meta-point about the stuff you’ve been reading here in Hot Pod: I work hard to think through the numbers that I run in the newsletter. Part of my reporting process involves reaching out to publishers directly and asking them for numbers. That move is imbued with a certain good faith in the publisher — at the very least, I’m asking them to not give me a completely fabricated number. And it’s not like I take their numbers at face value. Indeed, I try to qualify and contextualize it as much as I can, talking about where it’s hosted, how many episodes it runs against, evoking how long the timeline is. One of the defining narratives about the space is its lack of analytics, which is really a conversation about its lack of consensus evaluations, and so I do my best to lay down different pieces for the reader to draw their own conclusions about the publisher, their truthfulness, and how they’re packaging their numbers. I take to heart the critic Wesley Morris who, in a Longform interview, once talked about the act of criticism as not just taking what someone is giving you, but to also sauté it, spice it, process and do something more with it. To some extent, I see the mere act of making public what a publisher says about its numbers as putting them in the line of fire: if they’re lying to me, someone reading Hot Pod who knows better will speak, and I’ll take that response and go on from there.

So, I pay attention when I get something like the following from FiveThirtyEight and 30 for 30’s Jody Avirgan:

Avirgan goes on to draw from his experience at the data journalism site: “… I’ve learned this the hard way in my time at 538 — when you put a number to something people just latch on… the solution is not to stop using numbers, it’s better numbers, and we are getting there. Until then there should be asterisks everywhere.”

The natural reflex of my brain, my gut, and my shoulders is to tighten to this, and defensively hold up some internally-held belief that all my efforts at number contextualization in Hot Pod have in some way been sufficient. But whatever my beliefs, Avirgan is and will always be correct: there should always be asterisks everywhere, and then some, because we’re dealing with an eternally moving target — not just in these numbers, but in how these publishers want to be positively positioned in public. And so there should always be further effort in being diligent about finding new ways to express that larger context of a number not the total sum of all representation of a thing. So it is, and so I will continue to try my best.

Nevertheless, it should be noted that Avirgan’s tweet came as a response to another point — or, more accurately, another vent — that I was expressing online: if you want me to write about how well your podcast is doing, maybe don’t be cagey when asked for download numbers? Which, unfortunately, is a fairly common interaction that I field from week to week.

(Interestingly enough, this opened another line of conversation in the mentions: that downloads aren’t everything when it comes to measuring the achievements and health of a podcast. To that I say: no duh. I’m just saying that if you’re trying to say you’ve achieved something, give me something other than your word to work with. Jesus.)

(Also: this entire string of inquiry is somewhat pleasantly related to questions of epistemology, data, and the ultimate knowability of a thing or person or show. Apropos of nothing, I really liked David Fincher’s adaptation of Gone Girl.)

This has been Moderately Defensive but Eventually Repentant Fridays, with your host, Nick Quah.

SiriusXM gets into podcast exclusives, with supply from Feral Audio. Did you know that SiriusXM, the satellite radio company with almost 30M subscribers that’s arguably built on the star-strength of Howard Stern and whose presence in the consumer market is in good chunks tied to its relationship with car companies, has a podcast initiative? And that said podcast initiative comes in the form of an social audio app named Spoke, whose website suspiciously does not have SiriusXM branding on it? (So far that I can spot, anyway.)

Well, a portion of you might know, either because some of you are working with them in some capacity and because I briefly talked about the app in my pre-Sabbatical state of the union. But it’s all been weirdly under the radar, and so it’s interesting to hear that Spoke is apparently in the process of rolling out a new version of the app — and, to go with that, releasing some 12 exclusive episodes from several shows across the Feral Audio network.

Which is to say, it’s yet another instance of windowing, following TuneIn’s dalliances with Gimlet’s Uncivil, Stitcher’s arrangements with its own shows (Heaven’s Gate is dropping early episodes behind the wall), and Spotify doing whatever it’s doing with its original programming slate.

On the broader existence of windowing, I have mixed feelings. On the one hand, the whole trend largely amounts to a Balkanization of the podcast ecosystem, which on the whole isn’t really the most user-friendly outcome for both long-time listeners and newbies. On the other hand, I understand the strategic move to try and build enclosures, and I suppose if you’re something like SiriusXM — whose business is only partially about creative development but mostly about controlling distribution — this would be the card you play. You spend your whole life as a hammer, so you don’t screw: you hammer. On the third hand, I understand the appeal for publishers, who would get upfront cash moneys for agreeing to the arrangement and maybe whatever data you’d get from the platform in case the thing actually takes among listeners… provided, of course, you’re comfortable with the calculus you run of how this might square with the listeners you have and the listeners you want. But on the fourth hand, you have the whole thing about the way the potential rise of these platform plays undermine the fundamental open nature of the ecosystem, which has increasingly proven to suck more than rule on average. (See this Oatmeal comic.) And yet, on the fifth hand, the nihilism of late-stage capitalism.

(Hands? I should’ve said fingers. That’s a better metaphor.)

I think, on the whole, I’m increasingly perturbed by this whole windowing business — even if it does translate to some short-term gains for a few enterprising publishers. The nature of those short-term gains will almost always be tied to a few platforms, aspiring to be centralizing, being in a position of picking and choosing winners, and that might not mean much when said platform is in its early days, but it’ll mean a hell of a lot more should one of these guys accrue sufficient levels of power.

Now, don’t take this to mean that I’m advocating for an ecosystem that’s powered only by advertising or membership/donation support. I’m partial to a world in which podcast paywalls can be a functional revenue channel. I just think that kind of stuff should be localized to the show level, and not on some faceless platform level.

Prime(d). I’m really buzzing on this flash pod that just came out KUOW, the public radio station coming out of Puget Sound, a.k.a. that region that includes the greater Seattle area. (Sorta? Kinda?) The conceit is a reported response to the many, many stories we’re seeing now about various cities vying for the love, attention, and financial bounties of Amazon HQ2; in a sense, it’s a local station telling the story about what has happened to its home that might happen to someone else’s. A kind of distress signal, a PSA and a signal flare. As you know, this whole thing is in the goddamn Venn Diagram of my interests. Local podcasts? The tentacular ambitions of Jeff Bezos? The Pacific Northwest?? CAPITALISM??? Shit, gimme that ticket.