In “Caliphate’ and the Problem of Testimony,” Beba Cibralic (2020) argues that the New York Times (NYT) podcast Caliphate represents an epistemic failure that is similar in kind  to its failures in reporting on weapons of mass destruction in the lead up to the Iraq war. According to Cibralic, in both cases, reporters failed to fulfil their journalistic responsibilities to check the testimony of their sources, leading to the NYT making inexcusable false claims, and failing in its obligations to its readers … [please read below the rest of the article].

Image credit: Bernard Goldbach via Flickr / Creative Commons

While I am sympathetic to the idea that Caliphate falls short of exemplary journalism, I want to suggest that there are some important differences between these two cases. In particular, the failures of Caliphate are not failures of reporting, but rather failures to accurately represent the credibility of someone whose speech they were presenting. More generally, I worry that stressing the continuities between print journalism and podcasting leads to a mistaken picture of the epistemic role of podcasting, so I want to take this opportunity to think a little bit about the social epistemology of podcasting.

The Caliphate Podcast

The Caliphate podcast (released in April 2018) was produced by terrorism reporter Rukmini Callimachi, and producer Andy Mills. Although Caliphate is made by the NYT and is presented as reportage, it borrows heavily from true crime podcasts, especially Serial (formerly produced by This American Life, and now owned by the NYT). The series focuses on Callimachi’s “quest to understand ISIS”, and is organised around a number of narrative threads: interviews with Abu Huzayfah (a nom de guerre), a Canadian resident who claimed to have been a member of Islamic state, and to have taken part in killings in Syria (episodes 1 to 6 and 10), frontline reporting of the recapture of Mosul from ISIS (episodes 7 and 8), and an investigation of ISIS’s enslavement of Yazidi girls (episodes 9 parts one and two).[1] The interviews with Huzayfah were used as a narrative hook to tell a wider story about the psychology of online radicalisation and ISIS atrocities. Callimachi is presented as an expert on ISIS, but throughout the series she switches between functioning as narrator, expert, participant and audience surrogate.

In the first couple of episodes, Huzayfah is presented as a reliable witness. This shifts in episode 6, which digs into inconsistencies in Huzayfah’s story, raising the possibility that he is catfishing the reporters. This episode suggests that there is good evidence against the truth of parts of his story, but withholds judgement about whether he was in fact an ISIS fighter. In the concluding episode 10, the producers return to catch up with Huzayfah, and—somewhat surprisingly—the concerns are dropped: the producers continue to use Huzayfah as a lense for thinking about ISIS, and they appear to take some of the more dubious elements of his story seriously. The podcast was well-reviewed when it was released, and won several awards including a Peabody award.

In September of 2020, Canadian authorities arrested Huzayfah (whose legal name is Shehroze Chaudhry) on charges of perpetuating a terrorist hoax. [2]  The NYT launched an internal investigation which led to an editorial (published in December 2020) which concluded that: “the episodes of “Caliphate” that presented Mr. Chaudhry’s claims did not meet our standards for accuracy.” [3] This editorial highlighted the failure to assign the podcast an editor with expertise in terrorism, and the team’s lack of scepticism and rigor in reporting on Huzayfah. The podcast has been amended rather than retracted, and continues to be hosted by podcast providers and the NYT website in an edited form that includes a prefatory warning about accuracy and a new episode that digs into the failures with the reporting.

The Social Epistemology of Podcasting

When we start to think about the epistemology of new media, it is tempting to start with analogies to established new media. However, analogies between podcasting and print newspapers can only get us so far. Podcasting borrows elements from a number of different kinds of media (public, commercial, alternative, and amateur radio, blogging, and alternative print media, and has created several distinctive genres with their own epistemic norms and expectations (Crofts et al 2005, Berry 2006, Meduni 2007, Spinelli and Dann 2019).

In order to think about the distinctive social and communicative features of new media, research in media studies and communications is extremely valuable. A quick survey of the literature suggests a number of distinctive features of podcasts:

❧ Much like blogs, podcasts are typically free to listen, have historically had no gatekeepers, and have fairly low costs of production, allowing for a large number of producers, and for the cultivation of niche audiences (Crofts et al 2005, Berry 2006, 150, 2015, 173, Spinelli and Dann 2019, 8);

❧ Most podcasts operate on a model in which listeners subscribe to shows (rather than stations) which they listen to on-demand through an RSS feed (Berry 2006: 145), combining personalisation of shows with serendipity in what shows cover (Crofts et al 2005; Berry 2006, 156; Spinelli and Dann 2019, 8);

❧ Podcasts are an intimate medium that are typically listened to alone and involve high levels of trust and the formation of parasocial relationships between listeners and producers  (Berry 2006, 148; Spinelli and Dann 2019, 7);

❧ Podcasts are sometimes treated like radio and have some radiogenic features, but podcast producers are much more experimental than traditional radio (Berry 2006, 155-6).

By contrast newspapers have clear gatekeepers and legal regulation, aim at a general audience, have very limited personalisation, and have clear formal stylistic conventions.

Caliphate is an example of what we might call an inquiry-driven narrative non-fiction podcast, and we should interpret it in light of the conventions of this genre. Just as a detective novel dramatizes a fictional inquiry, an inquiry-driven podcast dramatizes an actual inquiry, transforming it into an aesthetic object, rich with tension, and surprises, and gradual progress towards an answer to the question. This structure is perhaps most familiar from Serial (which uses narrative techniques borrowed from This American Life), but it is also prominently used in RadioLab(which for the most part covers science and technology), and ReplyAll (which is a mixture of online culture and technology ghost stories). [4]^,[5]

Inquiry-driven podcasts typically focus on one presenter who is driven by a deep curiosity about some question or mystery, and the narrative structure is organised around the process of inquiry with tension being created by the paucity of evidence, the deliberate withholding of evidence, or the difficulty of making sense of contrasting perspectives. In some cases the narrative will be structured around splitting a complex question into sub-issues, and in others the story will progress by weighing up different possible answers. As a consequence, inquiry-driven podcasts will often present a complex and ambivalent perspective—or multiple perspectives—on the question under investigation, inviting the audience to engage in a kind of vicarious inquiry with the reporter. Serial makes this explicit: in season 1 episode 4, Sarah Koenig says “If you want to figure out this case with me, now is the time to start paying close attention. Because we have arrived, along with the detectives, at the heart of the thing.”[6] Not all inquiry-driven podcasts push the ambiguity quite as far as Serial, and some largely stick to developing one answer (You’re Wrong About is an inquiry-driven podcast that focuses on debunking misconceptions about public events), but the genre thrives on ambiguity and complexity.

Inquiry-driven narrative non-fiction podcasts have a number of epistemically significant features that challenge traditional norms of reporting. First, inquiry-driven podcasts display the backstage parts of journalism—including mistakes, dead-ends, and wrong answers—meaning that they may consider or even present as true claims that are partly or completely false. Often the final part of a podcast will iron out this complexity by summarising a complete answer to the question. Secondly, the narrative of an inquiry-driven podcast aims to provide not just true propositions, but also a cognitive structure for thinking about the topic being investigated (Spinelli and Dann 2019, C2) (on narrative testimony, see Fraser forthcoming).

Caliphate is interested in Huzayfah’s story not just for its own interest, but as a means to establish a general conception of how Muslim men living in North America become radicalised and involved in ISIS. This means that inquiry-driven narrative podcasts are doubly narrative: they present a narrativized inquiry, that is aimed at developing a narrative about the topic under investigation. The epistemic assessment of a podcast needs to ask both whether it makes true claims, and whether the narrative it constructs is epistemically and morally adequate. Thirdly, much like detective novels, inquiry-driven podcasts function as exemplary inquiries. This is not to say that the reporter(s) who are carrying out the inquiry need to be exemplars of intellectual virtue—a great deal of the drama in Serial emerges from Sarah Koenig working through the tensions between journalistic norms and her personal point of view (Spinelli and Dann 2019, C8)—but to say that inquiry-driven podcasts have an instructional role. A podcast might dramatize a badly conducted inquiry, but to do so responsibly, it ought to give space to the ways in which the inquiry has been badly carried out.

Cibralic’s Diagnosis of Caliphate

Cibralic argues that “while the weapons of mass destruction and ‘Caliphate’ reporting failures are different, the Times’ two misadventures share an important thread: each relied on faulty testimony” (Cibralic 2020, 33). Drawing on the work of John Greco, she distinguishes a number of different ways in which we might give uptake to testimony, depending on our access to the subject-matter at issue, and suggests that while journalists may not be able to double-check every claim made by their sources due to lack of access to the subject-matter of these claims, they ought to establish well-founded trust in their sources. In her view, Caliphate failed to establish well-founded trust in Huzayfah, and this represents a systematic failure in the newspaper, indicative of prioritisation of entertainment over accurate reporting.

I take it that Cibralic’s analogy between Caliphate and reporting on weapons of mass destruction indicates that she thinks that both cases involve what we might call laundering testimony. In example of laundered testimony, a reporter receives testimony that p from a source, fails to discharge her journalistic responsibilities regarding that source, and then repeats the claim that p in her reporting. Following Cibralic we might suggest that these responsibilities are disjunctive: if the reporter can double check the claim, then she should, if she cannot check the claim because it requires expertise or is otherwise inaccessible to her, she should establish that the source is a credible one. Laundering testimony comes in a few flavours:

i) a reporter might check a source’s credibility, find them unreliable but still repeats the claim;

ii) a reporter might non-culpably fail to check a source’s credibility (maybe it slips their mind), and;

iii) a reporter might fail to check a source’s credibility to avoid finding evidence that they are unreliable.

There are three problems with laundering testimony: it misrepresents by replacing an individual’s credibility with that of a media source, in many cases unjustifiably boosting the credibility associated with that claim, it generates a risk of making inexcusable false claims, and it obfuscates political manipulation (for example when political sources launder their talking points through putatively neutral media).[7]^,[8]

Laundering testimony is a real problem, but it doesn’t seem to be the central problem with Caliphate. Here’s a representative exchange from episode 3 of the podcast which covers Huzafyfah’s training (Note: All quotes are taken from the NYT’s transcript. Italicised text indicates a recording of an interview, and non-italicised text is later commentary.):

Callimachi: Right. So what happens next?

Huzayfah: Weapons, weapons training. We got — the very first thing we learned was parts of a rifle.

Callimachi: The training was a couple of weeks long, and again, this is something that I’ve seen both in the documents I have and in the interviews I’ve done with others. And it was, I would say, half weapons training.[9]

By interleaving interviews with commentary, the producers allow him to tell his own story, and throughout the series almost all of the claims about Huzayfah are made by him. This means that the analogy between a false claim printed in a news story and the false claims made in Caliphate is not quite right. If we wanted to look for an analogy from written media, we might think about a false quote or an interview where the interviewee makes false claims. The podcast producers haven’t laundered Huzayfah’s testimony, because they haven’t repeated what he said: he makes almost all of the relevant claims himself.

I say almost all because Callimachi’s commentary occasionally includes claims about Huzayfah. In the exchange above, we hear Callimachi say “the training was a couple of weeks long,” which is a direct claim about Huzayfah’s story. I think that there are reasonably good grounds to think of this statement not as a piece of testimony, but as a summary of what Huzayfah has said. The content of this statement is really “Huzayfah tells us that…”.[10] Inquiry-driven narrative podcasts typically heavily edit interviews to extract key claims, and when an interviewee hasn’t expressed an important point in a sufficiently pithy way, the producers will edit in a presenter summarising a part of the story.[11]

Pointing out that the majority of false statements about Huzayfah’s story were made by him, rather than by a journalist, might seem like hair-splitting, but there are important differences between a journalist making a false claim based on a source, and a journalist presenting a false claim made by a source.[12] It is quite possible to quote a source without presenting them as credible. A news article about the firing of an academic might make several well-supported claims about why the academic has been fired, followed by a quote from the university offering a competing explanation.

If the journalist’s explanations are presented as matters of fact, then it should be clear to the reader both than the university is not being presented as a credible source—in fact, the quoting the university might lower their credibility—and that the university’s quote has only been included to discharge the obligation to offer a right of reply. This means that a journalist knowingly prints a false quote without breaking any journalistic norms, if they present the source as unreliable. And, if a journalist prints a false quote from an unreliable source that they presented as reliable, they would be responsible for their presentation of the source, not for what the source said.

Distinguishing between a journalist making claims directly, and a journalist reporting on another’s speech either through quotation or an interview helps us see that these two cases involve different epistemic responsibilities. If a journalist makes a claim on the basis of a source’s testimony without checking that source’s competence, they are laundering testimony. If a journalist quotes a source making a claim, and irresponsibly presents them as more credible than they are, they are laundering the credibility of that source. This means that the important question for the journalists behind Caliphate is not whether they made false claims (but rather whether they presented Huzayfah in a way that unjustifiably and irresponsibly inflated his credibility.

The podcast’s presentation of Huzayfah is a little complex. In the first couple of podcasts he is presented as a reliable narrator, and Callimachi repeatedly claims that features of his story matches with her expert knowledge about ISIS recruits. However, episode 6 suggests a more nuanced attitude towards Huzayfah’s testimony. Callimachi underlines the importance of criticality, saying “we can’t take people at face value […] we take what they say and then you have to see if you can find corroboration.”[13] This episode brings in a bigger investigate team to dig into a number of problems with Huzayfah’s story: an inconsistent timeline, contradictory reports from his father, and the revelation that pictures he presented to evidence his time in Syria were taken at a different time. They raise the possibility that Huzayfah was catfishing them, but end up giving some credence to the claim that he was an ISIS fighter, based on reports from two unnamed US government sources and two putative ISIS sources who reported seeing Huzayfah.

The problem is that when the podcast revisits Huzayfah in the concluding episode 10, it sets these worries to one side. This episode features interviews with Huzayfah recorded one year after the original interviews focusing on how he is readjusting to life in Canada, his response to ISIS losing territory, and his investigation by the Canadian authorities (partly as a consequence of the podcast). These interviews presuppose that Huzayfah was in fact an ISIS fighter, and in her questions Callimachi appears to give credence to some of the more doubtful parts of Huzayfah’s story. There is no attempt to adjudicate whether he was a hoaxer, or to determine which parts of his story might be true.

The NYT’s post-mortem investigation identified several problems with the research that was done on the podcast. Some relevant evidence was missed: it appears that the Canadian government had already concluded that Huzayfah was a hoaxer before the podcast was aired, and several pictures which Huzayfah presented as evidence of his time in Syria were copied from publicly available sources on the internet. It also seems that the producers of the podcast didn’t take the inconsistencies in Huzayfah’s stories seriously enough, and assigned too much weight to the two putative ISIS sources. To be clear, the post-mortem did not uncover definitive evidence that Huzayfah didn’t travel to Syria to sign up with ISIS. Their conclusion was that the evidence that the podcast team had—the two US government sources, and the purported ISIS sources—was not sufficient to corroborate Huzayfah’s story or establish his credibility.

Cibralic is right to say that there are failures of journalistic responsibility in this case, but the problem is not laundering testimony by making claims made by an unreliable source. The central issue with Caliphate is that the producers took Huzayfah’s narrative too seriously, given the internal inconsistencies, evidence undermining parts of his story, and lack of direct corroboration. This is not to say that there was no evidence supporting his story. The issue is that the producers overplayed their hand by concluding the podcast with an episode that took the core of Huzayfah’s story seriously, rather than presenting a more ambivalent conclusion.

This failing doesn’t mean that the series is devoid of epistemic value. The picture that Caliphate constructs about internet radicalisation and the organisation of ISIS in Syria is based on much more than Huzayfah’s story, and may be an useful narrative for thinking about the psychology of ISIS fighters. I listened to the podcast knowing that many of Huzayfah’s claims were false, and I still think that I came away understanding more about ISIS.

Above, I pointed out above that inquiry-driven podcasts serve as exemplary inquiries, and the failure to follow up on doubts about Huzayfah’s story in the wrapping up episode suggests a second issue with the podcast. Credulity in the early episodes is understandable given Huzayfah’s compelling story, but by letting serious worries about the plausibly of this story slide, the podcast’s producers shift from credulity towards the intellectual vice of gullibility. Because they display the backstage parts of reporting, inquiry-driven podcasts need to grapple with the tensions between the journalistic norm of objectivity, and a narrative which is organised around a personal take on the issue being investigated. Callimachi’s personal sympathy with Huzayfah creates a tension with the journalistic norm to establish the credibility of sources that is worthy of further investigation, but is not addressed head on by the podcast.

Since they function as exemplary inquiries, when an inquiry-driven podcast gets something wrong, a correction needs to not just correct mistaken claims, but to understand the ways in which the inquiry went wrong (see Spinelli and Dann 2019 26-28 for a discussion of how Radiolab dealt with reporting failures in their reporting of ‘yelllow rain’ in cambodia). The NYT’s inquiry, editorial, and post-mortem episode can be seen as an attempt to carry out this kind of repair. The puzzling thing about the post-mortem (which may be partly behind Cibralic’s diagnosis of the problems of the podcast) is that the NYT’s editorial staff repeatedly try to present the failure as a failure of reporting.

Although there is a touch of the theatre of rigour about these editorials, it is understandable for a newspaper moving into podcasting to try to implement (or at least present) a coherent set of epistemic norms across all of its products in the interests of brand continuity. What this policy of continuity misses is the respects in which podcasting production has created distinctive genres, with their own styles, expectations, and epistemic norms.

Acknowledgements

Thanks to Jessica Bernard Black, Beba Cibralic, Aadil Kurji, and Laura Whateley.

References

Berry, Richard. 2006. “Will the iPod Kill the Radio Star? Profiling Podcasting as Radio.” Convergence 12 (2):143-162.

Berry, Richard. 2015. “A Golden Age of Podcasting? Evaluating Serial in the Context of Podcast Histories.” Journal of Radio & Audio Media 22 (2): 170-178.

Cibralic, Beba. 2020. “Caliphate and the Problem of Testimony.” Social Epistemology Review and Reply Collective 9 (12): 33-36.

Crofts, Sheri, John Dilley, Mark Fox, Andrew Retsema, Bob Williams 2005. “Podcasting: A New Technology in Search of Viable Business Models.” First Monday https://firstmonday.org/article/view/1273/1193.

Fraser, Rachel. Forthcoming. “Narrative Testimony.” Philosophical Studies.

McCracken, Ellen, editor. 2017. The Serial Podcast and Storytelling in the Digital Age. Taylor & Francis.

Spinelli, Martin and Lance Dann. 2019. Podcasting: The Audio Media Revolution. Bloomsbury Publishing USA.

[1] https://www.nytimes.com/2018/09/20/podcasts/caliphate-transcript-chapter-one-the-reporter.html.

[2] I’ll continue to refer to Chaudhry by his nom de guerre to avoid confusion with the transcript.

[3] https://www.nytimes.com/2018/09/20/podcasts/caliphate-transcript-prologue-the-mission.html.

[4] For in-depth discussions of Serial see (Berry 2015), (McCraken 2017), (Spinelli and Dann 2019 C8).

[5] See Spinelli and Dann 2019, C2 on narrative construction in Radiolab.

[6] https://serialpodcast.org/season-one/4/inconsistencies.

[7] Any media source that cares both about making true claims and avoiding errors will end up making some errors, but if they follow rigorous procedure these errors will be excusable, requiring a correction notice rather than an internal investigation.

[8] But not always: lots of people are more credible than lots of newspapers.

[9] https://www.nytimes.com/2018/09/20/podcasts/caliphate-transcript-chapter-three-the-arrival.html?action=click&module=audio-series-bar&region=header&pgtype=Article.

[10] In other cases, we find explicit summaries. Here’s a passage from episode 5:

Callimachi: So he tells us that, at this point, he’s in his dorm room. He’s moping around, depressed, having a hard time recovering from what he just did.

In other cases, summaries are marked by their use of the historic present tense. A couple of examples from episode 2:

Callimachi: First he’s doing, you know, just normal research, like you and I.

Callimachi: He ends up in forums, in chat rooms, where Al Qaeda and later ISISrecruiters are lurking.

[11] Radiolab often does this by fading out the audio of the interviewee, making clear that the presenter is summarizing a longer piece of testimony.

[12] I will talk about a journalist making claims rather than about a newspaper making claims to avoid complications concerning institutional assertion.

[13] https://www.nytimes.com/2018/09/20/podcasts/caliphate-transcript-chapter-six-paper-trail.html?action=click&module=audio-series-bar&region=header&pgtype=Article

Categories: Articles, Critical Replies

Tags: Beba Cibralic, Caliphate podcast, John Greco, Joshua Habgood-Coote, journalism, media, New York Times, podcasts, RadioLab, ReplyAll, Sarah Koenig, Serial, social epistemology, Testimony, This American Life