Voice Deepfake Detection Fails on Phone Calls. Here’s Why

In February, Paolo Molesini, then chairman of Fideuram, Intesa Sanpaolo’s private banking arm, received a WhatsApp message that appeared to come from the group’s chief executive, Carlo Messina. It asked for urgent help with an overseas transaction. A phone call followed from what seemed to be a senior partner at a prominent law firm, confirming the instruction. According to two sources who spoke to Reuters, the callers used AI to replicate the lawyer’s voice.^[1]

Molesini asked his finance department to make a series of transfers to foreign accounts, mainly in China and Hong Kong. The total was €95m. Fideuram spotted the irregularities quickly, and cooperation between authorities in China, Portugal and Italy recovered roughly €53m. The rest, the sources said, went through a chain of overseas accounts and into cryptocurrency. Intesa Sanpaolo and Fideuram declined to comment.^[1],[2]

The detail that matters is where the voice sat in the scheme. The message did the asking. The cloned voice did the confirming. A phone call to a trusted third party is exactly the check that payment teams are trained to make, and here the check was the attack. This is the shape of AI voice cloning fraud now: the voice is used to pass a verification step, not to tell the whole lie.

Voice deepfake detection is the task of deciding, from the audio alone, whether speech was spoken by a person or generated by a model. Research systems report error rates near 1% on standard benchmarks. This post explains how those systems work, why the number collapses on a real phone line, and what data a detector needs before its score means anything on the calls a bank actually receives.

A convincing clone now takes ten seconds of audio

The raw material for a clone has shrunk to a voicemail greeting. ElevenLabs released Eleven v4 on 28 September and wrote that its Instant Voice Clones “can now capture voices with high fidelity using just 10 seconds of audio”.^[3] Google’s Gemini 3.8 text-to-speech, announced on 23 September, recreates “consistent vocal profiles from just a 30-second audio sample”.^[4] Research systems go further. The authors of the SPADE dataset, discussed below, note that zero-shot synthesis can now clone a speaker from about 3 seconds of reference speech.^[5]

Commercial tools wrap that capability in checks. Google requires “a verbal consent recording from the voice owner that matches the reference speaker” before it creates a voice, watermarks every clip from its Gemini audio models with SynthID, attaches C2PA content credentials, and does not offer voice replication through AI Studio in Illinois, Texas, the EEA, the UK, Switzerland or India.^[4] Those are real controls. They also share a property: each one binds the person using that product. An attacker who runs an open-weight model on a laptop meets none of them, and the five generators used to build SPADE are all publicly available.^[5]

The law now asks generators to mark their audio

Regulation arrived in the same quarter. Article 50(2) of the EU AI Act requires providers of systems that generate synthetic audio, image, video or text to “ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated”.^[6] The transparency obligations apply from 2 August 2026. The Digital Omnibus on AI gave generative systems already on the market until 2 December 2026 to comply with the marking duty.^[7] Fines for breaching Article 50 reach €15m or 3% of worldwide annual turnover.^[8]

Vendors moved with it. Deepgram’s self-hosted release of 1 October watermarks its Flux text-to-speech audio, and the engine “will not start with Flux TTS enabled if the watermarker is missing”.^[9] Courts and legislators are working on the human side. On 30 September a Tokyo district court held that a performer’s voice is protected under publicity rights, in voice actor Kenjiro Tsuda’s case over AI-voiced TikTok videos, while dismissing his request to remove videos that had already been deleted.^[10] On 17 September Florida’s attorney general proposed legislation that would classify “using an undisclosed AI voice to obtain money” as communications fraud. It is a proposal, not yet a law.^[11]

All of this helps with honest generators and with prosecution afterwards. None of it marks audio made by someone who has decided to commit fraud. Three technologies are often discussed as one, and they answer different questions.

Sources for the table: Google’s SynthID page and C2PA’s explainer.^[12],[13] Detection is the only one of the three that does not need the attacker’s help, which is why the rest of this post is about it.

How voice deepfake detection works: a score, a threshold and an error rate

A deepfake voice detector, called a countermeasure in the anti-spoofing literature, is a binary classifier. Audio goes in. A front end turns it into features. Older systems used hand-built spectral features such as linear-frequency or constant-Q cepstral coefficients; current systems mostly run raw-waveform networks or a frozen self-supervised speech encoder such as wav2vec 2.0, XLS-R or WavLM, with a small trained back end on top. Out comes one number, a score for how likely the audio is to be synthetic.

The classifier learns from labelled pairs: bona fide speech, recorded from people, and spoofed speech, produced by known synthesis and voice-conversion systems. The ASVspoof challenges, which have set the field’s evaluation habits since 2015, build in one honest test. In the 2019 logical-access set, training and development data contain six attack systems, and the evaluation set uses thirteen attack conditions, most of them systems the detector never saw in training.^[14],[15] The 2021 edition kept those thirteen attacks and sent both real and fake audio through telephone systems, which turns out to matter.

To turn a score into a decision, someone picks a threshold t and flags every call whose score is at least t. Two errors follow. A genuine caller can be flagged (a false alarm), and a clone can pass (a miss). Raising t trades one for the other. The equal error rate (EER) is the error at the threshold where the two rates are equal.

Take ten genuine calls and ten studio-quality clones, scored by a detector. The scores are constructed for illustration. At a threshold of 0.50, two genuine callers score above it and are flagged, and two clones score below it and pass. Both error rates are 20%, so that is the EER. Now play the same ten clones down a phone line. In this illustration the channel pulls their scores toward the human side. The EER of the new mix rises to 30%, which sounds like modest damage. But the detector is still running at the 0.50 threshold it was tuned with, and at that threshold six of ten clones now pass. Getting the miss rate back to 20% would mean lowering the threshold to 0.28, which flags four genuine callers in ten.

Two lessons sit in that toy. First, EER describes a whole score distribution, but a deployed detector runs at one fixed threshold, so a shift in scores shows up first as misses. Second, base rates punish false alarms. Suppose, purely for illustration, a bank screens 100,000 calls a month and one in 10,000 is a clone. That is ten fakes. A detector running at a 1% false-alarm rate also flags about a thousand genuine customers. Choosing the threshold is a business decision, and it only holds if real traffic scores like the tuning set.

Detectors learn the generator, not the fakeness

September’s papers show how narrowly detectors learn. SPADE, from researchers at the University of Texas at Austin and collaborators, accepted to SLT 2026, builds partial deepfakes, real recordings with a few words regenerated, in 12 languages from up to five publicly available generation systems per language, and trains localizers that mark which segments were edited.^[5] A localizer trained on one generator and tested on the same one can be excellent: 0.31% EER for XTTS v2, 0.65% for VoiceCraft-X. Tested on a different generator, three of the five single-generator localizers fall to near chance.

The more useful experiment trains on four generators and tests on the fifth. With all five in training, the XTTS v2 error is 0.78%. With XTTS v2 held out, it is 14.65%. For F5-TTS the jump is from 4.62% to 37.73%. The authors’ summary is blunt: training on several generators “does not yield synergistic gains for identifying deepfakes from unseen systems”.^[5] These are segment-level errors on English partial edits, but the whole-utterance studies below point the same way.

A Tsinghua study, “What Survives the Codec Shift”, adds the threshold effect from the worked example.^[16] The codec here is a neural audio codec used to generate speech, not a phone codec: the paper trains on older vocoder-based fakes and six codec-based generators and holds out a seventh, DAC. One strong retrained baseline, a wav2vec 2.0 detector with an AASIST back end, scores 5.99% EER on the pooled test set and 15.69% on the unseen generator. At the threshold chosen on development data, it misses 74.3% of the unseen generator’s fakes. The authors’ own method lowers that to 27.9%, still more than one fake in four.

VietPrism, a Vietnamese corpus released on 24 September, shows the same dependence on generator and adds speaker similarity.^[18] It holds 993 hours of genuine speech from 1,262 verified speakers and pairs every clone with a genuine utterance from the same speaker saying the same words, so a detector cannot cheat on vocabulary or voice identity. Five pretrained multilingual detectors were tested without fine-tuning. One, an XLS-R-based model, scored 56.30% EER against the commercial MiniMax voices, worse than a coin, and 0.00% against two open generators. Another, DF-Arena 1B, rose from 16.3% to 33.6% EER as the clones’ similarity to the real speaker increased. The clones that matter most in fraud are, by construction, the most similar ones.

The phone line removes the evidence detectors were trained on

Now add the channel. A call on the public telephone network is typically sampled at 8 kHz, so nothing above 4 kHz survives. A codec such as G.711 or a mobile AMR codec then quantises what is left, VoIP links drop packets, and the receiving end fills the gaps with concealment frames that are themselves generated. Real calls also carry rooms, cars and speakerphones. Each step touches genuine and synthetic audio alike, and each can erase or imitate the small artefacts a detector learned to look for.

ASVspoof 2021’s logical-access set is the cleanest test of this, because it keeps the attacks fixed and changes only the channel. Its thirteen spoofing systems are the same as the 2019 evaluation set’s, but the audio goes through a private exchange using codecs at 8 or 16 kHz, or through a call from a mobile phone across the public network to a VoIP endpoint.^[15] Benjamin Hurt and Oscar O’Donnell trained frozen-encoder detectors on the 2019 data and tested them on both.^[17] Their best encoder, XLS-R, went from 1.41% EER to 8.50%, six times worse on the same fakes. HuBERT went from 1.09% to 13.88%. On In-the-Wild, a set of 37.9 hours of real recordings of celebrities and politicians, 17.2 hours of them deepfakes found online, the same detectors scored 18.30% to 46.03%.^[19] WavLM, the best of the three in-domain at 0.92%, was the worst there.

The paper’s broader finding is also a warning against waiting for a bigger model. Multilingual pretraining helped up to about a hundred languages and then stopped; a 577M-parameter encoder did not beat a 315M one out of domain. “Broader and larger models are not reliably better,” the authors conclude, within their fixed recipe.^[17]

SPADE’s noise tests help separate the parts of the channel. Low-pass filtering at 4 kHz, roughly telephone bandwidth, moved its English XTTS localizer from 0.31% to 2.25%: worse, but usable. Additive noise at 5 dB signal-to-noise ratio moved it to 15.42%, and simulated reverberation to 6.43%. A VoiceCraft-X localizer went from 0.65% clean to 31.78% at the same noise level. Training on more languages or more generators did not fix it.^[5] On this evidence, bandwidth alone is not what breaks detectors. Real calls stack band-limiting, codecs, loss and noise, and the detector meets the combination it never saw.

Asking a large audio-language model instead does not escape the problem. SEAR, a benchmark released on 30 September, tested six such models, including Gemini and GPT audio models, on ASVspoof audio.^[14] On the 2021 telephony set their zero-shot verdicts reached a macro-F1 between 14% and 45%. Asked which acoustic feature most strongly indicated spoofing, they stayed close to the 25% random-choice baseline. Their written forensic explanations scored 83–85% on BERTScore, a text-similarity metric, for every model. The rationales read well whether or not the model had found anything.

Accents and languages are a distribution shift too

The third axis is the speaker. In VietPrism, DF-Arena 1B scored 16.6% EER on Southern and 18.3% on Northern Vietnamese speakers, but 28.7% on the smaller Southwest group and 26.9% on overseas speakers. Code-switching between Vietnamese and English raised the commercial generator’s EER by up to 7.1 points for one detector.^[18] SPADE found cross-language transfer gentler than cross-generator transfer but uneven: Japanese was the hardest language for localizers trained elsewhere.^[5]

None of SPADE’s twelve languages is Indian, and the ASVspoof sets are English. That is a gap where some of the heaviest voice traffic now flows. TechCrunch reported on 28 September that DetectifAI, a San Francisco startup, handles more than 100,000 calls a month for financial institutions in India, calls placed by AI voice agents for debt collection and loan follow-up, with deepfake detection and speaker verification on every call, according to its founder.^[20] Hindi-English code-switching, Indian English accents and mobile networks are a long way from clean read English, and, as with any rare case, a model that has seen few such calls has little to go on, as the post on why AI fails on rare cases describes.

What a telephone voice deepfake detection dataset has to contain

The papers above share a remedy: measure, and train, on the conditions the detector will meet. For a bank’s phone line that turns into a specification with three axes and a few rules.

Two further rules come from the failures above. Similarity must be measured and stratified, because the closest clones are the ones that fool detectors and people. And the held-out test set has to stay private. A test set that has leaked into training measures recall, the problem described in the post on the benchmark nobody sees, and for a detector the leak can be as small as a few clips from a held-out generator slipping into the training pool.

Consent is the hard part of the paired half. Cloning a real person’s voice to build training data is itself voice cloning, and the voice owner has to agree to that use specifically. Perit’s published consent covers training and evaluating AI systems and is captured on the file before the contributor speaks; its standard licence excludes voice enrolment and verification.^[21],[22] A spoof-pair collection would therefore need its own consent wording that names cloning, its own licence scope, and a clear rule for what happens to the clones if a contributor withdraws.

The collection side maps onto work Perit already describes. Its data collection service records two-party conversations with each side on its own channel, accents recruited to quota and matched acoustic conditions for degradation studies, across nine live locales including Hindi, with every collection pinned to one region (EU, US or India).^[23] Whether a team needs new calls, labels on calls it already has, or both is the question collection, annotation or both works through. Perit has not published detector results on its own audio, and none are reported here. The protocol a team would run is straightforward: record genuine calls across the channel grid, generate consented clones from at least three generator families, transmit both, train on the blue cells of Figure 5, and report EER and fixed-threshold miss rates on the orange ones.

The check that failed was a phone call

Go back to Fideuram. Nothing in the reporting says what language the lawyer’s cloned voice spoke, which network carried the call, or which tool generated it. Those three unknowns are exactly the axes on which September’s papers show detectors breaking: an unseen generator, an unseen channel and an unfamiliar speaker. A detector that scores 1% on clean benchmark audio says little about that call.

Two things would have helped, and neither is a watermark. One is a payment rule that does not accept a voice, however familiar, as confirmation of a transfer, so the check runs through a channel the attacker does not control. The other is a detector whose threshold was set on audio that looks like the bank’s own traffic: real phone calls, in the languages its clients speak, against clones from generators it was never trained on. The first is a process change. The second is a dataset.

Questions teams ask about voice deepfake detection