The deepfake did not defeat a control. It satisfied one.
A finance employee approved fifteen transfers worth twenty-five million dollars after a video call with colleagues who were all synthetic. Nothing technical was breached. The process worked exactly as designed, and the design was the problem.
The reaction in most organisations to the Arup case was to ask whether detection software could be bought. It is the wrong first question, and the interesting thing is how reliably it is the first one asked.
In early 2024 an employee in the Hong Kong office of the engineering firm Arup made fifteen transfers totalling around twenty-five million dollars to accounts controlled by criminals. The instruction began with an email the employee was suspicious of. What removed the suspicion was a video call, on which the chief financial officer and several familiar colleagues appeared and spoke. Every one of them was synthetic, assembled from recordings of real meetings and public appearances. Arup’s own account, reported by CNN, was that no systems were compromised and no data was taken.
That last detail is the whole story. No control failed. A control was satisfied — the control was recognition, and recognition is not authentication.
The control that was actually in place
Written down, the payment process probably said something reasonable: unusual transfers require confirmation from an authorised approver. Operated, it meant something else — that a request is confirmed when a person who appears to be an authorised approver says so, on a channel where appearing to be someone has always been sufficient evidence of being them.
For as long as that has been true, the control worked. It stopped being true at the point where a convincing likeness of a named executive could be produced from material that named executive publishes as a matter of course. The control did not degrade. Its single unstated assumption expired.
This is the general shape, and it is worth stating separately from the technology, because it will outlive this particular attack: any control whose evidence is that something looked or sounded right is now a control with an expiry date. Voice on a phone call. A face on a video call. A signature block. A writing style. Each of them was a reasonable proxy for identity, each of them is now cheap to reproduce, and none of them was ever written down as the thing the control depended on.
Where the fix belongs
Detection is worth having and it is not the answer. It puts a probabilistic classifier in the position of deciding whether a real payment goes out, it is an arms race against a generator that improves faster than the detector, and it fails in the direction that matters — a false negative approves the transfer.
The fix belongs in the process, and specifically in separating the channel that carries the instruction from the channel that confirms it.
An unusual payment instruction arrives. What confirms it?
- Nothing is confirmed. The attacker controls both sides. This is the Arup shape, and it is the default in almost every organisation.
- The channel is now independent of the instruction. Cheap, unglamorous, and it defeats this class of attack outright.
- Confirmation is bound to the transaction, not to a conversation. The strongest of the three, and the one that survives the next generation of the attack.
The middle row is the entire remedy for the case above and it costs nothing. What makes it hard is not expense. It is that calling the chief financial officer back on a directory number, after they have personally appeared on a call to ask for something urgent, feels like an accusation. Controls that require a junior person to imply that a senior person may be an impostor do not survive contact with hierarchy unless the organisation has said, in advance and in writing, that performing the check is mandatory and never rude.
What to check this quarter
For each one, write down the assumption in a single sentence, in the form “this is safe because an attacker cannot produce X”. Then ask whether producing X now costs more than the transaction is worth. That test takes an afternoon, it requires no product, and it will find things that no amount of AI-specific policy would have surfaced — because the exposure is not in the organisation’s AI systems at all. It is in the ordinary processes that generative tools have quietly revalued.
What this does not tell you
We are not saying detection has no place. In high-volume consumer contexts where no callback is possible, a classifier may be the only control available, and something imperfect beats nothing. The argument is about order: process separation first, because it is deterministic and cheap, and detection second, as a signal rather than a gate.
We are also not saying that this is an attack on AI systems. It is not. Arup’s own position was that its systems were not compromised, and treating this as an AI security incident sends the review to the wrong team. It is a finance-process incident carried out with a synthetic-media tool, and the people who need to act sit in treasury and internal audit rather than in the AI programme.
Which is the point worth leaving with. The most likely way generative AI harms a large organisation in the next two years is not through a model it deployed. It is through a control it wrote in 1998 that assumed faces are hard to forge.