AI-assisted fraud: deepfakes and forged instructions

At a quarter to four on a Thursday, the finance lead at a forty-person machinery manufacturer received a voice note from the managing director. The voice was his: the same intonation, the same small pause mid-sentence, even the same faint background hum he always has. The message was short. He was at a trade fair, the line was poor, and an advance had to reach a new supplier before the bank cut-off. Account details arrived in writing right after. The finance lead hesitated for a moment, then sent it, because he knew that voice. Two days later the transfer was past recall. The real damage, though, was not the money. For the next three weeks nobody in that office could take an ordinary request from a real manager at face value again.

This article covers what AI actually changed about fraud, the three attack shapes that matter, why a forged instruction feels so credible, the most expensive target of all in payment and bank-detail requests, why telling people to be careful is not a defense, how to build a verification rule, the principle that instructions live in the system rather than in a channel, the inverted form of the urgency trap, the damage done to your customers by people impersonating you, what to do in the first hour after an incident, and the minimum arrangement a small team can realistically run.

What did AI actually change about fraud?

Asking for a transfer while impersonating an executive is not a new idea; it has been around for decades. What changed is not the idea but the cost. Producing a convincing forged instruction used to take work: learning the target's writing habits, composing flawless language, using the company's internal vocabulary correctly. That effort meant the attack only made economic sense against large organizations. The same effort now takes minutes, which makes a forty-person manufacturer a worthwhile target.

The second change is language. Awkward phrasing and odd salutations were, for a generation, the most reliable warning sign anyone had. That signal is gone. The third is what happens inside your own walls: when team members paste contract text, customer lists and internal threads into unsupervised AI tools, they hand attackers a shortcut. How to manage that side without banning everything is the subject of our piece on shadow AI inside the company.

Three attack shapes: voice, video and thread

The first is voice cloning. A few minutes of an executive speaking — a promotional video, a conference panel, a recorded webinar — is more than enough raw material. The voice-note format suits the attacker perfectly: it delivers the full weight of familiarity while giving the recipient no opportunity to ask a question.

The second is the video call. A short call with a poor connection and a freezing camera can be generated today, and it looks exactly as expected, because most corporate calls already look like that. The degradation reads as normal rather than suspicious. The brevity is not accidental either; the longer such a call runs, the more likely it is to fall apart.

The third, and in practice the most dangerous, is thread hijacking. The attacker does not send a new email. They appear inside a real conversation. Genuine messages from previous weeks sit underneath, the signature is correct, the subject line is one you wrote yourself. Only the bank details in the latest message have changed. This shape requires the least technology and has the highest success rate, which is why it deserves the most attention.

Why does a forged instruction feel so credible?

Credibility comes from context, not from audio quality. The attacker asks for something you were already expecting. During trade fair season they ask for a fair expense. At year end they ask for a closing payment. If you are negotiating with a new supplier, they use that supplier's name. Almost none of this comes from secret sources; it comes from material you published yourself in job ads, press notes, social posts and the job titles on your team's public profiles.

The second source of credibility is hierarchy. The request arrives from a place that is uncomfortable to question. In a small company that effect is stronger, because there is no procedural layer between the managing director and the person who moves money; they already message each other all day. The attacker imitates that informality precisely. How much of your data is exposed, and what AI tools do with it once it leaves, is a related question we cover in AI and customer data.

The most expensive target: payment and bank detail changes

Almost every forged instruction is aimed at one thing: the account the money lands in. The most common shape is a notice of changed bank details, arriving in the name of a real, existing supplier. The invoice is real, the amount is real, the work was really done; only the account number belongs to the attacker. This is why controls on the payment side should focus not on whether the invoice is genuine but on whether the payee details changed.

The single most effective rule here: a change of bank details is never completed by the message that requested it, only by a call you initiate. Which fields belong on a supplier record, and where the purchasing process should demand approval, is covered in supplier management and the purchasing process. Seeing quickly which account a payment actually reached also gives early warning, and matching bank movements to records automatically is explained in bank statement integration.

Why telling people to be careful is not a defense

Training a team to spot fakes is a race you have already lost. Generation quality improves every month; human attention does not. Awareness training can even backfire: the more confident people become that they can recognize a fake, the more they trust their own instinct and the more legitimate it feels to skip a procedural step. A team trying to detect forgeries is effectively betting on its own fallibility.

Trying to spot the deepfake is a losing race; the winning side is a single verification step that makes spotting unnecessary.

The better question is this: how do you build an arrangement in which the money stays put even if the instruction is fake? The answer looks at the process, not the person. A well-designed control does not care in the slightest whether the voice was real; it only asks whether a second channel confirmed the request. A control like that neutralizes even the most capable imitation and removes the burden of detection from your team entirely.

How do you build a verification rule?

A verification rule has to be short enough to remember and narrow enough not to slow down ordinary work. The components below are enough for most small companies.

• Channel separation: Verify on a different channel from the one the instruction arrived on; an emailed request is confirmed by phone, a phoned request is confirmed inside the system.
• Where the number comes from: Call back on the number held in your records or the contract, never the one printed in the message; that single sentence stops a large share of attempts outright.
• New payee rule: Tie verification to novelty rather than to size; any account receiving money for the first time needs a second person, whatever the amount.
• A genuine second approver: The approver must not be the person who received the instruction, and two people listening to the same voice note is not verification — the confirmation has to come independently.
• The right to pause without blame: Put in writing that an employee who holds a request will never be criticized for it, even when the request turns out to be genuine; without that line, hierarchy defeats every control you build.
• A logged confirmation: Record who verified, when, and on which number; if a dispute follows, you consult the record rather than anyone's memory.

These rules belong on a single written page. How to capture critical flows in a format people actually read is covered in process documentation and SOPs. Keeping the scope narrow matters just as much: a company that demands dual approval on every payment will find a way around its own rule within two weeks, and be left with a control that exists only on paper.

Instructions live in the system, not in a channel

Banning voice notes or instant messages is both impossible and unnecessary. One principle replaces the ban: no instruction is valid unless it exists as a record in the system. A voice note is a reminder, not an instruction; the instruction itself appears on the payment record, the purchase request or the workflow. This ends the argument about channels completely, because where the message came from stops mattering.

For that principle to hold, it has to be clear who can do what inside the system. Who can create a payment record, who can approve one, and who can change payee details are three separate rights and should be defined separately; we cover that separation in roles and permissions management.

Every change should leave a trace

When a supplier's bank details change, it should be visible who changed them, when, and on what basis. In systems without an audit trail this kind of edit happens silently and is usually noticed months later, when the money is long gone. How to set up change history on a record, and what to look at in it, is explained in the CRM audit log. On the contract side, signature flows with verified identity stand on far firmer ground than scanned images of wet signatures, and how those flows are built is covered in e-signature contract processes.

The urgency trap and its inversion

Classic warning lists put urgency at the top: be suspicious of anything that rushes you, demands secrecy and leans on authority. That is true, and it is now incomplete. Precisely because urgency became such a well-known signal, the more careful attackers do the opposite.

The new shape is calm and boring. First comes a small, ordinary, entirely legitimate request — a document, perhaps, or a minor confirmation. Once it is fulfilled, two things happen: the attacker learns where your controls sit, and they become a normal presence in the thread. The real instruction arrives three days later, from inside a conversation that now feels familiar. This is why suspicion cannot be pegged to urgency. The trigger has to be the transaction itself: if money or access is changing hands, verification runs no matter how relaxed the message sounds.

People impersonating you: the risk on the customer side

This subject is usually discussed in terms of the company's own losses, but the second and often larger risk sits with your customers. A fake invoice sent in your name, a page opened under your logo, a call made in your account manager's voice — these take your customer's money and your reputation at the same time. The customer will not remember their own missing control. They will remember your brand.

The counter to this is not defense but a conversation held in advance. Tell customers and suppliers in writing what you will never do: we never change our bank details by email, we never redirect a payment over instant messaging, we never ask you for a verification code. Repeated once a year, that short notice measurably lowers the success rate of anyone impersonating you, and more importantly it gives the customer permission to stop and ask.

The first hour after an incident

The first hour is the only window in which a transfer can realistically be pulled back. The order matters: call your bank and request a recall, ask that the receiving bank be notified as well, then make the formal reports. In parallel, reset the password and close the sessions on whichever account was likely compromised — but delete nothing from the mailbox, because those messages are the only basis any later investigation will have.

None of this can be left to whoever remembers it on the day. Anyone who moves money should have a one-page card at their desk showing who to call and in what order. If the incident becomes visible outside the company, how to handle the communication side is covered in crisis communication and brand crisis management.

Repair the control instead of finding a culprit

The least productive reflex after an incident is to declare the person who made the transfer responsible. The price of that reflex is steep: nobody escalates the next suspicious request, because being wrong now looks more expensive than staying quiet. The right question is not who did this but which step would have prevented it, and the answer is almost always one missing verification that can be added in a day.

The minimum arrangement for a small team

A six-person company has no security team and never will. Four steps still cover most of the exposure at a cost that is actually payable. First, out-of-band verification on any transaction where money or access changes hands. Second, a second approver for any account being paid for the first time. Third, two-step authentication on critical accounts and shared credentials kept in a proper vault rather than a spreadsheet. Fourth, an audit trail on bank detail changes in supplier and customer records.

None of these four is specific to AI. They are ordinary security hygiene, and what AI changed is only how urgently they are needed. The full set of baseline defenses is collected in cybersecurity for small businesses. The cheapest moment to put this in place is while nothing has happened; the most expensive is somewhere inside that first hour.

The strongest defense against forged instructions is keeping the instruction in the record rather than in a channel: when supplier details, quotes, orders, payments and approvals all sit in one place, no single message can move money on its own. Rocketly brings customer and supplier records, bookkeeping, task flows and user permissions into one system — open a free account and build your own verification flow.