Run history and retry: rescuing a failed workflow step

Thursday, 8:40 a.m. An admissions advisor opens the dashboard and sees that none of the thirty-eight applicants who filled in last night's form received a welcome message. The workflow does not look broken: the status reads active, the trigger fired thirty-eight times, and the first two steps completed. Only the third step failed. The channel carrying the message lost its authorisation at 11:10 p.m., and every run that started after that stopped at the same point. Nobody configured anything wrong. Nobody looked, and nobody retried.

This article is about the routine that recovers those thirty-eight runs. We start with where a run actually breaks and what question a run history has to answer. Then we work through which failures deserve a retry, why a retry must not send a second message, when a bulk retry is the right call, why half-finished runs pile up, the difference between fixing the workflow and rescuing a run, who should learn about a failure and how fast, where you rehearse a recovery before it touches anyone, and how all of this becomes a fifteen-minute weekly habit.

Where does a run actually break?

Automation does not split neatly into worked and did not work. A run moves step by step, and every step has its own way of failing. The trigger may never have fired at all: the form was taken down, or the system sending the webhook changed its endpoint. The trigger may have fired but the condition sent the run down an unexpected branch, because a field arrived empty and the else path ran instead of the if path. Or the run reached the right branch and the outside service refused the request at the moment of sending.

All three appear on the same screen with the same red badge, yet the fix for each is entirely different. The first is a configuration problem, the second is a data problem, the third is a transient connection problem. A good deal of the disappointment behind CRM programmes that never deliver what was promised hides exactly here on the automation side: the team can see that something failed, but has no record that tells them which kind of failure it was. Without that distinction only one reflex remains, and it is switching the workflow off.

What question should a run history answer?

A useful run record does not say an error occurred. It says: for this contact, at this time, at this step, for this reason, the run stopped. In practice you need to read four things: the event and person that started the run, how far the workflow progressed, the message returned by the step that stopped, and the total duration of the run. Without those four, every investigation turns into guesswork.

In Rocketly two screens carry this together. Analytics and logs show how long each step took and where it got stuck, while Run History and Retry keeps a record of every single execution so you can retry failed runs in bulk rather than one at a time. Duration matters more than it looks: a step that suddenly takes three times as long is the first sign of a problem that has not become an error yet. To see the human side of the same event, reading the run next to the customer's full history on a single card settles the question of whether someone already spoke to this person yesterday in a couple of seconds.

Which failures are worth retrying, and which are not?

Retrying is not free. Retrying a permanent failure a hundred times produces the same result a hundred times, burns through your quota with the receiving service, and buries the real problem under a pile of logs. So classify the failure before you retry it. The distinction below is sharp enough for daily use.

Failure type	Typical example	Retry?	Do this first
Transient connection	Timeout, service briefly unresponsive	Yes, with widening intervals	Nothing. Wait.
Expired authorisation	Channel connection has lapsed	Yes, but only after reconnecting	Reconnect the channel
Missing data	Empty phone field, invalid email format	No, pointless until the record is fixed	Fix the record or filter it out in the condition
Rule rejection	Recipient opted out, quiet-hours window	No, this is correct behaviour	Leave it. Accept the workflow's decision
Configuration error	Disconnected step, deleted template	Not before fixing it	Repair the workflow and publish the new version

This table earns its keep in operations. Inside order management workflows, "the carrier API did not respond" and "the address field is empty" wear the same red badge; retrying the first one in five minutes solves it, while retrying the second fifty times only manufactures noise.

Will a retry send a second message?

This is the real reason teams avoid bulk retries: the fear of messaging the same person twice. The fear is reasonable, but the answer is not to avoid retrying. The answer is a run that knows where it stopped. A retry has to resume from the step that failed rather than restarting the workflow from the top. Completed steps stay completed; only the stalled step runs again.

For steps that carry text there is a second safeguard. When you define an A/B variant on a step in the sequence step editor, the variant choice is fixed at run time, so a retry does not send a second message to the same person and does not switch the variant either. That also protects your measurement, since otherwise every retry would quietly corrupt the A/B result. The same discipline matters even more in campaigns that warm up leads who have gone quiet: two messages back to back to someone who has ignored you for months turns the campaign itself into a reason to unsubscribe.

When is a bulk retry the right call?

You can always retry a single run by hand, but sixty-three records accumulated overnight would take hours that way. A bulk retry is right once you have confirmed they all stopped for the same reason. The check is simple: group the failure messages in the history, and if one reason covers the overwhelming majority of the list, there is a common root.

Order matters too. Fix the root first, reconnect the channel or restore the missing permission, and retry afterwards. Working in the opposite order simply doubles the error count. Watch the timing as well: releasing hundreds of runs at once can exhaust the receiving service's quota in a minute. Rate and working-hours limits exist for this. When you have defined a ceiling on runs per hour or per day, a per-contact limit and a working-hours window, the recovery wave spreads itself out. This is most visible in appointment cancellation and rescheduling flows, where a recovery wave landing at dawn turns a helpful reminder into a middle-of-the-night notification.

Why do half-finished runs pile up?

Not every failure is loud. Some runs neither fail nor finish: they hang on a wait step, sit there expecting a reply, and are forgotten. Because they appear to be running, nobody notices them, and within weeks they accumulate. The result is a board where nobody knows how many workflows are genuinely alive.

Stalled Workflow Cleanup exists precisely for that in Rocketly: half-finished runs do not build up in the system, they are detected and swept. Automatic Diagnosis goes a step further and reports problems such as a disconnected step, a workflow that has never fired, or an unconnected channel, so you are not the one hunting for them. The same housekeeping applies on the advertising side; in retargeting setups, a workflow built months ago that quietly stopped reaching anyone costs more than one that fails loudly.

Fix the workflow, or rescue the run?

These two jobs get confused constantly. Rescuing a run touches the past: it aims to reach today the thirty-eight people who were left hanging last night. Fixing the workflow touches the future: it aims to prevent the same thing tomorrow night. After an incident you need both, but sequentially and separately.

The confusion usually plays out like this. The team panics, starts editing the workflow, changes a few steps, then retries the old runs, and the recovery now executes against a different workflow than the one that failed. Version History shrinks that risk: every save is kept as a version, so you can roll a broken workflow back to an earlier version in one click, and the state before the rollback is stored as well. You lose neither the fix nor what preceded it. The logic resembles restoring a record you deleted by accident: being able to go back protects you more reliably than being careful ever will.

Who should learn about a failure, and how fast?

The real loss in the opening scene was not the broken automation. It was nine and a half hours of nobody noticing. Half of a recovery routine is therefore alert design. What works is thresholds: a single failed run wakes nobody, but five consecutive failures on the same step, or an error rate above ten percent within an hour, reaches an owner.

Owner here is an organisational word, not a technical one. Every workflow needs a named person, and the alert should land on that person rather than on a channel everyone half-reads. An alert sent to everyone counts as an alert sent to nobody. Loose thresholds end in the same place: notification fatigue turns CRM alerts into background noise within weeks, and the one message that genuinely mattered disappears into it.

Where do you rehearse a recovery before it goes live?

A recovery is itself a change, and it carries its own risk. Before a bulk retry that will touch hundreds of people, you want to see that the repaired workflow behaves the way you think it does. A dry run test does that without publishing: it executes the workflow against sample data, shows step by step which steps would run, and sends no real messages.

Placing a Wait for Approval step in front of a risky action does the same job from another angle. The workflow pauses, a notification goes to the person you chose, and if the window expires the run does not proceed unapproved. Using both together substantially lowers the chance that your recovery becomes a second incident. In sequences that turn trials into paid accounts this rehearsal habit touches revenue directly: a step that stalls on day three of onboarding means a user who finishes the trial without ever setting the product up.

How does this become a weekly habit?

A recovery capability that goes unused is just another screen. Fifteen minutes a week is enough to make it real. The order is: group the last seven days of failed runs by error message, write down the root of the three largest groups, fix the roots that can be fixed, then bulk retry those groups. After that, look at workflows that have never fired at all. They raise no errors, and they are usually the most expensive silence on the board.

The fourth move is measurement: did the error rate fall against last week, and did the average step duration grow. Those two numbers are the cheapest single-glance read on the health of your automation. In software businesses this review usually connects straight to trial and expansion flows; within a CRM setup built for SaaS companies, a weekly run review is one of the earliest places churn signals show up. If you want to start reading your own run history this way, you can open a free Rocketly account and build your first workflow.

FAQ

Does a retry restart the workflow from the beginning?

No. A properly built retry resumes the run from the step where it stopped. Completed steps do not run again; only the stalled step and everything after it is executed. That is why retrying a run that already cleared the first two steps does not produce a second welcome message.

How many retries are reasonable?

For transient failures, three to five attempts with widening gaps covers most scenarios. If it has not recovered by then the problem is not transient, and continuing only burns quota. For permanent failures the right number is zero: fix the root first.

Which runs should never be retried?

The ones rejected by rule. If the recipient opted out, the quiet-hours window is closed, or the daily message cap is full, the workflow did not fail. It behaved exactly as designed. Forcing those through breaks your consent handling.

Does it make sense to retry an old run days later?

It depends on the step. Creating a task or applying a tag still makes sense days later. A time-bound reminder does not: a reminder sent after the appointment has passed does not help, it irritates. Before retrying, ask whether the message is still true.