Retries and Failures
A task that fails is no longer the end of the line. Process automation tries it again, with exponential backoff, and keeps a record of what went wrong. A task is attempted five times by default.
Before 2.0.0 a failing task was attempted once. ApplicationsApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. that treat a transition's failed state as a business
end-state, and act on the elements sitting in it, will now see those elements picked up again by a later attempt. Set
job.attempts to 1 to keep the old behaviour.
What a failure does to the element
A failing task moves its element into the transition's failed state right away, on the very first failure. What the retry adds is that the element does not have to stay there: an automatic retry is allowed to resume from the failed state, so a later attempt picks the element back up and carries it to the end state if it succeeds.
The failed state is therefore only a resting place once the attempts have run out. If you have screens or reports built on elements in that state, they will show elements that are still being worked on.
Attempts
This is the total number of attempts, not the number of retries: job.attempts = 3 means one initial attempt followed
by at most two more. Set it to 1 to switch retrying off for a transition. The value must be a positive integer, so a
job is always attempted at least once.
The attempt count travels with the job, so changing the option affects work submitted afterwards and not work already queued.
An error returned by a task element counts as a failure and uses up an attempt. Use errors for genuinely transient problems; a database that is briefly unavailable, a remote service that times out. For an outcome your process is meant to handle, give the transition an end state instead, see Designing Workflows.
How long between attempts
Attempts are spaced out further and further apart, so a transition that depends on something broken does not hammer it. The wait triples each time, starting at five seconds and never growing beyond a day. With the default of five attempts, the waits are 5s, 15s, 45s and 2m15s. After the tenth failed attempt the wait reaches its cap, and every further attempt follows a full day after the previous one.
These intervals are fixed and currently cannot be configured. The only control you have over the total time a job keeps
trying is job.attempts.
A job that is waiting stays on the queue but is not handed out before its wait has passed, so a backlog of failing jobs does not hold up healthy work on the same queue.
Failures inside a batch
An element that fails inside a batch does not fail the run. Its failure is recorded and the job carries on with the rest of its elements.
These elements are not retried individually. They are added to the next run of the same transition, so the retry is paced by the trigger rather than by the waits above: a schedule that fires hourly retries at most once an hour.
Following failures
Every failing job is recorded as a JobFailure, available in the process automation UI. It shows the element that
failed, the transition it belongs to, how many attempts have been made out of how many are allowed, when it last failed,
and when the next attempt is due.
For batched transitions, each failed task gets its own JobFailure. A batch with several failing elements creates one
record per failed task, never a single record for the entire batch.
Records are kept after a job succeeds, so JobFailure is also the failure history of the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution.. On a busy
applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. it is worth indexing, see
Database Indexes.
A failure is in one of three states:
| State | Meaning |
|---|---|
PendingRetry | Attempts remain and another one is coming. |
MaxAttempts | The attempts have run out. Nothing will retry this alone. |
Resolved | A later attempt succeeded. |
MaxAttempts is the state that needs a person: it means process automation has given up on that element. Watching for
it makes a good operational alert.
Retrying by hand
A JobFailure whose attempts ran out can be retried from the same screens, through the retry operation. It grants
one more attempt and queues the job again immediately.
Retrying does not change the element itself. It is still in the transition's failed state, which is exactly where the retry resumes from — so if the element was moved out of that state by hand in the meantime, the retry finds nothing to do and is skipped. Retrying a failure that is already resolved, or one that is already due for another attempt, does nothing.