Skip to main content

ProcessAutomation 2.0.0

· 9 min read
Dries Ryckbosch
Dries Ryckbosch
Engineer
Jorren Hendriks
Jorren Hendriks
R&D Engineer
caution

The beam is currently a release candidate. We will be collecting feedback on these before a full production-ready release is made.

ProcessAutomation v2 closes some feature gaps and stability issues from our v1 engine. Some of the new features include:

  • Automatic Retries: Built-in exponential backoff for failed tasks with configurable max attempts. (docs)
  • Batch Queuing: FlowEngine and Schedule now submit BatchJobs directly to the queue. (docs)
  • Native TimerJobs: Always available by default; feature toggles are no longer required.
  • Failure Tracking: A new JobFailure dataElement logs all Task and BatchJob failures.
  • Delayed Startup: WorkerNodes can wait for a startup delay or an explicit start command before they begin processing. (docs)
  • Angular UI Support: A new FeatureModule which can be imported into an angular application.
  • Architecture Upgrades: Lays groundwork for future capabilities like sub-processes and multiple begin-states.

The new architecture integrates Tasks, Batches, Timers and Failures into a single coherent system.

Resolved Issues​

  • Lost Queue Entries: TaskJob entries no longer vanish on error. They now persist with Executing, Complete, or Failed states.
  • Concurrency Limits: maxConcurrentJobs is removed from QueuedProcessor to eliminate confusion. The queue manages memory availability natively, while maxSubmittedJobs strictly controls processor submissions.
  • WorkerNode Stability: WorkerNodes are unified into a single element, preventing cascading crashes during service startup and shutdown.
  • Eager Triggers: Schedules and FlowEngines now use BatchJobs, pausing new work submissions until the active batch completes. Batch size and concurrency are tunable through options.

TomEE Jakarta Image​

We've observed issues in application workflows after upgrading to the Jakarta versions of TomEE (9+). The root cause was identified and a patch has been merged upstream. Staring with version 10.3.0 these issues are resolved. We recommend applications on TomEE 10 to upgrade immediately. Applications still running on TomEE 8 can now safely start migrating to Jakarta.

A full migration guide is available below. (Read more)

Migration guide​

To get started, upgrade the process-automation beam. This includes updates to the net.democritus:Expanders artifact, so make sure your other beams are up-to-date and compatible.

<expansionResource>org.normalizedsystems.beam:process-automation-beam::2.0.0-rc1</expansionResource>

Database upgrade​

Some new DataElements were added, some others were changed. The only data which needs to be migrated is the TaskJobQueue, the rest is generated at runtime.

  1. Prepare data for clean migration.
Prepare TaskJobQueue data (PSQL)
-- Clean up previously completed tasks
DELETE FROM PROCESS_AUTOMATION.TaskJobQueue WHERE status = 'Committed';
-- Reset remaining tasks to their initial state
UPDATE PROCESS_AUTOMATION.TaskJobQueue SET status = 'Queued';
  1. Apply schema upgrades
Schema upgrade (PSQL)
-- Drop removed element 'WorkerNodeService'
DROP TABLE IF EXISTS PROCESS_AUTOMATION.WorkerNodeService;

-- Replace 'name' field with 'node' and add 'service' field on 'WorkerNode'
ALTER TABLE PROCESS_AUTOMATION.WorkerNode
DROP COLUMN IF EXISTS name,
ADD COLUMN IF NOT EXISTS node VARCHAR(255),
ADD COLUMN IF NOT EXISTS service VARCHAR(255);

-- Rename element 'TaskJobQueue' to 'TaskJob'
ALTER TABLE IF EXISTS PROCESS_AUTOMATION.TaskJobQueue RENAME TO TaskJob;

-- Add 'batch_id', 'maxAttempts', 'attemptAfter' and 'createdAt' fields to 'TaskJob'
ALTER TABLE PROCESS_AUTOMATION.TaskJob
ADD COLUMN IF NOT EXISTS batch_id BIGINT,
ADD COLUMN IF NOT EXISTS maxAttempts INTEGER DEFAULT 1,
ADD COLUMN IF NOT EXISTS attemptAfter TIMESTAMP,
ADD COLUMN IF NOT EXISTS createdAt TIMESTAMP;

-- Add 'createdAt' field to 'TimerJob'
ALTER TABLE IF EXISTS PROCESS_AUTOMATION.TimerJob
ADD COLUMN IF NOT EXISTS createdAt TIMESTAMP;
  1. Create new tables. NOTE: check the expanded 000-init-database.sql script for the exact definition. Can be slightly different depending on your configuration.
Create tables (PSQL)
-- Create element 'BatchJob'
CREATE TABLE IF NOT EXISTS PROCESS_AUTOMATION.BatchJob (
id BIGINT NOT NULL,
completedJobs INTEGER,
createdAt TIMESTAMP,
failedJobs INTEGER,
key VARCHAR(255),
maxAttempts INTEGER,
payload VARCHAR(16384),
status VARCHAR(255),
timeout TIMESTAMP,
totalJobs INTEGER,
totalTasks INTEGER,
PRIMARY KEY (id)
);

-- Create element 'JobFailure'
CREATE TABLE IF NOT EXISTS PROCESS_AUTOMATION.JobFailure (
id BIGINT NOT NULL,
attempts INTEGER,
createdAt TIMESTAMP,
key VARCHAR(255),
lastFailureAt TIMESTAMP,
maxAttempts INTEGER,
name VARCHAR(255),
reference VARCHAR(255),
retryAfter TIMESTAMP,
source VARCHAR(255),
status VARCHAR(255),
target VARCHAR(255),
PRIMARY KEY (id)
);

-- Create element 'TimerJob'
CREATE TABLE IF NOT EXISTS PROCESS_AUTOMATION.TimerJob (
id BIGINT NOT NULL,
createdAt TIMESTAMP,
key VARCHAR(255),
payload VARCHAR(16384),
status VARCHAR(255),
timestamp TIMESTAMP,
type VARCHAR(255),
PRIMARY KEY (id)
);
  1. Add optimized indexes & drop old ones
Create indexes (PSQL)
-- DROP any existing indexes on PROCESS_AUTOMATION tables first.

CREATE INDEX IF NOT EXISTS idx_task_job_priority
ON PROCESS_AUTOMATION.TaskJob (queue, priority DESC)
WHERE status = 'Queued';

CREATE INDEX IF NOT EXISTS idx_task_job_status
ON PROCESS_AUTOMATION.TaskJob (status);

CREATE INDEX IF NOT EXISTS idx_task_job_batch
ON PROCESS_AUTOMATION.TaskJob (batch_id)
WHERE batch_id IS NOT NULL;

CREATE INDEX IF NOT EXISTS idx_timer_job_active_timestamp
ON PROCESS_AUTOMATION.TimerJob (status, timestamp)
WHERE status = 'Active';

CREATE INDEX IF NOT EXISTS idx_batch_job_key
ON PROCESS_AUTOMATION.BatchJob (key)
WHERE key IS NOT NULL;

Adding to AngularApp​

The new featureModule simply has to be imported by your AngularApp.

<angularApp xmlns="https://schemas.normalizedsystems.org/xsd/angularProjects/7/0/0">
<import module="net.democritus:feature-modules::process-automation" scope="expansion"/>
<!-- ... -->
</angularApp>

When using a custom menu the process-automation submenu can be added by importing the MENU_ROOT_ITEM_PROCESS_AUTOMATION menu item.

Retries and Changed Behavior​

Tasks now default to 5 attempts with a 5-second exponential backoff. If your application relies on the Failed state as an end-state for business logic, this might break existing behavior. Using the default setting will retry the transition up to 4 times, while the transition would only run exactly once before.

To opt out of automatic retries, set job.attempts to 1 on Application. We recommend allowing retries for most tasks to prevent workflows from getting permanently stuck in their failed state.

Option
job.attempts TransitionWorkflowComponentApplicationCascading

When defined, the transition will take at most this many attempts. (positive integer)

<options>
<job.attempts>1</job.attempts>
</options>

Tracing​

The process-automation-tracing resource was added to the beam. This enables logging for each transition. This adds logging to each transition such as:

Transition complete, target:<Order id=8489>, workflow:<Workflow name=OrderFlow>, transition:<Transition name=Fulfill>, from:<State name=Fulfilling>, to:<State name=Fulfilled>

For some cases this is too verbose or an alternative implementation exists. To disable the behavior add the exclude option to the application or #processAutomation.tracelog.disable tag to your profile.

Option
processAutomation.tracelog.exclude TransitionWorkflowComponentApplicationCascading

Disable trace logging for the underlying transitions.

<options>
<processAutomation.tracelog.exclude/>
</options>

Custom triggers​

The api for scheduling transitions has changed. Any custom triggers using this api should be refactored.

Before
// DocumentFlowEventHandler.java
checkSuccess(this.scheduler.scheduleTransition(TransitionContext.of(context, DocumentFlowTransition.PUBLISH).withTarget(target)));
After
// DocumentFlowEventHandler.java
this.scheduler.submitTransition(TransitionContext.of(context, DocumentFlowTransition.PUBLISH).withTarget(target)).throwOnError();

Custom task target​

Support for the target.class.custom option on TaskElement has been dropped in favor of the transition.parameter.class option on Transition. The option must be moved from the TaskElement to the Transition. This impacts the signature of the perform method which should be refactored. This change makes task implementations more consistent and decouples transition context from the task signature.

To automatically convert between the options, use the following transmuter:

Transmutation
MigrateCustomTargetTaskElement

Applies the transition.parameter.class option to all transitions where the task currently has the target.class.custom option. Removes the target.class.custom option from the task.

Before
public TaskResult<Void> perform(ParameterContext<MyCustomClass> targetParameter) {
Context context = targetParameter.getContext();
DataRef target = context.find(FunctionTargetContext.class).orElseThrow().getTarget();
CityDetails details = CityLocalAgent.getCityAgent(context).getDetails(target)
.getValueOrElseThrow(e -> new IllegalStateException("Could not retrieve Details"));
MyCustomClass myCustomClass = targetParameter.getValue();
// ...
return TaskResult.success();
}
After
public TaskResult<Void> perform(ParameterContext<CityDetails> targetParameter) {
Context context = targetParameter.getContext();
CityDetails details = targetParameter.getValue();
MyCustomClass myCustomClass = ValueParameterContext.retrieve(context, MyCustomClass.class)
.orElseThrow(() -> new IllegalArgumentException("Missing value parameter MyCustomClass"));
// ...
return TaskResult.success();
}

API Refactoring​

Internal APIs have been updated and require refactoring for custom code or expanders:

  • {Transition}Transitioner:
    • State updates now require TransitionParams and TransitionResult instead of StateUpdate.
    • The deprecated @EJB {DataElement}Local field and the executeWithTarget method are removed. Custom code in the anchors should use the {DataElement}{Field}TransitionService instead.
  • {Workflow}Transition:
    • The enum constructor changed from (String, int priority) to (String, Priority priority, int attempts).
    • getPriority() returns the Priority enum, use asInteger() to get the numeric value.
    • Use Priority.fromValue() to go from an integer to Priority.
  • {Workflow}Scheduler:
    • Use submitTransition instead of scheduleTransition.
    • Use forkTransition instead of performTransition.
    • The result of these methods is now a TaskJobResult instead of TaskResult<Void>
  • TaskJobQueue: DataElement renamed to TaskJob. All generated artifacts follow the same rename.
  • Runtime classes:
    • net.democritus.processautomation.TaskJob is moved and renamed to net.democritus.processautomation.task.TaskJobPayload.
    • net.democritus.processautomation.execution.TaskJobExecutor is moved to the net.democritus.processautomation.task.TaskJobExecutor.
    • A new net.democritus.processautomation.task.TaskJob class has been added
  • TaskJobQueueDriver gained the completeJob() and rejectJob() methods, which custom drivers must implement.
  • Jackson: Moved from logic/ext to logic/gen. Overlays of this class need to be moved. It is recommended to use Jackson.getObjectMapper() instead of Jackson.OBJECT_MAPPER.
  • WorkerNode: Consolidated with WorkerNodeService. The name field is replaced by a node and service fields to explicitly link it to a service. WorkerNodeService and WorkflowRecovery interfaces are updated accordingly (WorkflowRecovery requires a getServiceName() method). The finders findByNameEq, findExpiredOnly and findByIsLeaderEq are replaced by findByValues, findExpired, findLeader, findByServiceEq and findAwaitRecovery.
Before
WorkerNodeFindByNameEqDetails finder = new WorkerNodeFindByNameEqDetails();
finder.setName(WorkerNodeConfig.getWorkerNodeName());
After
WorkerNodeFindByValuesDetails finder = new WorkerNodeFindByValuesDetails();
finder.setNode(WorkerNodeConfig.getNode());

Removed options​

  • transition.noSelfLoop: Self loops are now always prevented.
  • workflow.priority: Use job.priority instead.
  • processAutomation.TimerJob.enable: TimerJob is always enabled.
  • schedule.maxFetchSize: Renamed to batch.maxFetchSize. The old name is still accepted as alias.