Skip to main content

Managing Executions

Out of the box the process-automation component provides the QueuedProcessor workflow execution type. By default, a processor which should work for most applicationsApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is provided. In some cases it can be needed to tweak the workflow configuration to suit your performance needs.

Queued Processor​

As its name suggests, the queued processor handles two concerns; Queueing and Processing. Both can be configured and can even handle completely different implementations than provided by default.

Executions can be added as a dataResource in an expander bundleExpander BundleA set of expanders sharing a repository, packaged into an expansion resource or archive., or directly in the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. through the settings directory. On this page we will provide examples for the settings directory, but it is trivial to apply this to a dataResource.

The default configuration looks as follows and is provided by the component. It is also possible to override this default, simply by defining a QueuedProcessor with the name default yourselves.

settings/queuedProcessor/default.xml
<workflowExecution type="processing::QueuedProcessor">
<name>default</name>
<maxSubmittedJobs>15</maxSubmittedJobs>
<queuePollingInterval>100</queuePollingInterval>
<shutdownTimeout>1m</shutdownTimeout>
<processing type="processing::Asynchronous"/>
<queueDriver type="processing::ApplicationPersistence">
<pollingIntervalMs>5000</pollingIntervalMs>
<localCacheCapacity>128</localCacheCapacity>
<cleanupInterval>60</cleanupInterval>
</queueDriver>
</workflowExecution>
ConfigurationDescription
maxSubmittedJobsMaximum number of jobs submitted to the underlying processor at once. Each submitted job runs on its own thread, so this is also the maximum number of jobs running concurrently.
queuePollingIntervalHow often, in milliseconds, the processor takes new jobs from the queue driver.
shutdownTimeoutPeriod in which the processor will wait for a graceful shutdown.

An example of a custom workflow execution can be as follows. It is used for a polling workflow with an ExecutorService referenced by jndi, and using an in-memory queue. This queue is non-persistent but since the execution is for polling it doesn't matter if some of the jobs are lost.

settings/queuedProcessor/polling.xml
<workflowExecution type="processing::QueuedProcessor">
<name>polling</name>
<processing type="processing::ExecutorService">
<jndiName>polling</jndiName>
</processing>
<queueDriver type="processing::InMemory"/>
</workflowExecution>

To use a non-default workflowExecution it needs to be added to the workflow configuration.

components/myComp/model/workflow/PollRecords.xml
<workflow>
<name>PollRecords</name>
...
<execution>polling</execution>
</workflow>

Work Schedules​

Not all workflows should run continuously. A queued processor allows you to define a work schedule to indicate when the processor may be active. An example of such schedule is shown below:

settings/queuedProcessor/withSchedule.xml
<workflowExecution type="processing::QueuedProcessor" name="withSchedule">
...
<workSchedules>
<workSchedule>
<startAt>06:00</startAt>
<until>12:00</until>
</workSchedule>
<workSchedule>
<startAt>15:00</startAt>
<until>19:00</until>
<timezone>Europe/Brussels</timezone> <!-- optional: specify timezone -->
</workSchedule>
<workSchedule>
<startAt>00:00</startAt>
<until>24:00</until> <!-- 24:00 results in 00:00 the next day -->
<weekDay>sunday</weekDay> <!-- optional: specify day of week it is active -->
</workSchedule>
</workSchedules>
</workflowExecution>

The configuration above will result in a schedule as shown below, assuming the default timezone is UTC and Brussels is UTC+2.

Is there a need for more advanced schedules? See how to use calculated/runtime schedules in the advanced work schedule guide.

Processing Types​

Processing is responsible for scheduling available tasks to an underlying resource. The implementation must always implement the net.democritus.processAutomation.TaskJobExecutorService interface. The component provides two types of processing by default; Asynchronous and ExecutorService.

  • Asynchronous utilizes the jakarta.ejb.Asynchronous api from Jakarta EE. It is supported by all applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. servers which adhere to the specification and should therefore always be available.

  • ExecutorService makes use of an injected jakarta.enterprise.concurrent.ManagedExecutorService. Some applicationsApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. such as TomEE provide a default, for others you may need to explicitly configure one.

    ConfigurationDescription
    jndiName(optional) The jndi name of the resource, as configured on the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. server.

Queue Types​

Each type of queue needs a QueueDriver implementation. An ApplicationPersistence and InMemory driver are provided by default. It is of course possible to use different queueing technologies by implementing a driver as an expansionResource.

  • ApplicationPersistence persists submitted jobs to a dataElement (TaskJob) in the processAutomation component. The advantage is that these jobs will not be lost on a restart. The downside is a bit more overhead to maintain state for these jobs.

    ConfigurationDescription
    pollingIntervalMsInterval in milliseconds in which the database is polled for new tasks
    localCacheCapacityThe number of jobs to keep in memory, to avoid waiting on the next polling tick
    cleanupIntervalNumber of minutes between runs of a cleanup removing all 'Complete' jobs (default: 60, disabled at 0)

    Jobs flow through two polling loops. The queue driver prefetches queued TaskJobs from the database into a local cache every pollingIntervalMs. This is the slower loop, since it needs a database roundtrip. When a job is submitted while the local cache is empty, it is placed in the cache directly, so it does not have to wait for the next database poll. The processor then takes jobs from this local cache every queuePollingInterval, as long as fewer than maxSubmittedJobs jobs are running.

  • InMemory is a non-persistent queue keeping all available jobs in memory. This provides a speed advantage with the downside of losing jobs when the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is restarted.

Policies​

Policies can be added to the configuration to specify what process-automation should do in case of errors. If no policies are defined, it will just keep trying to complete work indefinitely.

  • DailyRecoveryPolicy Starts WorkerNode recovery every day at a specified time and timezone.

    ConfigurationDescription
    startRecoveryAtAt what time to start recovery. (default: 00:00)
    timezoneWhich timezone to use for this time. (default: JVM timezone)

Policies are added to the policies of a queued processor:

settings/queuedProcessor/default.xml
<workflowExecution type="processing::QueuedProcessor">
<name>default</name>
...
<policies>
<policy type="processing::DailyRecoveryPolicy">
<startRecoveryAt>03:00</startRecoveryAt>
<timezone>Europe/Brussels</timezone>
</policy>
</policies>
</workflowExecution>

The following policies exist but are currently not implemented:

  • ErrorRateShutdownPolicy Shuts down a WorkerNodeService once its internal errors exceed the specified error-rate. Note that a TaskResult.error does not count towards this number, only internal errors (such as failing to commit) are counted.

    ConfigurationDescription
    whenErrorRateExceedsHow many errors per minute should trigger a shutdown.
  • ErrorRecoveryPolicy Starts WorkerNode recovery once any of its services shutdown due to an error. Note that the indicated default value applies only when the configuration does not.

    ConfigurationDescription
    afterMinutesHow many minutes to wait before initiating recovery. (default: 0)

Worker Node Startup​

A WorkerNode is created in the Init state and only starts processing once startup is allowed. By default this happens immediately. Use the PROCESS_AUTOMATION_STARTUP_DELAY environment variable to change this, for example to give other services time to become available after a deployment.

ValueBehavior
unset or 0WorkerNodes start immediately.
positive numberWorkerNodes start after this many seconds.
negative numberWorkerNodes never start automatically and wait for a start command.

A WorkerNode waiting in Init can also be started by hand with the start data operationData Operation (DataOperation). A type-safe action (e.g., Create, Update, Custom) executed on DataElement instances to modify data or trigger business logic., available in the process automation UI. This starts all WorkerNodes running on the same node.

For full control, custom code can call net.democritus.processautomation.WorkerNodeStartup.ready() to allow the WorkerNodes to start, for example once an applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution.-specific readiness check passes. WorkerNodeStartup.isReady() reports whether startup has been allowed.

Health Checks​

It is important to know if your applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is up and functional. In a containerized environment such as Docker, health checks can be added to improve stability. By adding the following option, we can expose an endpoint which returns a status based on the internal status of the WorkerNode and its services.

Option
workflow.healthcheck.endpoint Application

Adds a healthcheck http endpoint at the provided path. When setting this you must make sure it is not already occupied by another http handler. The endpoint will be mounted within the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. context, so the resulting path will be /$app/healthcheck in this example. If no value is specified it will default to /healthcheck.

<options>
<workflow.healthcheck.endpoint/>
</options>
<options>
<workflow.healthcheck.endpoint>/healthcheck</workflow.healthcheck.endpoint>
</options>

To use this with Docker, you would add the following line to your Dockerfile:

HEALTHCHECK --interval=10s --timeout=20s --retries=3 --start-period=1m \
CMD curl -sf http://localhost/myapp/healthcheck || exit 1

These values can be tweaked to suit your needs, but this should serve as a good starting point.