Managing Executions
Out of the box the process-automation component provides the QueuedProcessor workflow execution type. By default, a
processor which should work for most applicationsApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is provided. In some cases it can be needed to tweak the workflow
configuration to suit your performance needs.
Queued Processor
As its name suggests, the queued processor handles two concerns; Queueing and Processing. Both can be configured and can even handle completely different implementations than provided by default.
Executions can be added as a dataResource in an expander bundleExpander BundleA set of expanders sharing a repository, packaged into an expansion resource or archive., or directly in the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. through the settings
directory. On this page we will provide examples for the settings directory, but it is trivial to apply this to a
dataResource.
The default configuration looks as follows and is provided by the component. It is also possible to override this
default, simply by defining a QueuedProcessor with the name default yourselves.
<workflowExecution type="processing::QueuedProcessor">
<name>default</name>
<maxSubmittedJobs>15</maxSubmittedJobs>
<queuePollingInterval>100</queuePollingInterval>
<shutdownTimeout>1m</shutdownTimeout>
<processing type="processing::Asynchronous"/>
<queueDriver type="processing::ApplicationPersistence">
<pollingIntervalMs>5000</pollingIntervalMs>
<localCacheCapacity>128</localCacheCapacity>
<cleanupInterval>60</cleanupInterval>
</queueDriver>
</workflowExecution>
| Configuration | Description |
|---|---|
| maxSubmittedJobs | Maximum number of jobs submitted to the underlying processor at once. Each submitted job runs on its own thread, so this is also the maximum number of jobs running concurrently. |
| queuePollingInterval | How often, in milliseconds, the processor takes new jobs from the queue driver. |
| shutdownTimeout | Period in which the processor will wait for a graceful shutdown. |
An example of a custom workflow execution can be as follows. It is used for a polling workflow with an ExecutorService referenced by jndi, and using an in-memory queue. This queue is non-persistent but since the execution is for polling it doesn't matter if some of the jobs are lost.
<workflowExecution type="processing::QueuedProcessor">
<name>polling</name>
<processing type="processing::ExecutorService">
<jndiName>polling</jndiName>
</processing>
<queueDriver type="processing::InMemory"/>
</workflowExecution>
To use a non-default workflowExecution it needs to be added to the workflow configuration.
<workflow>
<name>PollRecords</name>
...
<execution>polling</execution>
</workflow>
Work Schedules
Not all workflows should run continuously. A queued processor allows you to define a work schedule to indicate when the processor may be active. An example of such schedule is shown below:
<workflowExecution type="processing::QueuedProcessor" name="withSchedule">
...
<workSchedules>
<workSchedule>
<startAt>06:00</startAt>
<until>12:00</until>
</workSchedule>
<workSchedule>
<startAt>15:00</startAt>
<until>19:00</until>
<timezone>Europe/Brussels</timezone> <!-- optional: specify timezone -->
</workSchedule>
<workSchedule>
<startAt>00:00</startAt>
<until>24:00</until> <!-- 24:00 results in 00:00 the next day -->
<weekDay>sunday</weekDay> <!-- optional: specify day of week it is active -->
</workSchedule>
</workSchedules>
</workflowExecution>
The configuration above will result in a schedule as shown below, assuming the default timezone is UTC and Brussels is UTC+2.
Is there a need for more advanced schedules? See how to use calculated/runtime schedules in the advanced work schedule guide.
Processing Types
Processing is responsible for scheduling available tasks to an underlying resource. The implementation must always
implement the net.democritus.processAutomation.TaskJobExecutorService interface. The component provides two types of
processing by default; Asynchronous and ExecutorService.
-
Asynchronousutilizes thejakarta.ejb.Asynchronousapi from Jakarta EE. It is supported by all applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. servers which adhere to the specification and should therefore always be available. -
ExecutorServicemakes use of an injectedjakarta.enterprise.concurrent.ManagedExecutorService. Some applicationsApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. such as TomEE provide a default, for others you may need to explicitly configure one.Configuration Description jndiName (optional) The jndi name of the resource, as configured on the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. server.
Queue Types
Each type of queue needs a QueueDriver implementation. An ApplicationPersistence and InMemory driver are provided by
default. It is of course possible to use different queueing technologies by implementing a driver as an
expansionResource.
-
ApplicationPersistencepersists submitted jobs to a dataElement (TaskJob) in the processAutomation component. The advantage is that these jobs will not be lost on a restart. The downside is a bit more overhead to maintain state for these jobs.Configuration Description pollingIntervalMs Interval in milliseconds in which the database is polled for new tasks localCacheCapacity The number of jobs to keep in memory, to avoid waiting on the next polling tick cleanupInterval Number of minutes between runs of a cleanup removing all 'Complete' jobs (default: 60, disabled at 0) Jobs flow through two polling loops. The queue driver prefetches queued TaskJobs from the database into a local cache every
pollingIntervalMs. This is the slower loop, since it needs a database roundtrip. When a job is submitted while the local cache is empty, it is placed in the cache directly, so it does not have to wait for the next database poll. The processor then takes jobs from this local cache everyqueuePollingInterval, as long as fewer thanmaxSubmittedJobsjobs are running. -
InMemoryis a non-persistent queue keeping all available jobs in memory. This provides a speed advantage with the downside of losing jobs when the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is restarted.
Policies
Policies can be added to the configuration to specify what process-automation should do in case of errors. If no policies are defined, it will just keep trying to complete work indefinitely.
-
DailyRecoveryPolicyStarts WorkerNode recovery every day at a specified time and timezone.Configuration Description startRecoveryAt At what time to start recovery. (default: 00:00) timezone Which timezone to use for this time. (default: JVM timezone)
Policies are added to the policies of a queued processor:
<workflowExecution type="processing::QueuedProcessor">
<name>default</name>
...
<policies>
<policy type="processing::DailyRecoveryPolicy">
<startRecoveryAt>03:00</startRecoveryAt>
<timezone>Europe/Brussels</timezone>
</policy>
</policies>
</workflowExecution>
The following policies exist but are currently not implemented:
-
ErrorRateShutdownPolicyShuts down a WorkerNodeService once its internal errors exceed the specified error-rate. Note that aTaskResult.errordoes not count towards this number, only internal errors (such as failing to commit) are counted.Configuration Description whenErrorRateExceeds How many errors per minute should trigger a shutdown. -
ErrorRecoveryPolicyStarts WorkerNode recovery once any of its services shutdown due to an error. Note that the indicated default value applies only when the configuration does not.Configuration Description afterMinutes How many minutes to wait before initiating recovery. (default: 0)
Worker Node Startup
A WorkerNode is created in the Init state and only starts processing once startup is allowed. By default this
happens immediately. Use the PROCESS_AUTOMATION_STARTUP_DELAY environment variable to change this, for example to
give other services time to become available after a deployment.
| Value | Behavior |
|---|---|
unset or 0 | WorkerNodes start immediately. |
| positive number | WorkerNodes start after this many seconds. |
| negative number | WorkerNodes never start automatically and wait for a start command. |
A WorkerNode waiting in Init can also be started by hand with the start data operationData Operation (DataOperation). A type-safe action (e.g., Create, Update, Custom) executed on DataElement instances to modify data or trigger business logic., available in the process
automation UI. This starts all WorkerNodes running on the same node.
For full control, custom code can call net.democritus.processautomation.WorkerNodeStartup.ready() to allow the
WorkerNodes to start, for example once an applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution.-specific readiness check passes. WorkerNodeStartup.isReady()
reports whether startup has been allowed.
Health Checks
It is important to know if your applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. is up and functional. In a containerized environment such as Docker, health checks can be added to improve stability. By adding the following option, we can expose an endpoint which returns a status based on the internal status of the WorkerNode and its services.
ApplicationAdds a healthcheck http endpoint at the provided path. When setting this you must make sure it is not already
occupied by another http handler. The endpoint will be mounted within the applicationApplicationThe overarching model container grouping all components, domain logic, and configurations for an enterprise solution. context, so the resulting path
will be /$app/healthcheck in this example. If no value is specified it will default to /healthcheck.
<options>
<workflow.healthcheck.endpoint/>
</options>
<options>
<workflow.healthcheck.endpoint>/healthcheck</workflow.healthcheck.endpoint>
</options>
To use this with Docker, you would add the following line to your Dockerfile:
HEALTHCHECK \
CMD curl -sf http://localhost/myapp/healthcheck || exit 1
These values can be tweaked to suit your needs, but this should serve as a good starting point.