Skip to main content

· 6 min read
Frédéric Hannes
Koen De Cock

Version 5.15.0 of the Expanders brings quite a few new features and improvements! This post will cover an overview of what's new and provide a migration guide to assist in updating. The full release notes can be found here.

Resources

The expansion resources below provide Expanders 5.15.0.

ResourceVersion
Expanders5.15.0
nsx-default-stack2022.15.0
rest-jaxrs-stack3.4.1
caution

Due to some structural changes in the way Struts is packaged, it is recommended to update to one of the listed stack resources. It is also recommended to update expanders-maven-plugin to at least version 2022.5.2.

Changes and improvements

Changes in GlobalOptionSettings

Several of the settings in GlobalOptionSettings have been replaced with options on the ApplicationInstance. The long-term goal is to make the GlobalOptionSettings, TechnicalInfrastructure, PresentationSettings and BusinessLogicSettings obsolete.

BeanInterfacePolicy

The beanInterfacePolicy setting could either be BOTH, LOCAL or REMOTE and controls which interfaces are present on the bean. However, in practice, only the LOCAL and BOTH settings were functional. The option ejb.interfaces.localOnly replaces beanInterfacePolicy='LOCAL', while beanInterfacePolicy='BOTH' becomes the default.

Security Options

The following flags have been migrated to options:

  • enforceHttpMethod: replaced by option struts.security.enforceHttpMethod (Prevents requests to struts control layer from using the wrong HTTP Method.)
  • useCsrfProtection: replaced by option struts.security.csrfProtection (Adds a security mechanism to protect against CSRF attacks.)
  • useJavaEESecurity: replaced by option struts.security.custom, or simply remove account Component from the Application. (Removes login mechanism from the struts-header.xml file.)

GenerateArtifactLabel

This feature added a postfix to the versions of Component artifacts in the project. This allows you to install artifacts or deploy them to the nexus so speed up subsequent maven builds by using e.g. the -Pslim profile.

The flag has been replaced by the option mvn.version.appendUniqueLabel. It also now appends the applicationInstance shortname and expanders version, instead of the names of the settings objects. This change is necessary if we want to move away from the settings elements.

Identity generation strategy for JPA

The options persistence.identifier, persistence.sequence.name and persistence.sequence.schema have been renamed from options previously marked as experimental. These options allow a specific identity generation strategy to be selected for a DataElement. The options are available at any level of the model above DataElement and cascaded down. They can be overriden on a lower level for granular control if needed.

The options support either auto-incremented identity columns or native sequences.

For more information, visit the documentation about id generation.

Struts2 6.0

Support for Struts2 6.0.3 was added to the expanders and is now the default version. From this version and onward, the expanders will start supporting a specific contemporary release version of Struts to improve the distribution of security patches.

There are some breaking changes both in how we handle Struts and the framework itself. See the migration guide for more information.

· 5 min read
Frédéric Hannes

Version 3 of the REST Expanders bundle brings quite a few new features and improvements! This post will cover an overview of what's new and provide a migration guide to assist in updating. The full release notes can be found here.

Resources

The expansion resources below provide REST Expanders 3.0.0.

ResourceVersion
rest-expanders3.0.0
rest-jaxrs-stack3.0.0

Changes and improvements

Java 17

The REST Expanders have fully embraced Java 17! The expanders now require JRE 17 to run, as they now target Java 17 bytecode when building. Aside from the expanders, also the code they generate will now also require a Java 17 target. This means that you can expect Java 17 specific to start appearing in expanded applications!

Transmutations & Validations

With the introduction of model transmutations and validations validations, the REST Expanders now provide their own transmuters and validations to support the development process.

Transmutations

There are two transmutations available, which can provision a model with the required elements and option to expose a default JAX-RS REST API:

  • On elements::Component, the transmutation CreateDefaultRestApi can be used to provision a Component and all of its DataElements a default API.
    mvn expanders:transmute-model -Dtransmutation=CreateDefaultRestApi -DtargetElement=componentName
  • On elements::DataElement, the transmutation AddDefaultRestApi can be used to provision a single DataElement with a default API, given that its component has the option enableJaxrs.
    mvn expanders:transmute-model -Dtransmutation=AddDefaultRestApi -DtargetElement=componentName::dataElementName

The transmutations are also accessible in the context menus in the µRadiant and take into account modifications to the provisioned elements and options where possible.

Validations

There are 5 new model validations that check for common mistakes made when using the options for the REST Expanders. More will be added in the future, but these will already cover quite a few issues.

· 13 min read
Koen De Cock

Version updates

ResourcePrevious versionNew Version
nsx-default-stack2021.9.02022.8.4
Expanders5.1.05.9.9
base-components2021.3.12022.7.1
web-styles2021.1.02022.4.1

Breaking Changes

  • The net.palver.util.Options.Option class no longer accepts null values. Option types exist to represent nullable values in a more typesafe way as either Some {value} or None. Some null breaks this contract. If you need to convert a value that can be null, use the Options.notNull(V) method
  • Removed name field from ProjectionPojoExpander with option nameNotWanted. The name field was still being set for DataElements with the nameNotWanted option, which was redundant. To revert this behaviour, use legacy.nameNotWanted.pojoName.
  • Deprecated JndiProperties and removed its behavior as it pertains only to obsoleted external RMI invocation
  • Deprecated net.democritus.mapping.Base64 in favor of java.util.Base64
  • Removed classes: FileInserter, ReportFile, SimpleHtmlFormat (for removed ReportExpander)

New Features

OpenID Connect

The account component now supports OpenID Connect, which allows you to authenticate users using any provider that follows the standard.

See here for more detailed information.

This page describes the setup and solutions for Google Cloud Platform, Microsoft Azure, Keycloak and Auth0.

XML representation of the Model

note

The format described here is the way the µRadiant exports the model files and how prime-core writes the files when writing model files programmatically. We will release a tool to automatically convert a model using maven in the near future.

note

Model files described in the old format are still compatible and can be used to expand with the newest version of the stack.

There have been a number of improvements to make the model XML files more readable and concise.

First, several irrelevant fields have been removed from the model. In addition, fields that are empty will not be exported. You can see the effect in the Component XML, where only the relevant information remains:

Before
<component name="account">
<version>1.0</version>
<fullName>BASE:account</fullName>
<businessOwner name="NSX"/>
<modelOwner name="NSX"/>
<firstAnalyst name="Paul"/>
<modelRepository name="BaseComponents"/>
<customRepository name="BaseComponents"/>
<description/>
<mainScreen/>
<componentDependencies>
<componentDependency name="account:utils">
<dependsOn name="utils" version="1.0"/>
<isManyToMany value="false"/>
</componentDependency>
</componentDependencies>
<componentOptions/>
</component>
After
<component name="account">
<version>1.0</version>
<fullName>BASE:account</fullName>
<componentDependencies>
<componentDependency name="account:utils">
<dependsOn name="utils"/>
<isManyToMany>false</isManyToMany>
</componentDependency>
</componentDependencies>
</component>

You may also notice that the boolean values are no longer represented as attributes, to be more in line with other values:

Before
<field name="name">
<fieldType>VALUE_FIELD</fieldType>
<description/>
<isInfoField value="true"/>
<isListField undefined="true"/>
<valueField name="ValueField:Account_name">
<valueFieldType component="" name="String"/>
</valueField>
</field>
After
<field name="name">
<fieldType>VALUE_FIELD</fieldType>
<isInfoField>true</isInfoField>
<isListField>false</isListField>
<valueField>
<valueFieldType component="" name="String"/>
</valueField>
</field>

Next, the options follow a more compact notation. Options are always key-value pairs, which allows us to use a more compact notation:

Before
<componentOptions>
<componentOption name="account:hasDataBaseSchema">
<value>ACCOUNT</value>
<componentOptionType name="hasDataBaseSchema"/>
</componentOption>
<componentOption name="account:isBaseComponent">
<value/>
<componentOptionType name="isBaseComponent"/>
</componentOption>
<componentOption name="account:baseComponents.isAccount">
<value/>
<componentOptionType name="baseComponents.isAccount"/>
</componentOption>
<componentOption name="account:configuration.properties">
<value/>
<componentOptionType name="configuration.properties"/>
</componentOption>
</componentOptions>
After
<options>
<hasDataBaseSchema>ACCOUNT</hasDataBaseSchema>
<isBaseComponent/>
<baseComponents.isAccount/>
<configuration.properties/>
</options>

Not coincidentally, this arrives together with a unified representation of option-types in the model. If you are interested in representing the option types of your expanders, optionally with some value constraints, read this.

Target Element of TaskElement

Historically, the target element of a TaskElement was represented by the targetClass field, which contained the qualified java class name of the projection of the target element:

<targetClass>net.demo.PersonDetails</targetClass>

It was later replaced with a more correct link to the DataElement:

<targetElement component="registration" name="Person"/>

The targetClass field, however, allowed 2 additional use-cases:

  • To use an alternative projection in the task
  • To create a task that takes a custom class as input

To support these cases, the following options have been introduced:

  • target.element.projection: Changes the projection used from Details to the provided value
  • target.class.custom: Overrides the parameter type with the value (takes a fully qualified name)
note

The custom input class does not work in workflows or any other integration of the TaskElement in generated code.

Model Validations

The model validations are a set of rules which can be used to generate reports for application models and generate a report detailing any rule violations.

The model validations were already available in some respect, but difficulty to use. It is now possible to run the validations as a maven goal using the expanders-maven-plugin.

In addition to this, the rules are now packaged in expansion-resources, which makes it easier to distribute them. It also makes it possible for teams to develop their own set of rules and include them in the report.

Check out the Model Validations tools page to get started.

If you are interested in creating your own rules, go to this page.

Better traceability of errors during model loading

Some exceptions during the model loading lacked crucial information to trace back the exception to it's cause.

This has been improved by keeping track of a model execution context. When an exception is thrown, the model execution context will be added to the exception message:

If you are interested in how models are loaded, you can find more information on this here.

net.democritus.elements.ElementNotFoundException: Cannot find Field(registration::Person::Status)

at FlowElementTreeToComposite.getStatusField (statusField=Status, targetElement=<DataElement registration::Person>)
at FlowElementTreeToComposite.mapTree (tree=<FlowElement registration::PersonFlow>)
at ComponentTreeToComposite.mapFlowElement (flowElement=<FlowElement registration::PersonFlow>)
at ComponentTreeToComposite.mapTree (tree=<Component registration>)
at ModuleCompositeModelLoader.loadModules (moduleType=<ModuleType elements::JeeComponent>)
at ExpansionCompositeModelConverter.convert (program=<Application demo::1.0.0>, programType=<ProgramType elements::JeeApplication>)
at ModelLoader.performStep (step=ConvertModelStep)
at ModelLoader.loadModel (expansionSettings=F:\NSF\workspace\demo\conf\expansionSettings.xml)

This reveals that the issue occurred:

  • during the ConvertModelStep, (Converts Tree representation to interconnected Composite representation. This step will look up every reference and replace it with an object reference. Hence missing faulty element reference will usually break here.)
  • while converting the registration component,
  • specifically when converting the PersonFlow FlowElement,
  • when the FlowElementTreeToComposite class tried to resolve the statusField

With this information, we can conclude that for some reason the field status does not exist on the target DataElement Person. Looking at the corresponding DataElement XML file will then probably reveal that we forgot to add the field.

ContextRetriever in Control Layer

For each request, the Context object is now provided by the ContextRetriever class. This class allows you to add additional context objects to the context, which will be available in each layer through the ParameterContext.

public class ContextRetriever {

public static Context getContext() {
Map<String, Object> session = ActionContext.getContext().getSession();
Context context = (Context) session.get("context");
if (context == null) {
context = Context.emptyContext();
}
if (context.getContext(UserContext.class).isEmpty()) {
//...
}
// @anchor:context:start
// @anchor:context:end
// anchor:custom-context:start
// anchor:custom-context:end
return context;
}
//...
}

An example of this is a TranslationContext. In this application, admin user can insert translations for different languages into the database for a number of values. When a user retrieves the data, the TranslationContext provides the correct language for each user.

The TranslationContext is assigned in the ContextRetriever based on information in the session.

// anchor:custom-context:start
context=context.extend(TranslationContext.getTranslationContext(context,session));
// anchor:custom-context:end
caution

Note that the Context object is immutable and needs to be reassigned after extending it.

Another example is a TenantContext in an application where all data is linked to a tenant. Users have a tenant assigned to them and when data is retrieved, it is filtered based on the tenant provided by the TenantContext.

Workflow

The EngineService table has 2 new buttons:

  • A 'Refresh' button to force a timer reset when settings have changed. This currently on works for single-node applications.
  • A 'Run Now' button to run the EngineService a single time.

To test the recovery mechanism, it is now possible to trigger the recovery with a http call to the workflow/workflow-recover-json endpoint. Provide the workflow reference as a parameter, e.g. workflow/workflow-recover-json?workflow.name=MessageProcessorFlow

There are also a number of performance improvements, which are listed below.

NG Knockout UI

There are some small improvements in the UI to help with customizations:

  • Use the applicationInstance/component/data option cruds.table.csvExportButton to add a button to the DataElement table which export instances as CSV, taking into account the selected finder.
  • Added Data Option view.buttons.commands, which adds buttons to NG pages to open command forms. Primarily useful for prototyping and quickly testing new DataCommand features
  • Added command-button-builder to add buttons to execute commands in various ways
  • Added optional parameters searchMethod and details to data-download-action.ts to provide search parameters when exporting data.
  • Added option size to popup, which can be set to large to make the popup bigger.

Second level caching in Hibernate 5

Add the option persistence.secondLevelCache to a DataElement to enable second level cache. Currently only available for Hibernate 5 and is activated with SearchDetails.setAllowCaching(true).

Read more about it here

Component Application Settings

It might be relevant to have a properties file in your application server to configure properties of your application.

In this case, you can add the option configuration.properties to one or more of your components. This will generate a <component>ApplicationSettings class, which reads <component>.ns.properties from the classpath and exposes the properties defined in the file.

This can be a good alternative to keeping configurable properties in the ParamTargetValue table. Especially in case of security.

This option is already used in account, to allow configuration of the authentication and authorization mechanics.

DetailsWithoutRefs projection

Fetching the details projection can at times be too resource-intensive.

This is caused by the resolution of DataRefs. Each LinkField requires an additional lookup to resolve the information for the DataRef, e.g. name.

For remedy this, the option projection.detailsWithoutRefs has been implemented. By adding this option to a DataElement, a detailsWithoutRefs projection will be generated. It contains the same fields as the details projection, but the linkFields are only represented as ids.

It can be useful to use this projection as target projection for a task. Or when retrieving a large amount of instances in a find() operation.

Custom Anchors

There are some new custom anchors in the expanders:

  • Several custom anchors have been added to methods generated by the Claims feature.
  • Anchors have been added to FindAction in the Control Layer
  • Added fetch-after-create-queries anchors to FinderBean.fetchData().
  • Moved the getData-strategy anchors in the Cruds classes to a separate method, so that the getData-after anchors are reached in case of a defined strategy.

Miscellaneous

Expansion

  • Add the expansion option expansion.failFast to your expansionSettings to fail at the first error.
  • Add the expansion option expanders.traceFeatures to enable feature tracing. Features tracing adds comments around each inserted feature, which allows you to trace the origins of inserted code in the generated artifacts.
  • Add option harvest.skip to a Component to skip harvest for that component. (primarily useful for model resources)

Cleanup

  • Made Cruds.getName() opt-in. Add transient.cruds.useGetName to add getName() to the Cruds classes.

Control Layer

  • Enable findAll to be used as searchMethod on the find-json struts endpoint

Finders

  • Added support for finders with in operator in combination with linkFields

Data Import/Export

  • It was no longer possible to expand export() and importFile() pipelines without the use of the includeCsvImport and includeCsvExport options. This functionality is now available separately with the io.import and io.export options for the DataElement.
  • Add the application option csv.import.merge, which changes the CSV import so that the imported fields are merged into the existing data, as opposed to the original implementation, which overwrites all fields.

Bug Fixes

Expanders

  • Fixed implicit name field not being initialized correctly
  • Adding property tomee.jpa.factory.lazy=true to persistence.xml when expanding for TomEE with Hibernate 5, to avoid CDI loading issues. This is advised by Apache in the TomEE documentation.
  • Added InterruptRecoverer error handling
  • Fix status capitalization in FlowStateTransitioner
  • Fixed workflow claiming not working with Se finders
  • Fixed recovery failing because recovered instances were not claimed
  • Expose messages from CommandResult
  • Fixed Context not being passed by StateTransitioner, causing compareAndSet() to fail if claims are used
  • Added checkClaims() to compareAndSet() method for claimable DataElements
  • Fixed InterruptRecovererExpander duplicating _FindByNotClaimed postfix if finder with postfix is not present in the model
  • Projector class did not add imports for types of local calculated fields.
  • Fixed artifactLabel not being included in the application root pom
  • Fixed FinderBean not using {DataElement}Data.ENTITY_NAME and not applying jpa.entity.name.format option
  • Fixed some copying steps (ext, harvest) not following active layer configuration (copying no{X}Layer files)
  • Fixed getStatus(), getStatusAsEnum() return type in case of noDataLayer
  • Fixed compareAndSetStatus() method in case of noDataLayer
  • Hide tabs in waterfall setup for elements to which the users does not have view access
  • Fixed expansion option hideAnchors failing on .properties expanders
  • before-harvest and after-harvest are now correctly ignored for base-components. This prevents issues with harvest files in base-components being cleaned during harvest.
  • Fixed ext directories for some layerImplementations not being generated because of missing variables when resolving the conditions

Base-Components

  • Removed start/stop engine buttons from workflow page, since they are automatically managed by the EngineStarterBean
  • Fixed automatic timer reset on timeout when settings are changed
  • Fix incorrect timewindow validation in last minute of window
  • Added error log when recovery state transitioning failed

Web-Styles

  • access-rights.js: Update approved variable before triggering granted
  • Fixed loading for observableInstanceBuilder.js when minified
  • Fixed tabs to always select the first tab on visibility changes until the user interacts with the tabs. From then on, the selected tab is preferred.
  • Fixed nullable DateTime input field
  • Fixed loading for nsx-knockout-utils.js and nsx-interaction-model.js
  • Change nsx-navbar-fixed.css to fix issues with responsiveness
  • Fix several require statements in old js files
  • Use nsx-navbar-static.css by default, since it is less buggy. (There is a patch option webstyles.patch.navbar-fixed to revert this)
  • Only load instances when tab in waterfall is visible, and hence a parent element is selected, to reduce page load.
  • Fixed missing oxide skin for tinymce
  • Fix to size bottom of big dialogs properly

Run-time

  • Made ClaimId serializable
  • Added error log when recovery state transitioning failed
  • Fixed Result class returning a null value on error. Instead, it now throws an exception if the getValue() is called on an error result. Implementations should always check the isError() or isSuccess() method first.
  • Changed Context.getContext() typing to allow parameterized context groups

Performance Improvements

Workflow

  • Workflow runtime configuration is now cached to reduce the number of database queries
  • The detailsWithoutRefs projection is used for several elements in workflow orchestration to reduce the number of database queries
  • Optimized getParamTargetValue() with skipCount
  • Increased recovery claim timeout to 5 minutes
  • Updating EngineNodeService with projection to reduce database queries
  • Optimized database query used to update engineService.lastRunAt
  • Ignore collector from EngineService

UI

  • Render waterfall tab only once an instance is selected

· One min read
Frédéric Hannes

Removed generated PATCH annotation

To annotate PATCH methods in the REST API, the expanders used the io.swagger.jaxrs.PATCH annotation, which is part of the Swagger library. The reason for using this, is that the JAX-RS libraries did not include this by default, since PATCH was a later addition to the HTTP standard.

Using this annotation caused an issue, as it is part of the Swagger libraries, creating an unwanted coupling, as well as an issue when the enableSwagger option was not used. To resolve this, a new expanded annotation was added as <componentName>.jaxrs.method.PATCH, which is referenced when the option enableSwagger is not used.

Recent versions of JAX-RS now include the annotation javax.ws.rs.PATCH, just like the annotations for other default HTTP methods. Therefor all dependencies on both the Swagger PATCH annotation, as well as the generated one have been replaced. As this deprecates the generated annotation, this has also been removed from the expanders.

To update custom code that uses either annotation, simply replace the import with javax.ws.rs.PATCH.

· 4 min read
Frédéric Hannes

RestFacade removal

The RestFacade class which was used previously in the control layer as a bridge between the Connector and Agent classes has been removed. This was an old artifact left from the original REST JAX-RS implementation for the NS application stack. This class was intended to provide some encapsulation, but it did not really encapsulate anything and violated data version transparency.

  • $dataElement.name$RestFacade.create$dataElement.name$(): This method was integrated into the $dataElement.name$Connector.create$dataElement.name$() method.
  • $dataElement.name$RestFacade.modify$dataElement.name$(): This method was integrated into the $dataElement.name$Connector.put$dataElement.name$() method.
  • $dataElement.name$RestFacade.update$dataElement.name$(): This method was integrated into the $dataElement.name$Connector.patch$dataElement.name$() method.
  • $dataElement.name$RestFacade.remove$dataElement.name$(): This method was integrated into the $dataElement.name$Connector.delete$dataElement.name$() method.
  • $dataElement.name$RestFacade.fetch$dataElement.name$(): This method was integrated into the $dataElement.name$Connector.get$dataElement.name$Details() method, where the SearchDetails is now also constructed.
  • $dataElement.name$RestFacade.fetch$dataElement.name$List(): This method was integrated into the $dataElement.name$Connector.get$dataElement.name$DetailsList() method, where the SearchDetails is now also constructed with support for sort parameters.
  • $dataElement.name$RestFacade.applyPagination(): This method was moved to a new class $dataElement.name$Pagination.
  • $dataElement.name$RestFacade.DEFAULT_PAGE: This constant was moved to a new class $dataElement.name$Pagination.
  • $dataElement.name$RestFacade.DEFAULT_PAGE_SIZE: This constant was moved to a new class $dataElement.name$Pagination.
  • $dataElement.name$RestFacade.MAX_PAGE_SIZE: This constant was moved to a new class $dataElement.name$Pagination.
caution

The $dataElement.name$Pagination class may still be revisited in the future to improve its design.

All existing custom methods that were placed in the RestFacade class should be either integrated into their respective callers or be properly encapsulated if required in its own custom class. To provide some backwards compatibility for transitioning, the expansion of the RestFacade class can still be enabled with the transient option jaxrs.transient.generateFacade.enable. All expanded endpoints will no longer use the RestFacade even if this transient option is defined.

caution

The transient option jaxrs.transient.generateFacade.enable was removed in rest-expanders version 3.0.0.

Sorting

Default Order

For some time data returned by the GETlist endpoints was always sorted ascending by database identifier by default if no other sorting was enforced in the DataElement's QuerySearch class. This functionality was removed after the querysearch-expanders split off from the rest-expanders, as for many requests it does not make sense to deteriorate query performance by always enforcing a default order.

Not having this default ordering for the REST apis created an issue with results returned by the query not always being consistently ordered in between multiple calls. As a result, paginated results do not always make sense as resources can will sometimes shift places in the results.

The querysearch-expanders introduced the option useDefaultOrder on the QuerySearch metamodel element, which is implicitly created by the rest-expanders. This option is now always added to that QuerySearch instance as well, so the default ordering by database identifier is always enforced for the REST API. Contrary to the original implementation, the ordering by database identifier is also enforced if other ordering is applied to the query, but it is added as the last order in the chain.

If for some reason it is required to disable this default ordering, this can still be achieved per individual DataElement by setting the default order to false on the queryBuilder in the QuerySearch class:

// anchor:custom-query-before-build:start
queryBuilder.useDefaultOrder(false);
// anchor:custom-query-before-build:end

Sorting parameter

This version of the REST expanders introduces sorting of results returned by the GETlist endpoint through a query parameter sortby. This parameter is now always added to the expanded endpoints. It can however be disabled by adding the option jaxrs.endpoint.getList.sorting.disable to the DataElement or for the entire Component by adding the option jaxrs.endpoint.getList.sorting.disable to the Component.

Miscellaneous

  • All ValidationError constants in the ValidationError class are now marked final. They should always be fixed. Any deviations should be registered as new error codes.
  • The pagination anchors were removed from the applyPagination() method, as pagination parameters can now be set through expansion options.

· One min read
Frédéric Hannes

SQL script location

Starting with version 2.0.0, the database setup script is no longer expanded to [APPLICATION_ROOT]/provision_db.sql, not is now available as [APPLICATION_ROOT]/database/000-init-database.sql. This change was made as there is now also the initial data script, which is expanded to [APPLCIATION_ROOT]/database/001-app-init-data.sql.

· 5 min read
Herwig Mannaert

The use of transactions in software systems is quite a fundamental and non-trivial issue. Every one knows the obvious example of retrieving money from a cash machine. Either your account is debited and you receive the money, or none of the above happens. In case the machine fails to dispense the money, the transaction requires a rollback of the debiting. However, it is not always that straightforward. Though the use of transactions providing automated rollbacks is often desirable to end-users and system analysts, it may well be unnecessary complicated or even plainly impossible to implement for the software developer.

In order to enable scaling and to avoid deadlocks, transaction processing systems typically deal with large numbers of small things, and strive to come in, do the work, and get out. This means that it is in general not feasible to use technical transactions to provide end-to-end transactional integrity for customer transactions. For instance, guaranteeing the customer transaction of a money transfer is in general not implemented through a technical transaction encompassing the debiting of the source bank account and the crediting of the destination bank account, which may reside in another bank across the globe. It is typically guaranteed by taking the transfer request as fast as possible into a secure database, followed by a number of sequential processing steps or individual transactions. This flow may even require human intervention to make sure that the customer transaction, i.e., the transfer request stored in the secure database, is executed properly.

The use of end-to-end technical transactions would often cause lots of problems in the real world. For instance, defining a technical transaction around the reservation of a flight ticket, a rental car, and a hotel room, could seem desirable to indulge the customer, but would soon lead to the simultaneous locking of all reservation systems around the world. Avoiding such deadlocks is closely related to the NS theorem Separation of States. This theorem states that after every task or action, the result state should be stored in a corresponding and appropriate data entity. This will enable the independent execution, and therefore evolution, of the implementation of the processing task, and the error handling of the task. Besides better evolvability, this strategy improves the tracing and handling of errors, and reduces the chances of having locked resources.

Another often overlooked issue in transaction management is the complexity of the rollback. While a previous write operation in the same database could be compensated by a relatively simple rollback, developers would have to implement dedicated compensating transactions in case the various steps imply changes across different systems. These compensating actions are not always straightforward or even possible. For instance, in case one of the steps has triggered a physical action, like shipping a product or opening a switch or water tap, it is simply impossible to rollback. Unless mopping the floor is part of the compensating transaction.

These issues are also quite relevant when client systems invoke an API (Application Programming Interface) of a target system. Examples of such client systems are business process management systems invoking services from underlying transactional systems, or user interface front-ends developed upon the service APIs provided by back-end systems. The client systems often desires a customer transaction encompassing several service calls on the target system. This requires client and target system to agree on one or multiple scenarios, that define the responsibilities and behavior at both ends. We discuss here the various possibilities.

  • The client system defines the transaction and the target system is expected to honor the transaction. By invoking some start-transaction interface, the client system expects the target system to be able to rollback everything that is being processed until the stop-transaction interface is called. This is a very tough, cumbersome, and nearly impossible requirement to fulfil for the target system.

  • The client system attempts to both manage and honor the transaction. Within the scope of a desired transaction, the client system keeps track of the various changes it incurs in the target system, and will rollback and/or compensate the various changes that have already been performed in case something fails.

  • The target system provides an explicit aggregated service API for certain customer transactions required by the client system. In this case, the target system attempts to manage and honor the transactional integrity for a specific set of customer transactions, thereby providing appropriate rollbacks and/or compensating actions for these specified transactions.

  • The target system provides an asynchronous request API for the aggregated customer transaction required by the client system. The submitted request will be stored immediately in the database, and the target system can start the processing flow performing the various tasks, compensating already performed tasks if required, thereby even supporting manual interventions. The client system can choose to perform other tasks while checking regularly for an answer, or opt to wait or block until the aggregated service is fully executed. The latter option would emulate a synchronous service call at the client side.

· 4 min read
Frédéric Hannes

CommandExtension linking

In previous versions of the main expanders, the commands were injected into the logic bean as a feature. Since we do not nest features, this made it impossible to implement commands from another expander bundle, such as the rest-expanders. This feature was integrated into the bean expander, allowing other bundles to now implement commands.

To solve this original issue, the rest-expanders injected some code into the custom anchors of the commands in the logic bean. This was never intended to be a permanent solution, but a temporary workaround to address this issue. Version 1.15.0 of the rest-expanders has a minimum dependency of version 4.12.0 of the main expanders, to be able to transition to actual feature injection for this purpose. As the previous code has now ended up in the harvest files of every project, the workaround has been inverted to remove the workaround code that was injected.

An example of the old injected code:

// anchor:custom-command-createEntity:start
// Default implementation generated by the REST expanders
commandResult = executeCreateEntityCommand(commandParameter);
// anchor:custom-command-createEntity:end

The code will be removed automatically if the endpoint still exists in the model and no code (other than whitespace) has been added in between the start anchor and the injected code.

The previous execute methods were added to avoid the code from being altered easily and make the transition more difficult. These have now been removed and the code they contained is now injected as part of the feature.

An example of the new injected feature code, which was previously in the execute method:

// @anchor:command-createEntity:start
commandResult = new CreateEntityCommandExtension().execute(commandParameter);

if (commandResult == null || commandResult.isError()) {
sessionContext.setRollbackOnly();
}
// @anchor:command-createEntity:end

Exposing diagnostics

We are no longer exposing values in Diagnostic object directly through error payloads in the REST API. The reasoning behind this is that we do not want to expose diagnostic information directly to the end-user. As an alternative, all exceptions that are handled by the REST API that contain diagnostics now have their diagnostics output into the logs along with the information that was already logged previously. Given that this pertains to 5xx errors, these are technical errors that should not occur in production and will likely be handled by a developer. Any issues with input of data should already have been handled by the validation system. These errors which return diagnostics are only thrown when something unexpected has occurred. The diagnostics have more value to a developer rather than the end-user and are more easily accessed through the application logs.

A transient option jaxrs.transient.exposeDiagnostics.enable has been supplied to revert the behavior of the expanders for backwards compatibility. But keep in mind that this option will be removed in the future and has only been supplied for short-term compatibility if needed, allowing for planning to upgrade the codebase. The option will once again expose the diagnostics through the API, but they will also appear in the application logs.

caution

The transient option jaxrs.transient.exposeDiagnostics.enable was removed in rest-expanders version 2.9.0.

The following expanders have been affected:

  • CreateFailedExceptionHandlerExpander
    • File: CreateFailedExceptionHandler.java
    • Changes:
      • Import removed: net.democritus.sys.Diagnostic
      • Objects returned by exception.getDiagnostics() no longer added to errorModel in mapException(exception) method
  • DeleteFailedExceptionHandlerExpander
    • File: DeleteFailedExceptionHandler.java
    • Changes:
      • Import removed: net.democritus.sys.Diagnostic
      • Objects returned by exception.getDiagnostics() no longer added to errorModel in mapException(exception) method
  • ModificationFailedExceptionHandlerExpander
    • File: ModificationFailedExceptionHandler.java
    • Changes:
      • Import removed: net.democritus.sys.Diagnostic
      • Objects returned by exception.getDiagnostics() no longer added to errorModel in mapException(exception) method
  • SearchResultExceptionHandlerExpander
    • File: SearchResultExceptionHandler.java
    • Changes:
      • Import removed: net.democritus.sys.Diagnostic
      • Objects returned by exception.getDiagnostics() no longer added to errorModel in mapException(exception) method
  • ServerErrorModelExpander
    • File: ServerErrorInfoModel.java
    • Changes:
      • Import removed: java.util.ArrayList
      • Import removed: java.util.List
      • Field removed: List<String> diagnostics
      • Method removed: List<String> getDiagnostics()
      • Method removed: void setDiagnostics(List<String>)
      • Method removed: void addDiagnostic(String)

Miscellaneous

  • The ValidationException class was moved from the shared layer to the control layer. It was never intended to be used outside the control layer.