December 2023 R&D update

On the 7th of December, we presented an overview of some of the major developments over the last half year in the NSX R&D team. This post is a guiding article for the presentation, which summarizes the notable topics that we touched on. We now also include an overview of planned development over the course of the coming half year.

Download the presentation

Angular Expanders

The angular-expanders underwent a lot of changes in the last 6 months and has grown more stable. The AngularProjects metamodel was created (1) to facilitate the expansion to a different root folder than the JEE application stack and (2) to better describe front-end applications. The metamodel consists of:

• AngularApp: The program of the model. Contains attributes like version and prefix. Can be compared to Application.
• FeatureModule: The module of the model. Is used to expand a logical folder structure. Can be compared to Component.
• DataConnector: Defines a connection to a DataElement and its CRUD operations.
• DataView: Defines the views for CRUD operations.

Together with the front-end developers of our ongoing projects a new folder structure has been defined. This will make sure that every project looks the same and people can quickly navigate throughout as we are used to with our back-end applications.

In order to adhere to the flat folder structure convention of front-end development, we moved on from the use of gen and ext folders to differentiate custom code. Instead, the expansion/harvest process now uses a manifest file to list all files that get expanded. This means that ext files can be placed everywhere now, however we do restrict the use of this to maintain the project familiarity as mentioned before.

As we are introducing a new front-end technology, we also want to give a new look and feel together with a better user experience to the projects. Tom has been busy creating new designs for this and these are gradually being added to the expanded projects.

Currently, projects using the angular-expanders rely heavenly on custom views and styles. An effort is being made for these projects to not diverge too much, a weekly front-end meeting has been introduced where our developers can:

• explain upcoming features and get feedback on them
• ask if other people have already created certain components
• define the path forward for the angular-expanders

To further facilitate this, a way to share components/services/directives has been introduced to minimize double work. This is accompanied by documentation where you can try out the elements.

Enum types

EnumTypes are ValueTypes where you enumerate the possible values of the types in the model. They can be added to a Component and used as the type for a ValueField (like the SimpleValueTypes).

EnumTypes are expanded as enums in the shared layer.

Read more about the EnumTypes here.

Experimental

The enumTypes metamodel is currently in an experimental phase. Once we're happy with the result, it will be merged into elements.

Front-end implementations for angular and knockout are work in progress.

Expander development

Mapping file improvements

Mapping files now have a new syntax to make it easier to define import for an artifact.

Up until now, imports have been a difficult thing to get right in an Expander. This is because imports have a number of requirements that are hard to represent in mapping files:

• Deduplication: Ideally, imports do not have any duplicate imports. In typescript, this is even a prerequisite.
• Conditional imports: Imports need to be declared at the start of the artifact. Yet, the conditions when to add them are often processed deeper in the mapping file.
• Redundant/relative imports: Sometimes, imports are redundant because they have the same package as the artifacts. In javascript-like imports, the coordinates can be relative to the current artifact location.

The mapping files now support usage statements, which declare that a resource is used and needs to be imported. These usages are then collected and processed to generate a set of imports.

There is support for java and requirejs imports and the possibility to add your own import resolvers. In the future, a typescript import resolver will be added as well.

See Mapping Imports.

Expanders Assert

Expanders-assert is a new testing library for Expanders based on JUnit 5 and AssertJ. In the long run, it is intended to replace the expanders-test-utils library.

The focus is on making tests more predictable. With the expanders-test-utils library, tests can sometimes run into issues where the cause is hard to find. This can be the result of issues in the ClassPath, problems building the correct ExpansionContext, find elements in the model etc.

These problems only get more pressing now that new metamodels are starting to be created more and more.

Expanders-assert improves the stability of Expanders tests by changing how dependencies are loaded to build a test model. It also provides a slightly more modern API to write tests with.

You can read more about expanders-assert here.

Workflow with ProcessAutomation

A preview release (0.x.x) has become available for ProcessAutomation, the evolution of the Workflow component. The primary improvements over the workflow component are: Model based state-machines instead of database configuration; Queue-backed scheduling to support a wider range of execution models; And support for many types of Triggers and Events.

There is a well-defined migration path for applications which are currently using the workflow component. The first two steps of the migration can even be adopted while you keep using the workflow component.

The intent of the preview release is to find apparent issues with the design early on if any. We also don't guarantee no breaking changes until the first official release 1.x.x.

Modeling behavior with triggers

In addition to the FlowEngines used in the Workflow component, it is possible to model additional triggers with process automation. Out of the box we provide various triggers with OnCreate, OnModify, OnTransition and Schedule currently provided. This will lead to a more functional workflow model and also a more responsive application.

Triggers and events make full use of the underlying queue in process automation which makes task execution quick and predictable. More info on triggers is available in the documentation.

Fully extendable

The entire component is designed to be extendable. This means it is possible to add custom events or triggers. Not only is it functionally extendable but also the technical implementation allows for extension. An example of this is a custom queue driver. Two queues are provided by default, an Application Persistence queue and an In Memory queue. If you would like to use another queueing technology, you will only need to provide a QueueDriver for this technology.

NSX IntelliJ plugin

Some nice quality of life features have recently been added to the NSX-Support IntelliJ Plugin.

Reordering Java imports

You may have encountered issues before with missing imports after harvesting your application. This can occur due to the way intelliJ adds imports to your files. With the updates all manually added imports will now be moved to the anchor:custom-imports block when a file is saved.

Improved expander templates

The expander templates have been updated to include all the latest and greatest expander features. Things like <artifact> and @imports or support for the TestModelBuilder and expanders-assert with Junit5 are now all available when generating new expanders.

Debugging support for expander mapping

It can sometimes be difficult to figure out what exactly is added to the expansion context by a mapping file. The NSX support plugin now provides debugging support for mapping files. This is enabled simply by adding breakpoints to the Mapping.xml you want to inspect, and the editor will pause test execution when the selected mapping rule is evaluated. More info on the setup is available in the expander development documentation.

REST Expanders 4.x

QuerySearch Metamodel

With the introduction of version 3 of rest-expanders, it migrated to using the QuerySearch metamodel implicitly. This means that all expanders related to QuerySearch were reworked to use the QuerySearch metamodel instead of DataElement. The model was implicitly added when loading, based on existing options on DataElement. This resulted in it being 100% backwards compatible. The reason for this transition phase was to have time to refine the metamodel for QuerySearch, while already being able to prepare expanders to make use of it.

With version 4, we finally made the transition to using the QuerySearch model explicitly, because the model is now stable enough to be used. This transition was facilitated by introducing a transmuter that transforms the model in the same way it was previously done implicitly. This allows us to make the transition without making any changes to any expanders.

This transition also allows developers to add options to elements from the QuerySearch metamodel, which enables the introduction of new features in expanders. One such feature is the option jaxrs.querySearch.exposeFilter on QueryFilterField, which expands the code to expose the filter field on the GETlist endpoint in an expanded API.

Implicit metamodel

It has been a long term goal to create a metamodel to model REST APIs. In version 4 we're taking the first steps towards this goal. A metamodel is now in development and already implicitly in use (like QuerySearch was previously), so it can be developed without breaking backwards compatibility with existing applications.

One of the first major improvements that were introduced with this metamodel is that now all server error types are defined in a model, reducing the complexity of the related expanders greatly.

Jakarta support

The improvements to mapping files allow us to facilitate the transition to Jakarta EE. All imports in rest-expanders that must be changed from javax. to jakarta. have been migrated to the new imports system and technology mappings were added to make it compatible with both Java EE and Jakarta EE. There's still some changes requires to fully support Jakarta in rest-expanders, but this puts us very close to the finish line.

QuerySearch joins

The improve some functionality in rest-expanders, support to define explicit joins was added to QuerySearch. This makes it possible to define LEFT INNER and LEFT OUTER joins on tables involved in the query's FROM clause.

JEE stack improvements

Java 21 support

The new LTS of Java, version 21 was released on September 19th. This version introduced a couple of interesting new features such as pattern matching for switches and record patterns.

In order to allow these new features to be used in our JEE applications, we added support for Java 21 as soon as possible, transitioning to a Java 21 JVM in our docker images and supporting it as a compiling target as soon as a release of Temurin and a compatible version of TomEE were available. This has allowed us to create new applications with Java 21 and migrate existing applications to Java 21 in just over one month of its release.

Housekeeping

Over the years a lot of legacy features have been left behind in our JEE stack expanders to retain maximum backwards compatibility. Though our expanders allow us to reduce the cost of maintaining these, it does still incur a cost. Usually we clean up these features at a gradual pace, but over the last half year we did a serious cleaning of many deprecated features and code that was no longer in use.

• Before we moved to Java 8, we used a custom implementation Options.Option<T> to wrap non-null values. This was replaced with Optional<T>, as our minimum target for applications is now Java 8.
• UserContext has been part of many deprecated constructs in our expanded code. The reason for this is that is was replaced with Context at some point to allow context information to be represented individually for different features and concerns. We first deprecated UserContext in 2018 as main context transport object and have now removed most code where it was used as such.
• The CrudsResult<T> class was rebuilt and made immutable (to the extent possible without losing backwards compatibility). This is the first of the result classes that will be reworked in order to clean up the implementation and make them more consistent. This class was intended to be a simple wrapper with a dedicated purpose. The new implementation enforces more constraints to guide developers to the correct usage of the class.
• Mindmap was removed from nsx-runtime. This was an old feature that is no longer in use.
• We have accumulated a lot of methods marked with @Deprecated in nsx-runtime and expanded code. Most of these that were deprecated for more than a year (usually 3 or more) were removed.
• We've removed a lot of deprecated features in expanded code that we marked with legacy options. These features have typically been phased out for years and have a better alternative and a reasonable upgrade path.
• The groupIds for all maven modules in the JEE applications have been changed to be consistent and unique to individual applications, which was important for re-usable components, which are in a sense a unique instance per application they are expanded for.
• We replaced the implementation of FinderBean to no longer use reflection and require all finders to be defined in the model. This was placed behind an opt-in option for quite some time and used in all new projects, but has now been switched to be the default implementation.

Upcoming features in 2024H1

REST Expanders

Development on the REST metamodel which is added implicitly will be resumed. The goal is to move all information that is currently being gathered by multiple expanders into a single data structure represented by the model. This model once refined can then later be made explicit as was done for QuerySearch. The goal is to move the APIs out of components when the model in rest-expanders 5 with the explicit model, so the APIs are coupled to an application and APIs can be created for any component in multiple versions.

Work has also started on expanding Java REST clients for the API generated with rest-expanders. The introduction of the new metamodel will be a key part of this, as a lot of information about the API is needed by both the server and client expanders.

Replacing Struts2

Our JEE stack has historically used Struts2 to provide a REST API for our expanded Knockout-based frontend to communicate with. This API works well, but also needs some design improvements. We have started development on a new API for our generated UIs which is more uniform and structured, based on the JSON:API format.

We intend to finish the implementation of this new API, as well as provide integration with both our Knockout-based UI and a new fully-generated Angular UI. Through this we aim to provide a more manageable migration path for applications to move from the old UI to a new Angular UI.

Angular UI

Contribution

The use of Angular libraries and its documentation will be further developed. The current process for contribution to our "angular ecosystem" will be streamlined and optimized.

New designs

We've laid the foundation to implement a whole new custom design for our Angular UI that will provide a clean and unique identity to the applications we develop. In the near future we will further improve on this by supporting more types of screens and by adding new features to the views we already support.

Backend clients

Currently, the angular-expanders rely heavily on the endpoints expanded by the rest-expanders. We are working on a replacement for the current Struts2 API with a JAX-RS API. The latter will be supported by the angular-expanders and a way to have interoperability with the current behaviour needs to be defined.

Expander Tags

Expanders tags are a design concept to simplify the conditions of Expanders. The idea is to have each Expander define a set of tags. The Expander will only be executed is all tags are available on the element.

E.g. a RemoteExpander could have the tags ejb3 and remote. If either ejb3 or remote is missing from a DataElement, the Remote class will not be expanded.

Tags will also have rules to derive tags from other tags. E.g. hibernate => jpa means that the jpa tag is always included if the hibernate is present, since hibernate is an implementation of the jpa standard.

Rework of implicit name

If a DataElement does not have a field called name and does not have the nameNotWanted option, the Expanders generate an implicit name field. This name field is then filled in with a default value based on teh database id.

This functionality is problematic in a few ways:

• It is implicit, meaning there is no trace in the model. For developers new to NS, this can be very unpredictable.
• The Expanders each check whether they need to add an implicit name. As a result the behaviour is inconsistent.
• The name field also requires a findByNameEq finder. But this finder points towards a field that does not exist.

We will be streamlining this logic. The solution will probably be to create a transmuter that adds the name field explicitly, only with options to keep the behaviour the same. From then on, not adding a name field to the model has the effect of... not having a name field.

Redesigning DataCommand

DataCommands are currently used for a lot of tasks they were not designed for. We will design a more suitable solution to model actions on a dataElement. We also want to make it possible to have a better integration with the expanded control and front-end for this as well.

ProcessAutomation 1.0.0

The preview release has been going well for a while now. There are some missing features like retry mechanisms and time-window groups which we want to introduce still. These features will be integrated with the process automation component and tested before the first full release.

µRadiant 2.x

Plugin system

With the advent of new metamodels, we also need a more scalable solution of supporting those metamodels in the µRadiant. The current solution is to push changes to the micro-radiant repository.

The plugin system will allow developers to write their own plugin for the µRadiant and distribute them to other developers wanting to use their metamodels.

UI overhaul

Writing a new plugin system also required the components of the micro-radiant to be rewritten, because the plugins will be using components of the micro-radiant to implement their logic. Hence, it is important these are well-designed and version transparent.

With this refactoring also comes an overhaul of the styling and UI of the micro-radiant.

Tooling support for Mac

Currently we provide support with packaging of our tooling for both Windows and Ubuntu. We've noticed an influx of people using Mac devices to work with our tooling. Therefor we intend to provide better support for this platform, as we will then be covering the majority of desktop platforms.

NS Initializer modules

The NS Initializer is a tool based on expanders that can generate all needed files to start various new projects. It is currently required to model a project definition and write all expanders needed to generate the project. We want to improve this setup by adding support for modules, so parts of an initializer can be reused to build new initializer. A good example of this would be the initializer for an application being partially reused for an initializer that generates an application with a REST API. Then the common expanders do not have to be duplicated.