Thanks to the expand and build in the micro-radiant, our expansion framework, automagically, delivers a whole set of code. It supplies you with modern user interfaces, it manages data persistency, exposes an API and much more.

But once in a while, a more daunting task might require you to extend the stack with unsupported functionality... Maybe your client wants to export his data in a proprietary report structure, or you need to integrate with an obscure authentication framework, or maybe your client wants all action buttons validated with a PIN code.

To ensure these custom additions (and for that matter all code) remain maintainable, we track each project with automated measures. In this post, we explain how we measure each project and how these measures help you to keep your project in good shape.

Gathering data throughout the build pipeline​

At NSX, we provide each project with a default build pipeline. The pipeline integrates with a whole set of tools that capture dependency health, calculate test coverage, define code metrics and analyze your code structure.

The different stages in the pipeline might call an external system or produce a report artifact. All this information is afterwards aggregated and linked to your application in our central intelligence platform: Prime Radiant.

A deeper look into our pipeline​

The lifecycle of a pipeline consists of several stages that produce measures:

• Build: Executes the mvn package command. This uses the CycloneDX plugin to generate a bom.json (Bill of Materials).
• Report: Pushes the bom.json to DependencyTrack for security analysis. It also triggers SonarQube for code quality and gets a sonar-xml.report with high-level statistics back.
• NS-Audit: Calls our custom NS Audit Tool that validates code on NS principle violations and generates an audit.json result.
• Analyze: This is where the Project Analyzer comes in. It scrapes the source code, counts files, and interprets the models. It reads the bom.json, audit.json, and sonar-report.xml and pushes everything to Prime Radiant.

Project Analyzer​

The analyzer is a central Java application that we developed in-house. It is responsible for parsing all measure data available in the pipeline, calculating additional measures, and sending everything to Prime Radiant.

Some of the measures that it calculates include:

• Codesize: Measuring the total footprint of the codebase.
• Model size: Reading the application model and determining the amount of model elements of each type (Data Elements, Tasks, Flows, etc.).
• Detecting extensions and insertions: Identifying specific points in the source code where you added custom code or extended the framework.
• Overlays: Looking for overlays to see where the generated templates are overwritten.

While the project analyzer provides most of the measure data, some project-independent data is gathered by the Prime Radiant directly. One of such example is pulling vulnerability data from an open platform to enrich our internal dependency health measures.

What do we measure?

For applications, we split our measures into two main categories: Size Metrics and Quality Metrics.

Measures that are shared with other software artifact types are linked to the generic element AssemblyUnitBatch, application-specific measures are linked to ApplicationVersion.

Plugins, Expansion Resources and libraries have their own set of measures which are out of scope of this article.

Size Metrics​

Size metrics describe the size of your codebase and model. These are useful for:

1. Tracking growth over time: See how your application size is evolving (e.g. How many new data elements were introduced in the last year to monitor application's growth.)
2. Defining "peer" applications: Find similar projects to use as a benchmark. (e.g., If an application with the same amount of data elements has three times the workflows, maybe you should model your flows more granularly).
3. Normalizing quality metrics: Use size to put quality in context. (e.g. How much custom code do you have per data element, and does it remain stable as you grow?).

Quality Metrics​

Quality metrics check how solid the code is. We group these measures into three areas: Custom code, quality and dependencies.

1. Custom Code & Overlays​

These measures quantify how an application extends the generated stack. We monitor these to identify projects that are becoming overly customized, which increases the complexity of future upgrades and maintenance.

• CustomCodeMeasure: We track nrOfExtensions (files) and nrOfInsertions (files with code added within anchors). We also monitor totalSizeExtensions and totalSizeInsertions in bytes to assess the volume of custom implementation logic.

• OverlayMeasure: This tracks overrides of standard files via nrOfOverlays and totalSizeOverlays. A high overlay count often indicates that the core framework lacks a specific feature, leading to project-specific workarounds.

2. Code Quality (SonarQube)​

Code quality metrics are primarily derived from SonarQube, focusing on three technical domains: Reliability ( functional correctness), Maintainability (ease of modification), and Security (resistance to exploits).

Issue counts​

The aggregated issue counts help you validate code stability over time and can be used to benchmark your application against others in the Prime Radiant

• maintainabilityIssues: The count of technical patterns that increase complexity and make the codebase more difficult to modify.
• reliabilityIssues: The count of logic errors that increase the probability of runtime failures.
• securityIssues: The count of security flaws that represent potential points of exploitation.
• Severity Breakdown: Issues are categorized as lowSeverityIssues, mediumSeverityIssues, or highSeverityIssues based on internal rankings of SonarQube.

Ratings​

A rating tries to quantify the quality of your application in a single number (based on all detected issues). Any rating deviating from the best grade (1) might be a good starting point to start improving your application.

• debt: The estimated effort (in minutes) required to remediate all open maintainability issues.
• maintainabilityRating: A grade (1-5) based on the Debt Ratio (the cost to fix the code vs. the cost to rewrite it).
• reliabilityRating: A grade (1-5) derived from the severity of the most critical open bug.
• securityRating: A grade (1-5) based on the severity of the most critical open vulnerability.

Code complexity​

• duplication: The percentage of repeated code blocks.
• cyclomaticComplexity: A mathematical measure of the number of linearly independent paths through the code. It reflects how difficult the code is to unit test.
• cognitiveComplexity: A measure of how difficult the code is to understand for a human. It penalizes nested loops and complex logic structures.

3. Dependency measures​

Managing dependencies is an important factor in application quality. Every external library added increases the maintenance surface of the application and introduces external risks.

At NSX, we use Renovate Bot to automate update tracking. However, updates still require careful (and manual ) management. Breaking changes may require refactors, and version updates may unintentionally impact internal implementations.

Dependencies also extend the application's attack surface. Vulnerabilities within third-party libraries become direct security risks for the project.

We monitor:

• nrVulnerabilities: The total vulnerability count, categorized from nrCritical to nrInfo. The urgency category is based on public CVE scores.
• nrOutdatedVersions: The number of dependencies currently behind the latest (known) release.

What's next?

Analyzing our projects and storing this information in a structured way is just the first step. As a chief engineer, it is up to you to watch over your application's health.

The Prime Radiant offers you a view to check, from time to time, how your application is doing. In the daily run of a project, it is surprisingly easy to forget best practices, choose a quick shortcut, or leave a "dirty patch" that never gets fixed.

Tracking these measures allows you to reflect on this evolution and make time for a necessary refactor in a specific component or layer (i.e. if you went a little wild with UI customizations).

To help you with this, the Prime Radiant team is working on:

1. Automated notifications: Get alerts via mail, Slack, or build failures when your project crosses defined measure thresholds.
2. Dashboard views: Compare your application with other projects to detect trends or evolutions that might be less than ideal.

Below is a dashboard preview showing an application with an unusually high number of extensions compared to its data elements:

Software data is scattered across multiple tools (such as Git, Jira, Jenkins, Maven, SonarQube, etc.), making it challenging to have an integrated overview of one (or many) applications. Next to this, software systems become more pervasive to manage and controlling the end-to-end production process of every application in your organization can be a daunting task. Therefore, it is our goal at NSX to build the foundations for a true software factory, to manage and control the building and assembly of software systems.

The Prime Radiant provides a single analysis platform and control layer to optimize NSX factory's operations and output. (i.e. application and expander developments) It works by automatically collecting, aggregating, and integrating information from our different DevOps tools.

The result is a unified view of the development lifecycle across all applications. This not only supports data analysis but also offers the potential to control our factory: adding new expansion resources, patching vulnerabilities, or deploying new application versions, all directed from the Prime Radiant.

In the next sections, we'll describe the core functionality of the Prime Radiant. For each functionality we show a conceptual data model that contains the relevant data concepts.

Please note that all diagrams are showing a simplified version of the actual implementation

Managing Assembly Units​

A fundamental concern of the Prime Radiant is capturing and identifying all software created within the NSX factory. An Assembly Unit defines every piece of software or technology produced, allowing us to manage the factory’s artifacts comprehensively.

An Assembly Unit can be an Application (like a JEE System) , an Expansion Resource (e.g. Angular front-end expander) , a Library (providing basic utilities) , or a Developer Tool (plugins and command-line tools), or a unit combining any of the former.

For every unit, we keep track of the source code by mapping its associated Repository.

Tracking Versions​

To analyze and optimize applications over time, we must track changes and their impact. Versioning is fundamental to tracking software changes. As a result, each software element is mapped to a specific versioned entity, like an AssemblyUnitBatch or ApplicationVersion. This is crucial because it allows us to link dependencies, metrics, and vulnerabilities to its specific state at a certain moment.

Monitoring metrics​

A core function of a control system is tracking data over time. The prime-radiant logs crucial metrics for applications, expansion resources, libraries etc. We have defined a core set of shared metrics (such as Test Coverage and Code Quality) that apply to all Assembly Units. Additionally, we implement more specific metrics tailored to the unique requirements of each Assembly Unit type.

For example, for the unit type Applications, we monitor the amount and size of custom code (extensions/insertions), relative to the generated software to ensure maintainability and evolvability.

Deployments​

Deploying applications and managing the underlying infrastructure is a part of every development lifecycle. The Prime Radiant tracks Application Deployments to record information such as server specifications, OS versions, cloud provider type, and domains.

Build pipelines​

Each version of an Assembly Unit is traced back to a specific build produced by a pipeline. The pipelines are defined with PipelineTargets (steps) that make use of various BuildTechnologies, and ultimately produce versioned Resources (libraries, archives, or executables).

Modeling business requirements​

Creating software is not an end goal; it's done to fulfill specific business needs, making business context paramount. In the Prime Radiant, this context is modeled and captured.

Every Application, is linked to a Domain (e.g. car rental) and part of a larger Digital Platform (such as 'euRent platform').

This platform is described using Business Capabilities (e.g., invoicing, scheduling, maintenance) and Functional Requirements (e.g., canceling a booking).

This approach allows the software manufacturing control system to link development efforts directly to business value.

Angular Expanders​

The angular-expanders underwent a lot of changes in the last 6 months and has grown more stable. The AngularProjects metamodel was created (1) to facilitate the expansion to a different root folder than the JEE application stack and (2) to better describe front-end applications. The metamodel consists of:

• AngularApp: The program of the model. Contains attributes like version and prefix. Can be compared to Application.
• FeatureModule: The module of the model. Is used to expand a logical folder structure. Can be compared to Component.
• DataConnector: Defines a connection to a DataElement and its CRUD operations.
• DataView: Defines the views for CRUD operations.

Together with the front-end developers of our ongoing projects a new folder structure has been defined. This will make sure that every project looks the same and people can quickly navigate throughout as we are used to with our back-end applications.

In order to adhere to the flat folder structure convention of front-end development, we moved on from the use of gen and ext folders to differentiate custom code. Instead, the expansion/harvest process now uses a manifest file to list all files that get expanded. This means that ext files can be placed everywhere now, however we do restrict the use of this to maintain the project familiarity as mentioned before.

As we are introducing a new front-end technology, we also want to give a new look and feel together with a better user experience to the projects. Tom has been busy creating new designs for this and these are gradually being added to the expanded projects.

Currently, projects using the angular-expanders rely heavenly on custom views and styles. An effort is being made for these projects to not diverge too much, a weekly front-end meeting has been introduced where our developers can:

• explain upcoming features and get feedback on them
• ask if other people have already created certain components
• define the path forward for the angular-expanders

To further facilitate this, a way to share components/services/directives has been introduced to minimize double work. This is accompanied by documentation where you can try out the elements.

Enum types​

EnumTypes are ValueTypes where you enumerate the possible values of the types in the model. They can be added to a Component and used as the type for a ValueField (like the SimpleValueTypes).

EnumTypes are expanded as enums in the shared layer.

Read more about the EnumTypes here.

Expander development​

Mapping file improvements​

Mapping files now have a new syntax to make it easier to define import for an artifact.

Up until now, imports have been a difficult thing to get right in an Expander. This is because imports have a number of requirements that are hard to represent in mapping files:

• Deduplication: Ideally, imports do not have any duplicate imports. In typescript, this is even a prerequisite.
• Conditional imports: Imports need to be declared at the start of the artifact. Yet, the conditions when to add them are often processed deeper in the mapping file.
• Redundant/relative imports: Sometimes, imports are redundant because they have the same package as the artifacts. In javascript-like imports, the coordinates can be relative to the current artifact location.

The mapping files now support usage statements, which declare that a resource is used and needs to be imported. These usages are then collected and processed to generate a set of imports.

There is support for java and requirejs imports and the possibility to add your own import resolvers. In the future, a typescript import resolver will be added as well.

See Mapping Imports.

Expanders Assert​

Expanders-assert is a new testing library for Expanders based on JUnit 5 and AssertJ. In the long run, it is intended to replace the expanders-test-utils library.

The focus is on making tests more predictable. With the expanders-test-utils library, tests can sometimes run into issues where the cause is hard to find. This can be the result of issues in the ClassPath, problems building the correct ExpansionContext, find elements in the model etc.

These problems only get more pressing now that new metamodels are starting to be created more and more.

Expanders-assert improves the stability of Expanders tests by changing how dependencies are loaded to build a test model. It also provides a slightly more modern API to write tests with.

You can read more about expanders-assert here.

Workflow with ProcessAutomation​

A preview release (0.x.x) has become available for ProcessAutomation, the evolution of the Workflow component. The primary improvements over the workflow component are: Model based state-machines instead of database configuration; Queue-backed scheduling to support a wider range of execution models; And support for many types of Triggers and Events.

There is a well-defined migration path for applications which are currently using the workflow component. The first two steps of the migration can even be adopted while you keep using the workflow component.

The intent of the preview release is to find apparent issues with the design early on if any. We also don't guarantee no breaking changes until the first official release 1.x.x.

Modeling behavior with triggers​

In addition to the FlowEngines used in the Workflow component, it is possible to model additional triggers with process automation. Out of the box we provide various triggers with OnCreate, OnModify, OnTransition and Schedule currently provided. This will lead to a more functional workflow model and also a more responsive application.

Triggers and events make full use of the underlying queue in process automation which makes task execution quick and predictable. More info on triggers is available in the documentation.

Fully extendable​

The entire component is designed to be extendable. This means it is possible to add custom events or triggers. Not only is it functionally extendable but also the technical implementation allows for extension. An example of this is a custom queue driver. Two queues are provided by default, an Application Persistence queue and an In Memory queue. If you would like to use another queueing technology, you will only need to provide a QueueDriver for this technology.

NSX IntelliJ plugin​

Some nice quality of life features have recently been added to the NSX-Support IntelliJ Plugin.

Reordering Java imports​

You may have encountered issues before with missing imports after harvesting your application. This can occur due to the way intelliJ adds imports to your files. With the updates all manually added imports will now be moved to the anchor:custom-imports block when a file is saved.

Improved expander templates​

The expander templates have been updated to include all the latest and greatest expander features. Things like <artifact> and @imports or support for the TestModelBuilder and expanders-assert with Junit5 are now all available when generating new expanders.

Debugging support for expander mapping​

It can sometimes be difficult to figure out what exactly is added to the expansion context by a mapping file. The NSX support plugin now provides debugging support for mapping files. This is enabled simply by adding breakpoints to the Mapping.xml you want to inspect, and the editor will pause test execution when the selected mapping rule is evaluated. More info on the setup is available in the expander development documentation.

REST Expanders 4.x​

QuerySearch Metamodel​

With the introduction of version 3 of rest-expanders, it migrated to using the QuerySearch metamodel implicitly. This means that all expanders related to QuerySearch were reworked to use the QuerySearch metamodel instead of DataElement. The model was implicitly added when loading, based on existing options on DataElement. This resulted in it being 100% backwards compatible. The reason for this transition phase was to have time to refine the metamodel for QuerySearch, while already being able to prepare expanders to make use of it.

With version 4, we finally made the transition to using the QuerySearch model explicitly, because the model is now stable enough to be used. This transition was facilitated by introducing a transmuter that transforms the model in the same way it was previously done implicitly. This allows us to make the transition without making any changes to any expanders.

This transition also allows developers to add options to elements from the QuerySearch metamodel, which enables the introduction of new features in expanders. One such feature is the option jaxrs.querySearch.exposeFilter on QueryFilterField, which expands the code to expose the filter field on the GETlist endpoint in an expanded API.

Implicit metamodel​

It has been a long term goal to create a metamodel to model REST APIs. In version 4 we're taking the first steps towards this goal. A metamodel is now in development and already implicitly in use (like QuerySearch was previously), so it can be developed without breaking backwards compatibility with existing applications.

One of the first major improvements that were introduced with this metamodel is that now all server error types are defined in a model, reducing the complexity of the related expanders greatly.

Jakarta support​

The improvements to mapping files allow us to facilitate the transition to Jakarta EE. All imports in rest-expanders that must be changed from javax. to jakarta. have been migrated to the new imports system and technology mappings were added to make it compatible with both Java EE and Jakarta EE. There's still some changes requires to fully support Jakarta in rest-expanders, but this puts us very close to the finish line.

QuerySearch joins​

The improve some functionality in rest-expanders, support to define explicit joins was added to QuerySearch. This makes it possible to define LEFT INNER and LEFT OUTER joins on tables involved in the query's FROM clause.

JEE stack improvements​

Java 21 support​

The new LTS of Java, version 21 was released on September 19th. This version introduced a couple of interesting new features such as pattern matching for switches and record patterns.

In order to allow these new features to be used in our JEE applications, we added support for Java 21 as soon as possible, transitioning to a Java 21 JVM in our docker images and supporting it as a compiling target as soon as a release of Temurin and a compatible version of TomEE were available. This has allowed us to create new applications with Java 21 and migrate existing applications to Java 21 in just over one month of its release.

Housekeeping​

Over the years a lot of legacy features have been left behind in our JEE stack expanders to retain maximum backwards compatibility. Though our expanders allow us to reduce the cost of maintaining these, it does still incur a cost. Usually we clean up these features at a gradual pace, but over the last half year we did a serious cleaning of many deprecated features and code that was no longer in use.

• Before we moved to Java 8, we used a custom implementation Options.Option<T> to wrap non-null values. This was replaced with Optional<T>, as our minimum target for applications is now Java 8.
• UserContext has been part of many deprecated constructs in our expanded code. The reason for this is that is was replaced with Context at some point to allow context information to be represented individually for different features and concerns. We first deprecated UserContext in 2018 as main context transport object and have now removed most code where it was used as such.
• The CrudsResult<T> class was rebuilt and made immutable (to the extent possible without losing backwards compatibility). This is the first of the result classes that will be reworked in order to clean up the implementation and make them more consistent. This class was intended to be a simple wrapper with a dedicated purpose. The new implementation enforces more constraints to guide developers to the correct usage of the class.
• Mindmap was removed from nsx-runtime. This was an old feature that is no longer in use.
• We have accumulated a lot of methods marked with @Deprecated in nsx-runtime and expanded code. Most of these that were deprecated for more than a year (usually 3 or more) were removed.
• We've removed a lot of deprecated features in expanded code that we marked with legacy options. These features have typically been phased out for years and have a better alternative and a reasonable upgrade path.
• The groupIds for all maven modules in the JEE applications have been changed to be consistent and unique to individual applications, which was important for re-usable components, which are in a sense a unique instance per application they are expanded for.
• We replaced the implementation of FinderBean to no longer use reflection and require all finders to be defined in the model. This was placed behind an opt-in option for quite some time and used in all new projects, but has now been switched to be the default implementation.

Upcoming features in 2024H1​

REST Expanders​

Development on the REST metamodel which is added implicitly will be resumed. The goal is to move all information that is currently being gathered by multiple expanders into a single data structure represented by the model. This model once refined can then later be made explicit as was done for QuerySearch. The goal is to move the APIs out of components when the model in rest-expanders 5 with the explicit model, so the APIs are coupled to an application and APIs can be created for any component in multiple versions.

Work has also started on expanding Java REST clients for the API generated with rest-expanders. The introduction of the new metamodel will be a key part of this, as a lot of information about the API is needed by both the server and client expanders.

Replacing Struts2​

Our JEE stack has historically used Struts2 to provide a REST API for our expanded Knockout-based frontend to communicate with. This API works well, but also needs some design improvements. We have started development on a new API for our generated UIs which is more uniform and structured, based on the JSON:API format.

We intend to finish the implementation of this new API, as well as provide integration with both our Knockout-based UI and a new fully-generated Angular UI. Through this we aim to provide a more manageable migration path for applications to move from the old UI to a new Angular UI.

Angular UI​

Contribution​

The use of Angular libraries and its documentation will be further developed. The current process for contribution to our "angular ecosystem" will be streamlined and optimized.

New designs​

We've laid the foundation to implement a whole new custom design for our Angular UI that will provide a clean and unique identity to the applications we develop. In the near future we will further improve on this by supporting more types of screens and by adding new features to the views we already support.

Backend clients​

Currently, the angular-expanders rely heavily on the endpoints expanded by the rest-expanders. We are working on a replacement for the current Struts2 API with a JAX-RS API. The latter will be supported by the angular-expanders and a way to have interoperability with the current behaviour needs to be defined.

Expander Tags​

Expanders tags are a design concept to simplify the conditions of Expanders. The idea is to have each Expander define a set of tags. The Expander will only be executed is all tags are available on the element.

E.g. a RemoteExpander could have the tags ejb3 and remote. If either ejb3 or remote is missing from a DataElement, the Remote class will not be expanded.

Tags will also have rules to derive tags from other tags. E.g. hibernate => jpa means that the jpa tag is always included if the hibernate is present, since hibernate is an implementation of the jpa standard.

Rework of implicit name​

If a DataElement does not have a field called name and does not have the nameNotWanted option, the Expanders generate an implicit name field. This name field is then filled in with a default value based on teh database id.

This functionality is problematic in a few ways:

• It is implicit, meaning there is no trace in the model. For developers new to NS, this can be very unpredictable.
• The Expanders each check whether they need to add an implicit name. As a result the behaviour is inconsistent.
• The name field also requires a findByNameEq finder. But this finder points towards a field that does not exist.

We will be streamlining this logic. The solution will probably be to create a transmuter that adds the name field explicitly, only with options to keep the behaviour the same. From then on, not adding a name field to the model has the effect of... not having a name field.

Redesigning DataCommand​

DataCommands are currently used for a lot of tasks they were not designed for. We will design a more suitable solution to model actions on a dataElement. We also want to make it possible to have a better integration with the expanded control and front-end for this as well.

ProcessAutomation 1.0.0​

The preview release has been going well for a while now. There are some missing features like retry mechanisms and time-window groups which we want to introduce still. These features will be integrated with the process automation component and tested before the first full release.

µRadiant 2.x​

Plugin system​

With the advent of new metamodels, we also need a more scalable solution of supporting those metamodels in the µRadiant. The current solution is to push changes to the micro-radiant repository.

The plugin system will allow developers to write their own plugin for the µRadiant and distribute them to other developers wanting to use their metamodels.

UI overhaul​

Writing a new plugin system also required the components of the micro-radiant to be rewritten, because the plugins will be using components of the micro-radiant to implement their logic. Hence, it is important these are well-designed and version transparent.

With this refactoring also comes an overhaul of the styling and UI of the micro-radiant.

Tooling support for Mac​

Currently we provide support with packaging of our tooling for both Windows and Ubuntu. We've noticed an influx of people using Mac devices to work with our tooling. Therefor we intend to provide better support for this platform, as we will then be covering the majority of desktop platforms.

NS Initializer modules​

The NS Initializer is a tool based on expanders that can generate all needed files to start various new projects. It is currently required to model a project definition and write all expanders needed to generate the project. We want to improve this setup by adding support for modules, so parts of an initializer can be reused to build new initializer. A good example of this would be the initializer for an application being partially reused for an initializer that generates an application with a REST API. Then the common expanders do not have to be duplicated.

Alerting component​

We created the alerting component as a new base component, which properly tackles the concern of sending notifications. This was attempted in the past with the notification base component, but the model for this was overfitted for emailing and in practice was used to send full-size emails, rather than just simple notifications.

The alerting component provides a pub-sub system that can be used to route alerts to specific channels, which are instances of channel types such as the default included email, Slack, Pushover and logging channel types. The entire process is driven by the workflow engine so alerts can be efficiently processed and sent to their destination.

This component is also an experiment to further improve our understanding of what it takes to encapsulate a concern in a shareable component, while also providing curated extension points to allow for additional external functionality. With the implementation, it is possible to add additional channel types among other things, that are defined in a a separate component. All default implementations are also separated off into an implementation component.

Workflow overhaul​

We are working on overhauling the entire workflow engine. The goal is to not only improve the capabilities of the engine itself, but also model every state machine so more code can be expanded, configuration becomes much less of a hassle, and we can generate better insights into the state machine with diagrams in the μRadiant.

Right now we've created a metamodel that can be used to fully model workflows in an application, an importer application that can import existing workflow configuration from an SQL database and transform it into a model and integrated tooling for the new system into the μRadiant.

Next steps for the workflow will involve replacing the engine itself with a new implementation. This new implementation should be capable of more efficiently controlling the execution of tasks. Where tasks are now triggered through a polling mechanism that checks the state of an element, it will also be possible to use different types of triggers in the new system, allowing for time- or message-based triggering of tasks as well.

New ValueTypes system​

The ValueFieldType element has been used in the past to define the type of fields in a DataElement that contain a value, such as String or Integer. It was also possible to use these to define new custom types. This system imposed some restrictions that made it very difficult to further expand on its functionality. Among other things, it was rather difficult to use for other languages (other than Java) and required every expander related to Fields to provide its own interpretation about what to do with each type.

To resolve all of these issues, we've introduced a new system for these types, simply named ValueType. This will co-exist with ValueFieldType for some time, but a lot of functionality has already been ported over to the new system and a compatibility layer ensures that this is transparent for most applications.

At this time we've implemented a subtype of the ValueType, called SimpleValueType. This is intended to describe "simple" types, as in types that describe a single value that can be represented by a primitive type such as String, or in rare cases a boxed type of some sort that takes a single value. In the future, we do intend to expand this system into more complex type representations, such as enumerated types, aggregate types (multi-valued) and collection types.

For each SimpleValueType, it is possible to define NativeType instances, which describe some sort of implementation that is tied to technology (such as Java for example) and a concern. This way we can move this information out of the various different expanders, while also allowing for these types to be extended for new technologies and concerns. The system also provides a whole range of capabilities to define constraints on data and redefine types based on other types.

Expander development​

Mapping file syntax​

Mapping files are xml files that are used to map a model representation to a data structure that can be used by an expander to generate files. These mappings are mostly described using the OGNL language, but we've found that some transformations were difficult to do in the mapping files using the available functionality and the OGNL language. Examples of this include flattening collections (often used in flat-map operations) and filtering collections.

In addition to the new functionality of the mapping file structure, support was added to pass OGNL expressions as lambda expressions to Java code.

DataResources​

The _data.xml file is now deprecated. These file(s) were required to define what data resources are provided in an expansion resource. Now the type of DataResources must be defined in the file containing the resource itself and the expanders-maven-plugin will automatically pick this up. This reduces the complexity in defining DataResources considerably.

Expansion diff tool​

We published the expansion diff tool, which can be used to easily find the differences in code expanded from different versions of expanders. The tool takes in two files with expansion settings, expands the project with both of them and then creates a diff between the expanded files.

However, the use of ontological refinement to define classes entails several issues, both on a conceptual and technical level. First, taxonomies are essentially multi-dimensional, i.e., they can be based on multiple criteria. For example, one could refine or categorize a human person based on gender, nationality, or age. Constructing a taxonomy tree based on various criteria could lead to a combinatorial explosion, as nearly all combinations would be possible, e.g., BelgianMaleChild and GermanFemaleAdult. Another option would be to ignore the multi-dimensional nature of the taxonomy, and to use a single primary criterion at a certain level. This approach could limit the anthropomorphic meaning of the classes, as can be illustrated by the distinction between mammals and fish based on the procreation mechanism, e.g., whales and dolphins are no fish although their appearance and behavior is typical for fish. Therefore, it seems often preferable to use references to external classes, such as gender, nationality and age, instead of using inheritance to create subclasses based on taxonomy trees.

Second, the introduction of additional attributes does not always correspond with ontological refinement. Though it is indeed true that a more specific concept exhibits in general more specific attributes, this is not the only mechanism. For instance, a point in 3-dimensional space has an additional attribute with respect to a 2-dimensional point, but it is a generalization instead of a refinement. It is not that rare that attributes of a general concept get default values for more specific concepts and can therefore be omitted. In such cases, programmers are tempted to define the more general concept as a subclass, which is once again detrimental to the anthropomorphic meaning of the classes.

Besides the underlying conceptual issues, the use of inheritance in a traditional object-oriented language creates several technical issues in the context of software development. These issues are mainly related to the introduction of technical coupling between the various classes. Every subclass in the inheritance tree has unrestricted access to the variables and methods of the superclass(es). This coupling is essentially hidden, i.e., the inherited attributes and methods cannot be seen in the class source code, and may be spread out over many classes across many inheritance levels. The fact that modern IDE tools visualize inherited attributes and methods does not really change that. A simple typo can remain invisible during compilation as the compiler may find an actual attribute or method with that name somewhere in the inheritance hierarchy, resulting in the use of the wrong attribute or method. The fact that some languages do not allow multiple inheritance and authors often advise to limit the number of levels in multilevel inheritance, is a clear testimony to the dangers of this technical coupling.

We have often argued that the principle of separation of concerns implies that object-oriented classes cannot contain both data and behavior from a functional point of view. A class needs to either represent data with some auxiliary utility methods, or to represent a behavioral action with some auxiliary utility data attributes. Therefore, we need to distinguish between data and behavior to further elaborate the preferred way to deal with inheritance or ontological refinement.

For behavior functions or methods, we believe that the use of polymorphism is both appropriate and sufficient to provide the required functionality. The mechanism enables the software engineer to provide a variety of implementations that may represent ontological refinement, while realizing at the same time action version transparency for different versions and variants. For instance, an Encrypter interface can encapsulate various specific ways of encrypting a message, while a Publisher interface may be used as a common concept for different ways of message delivery. Every implementation class can be passed in a way that makes the presence of the behavior explicit and allows the programmer to invoke that behavior.

Concerning data attributes or references, we want to stress again that it is often preferable to use external taxonomy entities to avoid the combinatorial explosion of inheritance trees due to the multi-dimensional nature of the taxonomies. A person could for instance have separate taxonomy entities for gender, nationality, and age, or a car could have separate taxonomy entities for the car brand, vehicle type, and propulsion type. Nevertheless, in many cases ontological refinement could be preferable as a number of common attributes may reflect an important common notion or concept that needs to be made explicit. For instance, a home could be an important concept with common attributes that needs to be refined for a house or an apartment. Or a legal person, being a person or legal entity. We argue here once again that the use of polymorphism is both appropriate and sufficient to support such a concept. By providing an interface to represent the common concept, such as a home, the programmer is able to identify, handle, pass, and access the various attributes of this common concept through all ontological refinements.

Information systems still use to a large extent relational databases. To support ontological refinement, we distinguish traditionally three possible options for the database tables as reflected in the drawing below.

1. One super-table containing the union of all subclass attributes.
2. Dedicated tables for every subclass, duplicating the common attributes.
3. Main table with common attributes, and dedicated tables with specific attributes for subclasses.

Option a does not have an explicit representation for the various subclasses, and would lead to lots of empty entries in the various instance rows. Moreover, the introduction of new subclasses – and even new inheritance levels – would lead to an ever expanding structure of the single table.

Option c seems to be the most efficient in terms of database columns or attributes. However, in terms of instances of columns or attributes, it is not. With respect to option b, it requires for every subclass instance an additional type attribute and link key. In addition, it would not be evident to assign an anthropomorphic name to the various dedicated tables. And finally, the software dealing with the persistency data would have to perform some dereferencing logic to retrieve all the attributes, which would not scale well with the introduction of new subclasses or inheritance levels.

Option b seems to have redundant columns for the common attributes, but has less redundancy and more efficiency in terms of instances of database entries. It does not need dereferencing logic, and integrates easily with a polymorphism implementation, where the common concept and attributes are represented as an interface. The fact that the refinement relation is not explicit seems reasonable as the concept is only present in object-oriented classes and not in relational databases.

Based on this reasoning, we have initiated support in Normalized Systems tooling for inheritance or ontological refinement using an implementation that is based on polymorphism for the software classes, and utilizes option b for the organization of the database tables.

With the passing of 2022, we want to look back at many of the notable changes coming out of R&D. Several milestones in the development of our modelling and code generation frameworks have opened up many new possibilities for innovation and have accelerated development even further.

Model validations​

We initially developed model validations as a way to quickly identify some commonly occurring issues in our JEE application models. The initial implementation was therefore coupled to the metamodel for those applications.

Driven by innovations in our metamodel architecture and customer demand, we have completely overhauled the implementation of the model validation framework. It can now be used to validate any of our metamodels and the definition of new validations has been greatly simplified and integrated with our expansion resources. This allows us to specify validations that are relevant within specific scopes, whether it be relevant to a specific expander bundle or even individual projects.

Previously we integrated the validations into our build pipelines. To make them more accessible we now also integrated them into our modelling tool, the μRadiant. This makes the validation results visible at a glance.

Model transmutations​

Model transmutations are a new way of applying transformations to NS models. These transformations are defined as part of expansion resources, just like the new model validations, and are idempotent.

In the past, we've experimented with applying transformations dynamically before expanding a model into artifacts. Though we do believe that there are some valid use-cases for these kinds of transformations, we've found that they introduce unintended coupling between the transformation and the domain of the model. Many features added to a model, will often also have to be applied to the transformation. This means that the implementation of the transformation would have to be modified to accommodate an unrelated concern.

Another obvious issue with dynamically adapting the model at runtime is that the final model is obscured from the developer, which makes it harder to understand how it behaves. This issue is addressed by the model transmutations by writing the transformed model back to the model files, essentially updating the model permanently. As the transformation is idempotent and requires a manual action to execute, it can be reapplied when required.

Similar to the model validations, we've also integrated the transmutations into the μRadiant modelling tool, to make them more accessible. Though they can both also be executed from command-line.

Improved model loading​

The ModelLoader system we developed to load our models has also been fully reworked. The modularity of the system has been greatly improved, and we've introduced the ability to introduce additional processing steps in between each step of the model loading process through the introduction of ModelLoadingListeners. These listeners can be defined in expansion resources and attached at the end of a specific step in the loading process. They receive the result of that step, which they can then process to perform any number of operations.

Expanders​

Over the last year, we've put considerable effort into cleaning up the codebase of expanded JEE applications. Some of these changes include:

• Replacing the use of Vector<T> with ArrayList<T> in places where it was still used.
• Removing redundant (un)boxing operations such as new Long(String), Long.longValue() and new Long(0).
• Pruning many legacy methods.
• Patching EngineService.collector so it is no longer required for workflows to function.
• Removing support for Java 7 and introducing support for Java 17.

For workflows, caching of the workflow configuration was added to improve performance. The DetailsWithoutRefs projection was also added to improve performance in cases where link fields do not have to be retrieved from the database. Finders now come with an "in"-operation and the option configuration.properties was added to include a properties file on the classpath for configuration in individual components.

Exceptions during the model loading and expansion processes have been reworked to provide more concise and useful information about what exact part of the model caused the exception.

Improvements in the metamodel infrastructure allow for faster introduction of new features in various models. Therefore we were able to add a new concept for unique constraints to DataElements. These replace the options uniqueKey (Expanders) and isUnique (rest-expanders), to indicate that a field or a combination of fields should be unique in the database. Both rest-expanders and sql-expanders will introduce support for these constraints during 2023.

OptionTypes​

Another key innovation has been the introduction of a single OptionType meta element, to replace all element-specific ones, such as DataOption or FieldOption. Unifying these option types also allowed us to add support for them to all elements.

The new OptionType element has also been extended with a considerable set of properties:

• What elements the option can be applied to.
• Whether the option can or should have a value.
• Regex validation of the option value.
• Whether the option should be cascaded to child elements.
• Deprecation message and expiration date.
• Name aliases to allow renaming of options.

TestModelBuilder​

Up to this point, the ModelSpecBuilder class has been used to generate composite projects in expander tests, that can be used to test code against. Now we've introduced the TestModelBuilder, which does not only convert the specs to a composite projection. It also applies model transmutations, listeners, and other registered post-processing steps for the model.

Meta-metamodel​

We've designed a new metamodel that can be used to describe ontologies in a way that is not specific to our own technical implementation and allows for easier exchange of models. This new metamodel is capable of representing bidirectional relationships better than our current metamodel. We also introduced support for inheritance through polymorphic interfaces, to support this functionality without introducing the high coupling issues associated with classic inheritance implementations.

To bring this new model into practice, we've defined a transmutation which can transform the model into our current metamodel for which we already have technical infrastructure. This transformation of the model allows us to practically transition to the new metamodel without immediately having to retool our infrastructure.

μRadiant​

For many years, our main development tool has been the Prime Radiant. A tool based on our expanded JEE applications, that could be used to modify NS models. Though it was very successful in accelerating development of new NS projects, we felt that we could offer a better experience to our developers with a new tool that is purpose-built for the task of creating and developing with NS models.

The μRadiant is this new tool, built on technologies such as the Javalin framework and Angular. Though it is not a classic JEE application, it is built following the principles of the NS theory and leverages our expander technology where suitable. We were able to rapidly mach and exceed the capabilities of the Prime Radiant, which has turned the μRadiant into an all-out powerhouse in NS modelling and development.

Some notable features introduced in the μRadiant are:

• The integration of model validations and transmutations.
• A visual model representation with some interaction and editing capabilities that cover the majority of use-cases of our previous visual modelling tool.
• Integration with the NS Initializer tool to easily set up various new projects.
• Support for the development and use of new metamodels.

NS Scripting​

As is common in software development, we found ourselves creating many scripts to facilitate the use of both our own and third-party tooling. Supporting the different scripts/interpreters such as Bash, Batch and Powershell was rapidly becoming an issue. We therefore set out to develop our own scripting language, which we ended up basing on the Kotlin language due to its strong support for higher-order functions allowing us to define a simple API to define scripts with.

NS Scripting implements scoping, inspired by the DSL of Jenkins scripts, to define an API that is clear to understand and performs operations in a consistent manner. It is not our goal to reinvent the wheel and create a new general-purpose scripting engine. KotlinScript itself and many other technologies such as Python and Groovy already have a strong offering to cover this use-case. Our goal is to build a scripting environment that simplifies the common uses of our technology and other supported tools on all platforms.

NS Initializer​

The NS Initializer has been a key product to both simplify setting up new NS projects and standardizing the structure of projects and integration with infrastructure. Initially based simply on a string replacement mechanism that injected variables into template files, the initializer now fully leverages our expander infrastructure to generate artifacts and has projects definitions based on its own metamodel. This has allowed us to both dynamically generate its command-line interface, and integrate it into the μRadiant for setting up new projects.

In 2022 we added a new project type to the initializer to generate metamodel projects, which has since also been reworked to generate projects based on our new ontology metamodel. Recently we also added an initializer to generate Keycloak projects with support for a custom theming library, which shows that its uses can easily be extended beyond the scope of NS-specific projects.

One of the next big steps for the project include the introduction of project modules. This will allow us to separate out some specific functionalities into modules, such as generating CI/CD scripts for our own environment. These modules can then be combined into different project types, allowing re-use and extending of initializers to cover a multitude of use-cases.

Docker images​

We supply Docker base images to run our JEE applications. We have reworked these images with a clear versioning strategy and encapsulating all required technologies. There are images with fixed versions of TomEE and the image itself, as well as other tags that supply the latest image for a version of TomEE, a major version of TomEE or just the latest image with the latest version of all technologies. Additionally, we are also providing variants for different versions of JRE, but we have already phased out images for Java 8 and will phase out Java 11 soon, as JRE 17 should run the applications targeting older versions as well.

Product development​

To support our growing number of users both internally and externally, we have put great effort into improving the user-experience and accessibility of our tools. This includes providing better documentation and improving the setup process, through dedicated Windows installers, Ubuntu packages and Chocolatey packages.

New Foundation​

In line with our efforts to improve the accessibility of our products, we've also put considerable effort into improving our Foundation website and the documentation and content published on it. We've removed all deprecated content that was still present on the website, and we've rewritten and improved everything else. The new platform also has a far more effective search engine, which has weighted results to improve the relevance of the search results. We've now integrated many of the externally published documentation as well and integrated the Maven plugin documentation into the search engine.

In order to enable scaling and to avoid deadlocks, transaction processing systems typically deal with large numbers of small things, and strive to come in, do the work, and get out. This means that it is in general not feasible to use technical transactions to provide end-to-end transactional integrity for customer transactions. For instance, guaranteeing the customer transaction of a money transfer is in general not implemented through a technical transaction encompassing the debiting of the source bank account and the crediting of the destination bank account, which may reside in another bank across the globe. It is typically guaranteed by taking the transfer request as fast as possible into a secure database, followed by a number of sequential processing steps or individual transactions. This flow may even require human intervention to make sure that the customer transaction, i.e., the transfer request stored in the secure database, is executed properly.

The use of end-to-end technical transactions would often cause lots of problems in the real world. For instance, defining a technical transaction around the reservation of a flight ticket, a rental car, and a hotel room, could seem desirable to indulge the customer, but would soon lead to the simultaneous locking of all reservation systems around the world. Avoiding such deadlocks is closely related to the NS theorem Separation of States. This theorem states that after every task or action, the result state should be stored in a corresponding and appropriate data entity. This will enable the independent execution, and therefore evolution, of the implementation of the processing task, and the error handling of the task. Besides better evolvability, this strategy improves the tracing and handling of errors, and reduces the chances of having locked resources.

Another often overlooked issue in transaction management is the complexity of the rollback. While a previous write operation in the same database could be compensated by a relatively simple rollback, developers would have to implement dedicated compensating transactions in case the various steps imply changes across different systems. These compensating actions are not always straightforward or even possible. For instance, in case one of the steps has triggered a physical action, like shipping a product or opening a switch or water tap, it is simply impossible to rollback. Unless mopping the floor is part of the compensating transaction.

These issues are also quite relevant when client systems invoke an API (Application Programming Interface) of a target system. Examples of such client systems are business process management systems invoking services from underlying transactional systems, or user interface front-ends developed upon the service APIs provided by back-end systems. The client systems often desires a customer transaction encompassing several service calls on the target system. This requires client and target system to agree on one or multiple scenarios, that define the responsibilities and behavior at both ends. We discuss here the various possibilities.

• The client system defines the transaction and the target system is expected to honor the transaction. By invoking some start-transaction interface, the client system expects the target system to be able to rollback everything that is being processed until the stop-transaction interface is called. This is a very tough, cumbersome, and nearly impossible requirement to fulfil for the target system.

• The client system attempts to both manage and honor the transaction. Within the scope of a desired transaction, the client system keeps track of the various changes it incurs in the target system, and will rollback and/or compensate the various changes that have already been performed in case something fails.

• The target system provides an explicit aggregated service API for certain customer transactions required by the client system. In this case, the target system attempts to manage and honor the transactional integrity for a specific set of customer transactions, thereby providing appropriate rollbacks and/or compensating actions for these specified transactions.

• The target system provides an asynchronous request API for the aggregated customer transaction required by the client system. The submitted request will be stored immediately in the database, and the target system can start the processing flow performing the various tasks, compensating already performed tasks if required, thereby even supporting manual interventions. The client system can choose to perform other tasks while checking regularly for an answer, or opt to wait or block until the aggregated service is fully executed. The latter option would emulate a synchronous service call at the client side.

• Fire and forget: a workflow task triggers a secondary flow, which can be processed in an independent way.

• A workflow task triggers a secondary flow, but the task in the primary flow can only be considered complete when processing is completed in the secondary flow.

In the second scenario, one should be cautious about possible dependencies and the corresponding coupling. More specifically, coupling may surface in two ways.

• In case the primary workflow and/or task is polling the secondary flow that it has triggered, this could lead to lots of threads and coupling in time between software modules.

• In case the secondary workflow has to report back to the primary flow upon completion, this could lead to a bidirectional dependency between the two workflows. This could result in a technical issue in case the two workflows belong to a different software component.

In order to avoid coupling and even possible locking in time, NS theory suggests that the secondary workflow needs to report back upon completion. However, to avoid bidirectional coupling between the code of both workflows, this should be done using a loosely coupled mechanism. An example of such a loosely coupled mechanism would be a service interface on the primary workflow. This would allow the primary flow to pass the URL of that service interface as a callback function to the secondary flow, avoiding tight coupling in the source base.

Based on the confirmation or completion message of the secondary workflow, the primary flow will set the corresponding status on the target data element of the primary flow. In this way, the secondary workflow does not need any knowledge of the possible states of the primary flow. In case multiple subflows need to be processed and completed before the primary flow is able to continue, it is again (a task in) the primary flow that needs to have the appropriate knowledge to combine the information provided by the various subflows.