2022, looking back at a year of R&D

2022 was a fruitful year for R&D at NSX. Adapting to life during the prior two years marked by the COVID-19 pandemic was certainly something the software industry excelled at. Nevertheless, being back in the office and having face-to-face interactions and the open exchange of ideas has also been greatly beneficial to our work.

With the passing of 2022, we want to look back at many of the notable changes coming out of R&D. Several milestones in the development of our modelling and code generation frameworks have opened up many new possibilities for innovation and have accelerated development even further.

Download the presentation

Model validations

We initially developed model validations as a way to quickly identify some commonly occurring issues in our JEE application models. The initial implementation was therefore coupled to the metamodel for those applications.

Driven by innovations in our metamodel architecture and customer demand, we have completely overhauled the implementation of the model validation framework. It can now be used to validate any of our metamodels and the definition of new validations has been greatly simplified and integrated with our expansion resources. This allows us to specify validations that are relevant within specific scopes, whether it be relevant to a specific expander bundle or even individual projects.

Previously we integrated the validations into our build pipelines. To make them more accessible we now also integrated them into our modelling tool, the μRadiant. This makes the validation results visible at a glance.

Model transmutations

Model transmutations are a new way of applying transformations to NS models. These transformations are defined as part of expansion resources, just like the new model validations, and are idempotent.

In the past, we've experimented with applying transformations dynamically before expanding a model into artifacts. Though we do believe that there are some valid use-cases for these kinds of transformations, we've found that they introduce unintended coupling between the transformation and the domain of the model. Many features added to a model, will often also have to be applied to the transformation. This means that the implementation of the transformation would have to be modified to accommodate an unrelated concern.

Another obvious issue with dynamically adapting the model at runtime is that the final model is obscured from the developer, which makes it harder to understand how it behaves. This issue is addressed by the model transmutations by writing the transformed model back to the model files, essentially updating the model permanently. As the transformation is idempotent and requires a manual action to execute, it can be reapplied when required.

Similar to the model validations, we've also integrated the transmutations into the μRadiant modelling tool, to make them more accessible. Though they can both also be executed from command-line.

Example

The rest-expanders bundle requires a considerable amount of options to be integrated into an application model. To provision default implementations, which cover the majority of use-cases, a project called rest-adder was introduced. This application could transform an application model automatically to add the needed elements and options. This project has now been replaced by transmutations that achieve the same thing.

Improved model loading

The ModelLoader system we developed to load our models has also been fully reworked. The modularity of the system has been greatly improved, and we've introduced the ability to introduce additional processing steps in between each step of the model loading process through the introduction of ModelLoadingListeners. These listeners can be defined in expansion resources and attached at the end of a specific step in the loading process. They receive the result of that step, which they can then process to perform any number of operations.

Example

One such use-case is the current implementation of the QuerySearch metamodel. This model is under construction, but is already in use to drive the expanders in the querysearch-expanders bundle. ModelLoadingListeners are used to transform the original options that were introduced to enable QuerySearch into the metamodel. This makes it possible for us to refine the model and already migrate the technical implementation in a way that does not impact the projects that use these expanders. Once the migration is complete, this listener can be replaced with a transmutation to upgrade existing projects to the new metamodel.

Expanders

Over the last year, we've put considerable effort into cleaning up the codebase of expanded JEE applications. Some of these changes include:

• Replacing the use of Vector<T> with ArrayList<T> in places where it was still used.
• Removing redundant (un)boxing operations such as new Long(String), Long.longValue() and new Long(0).
• Pruning many legacy methods.
• Patching EngineService.collector so it is no longer required for workflows to function.
• Removing support for Java 7 and introducing support for Java 17.

For workflows, caching of the workflow configuration was added to improve performance. The DetailsWithoutRefs projection was also added to improve performance in cases where link fields do not have to be retrieved from the database. Finders now come with an "in"-operation and the option configuration.properties was added to include a properties file on the classpath for configuration in individual components.

Exceptions during the model loading and expansion processes have been reworked to provide more concise and useful information about what exact part of the model caused the exception.

Improvements in the metamodel infrastructure allow for faster introduction of new features in various models. Therefore we were able to add a new concept for unique constraints to DataElements. These replace the options uniqueKey (Expanders) and isUnique (rest-expanders), to indicate that a field or a combination of fields should be unique in the database. Both rest-expanders and sql-expanders will introduce support for these constraints during 2023.

OptionTypes

Another key innovation has been the introduction of a single OptionType meta element, to replace all element-specific ones, such as DataOption or FieldOption. Unifying these option types also allowed us to add support for them to all elements.

The new OptionType element has also been extended with a considerable set of properties:

• What elements the option can be applied to.
• Whether the option can or should have a value.
• Regex validation of the option value.
• Whether the option should be cascaded to child elements.
• Deprecation message and expiration date.
• Name aliases to allow renaming of options.

TestModelBuilder

Up to this point, the ModelSpecBuilder class has been used to generate composite projects in expander tests, that can be used to test code against. Now we've introduced the TestModelBuilder, which does not only convert the specs to a composite projection. It also applies model transmutations, listeners, and other registered post-processing steps for the model.

Meta-metamodel

We've designed a new metamodel that can be used to describe ontologies in a way that is not specific to our own technical implementation and allows for easier exchange of models. This new metamodel is capable of representing bidirectional relationships better than our current metamodel. We also introduced support for inheritance through polymorphic interfaces, to support this functionality without introducing the high coupling issues associated with classic inheritance implementations.

To bring this new model into practice, we've defined a transmutation which can transform the model into our current metamodel for which we already have technical infrastructure. This transformation of the model allows us to practically transition to the new metamodel without immediately having to retool our infrastructure.

μRadiant

For many years, our main development tool has been the Prime Radiant. A tool based on our expanded JEE applications, that could be used to modify NS models. Though it was very successful in accelerating development of new NS projects, we felt that we could offer a better experience to our developers with a new tool that is purpose-built for the task of creating and developing with NS models.

The μRadiant is this new tool, built on technologies such as the Javalin framework and Angular. Though it is not a classic JEE application, it is built following the principles of the NS theory and leverages our expander technology where suitable. We were able to rapidly mach and exceed the capabilities of the Prime Radiant, which has turned the μRadiant into an all-out powerhouse in NS modelling and development.

Some notable features introduced in the μRadiant are:

• The integration of model validations and transmutations.
• A visual model representation with some interaction and editing capabilities that cover the majority of use-cases of our previous visual modelling tool.
• Integration with the NS Initializer tool to easily set up various new projects.
• Support for the development and use of new metamodels.

NS Scripting

As is common in software development, we found ourselves creating many scripts to facilitate the use of both our own and third-party tooling. Supporting the different scripts/interpreters such as Bash, Batch and Powershell was rapidly becoming an issue. We therefore set out to develop our own scripting language, which we ended up basing on the Kotlin language due to its strong support for higher-order functions allowing us to define a simple API to define scripts with.

NS Scripting implements scoping, inspired by the DSL of Jenkins scripts, to define an API that is clear to understand and performs operations in a consistent manner. It is not our goal to reinvent the wheel and create a new general-purpose scripting engine. KotlinScript itself and many other technologies such as Python and Groovy already have a strong offering to cover this use-case. Our goal is to build a scripting environment that simplifies the common uses of our technology and other supported tools on all platforms.

NS Initializer

The NS Initializer has been a key product to both simplify setting up new NS projects and standardizing the structure of projects and integration with infrastructure. Initially based simply on a string replacement mechanism that injected variables into template files, the initializer now fully leverages our expander infrastructure to generate artifacts and has projects definitions based on its own metamodel. This has allowed us to both dynamically generate its command-line interface, and integrate it into the μRadiant for setting up new projects.

In 2022 we added a new project type to the initializer to generate metamodel projects, which has since also been reworked to generate projects based on our new ontology metamodel. Recently we also added an initializer to generate Keycloak projects with support for a custom theming library, which shows that its uses can easily be extended beyond the scope of NS-specific projects.

One of the next big steps for the project include the introduction of project modules. This will allow us to separate out some specific functionalities into modules, such as generating CI/CD scripts for our own environment. These modules can then be combined into different project types, allowing re-use and extending of initializers to cover a multitude of use-cases.

Docker images

We supply Docker base images to run our JEE applications. We have reworked these images with a clear versioning strategy and encapsulating all required technologies. There are images with fixed versions of TomEE and the image itself, as well as other tags that supply the latest image for a version of TomEE, a major version of TomEE or just the latest image with the latest version of all technologies. Additionally, we are also providing variants for different versions of JRE, but we have already phased out images for Java 8 and will phase out Java 11 soon, as JRE 17 should run the applications targeting older versions as well.

Product development

To support our growing number of users both internally and externally, we have put great effort into improving the user-experience and accessibility of our tools. This includes providing better documentation and improving the setup process, through dedicated Windows installers, Ubuntu packages and Chocolatey packages.

New Foundation

In line with our efforts to improve the accessibility of our products, we've also put considerable effort into improving our Foundation website and the documentation and content published on it. We've removed all deprecated content that was still present on the website, and we've rewritten and improved everything else. The new platform also has a far more effective search engine, which has weighted results to improve the relevance of the search results. We've now integrated many of the externally published documentation as well and integrated the Maven plugin documentation into the search engine.