June 2023 R&D update

On the 15th of June, we presented an overview of some of the major developments over the last half year in the NSX R&D team. This post is a guiding article for the presentation, which summarizes the notable topics that we touched on.

Download the presentation

Alerting component

We created the alerting component as a new base component, which properly tackles the concern of sending notifications. This was attempted in the past with the notification base component, but the model for this was overfitted for emailing and in practice was used to send full-size emails, rather than just simple notifications.

The alerting component provides a pub-sub system that can be used to route alerts to specific channels, which are instances of channel types such as the default included email, Slack, Pushover and logging channel types. The entire process is driven by the workflow engine so alerts can be efficiently processed and sent to their destination.

This component is also an experiment to further improve our understanding of what it takes to encapsulate a concern in a shareable component, while also providing curated extension points to allow for additional external functionality. With the implementation, it is possible to add additional channel types among other things, that are defined in a a separate component. All default implementations are also separated off into an implementation component.

Workflow overhaul

We are working on overhauling the entire workflow engine. The goal is to not only improve the capabilities of the engine itself, but also model every state machine so more code can be expanded, configuration becomes much less of a hassle, and we can generate better insights into the state machine with diagrams in the μRadiant.

Right now we've created a metamodel that can be used to fully model workflows in an application, an importer application that can import existing workflow configuration from an SQL database and transform it into a model and integrated tooling for the new system into the μRadiant.

Next steps for the workflow will involve replacing the engine itself with a new implementation. This new implementation should be capable of more efficiently controlling the execution of tasks. Where tasks are now triggered through a polling mechanism that checks the state of an element, it will also be possible to use different types of triggers in the new system, allowing for time- or message-based triggering of tasks as well.

New ValueTypes system

The ValueFieldType element has been used in the past to define the type of fields in a DataElement that contain a value, such as String or Integer. It was also possible to use these to define new custom types. This system imposed some restrictions that made it very difficult to further expand on its functionality. Among other things, it was rather difficult to use for other languages (other than Java) and required every expander related to Fields to provide its own interpretation about what to do with each type.

To resolve all of these issues, we've introduced a new system for these types, simply named ValueType. This will co-exist with ValueFieldType for some time, but a lot of functionality has already been ported over to the new system and a compatibility layer ensures that this is transparent for most applications.

At this time we've implemented a subtype of the ValueType, called SimpleValueType. This is intended to describe "simple" types, as in types that describe a single value that can be represented by a primitive type such as String, or in rare cases a boxed type of some sort that takes a single value. In the future, we do intend to expand this system into more complex type representations, such as enumerated types, aggregate types (multi-valued) and collection types.

For each SimpleValueType, it is possible to define NativeType instances, which describe some sort of implementation that is tied to technology (such as Java for example) and a concern. This way we can move this information out of the various different expanders, while also allowing for these types to be extended for new technologies and concerns. The system also provides a whole range of capabilities to define constraints on data and redefine types based on other types.

Expander development

Mapping file syntax

Mapping files are xml files that are used to map a model representation to a data structure that can be used by an expander to generate files. These mappings are mostly described using the OGNL language, but we've found that some transformations were difficult to do in the mapping files using the available functionality and the OGNL language. Examples of this include flattening collections (often used in flat-map operations) and filtering collections.

In addition to the new functionality of the mapping file structure, support was added to pass OGNL expressions as lambda expressions to Java code.

DataResources

The _data.xml file is now deprecated. These file(s) were required to define what data resources are provided in an expansion resource. Now the type of DataResources must be defined in the file containing the resource itself and the expanders-maven-plugin will automatically pick this up. This reduces the complexity in defining DataResources considerably.

Expansion diff tool

We published the expansion diff tool, which can be used to easily find the differences in code expanded from different versions of expanders. The tool takes in two files with expansion settings, expands the project with both of them and then creates a diff between the expanded files.