16 posts tagged with "expanders"

Changes and Improvements​

New ValueType System​

ValueFieldTypes are now obsolete and will be replaced with ValueTypes.

You can use the MigrateValueFieldsToValueType transmutation to migrate your ValueFields to the new ValueType representation.

The new ValueTypes support adding restrictions to some basic ValueTypes, like String patterns. So, it will be useful to check out the supported restrictions and see where they can be used in the model.

<field name="foo">
  <valueField>
    <type name="Text"/>
  </valueField>
</field>

Mapping Xml Improvements​

Some improvements have been added to simplify mapping files.

Filter and Distinct in List​

First, the <list> tag has 2 new child tags:

• <filter name="filterName" eval="condition">: Evaluate the condition. If the condition is false, the item will not appear in the list.
• <distinct name="distinctionName" eval="value">: Evaluate the value. Items with duplicate values will be de-duplicated.

These feature were previously already available as the filter and unique attributes on the <list> tag. However, these tags can be interleaved with <let> tags to simplify some of the logic.

E.g. this:

<list name="oneToManyLinkFields" eval="dataElement.fields" param="field"
        filter="field.linkField neq null and not field.linkField.linkFieldType.isMultivalued">
    <value name="name" eval="field.name"/>
    <value name="element" eval="field.linkField.linkFieldType.targetElement.name"/>
</list>

becomes:

<list name="oneToManyLinkFields" eval="dataElement.fields" param="field">
    <filter name="isLinkField" eval="field.linkField neq null"/>
    <let name="linkFieldType" eval="field.linkField.linkFieldType"/>
    <filter name="isOneToMany" eval="not linkFieldType.isMultivalued"/>
    <value name="name" eval="field.name"/>
    <value name="element" eval="linkFieldType.targetElement.name"/>
</list>

NotEmpty and Flatten in ognl​

2 methods have been attached to lists in OGNL statements in the mapping files:

<mapping>
    <!-- List#notEmpty returns true iff the list has no items -->
    <value name="hasDataElements" eval="component.dataElements.notEmpty"/>
    <!-- List#flatten will turn a list-of-lists List<List<T>> into a flat list List<T> -->
    <value name="allDataElements" eval="application.components.{ dataElements }.flatten"/>
</mapping>

String manipulation in OGNL​

Several properties have been attached to Strings to give them similar features to the StringTemplates:

<value name="fieldCapitalized" eval="field.name.firstToUpper"/>

Lambdas in OGNL​

OGNL does not support java 8 lambda syntax. And while OGNL did have its own version of lambdas, these did not work together with java 8 functional interfaces. This has been fixed for the Function, Predicate and Supplier functions.

So now, you can use the OGNL lambda syntax with, for example, Optional:

<value name="valueName" eval="someOptionalValue.map(:[ #this.name ]).orElse('no value')"/>

This version of Expanders is quite impactful. It has a breaking change that could potentially affect several projects if they did not apply the NS theory correctly. This change is the removal of the detailed constructor in the projection POJO classes. We carefully considered the impact of this change and this topic describes the motivation for that and other changes.

Resources​

The expansion resources below provide Expanders 5.19.0.

Changes and improvements​

Removal of detailed constructor​

The projection POJO classes such as <dataElement>Details and <dataElement>Info have always had two constructors. The primary constructor is the constructor with no arguments. There was however also a secondary constructor with the database id and all fields of the projection as arguments. We have now removed the secondary constructor, as there is no technical reason for this to exist anymore.

The reason for removing the secondary constructor is that it violates Data Version Transparency, the first theorem of the NS theory. This theorem states that a changes in a data structure should not affect processing functions that consume them. This constructor, though generated by Expanders, would change whenever the relevant projection or fields in the DataElement change.

MyElementDetails myElement = new MyElementDetails(0L, "some name", "and a value");
...
CrudsResult<Void> result = createMethod(context.withParameter(new MyElementDetails(0L, "please don't", "do this")));

The legacy option legacy.projectionDetailedConstructor.reenable was added to re-enable the old behavior where the secondary constructor was generated. This option will expire on the 1st of September 2023.

Pruning deprecated imports​

Some expanders still contained some deprecated imports for code that was reworked or removed. These have now been removed, so the relevant classes could be deprecated or removed in nsx-runtime.

In version 5.16.0 of Expanders, we removed several expired legacy options. One such option was legacy.searchDataRefMethods.reenable. All related functionality has been removed from the expanders. As such, all related classes have now been removed from nsx-runtime in version 2023.0.0. The remaining related imports have been removed from all expanders for the following classes:

• net.democritus.support.ejb3.SearchDataRefToSearchDetailsMapper
• net.democritus.support.Paging
• net.democritus.sys.SearchDataRef

In 2020 the implementation of the FileUploadFeature, used to upload files through the generated UI, was improved and imports for two now deprecated classes in nsx-runtime were left behind in the Enterer classes. These imports have now been removed, though the classes are at this time still present in nsx-runtime:

• net.democritus.io.NioFileCopier
• net.democritus.io.SimpleUriFormat

Changes and improvements​

GetProjection​

The behaviour of the getProjection method in the element cruds has been changed to no longer return an empty object when the requested element could not be found. Instead it now returns a CrudsError to allow for better error handling.

• The option experimental.cruds.failOnProjectionNotFound has been removed, as it is now the default behaviour.
• A legacy option legacy.cruds.getProjection.emptyOnError was added to revert to old behaviour. (Expires 2023-08-01)

This change shouldn't affect your application if errors were handled previously. If the application assumes the returned CrudsResult to be successful you either:

• Handle the error explicitly, by checking CrudsResult.isError
• Or by providing a default value, for example with CrudsResult.getValueOrElseGet(Projection::new)

Frontend support for Null fieldoperator​

Finders specifying the Null fieldoperator will now be shown as a checkbox in the knockout UI to enable or disable the null check when searching. The label for this checkbox can be customized by overriding the $component.$dataElement.$field.null translation.

Changes and improvements​

UniqueConstraint​

This Expanders version introduces a new Element UniqueConstraint. Each DataElement can have one or more of these UniqueConstraints.

Each one lists a number of Fields. The constraint will generate a check to make sure it is not possible to create 2 instances. with the same values for these Fields.

It replaces the option uniqueKey and improves on it:

• You can have more than one UniqueConstraint.
• The check uses a dedicated query instead of a Finder, so that it will work even for null values.
• No need to explicitly define a Finder.
• No more loosely typed String to define the Fields.

Removed Options​

Some of the Options have expired and have been removed:

• legacy.logicDataMethods.reenable
• legacy.oldCollectionMethods.reenable
• legacy.oldSearchMethods.reenable
• legacy.relationRetrievalMethods.reenable
• legacy.searchDataRefMethods.reenable
• legacy.struts.defaultStyle

Version 5.15.0 of the Expanders brings quite a few new features and improvements! This post will cover an overview of what's new and provide a migration guide to assist in updating. The full release notes can be found here.

Resources​

The expansion resources below provide Expanders 5.15.0.

Changes and improvements​

Changes in GlobalOptionSettings​

Several of the settings in GlobalOptionSettings have been replaced with options on the ApplicationInstance. The long-term goal is to make the GlobalOptionSettings, TechnicalInfrastructure, PresentationSettings and BusinessLogicSettings obsolete.

BeanInterfacePolicy​

The beanInterfacePolicy setting could either be BOTH, LOCAL or REMOTE and controls which interfaces are present on the bean. However, in practice, only the LOCAL and BOTH settings were functional. The option ejb.interfaces.localOnly replaces beanInterfacePolicy='LOCAL', while beanInterfacePolicy='BOTH' becomes the default.

Security Options​

The following flags have been migrated to options:

• enforceHttpMethod: replaced by option struts.security.enforceHttpMethod (Prevents requests to struts control layer from using the wrong HTTP Method.)
• useCsrfProtection: replaced by option struts.security.csrfProtection (Adds a security mechanism to protect against CSRF attacks.)
• useJavaEESecurity: replaced by option struts.security.custom, or simply remove account Component from the Application. (Removes login mechanism from the struts-header.xml file.)

GenerateArtifactLabel​

This feature added a postfix to the versions of Component artifacts in the project. This allows you to install artifacts or deploy them to the nexus so speed up subsequent maven builds by using e.g. the -Pslim profile.

The flag has been replaced by the option mvn.version.appendUniqueLabel. It also now appends the applicationInstance shortname and expanders version, instead of the names of the settings objects. This change is necessary if we want to move away from the settings elements.

CounterDefault​

The counterDefault settings on GlobalOptionSettings used to support a GLOBAL value. This would cause all of the id's to be set programmatically based on a single table.

This has been removed as there are better alternatives (see below).

Identity generation strategy for JPA​

The options persistence.identifier, persistence.sequence.name and persistence.sequence.schema have been renamed from options previously marked as experimental. These options allow a specific identity generation strategy to be selected for a DataElement. The options are available at any level of the model above DataElement and cascaded down. They can be overridden on a lower level for granular control if needed.

The options support either auto-incremented identity columns or native sequences.

For more information, visit the documentation about id generation.

Struts2 6.0​

Support for Struts2 6.0.3 was added to the expanders and is now the default version. From this version and onward, the expanders will start supporting a specific contemporary release version of Struts to improve the distribution of security patches.

There are some breaking changes both in how we handle Struts and the framework itself. See the migration guide for more information.

Version updates​

Breaking Changes​

• The net.palver.util.Options.Option class no longer accepts null values. Option types exist to represent nullable values in a more typesafe way as either Some {value} or None. Some null breaks this contract. If you need to convert a value that can be null, use the Options.notNull(V) method
• Removed name field from ProjectionPojoExpander with option nameNotWanted. The name field was still being set for DataElements with the nameNotWanted option, which was redundant. To revert this behaviour, use legacy.nameNotWanted.pojoName.
• Deprecated JndiProperties and removed its behavior as it pertains only to obsoleted external RMI invocation
• Deprecated net.democritus.mapping.Base64 in favor of java.util.Base64
• Removed classes: FileInserter, ReportFile, SimpleHtmlFormat (for removed ReportExpander)

New Features​

OpenID Connect​

The account component now supports OpenID Connect, which allows you to authenticate users using any provider that follows the standard.

See here for more detailed information.

This page describes the setup and solutions for Google Cloud Platform, Microsoft Azure, Keycloak and Auth0.

XML representation of the Model​

There have been a number of improvements to make the model XML files more readable and concise.

First, several irrelevant fields have been removed from the model. In addition, fields that are empty will not be exported. You can see the effect in the Component XML, where only the relevant information remains:

<component name="account">
  <version>1.0</version>
  <fullName>BASE:account</fullName>
  <businessOwner name="NSX"/>
  <modelOwner name="NSX"/>
  <firstAnalyst name="Paul"/>
  <modelRepository name="BaseComponents"/>
  <customRepository name="BaseComponents"/>
  <description/>
  <mainScreen/>
  <componentDependencies>
    <componentDependency name="account:utils">
      <dependsOn name="utils" version="1.0"/>
      <isManyToMany value="false"/>
    </componentDependency>
  </componentDependencies>
  <componentOptions/>
</component>

<component name="account">
  <version>1.0</version>
  <fullName>BASE:account</fullName>
  <componentDependencies>
    <componentDependency name="account:utils">
      <dependsOn name="utils"/>
      <isManyToMany>false</isManyToMany>
    </componentDependency>
  </componentDependencies>
</component>

You may also notice that the boolean values are no longer represented as attributes, to be more in line with other values:

<field name="name">
  <fieldType>VALUE_FIELD</fieldType>
  <description/>
  <isInfoField value="true"/>
  <isListField undefined="true"/>
  <valueField name="ValueField:Account_name">
    <valueFieldType component="" name="String"/>
  </valueField>
</field>

<field name="name">
  <fieldType>VALUE_FIELD</fieldType>
  <isInfoField>true</isInfoField>
  <isListField>false</isListField>
  <valueField>
    <valueFieldType component="" name="String"/>
  </valueField>
</field>

Next, the options follow a more compact notation. Options are always key-value pairs, which allows us to use a more compact notation:

<componentOptions>
  <componentOption name="account:hasDataBaseSchema">
    <value>ACCOUNT</value>
    <componentOptionType name="hasDataBaseSchema"/>
  </componentOption>
  <componentOption name="account:isBaseComponent">
    <value/>
    <componentOptionType name="isBaseComponent"/>
  </componentOption>
  <componentOption name="account:baseComponents.isAccount">
    <value/>
    <componentOptionType name="baseComponents.isAccount"/>
  </componentOption>
  <componentOption name="account:configuration.properties">
    <value/>
    <componentOptionType name="configuration.properties"/>
  </componentOption>
</componentOptions>

<options>
  <hasDataBaseSchema>ACCOUNT</hasDataBaseSchema>
  <isBaseComponent/>
  <baseComponents.isAccount/>
  <configuration.properties/>
</options>

Not coincidentally, this arrives together with a unified representation of option-types in the model. If you are interested in representing the option types of your expanders, optionally with some value constraints, read this.

Target Element of TaskElement​

Historically, the target element of a TaskElement was represented by the targetClass field, which contained the qualified java class name of the projection of the target element:

<targetClass>net.demo.PersonDetails</targetClass>

It was later replaced with a more correct link to the DataElement:

<targetElement component="registration" name="Person"/>

The targetClass field, however, allowed 2 additional use-cases:

• To use an alternative projection in the task
• To create a task that takes a custom class as input

To support these cases, the following options have been introduced:

• target.element.projection: Changes the projection used from Details to the provided value
• target.class.custom: Overrides the parameter type with the value (takes a fully qualified name)

Model Validations​

The model validations are a set of rules which can be used to generate reports for application models and generate a report detailing any rule violations.

The model validations were already available in some respect, but difficulty to use. It is now possible to run the validations as a maven goal using the expanders-maven-plugin.

In addition to this, the rules are now packaged in expansion-resources, which makes it easier to distribute them. It also makes it possible for teams to develop their own set of rules and include them in the report.

Check out the Model Validations tools page to get started.

If you are interested in creating your own rules, go to this page.

Better traceability of errors during model loading​

Some exceptions during the model loading lacked crucial information to trace back the exception to it's cause.

This has been improved by keeping track of a model execution context. When an exception is thrown, the model execution context will be added to the exception message:

If you are interested in how models are loaded, you can find more information on this here.

net.democritus.elements.ElementNotFoundException: Cannot find Field(registration::Person::Status)

      at FlowElementTreeToComposite.getStatusField (statusField=Status, targetElement=<DataElement registration::Person>)
      at FlowElementTreeToComposite.mapTree (tree=<FlowElement registration::PersonFlow>)
      at ComponentTreeToComposite.mapFlowElement (flowElement=<FlowElement registration::PersonFlow>)
      at ComponentTreeToComposite.mapTree (tree=<Component registration>)
      at ModuleCompositeModelLoader.loadModules (moduleType=<ModuleType elements::JeeComponent>)
      at ExpansionCompositeModelConverter.convert (program=<Application demo::1.0.0>, programType=<ProgramType elements::JeeApplication>)
      at ModelLoader.performStep (step=ConvertModelStep)
      at ModelLoader.loadModel (expansionSettings=F:\NSF\workspace\demo\conf\expansionSettings.xml)

This reveals that the issue occurred:

• during the ConvertModelStep, (Converts Tree representation to interconnected Composite representation. This step will look up every reference and replace it with an object reference. Hence, missing faulty element reference will usually break here.)
• while converting the registration component,
• specifically when converting the PersonFlow FlowElement,
• when the FlowElementTreeToComposite class tried to resolve the statusField

With this information, we can conclude that for some reason the field status does not exist on the target DataElement Person. Looking at the corresponding DataElement XML file will then probably reveal that we forgot to add the field.

ContextRetriever in Control Layer​

For each request, the Context object is now provided by the ContextRetriever class. This class allows you to add additional context objects to the context, which will be available in each layer through the ParameterContext.

public class ContextRetriever {

  public static Context getContext() {
    Map<String, Object> session = ActionContext.getContext().getSession();
    Context context = (Context) session.get("context");
    if (context == null) {
      context = Context.emptyContext();
    }
    if (context.getContext(UserContext.class).isEmpty()) {
      //...
    }
    // @anchor:context:start
    // @anchor:context:end
    // anchor:custom-context:start
    // anchor:custom-context:end
    return context;
  }
  //...
}

An example of this is a TranslationContext. In this application, admin user can insert translations for different languages into the database for a number of values. When a user retrieves the data, the TranslationContext provides the correct language for each user.

The TranslationContext is assigned in the ContextRetriever based on information in the session.

// anchor:custom-context:start
context=context.extend(TranslationContext.getTranslationContext(context,session));
// anchor:custom-context:end

Another example is a TenantContext in an application where all data is linked to a tenant. Users have a tenant assigned to them and when data is retrieved, it is filtered based on the tenant provided by the TenantContext.

Workflow​

The EngineService table has 2 new buttons:

• A 'Refresh' button to force a timer reset when settings have changed. This currently on works for single-node applications.
• A 'Run Now' button to run the EngineService a single time.

To test the recovery mechanism, it is now possible to trigger the recovery with a http call to the workflow/workflow-recover-json endpoint. Provide the workflow reference as a parameter, e.g. workflow/workflow-recover-json?workflow.name=MessageProcessorFlow

There are also a number of performance improvements, which are listed below.

NG Knockout UI​

There are some small improvements in the UI to help with customizations:

• Use the applicationInstance/component/data option cruds.table.csvExportButton to add a button to the DataElement table which export instances as CSV, taking into account the selected finder.
• Added Data Option view.buttons.commands, which adds buttons to NG pages to open command forms. Primarily useful for prototyping and quickly testing new DataCommand features
• Added command-button-builder to add buttons to execute commands in various ways
• Added optional parameters searchMethod and details to data-download-action.ts to provide search parameters when exporting data.
• Added option size to popup, which can be set to large to make the popup bigger.

Second level caching in Hibernate 5​

Add the option persistence.secondLevelCache to a DataElement to enable second level cache. Currently only available for Hibernate 5 and is activated with SearchDetails.setAllowCaching(true).

Read more about it here

Component Application Settings​

It might be relevant to have a properties file in your application server to configure properties of your application.

In this case, you can add the option configuration.properties to one or more of your components. This will generate a <component>ApplicationSettings class, which reads <component>.ns.properties from the classpath and exposes the properties defined in the file.

This can be a good alternative to keeping configurable properties in the ParamTargetValue table. Especially in case of security.

This option is already used in account, to allow configuration of the authentication and authorization mechanics.

DetailsWithoutRefs projection​

Fetching the details projection can at times be too resource-intensive.

This is caused by the resolution of DataRefs. Each LinkField requires an additional lookup to resolve the information for the DataRef, e.g. name.

For remedy this, the option projection.detailsWithoutRefs has been implemented. By adding this option to a DataElement, a detailsWithoutRefs projection will be generated. It contains the same fields as the details projection, but the linkFields are only represented as ids.

It can be useful to use this projection as target projection for a task. Or when retrieving a large amount of instances in a find() operation.

Custom Anchors​

There are some new custom anchors in the expanders:

• Several custom anchors have been added to methods generated by the Claims feature.
• Anchors have been added to FindAction in the Control Layer
• Added fetch-after-create-queries anchors to FinderBean.fetchData().
• Moved the getData-strategy anchors in the Cruds classes to a separate method, so that the getData-after anchors are reached in case of a defined strategy.

Miscellaneous​

Expansion

• Add the expansion option expansion.failFast to your expansionSettings to fail at the first error.
• Add the expansion option expanders.traceFeatures to enable feature tracing. Features tracing adds comments around each inserted feature, which allows you to trace the origins of inserted code in the generated artifacts.
• Add option harvest.skip to a Component to skip harvest for that component. (primarily useful for model resources)

Cleanup

• Made Cruds.getName() opt-in. Add transient.cruds.useGetName to add getName() to the Cruds classes.

Control Layer

• Enable findAll to be used as searchMethod on the find-json struts endpoint

Finders

• Added support for finders with in operator in combination with linkFields

Data Import/Export

• It was no longer possible to expand export() and importFile() pipelines without the use of the includeCsvImport and includeCsvExport options. This functionality is now available separately with the io.import and io.export options for the DataElement.
• Add the application option csv.import.merge, which changes the CSV import so that the imported fields are merged into the existing data, as opposed to the original implementation, which overwrites all fields.

Bug Fixes​

Expanders

• Fixed implicit name field not being initialized correctly
• Adding property tomee.jpa.factory.lazy=true to persistence.xml when expanding for TomEE with Hibernate 5, to avoid CDI loading issues. This is advised by Apache in the TomEE documentation.
• Added InterruptRecoverer error handling
• Fix status capitalization in FlowStateTransitioner
• Fixed workflow claiming not working with Se finders
• Fixed recovery failing because recovered instances were not claimed
• Expose messages from CommandResult
• Fixed Context not being passed by StateTransitioner, causing compareAndSet() to fail if claims are used
• Added checkClaims() to compareAndSet() method for claimable DataElements
• Fixed InterruptRecovererExpander duplicating _FindByNotClaimed postfix if finder with postfix is not present in the model
• Projector class did not add imports for types of local calculated fields.
• Fixed artifactLabel not being included in the application root pom
• Fixed FinderBean not using {DataElement}Data.ENTITY_NAME and not applying jpa.entity.name.format option
• Fixed some copying steps (ext, harvest) not following active layer configuration (copying no{X}Layer files)
• Fixed getStatus(), getStatusAsEnum() return type in case of noDataLayer
• Fixed compareAndSetStatus() method in case of noDataLayer
• Hide tabs in waterfall setup for elements to which the users does not have view access
• Fixed expansion option hideAnchors failing on .properties expanders
• before-harvest and after-harvest are now correctly ignored for base-components. This prevents issues with harvest files in base-components being cleaned during harvest.
• Fixed ext directories for some layerImplementations not being generated because of missing variables when resolving the conditions

Base-Components

• Removed start/stop engine buttons from workflow page, since they are automatically managed by the EngineStarterBean
• Fixed automatic timer reset on timeout when settings are changed
• Fix incorrect timewindow validation in last minute of window
• Added error log when recovery state transitioning failed

Web-Styles

• access-rights.js: Update approved variable before triggering granted
• Fixed loading for observableInstanceBuilder.js when minified
• Fixed tabs to always select the first tab on visibility changes until the user interacts with the tabs. From then on, the selected tab is preferred.
• Fixed nullable DateTime input field
• Fixed loading for nsx-knockout-utils.js and nsx-interaction-model.js
• Change nsx-navbar-fixed.css to fix issues with responsiveness
• Fix several require statements in old js files
• Use nsx-navbar-static.css by default, since it is less buggy. (There is a patch option webstyles.patch.navbar-fixed to revert this)
• Only load instances when tab in waterfall is visible, and hence a parent element is selected, to reduce page load.
• Fixed missing oxide skin for tinymce
• Fix to size bottom of big dialogs properly

Run-time

• Made ClaimId serializable
• Added error log when recovery state transitioning failed
• Fixed Result class returning a null value on error. Instead, it now throws an exception if the getValue() is called on an error result. Implementations should always check the isError() or isSuccess() method first.
• Changed Context.getContext() typing to allow parameterized context groups

Performance Improvements​

Workflow

• Workflow runtime configuration is now cached to reduce the number of database queries
• The detailsWithoutRefs projection is used for several elements in workflow orchestration to reduce the number of database queries
• Optimized getParamTargetValue() with skipCount
• Increased recovery claim timeout to 5 minutes
• Updating EngineNodeService with projection to reduce database queries
• Optimized database query used to update engineService.lastRunAt
• Ignore collector from EngineService

UI

• Render waterfall tab only once an instance is selected