MasterdataController

Contents 1 Introduction 2 Prerequisites 3 Defining a Master Data Model 3.1 Creating a Master Data Model Definition 3.2 Registering a Master Data Model 3.3 Creating the Master Data Model DataSource 3.3.1 Creating Test Cases 4 IndexedDB Masterdata Database 5 Masterdata Model API 5.1 withID 5.2 find 5.2.1 Find with Single Criterion 5.2.2 Find with Multiple Criterion (AND) 5.2.3 Find with Multiple Criterion (OR) 5.2.4 Find with Multiple Criterion (AND + OR) 5.2.5 Find with Limit 5.2.6 Find and OrderBy 5.2.7 Find with Inner Criteria 5.3 orderedBy 6 Performance Considerations 6.1 Creating indices 6.2 Effective Index Usage 6.2.1 Criteria Execution Process 6.2.2 User Model Example 6.3 High Volumes Configuration 7 Masterdata Listeners 8 Masterdata Model Configuration 8.1 Masterdata Model Application Dictionary Definition 8.2 Model Requirements to Support Remote Mode 8.3 Model Configuration 8.4 Remote Mode Requests

Introduction

Openbravo now includes a new component that manages the master data models, called MasterdataController.

This is a read-only component that allows to define and provides access to all of the master data models. At the same time, the master data models define an API, which is detailed in the present document, and that can be used to retrieve their data.

Prerequisites

This article assumes you have knowledge on some of the basic IndexedDB concepts. If this is not the case we recommend to start by taking a look at this guide.

Defining a Master Data Model

To define a Master Data Models the following steps should be followed:

• Create the model client-side definition
• Register the model with the MasterdataController
• Create the model source of data

Creating a Master Data Model Definition

The client side definition of a master data model is done by defining a Javascript class that extends OB.App.Class.MasterdataModel:

 
  class SalesRepresentative extends OB.App.Class.MasterdataModel {
    constructor() {
      super();
      this.indices = [
        new OB.App.Class.Index({
          name: '_identifier_idx',
          properties: [{ property: '_identifier' }]
        })
      ];
    }
 
    getName() {
      return 'SalesRepresentative';
    }
  }

Note that here we define the indices of the model which we will later explain in this section.

Registering a Master Data Model

To register a master data model we make use of the MasterdataController and its registerModel function:

 
  OB.App.MasterdataController.registerModel(SalesRepresentative);

The registerModel function receives the class name of the model to be registered.

Once the model has been registered the MasterdataController will expose it so we can access it using OB.App.MasterdataModels:

 
  OB.App.MasterdataModels.SalesRepresentative;

It is strongly recommended to create and register the model in the same function. Thus, we are not exposing the model class definition and the model can be accessed from a single central point (with OB.App.MasterdataModels):

 
(function SalesRepresentativeDefinition() {
  class SalesRepresentative extends OB.App.Class.MasterdataModel {
    constructor() {
      super();
      this.indices = [
        new OB.App.Class.Index({
          name: '_identifier',
          properties: [{ property: '_identifier' }]
        })
      ];
    }
 
    getName() {
      return 'SalesRepresentative';
    }
  }
 
  OB.App.MasterdataController.registerModel(SalesRepresentative);
})();

Creating the Master Data Model DataSource

Each master data model is linked to a backend component that takes care of providing the model data (records). This component is a Java class that extends the MasterDataProcessHQLQuery class.

A class extending MasterDataProcessHQLQuery should:

• Declare the @MasterDataModel qualifier to link it to the model client (Javascript) definition.
• Implement the getMasterDataModelProperties method which return a list of the properties that defines the master data model.
• MasterDataProcessHQLQuery extends ProcessHQLQuery. It should implement the methods inherited from that class which are required to retrieve the data properly.

The code snippet below shows an example of the structure of a MasterDataProcessHQLQuery:

 
@MasterDataModel("SalesRepresentative")
public class SalesRepresentative extends MasterDataProcessHQLQuery {
  public static final String salesRepresentativePropertyExtension = "OBPOS_SalesRepresentativeExtension";
 
  @Inject
  @Any
  @Qualifier(salesRepresentativePropertyExtension)
  private Instance<ModelExtension> extensions;
 
  @Override
  protected List<HQLPropertyList> getHqlProperties(JSONObject jsonsent) {
    ...
  }
 
  @Override
  protected List<String> getQuery(JSONObject jsonsent) throws JSONException {
   ...
  }
 
  @Override
  protected boolean bypassPreferenceCheck() {
    ...
  }
 
  @Override
  protected Map<String, Object> getParameterValues(JSONObject jsonsent) throws JSONException {
    ...
  }
 
  @Override
  public List<String> getMasterDataModelProperties() {
    return getPropertiesFrom(extensions);
  }
}

The getPropertiesFrom is a method that retrieves the HQL properties of the model from its extensions. If a master data model does not have extensions and its properties are directly retrieved from the related DAL class, there is also another method available with the same name but it receieves the DAL class as argument instead.

Creating Test Cases

Ideally, when creating a new MasterDataProcessHQLQuery it must be accompanied by jUnit test cases to check that the expected records are fetched when fully or incrementally refreshing of the model.

The MasterdataModelTestHelper utility class can be used to build this kind of test cases. This test class should be instantiated by providing the name used to annotate the MasterDataProcessHQLQuery with the @MasterDataModel annotation:

MasterdataModelTestHelper salesRepresentativeModel = new MasterdataModelTestHelper("SalesRepresentative")

And then it can be used to test the records retrieved either with a full refresh...

Map<String, Object> contextParameters = Map.of("organization", VALL_BLANCA_STORE);
salesRepresentativeModel.fullRefresh(contextParameters);

... or with an incremental refresh providing the date of the last update

Date lastUpdate = DateUtils.hoursAgo(1);
salesRepresentativeModel.incrementalRefresh(contextParameters, lastUpdate);

See a real example of use here.

IndexedDB Masterdata Database

All the information retrieved from the master data models is stored in an IndexedDB database managed by the MasterdataController. In particular, the data is persisted on an single object store for each model.

The MasterdataController is also in charge to populate and keep this database updated when the data is loaded in Web POS. See here to learn about the different modes to load the master data.

In addition, the MasterdataController has a recreation mechanism that detects model changes (both in the client and in the backend side). This can happen, for example, if after a module upgrade a new property is added into a model. In that case, on login the MasterdataController will drop the database entirely and recreate it from scratch using the new model definition(s).

Masterdata Model API

All of the master data models provide an API that is the mechanism that should be used to access to their data. Note that the functions of this API perform asynchronous tasks to retrieve the information from IndexedDB and for this reason they are declared as async functions.

Next all these master data model API functions are detailed.

withID

This function is used to retrieve an object of the master data model by its ID.

 
  const salesRepresentative = await OB.App.MasterdataModels.SalesRepresentative.withId(id);

find

This function is used to retrieve all the objects that match a criteria. To define a criteria the Criteria API is used. This API provides a set of functions that can be easily combined using function chaining:

• limit: define the maximum number of objects to be returned. If not used, a default limit of 300 is applied.
• operator: sets the criteria main operator: and or or.
• criterion: adds a criterion which is composed with three elements: a property, a value and a comparison. The available comparisons are:
  • equals (default comparison if not provided)
  • in
  • includes
  • not
  • notIn
  • greaterThan
  • greaterOrEqualThan
  • lowerThan
  • lowerOrEqualThan
• orderBy: defines the properties to order by the returned objects and the sorting mode for each one: asc or desc.
• multiCriterion: allows to combine several criterion elements with an operator.
• innerCriteria: adds a child (inner) criteria to the main criteria
• build: builds the criteria object that is expected by the find function.

Next some basic usage examples are shown:

Find with Single Criterion

Find the object of the Sales Representative model with user name equal to vallblanca:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion(new OB.App.Class.Criterion('username', 'vallblanca'))
      .build();
  const salesRepresentative = await OB.App.MasterdataModels.SalesRepresentative.find(criteria)[0];

which is equivalent to:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion(new OB.App.Class.Criterion('username', 'vallblanca', 'equals'))
      .build();
  const salesRepresentative = await OB.App.MasterdataModels.SalesRepresentative.find(criteria)[0];

because equals is used by default if no comparison is used.

In the previous example the Criteria has one query property: username.

Find with Multiple Criterion (AND)

The and operator is the default one, so we do not need to specify it:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion('active', false)
      .criterion('username', ['vallblanca', 'demouser'], 'in')
      .build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

Find with Multiple Criterion (OR)

If we want to use the or operator we need to set out it with the operator function:

 
  const criteria = new OB.App.Class.Criteria()
      .operator('or')
      .criterion('active', false)
      .criterion('username', ['vallblanca', 'demouser'], 'in')
      .build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

Find with Multiple Criterion (AND + OR)

We can combine operators using multiCriterion:

 
  const criteria = new OB.App.Class.Criteria()
      .multiCriterion(
        [
          new OB.App.Class.Criterion('username', 'vallblanca', 'not'),
          new OB.App.Class.Criterion('name', 'L', 'includes')
        ],
        'or'
      )
      .criterion('active', true)
      .build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

Find with Limit

The API supports defining the maximum number of objects that can be returned with find by using the limit function:

 
  const criteria = new OB.App.Class.Criteria().limit(5).build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

Note that if no limit is provided (the limit function is not used), the infrastructure sets a default limit of 300 elements.

Find and OrderBy

We can also sort (ascending or descending) the results of a search with find using orderBy:

 
  const criteria = new OB.App.Class.Criteria().limit(5).orderBy('name', 'asc').build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

Observe that it also exists an orderBy function that can be directly invoked from any Masterdata model.

Find with Inner Criteria

An inner criteria can be used when it is needed to define more complex conditions:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion('active', true)
      .innerCriteria(
        new OB.App.Class.Criteria()
          .multiCriterion(
            [
              new OB.App.Class.Criterion('username', null, 'not'),
              new OB.App.Class.Criterion('name', 'vallblanca', 'not')
            ],
            'and'
          )
          .multiCriterion(
            [
              new OB.App.Class.Criterion('_identifier', null, 'not'),
              new OB.App.Class.Criterion('name', 'L', 'includes')
            ],
            'and'
          )
          .operator('or')
      )
      .build();
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

orderedBy

The master data models also have a function that allows to directly retrieve their objects sorted by some specific properties:

 
  const salesRepresentatives = await OB.App.MasterdataModels.SalesRepresentative.orderedBy(['_identifier', 'active'], ['asc','desc']);

Performance Considerations

When writing queries using the API explained in the previous section it is very important to take into account some restrictions imposed by the IndexedDB design. In general, if we want to make a performant query against a model, we should try to ensure that the query uses a proper index.

Creating indices

To create a new index we make use of the OB.App.Class.Index class whose constructor receives two properties:

• name: the index name. It should be unique for all the indices of the same model. For this reason, in case of creating an index in a custom module it is a good practice to add the module dbprefix at the beginning of the name
• properties: an array with the names of the properties to be indexed. Each element in this array has:
  • property: the name of the indexed property
  • isBoolean: this property must be set as true for boolean properties.
  • isNullable: this property must be set as true for those properties that can have null values.

We can register indices of the model when creating the model or we can extend an existing model and register new indices on it using the addIndices function as below:

 
  OB.App.MasterdataModels.SalesRepresentative.addIndices(
    new OB.App.Class.Index({
      name: 'name_idx',
      properties: [{ property: 'name' }]
    })
  );

By default all the models are created with a default index by their id property, therefore it is not necessary to explicitly create an index by id.

Effective Index Usage

Now the we know how to create an index, we need to understand when they should be created. To this we need to learn how our Criteria IndexedDB queries are executed.

Criteria Execution Process

The process to execute a Criteria IndexedDB is the following:

1. First the results are filtered using a query at database level, if possible. To execute this query we need an index. The infrastructure will try to find an index which have the same properties (in the same order) as the query properties of the criteria. Remember: the query properties are the first level properties that use the equals comparison.
   1. If an index with exactly all the query properties (in the same order) is found, then that index will be used
   2. If not found but there is an index with the first n query properties (in the same order), then that index will be used.
   3. If no index can be found, then the query will fail, throwing an error.
2. Once we have filtered the results using the selected index, the rest of the criteria will be applied (non query properties: comparisons not using equal and/or inner criteria). This can be the most expensive part of the query as this filtering is done in memory through an IndexedDB cursor.
3. When all the objects matching the criteria are found or the query limit is reached, then they will be returned. If the result should be ordered, results will be ordered in memory before being returned.

User Model Example

Let's illustrate the process described above with an example. Suppose that we have a dummy model called User. An example object of this model would be as follows:

 
  {
    id: '3073EDF96A3C42CC86C7069E379522D2',
    name: 'Vall Blanca Store User',
    username: 'vallblanca',
    type: 'A',
    birthDate: '1980-07-08T00:00:00.000Z',
    credit: 23.09,
    active: true,
    expired: false
  }

We register our model with an index using the MasterdataController:

 
class User extends OB.App.Class.MasterdataModel {
  constructor() {
    super();
    this.indices = [
      new OB.App.Class.Index({
        name: 'active_type_idx',
        properties: [
          { property: 'active', isBoolean: true },
          { property: 'type' }
        ]
      }),
      new OB.App.Class.Index({
        name: 'username_idx',
        properties: [
          { property: 'username', isNullable: true}
        ]
      })
    ];
  }
}
OB.App.MasterdataController.registerModel(User);

Then, let's suppose that we want to execute the following query:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion(new OB.App.Class.Criterion('active', true))
      .criterion(new OB.App.Class.Criterion('type', 'B'))
      .criterion(new OB.App.Class.Criterion('credit', 25, 'lowerThan'))
      .orderBy('name', 'asc')
      .build();
  const users = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

The previous query has:

• Two query properties: active and type (as they use the equals comparison.
• A non query property: credit (using lowerThan operator)

Therefore, the query will be executed as follows:

1. The User model has an index that exactly matches with the query properties: active_type_idx. This index will be used to filter the results with a database query in order to retrieve the active users of type 'B'.
2. For each of the filtered result, an in-memory validation will be done to filter the results that have a 'credit lower than 25.
3. The filtered results are ordered by name.

Note that if we had defined the query in this way:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion(new OB.App.Class.Criterion('type', 'B'))
      .criterion(new OB.App.Class.Criterion('active', true))
      .criterion(new OB.App.Class.Criterion('credit', 25, 'lowerThan'))
      .orderBy('name', 'asc')
      .build();
  const users = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

The infrastructure would not have been able to find an index to execute the query, because we do not have an index by the query properties in the same order. In this case, the query would eventually fail.

Of course to execute the previous query we could add an index type_active_idx into the model to match with the query properties but this is not recommended because adding a new index implies an increase on the storage size used by the application. If the model has a lot of objects then the index size can be big. So in this case we recommended to reorder the query properties defined in the query to match with an existing index.

Finally, if we wanted to execute this query:

 
  const criteria = new OB.App.Class.Criteria()
      .criterion(new OB.App.Class.Criterion('active', true))
      .criterion(new OB.App.Class.Criterion('type', 'B'))
      .criterion(new OB.App.Class.Criterion('username', 'demouser'))
      .criterion(new OB.App.Class.Criterion('credit', 25, 'lowerThan'))
      .orderBy('name', 'asc')
      .build();
  const users = await OB.App.MasterdataModels.SalesRepresentative.find(criteria);

In this case the query would be executed using the active_type_idx index also. This is because although the index does not have all the query properties (it is missing the username property), the infrastructure is able to do a key range filtering using the matching properties (active and type). And the rest of the criteria properties would be filtered in memory.

High Volumes Configuration

In case a model has a high volume of data, it could happen that having an index is not enough to make a query work in a performant way.

If we were in such case, each of the master data models provides a mechanism that allows to execute the queries using an in-memory cache which in general brings a faster execution of the data filtering. To enable this mechanism it should be specified which object properties will be kept in the model cache.

It is possible to enable the cache at the same moment of registering a new model with the searchProperties property:

 
class User extends OB.App.Class.MasterdataModel {
  constructor() {
    super();
    this.indices = [
      new OB.App.Class.Index({
        name: 'active_type_idx',
        properties: [
          { property: 'active', isBoolean: true },
          { property: 'type' }
        ]
      }),
      new OB.App.Class.Index({
        name: 'username_idx',
        properties: [
          { property: 'username', isNullable: true}
        ]
      })
    ];
    this.searchProperties = ['active','name'];
  }
}
OB.App.MasterdataController.registerModel(User);

Or it is possible to extend existing models to include new properties into the cache using the addSearchProperties function:

 
OB.App.MasterdataModels.SalesRepresentative.addSearchProperties(['type', 'username']);

When the cache mechanism is enabled for a model, the internal process to execute the query against that model is slightly different:

1. First the query mode is selected (index or cache):
   1. If an index with exactly all the query properties (in the same order) is found, then that index will be used
   2. If not found and all the properties referenced in the query are present in the cache, then the results will be filtered with the cache
   3. If it is not possible to use the cache, but there is an index with the first n query properties (in the same order), then that index will be used.
   4. If no index can be found, then the query will fail, throwing an error.
2. If we have filtered the results using an index, the filter by non query properties should be done.
3. When all the objects matching the criteria are found or the query limit is reached, then they will be returned. If the result should be ordered, results will be ordered in memory before being returned.

Please note that enabling the cache mechanism has some implications:

• The cache is kept in memory. Therefore the memory usage of the application increases after enabling it.
• The cache is updated automatically with the model data stored in the IndexedDB Masterdata Database.
• The data in the cache is lost if the application is refreshed (F5). After that, the cache is automatically populated with all the data.
• The cache can not be used while it is being populated/updated.

Masterdata Listeners

It is possible to register a post update hook, to execute a function when specified masterdata models are updated.

• It will NOT launch the hooks on login:
  • full on login
  • incremental on login
• only will check for changes after login:
  • incremental refresh button
  • incremental refresh in background by timer
  • masterdata endpoint

• When the hook calls to the updateFunction, it will pass a paramenter that specified what was the type of refresh that launched it:
  • incrementalMasterdataRefresh -> refresh masterdata button  : updateFunction called once when finish all models
  • incrementalBackgroundSave -> refresh masterdata in background : updateFunction called once when finish all models
  • save -> masterdata endpoint  : updateFunction called once when finish each model

• If you listen to many models, and only want to execute the updateFunction once, consider the use of debounce when the update was from the masterdata endpoint (save)

Example code to register a masterdata post hook:

 
OB.App.MasterdataController.postUpdateHookRegister({
  modelsToListen: ['Keymap','KeymapVariant'],
  updateFunction: async refreshMasterdataMethod => {
    if (refreshMasterdataMethod === 'save') {
      // masterdataEndpoint messages, with the debounce we execute only once, instead once per model
      await updateFunctionDebounced();
    } else {
      await updateFunction();
    }
  }
});

Masterdata Model Configuration

Masterdata Model Application Dictionary Definition

It is possible to configure the mode which the data of a master data model is retrieved. To be able to set up this configuration first the model must be registered in the application dictionary through the Master Data Model window.

In this window the following fields must be populated:

• Module: the module that the model belongs to
• Application: the application that the model belongs to. When login into an application, it will be only loaded the configurations of the master data models registered for that application or any of its dependencies.
• Search Key: the search key of the model, it must be the same string used to annotate with @MasterDataModel the corresponding MasterDataProcessHQLQuery. Although it is not enforced that this value to start with the module DB prefix, it is strongly recommended to set this value starting with the module DB prefix in order to avoid collisions between models.
• Name: a more descriptive identifier for the master data model.
• Implemented Modes: the modes in which a model can work. It accepts two values:
  • Local: the model can only query the data from the browser’s local storage
  • Local and Remote: the model supports both querying the data from the browser’s local storage and remotely by requesting the data to the backend.
• Default Mode: the mode which a master data model works by default. The accepted values are:
  • Local: The model gets the data from the browser’s local storage
  • Remote: The model gets the data remotely from the backend. It can only be used if Implemented Modes is Local and Remote.
• Description: text that explains the purpose of the model

Model Requirements to Support Remote Mode

To make a master data model support remote the corresponding MasterDataProcessHQLQuery has to meet some requirements:

1. It must implement the getHqlProperties method
2. The HQL queries must contain the $filtersCriteria filter on its where clause and the order by clause must be $orderByCriteria

Model Configuration

By default all the master data models work in Local mode. But it is possible to change this behavior at configuration level, using the Master Data Model Configuration window.

Here the fields to populate are:

• Organization: the organization for which the model configuration applies
• Master Data Model: the model to be configure
• Mode: the configured mode. One of two values can be selected here:
  • Local: the model gets the data from the browser’s local storage
  • Remote: the model gets the data remotely from the backend

Remote Mode Requests

As explained above, when a model is configured to work in remote mode, the master data is retrieved by querying the information to the backend through HTTP POST requests. This requests are executed within the timeout configured for the org.openbravo.mobile.core.master.MasterDataLoader mobile service.