Reporting Server/Generating Data Loader App

Contents 1 Data Loader Application 2 Generating the data loader application 3 Data Loader Application Content 4 Running the Data Loader Application 5 Running Individual Load Steps

The update and load scripts are created during development. At runtime the scripts run within a specific application, called the data loader application. The architecture of the reporting solution in production is shown below.

The data loader application runs on the reporting server and is responsible for updating the reporting database/datawarehouse.

Data Loader Application

The data loader application is a standard java server side application which is controlled through ant commands.

The data loader app is generated from the definition in the application dictionary. The main difference between running the individual scripts from the application dictionary and the data loader app is that the data loader app runs the scripts in a certain order. This is defined by the script order field of the load script definition. The data loader app will run scripts and logic in parallel if possible, depending on the number of processors.

Generating the data loader application

The data loader application is generated from the build/development environment. The generation needs a correctly set reporting.properties file. The generation is done with the following ant command which needs to be executed from within the modules/org.openbravo.reporting.tools folder from the build environment.

ant generate.dataloader.application -DtargetDir=[TARGET_DIR]

Replace the [TARGET_DIR] with the relevant target folder.

After generating check the log4j2.xml file (copy it from the log4j2.xml.template file) and configure the log appropriately.

Data Loader Application Content

When you check the content of the data loader application you will see a folder structure like shown below.

A short description for the main files:

• build.xml: provides the ant commands to load the data into the reporting database
• reporting.properties: is copied from the modules/org.openbravo.reporting.tools/reporting.properties. If you didn't have this file setup then you need to copy the reporting.properties.template file and set it appropriately
• log4j2.xml.template: copy this to log4j2.xml and configure its content to get logging tuned to your requirements.
• sql-script: folder containing the sql commands from the load scripts. The materialized views are not included here as that script is done directly in java.
• lib: folder with jar libraries, dependencies of the data loader application.

org.openbravo.reporting.tools.dataloader.jar: the dataloader application itself, if you have custom load classes then these should also be included in this jar file.

• data-load-definition.config: important file containing the steps and their execution order. Below a sample of the file:

{"steps": [
  {
    "id": "2D670B9F28304A65B1063B42D2961C10",
    "name": "bp_group",
    "identifier": "obrtmd_bp_group",
    "tableName": "obrtmd_bp_group",
    "scriptFileName": "obrtmd_bp_group.sql",
    "className": "org.openbravo.reporting.tools.dataloader.StandardDataLoader",
    "modulePrefix": "OBRTMD",
    "reloadOrder": 20,
    "materializedView": false,
    "alwaysFullyReload": true,
    "executeOnce": false
  },
  {
    "id": "AE40DC98475D408A8761936703B543E5",
    "name": "channel",
    "identifier": "obrtmd_channel",
    "tableName": "obrtmd_channel",
    "scriptFileName": "obrtmd_channel.sql",
    "className": "org.openbravo.reporting.tools.dataloader.StandardDataLoader",
    "modulePrefix": "OBRTMD",
    "reloadOrder": 20,
    "materializedView": false,
    "alwaysFullyReload": true,
    "executeOnce": false
  },
  ...
]}

Running the Data Loader Application

To run the data loader application execute the following ant command from within the data loader application folder:

ant load.data

Typically the data loader application runs through the cron_tab. For this usage the data loader application provides the update_script file which can be included in the crontab. For more details check out this wiki page on the server configuration.

Running Individual Load Steps

It is also possible to run individual load steps. For this you need to know the identifier of the load step. You can find this in the data-load-definition.config file, the json field is called identifier. For example to load/run only the BP_Category script do the following action:

ant load.data -Dstep=obrtmd_bp_group