View source | Discuss this page | Page history | Printable version   
Toolbox
Main Page
Upload file
What links here
Recent changes
Help

PDF Books
Add page
Show collection (0 pages)
Collections help

Search

Module:External Data Integration/Developers Guide/How To Implement A EDL Type Of Data

Contents

Introduction

In this document is described the required steps to implement a new EDL Type of Data. The Type of Data of a process determines the type of data and its structure to be processed by the EDL Process. It determines the type of each Item and of each Batch of Items. This how-to takes as example the JSON type of data included in the module.

Type of Data definition

The definition in the Application Dictionary consists on adding a new item to the EDL Types of Data List Reference.

Java Implementation

To implement a new Type of Data it is required to extend 3 different abstract classes:

DataProcessor
It manages the raw data as it is expected to be received.
AsynchronousProcessor
Used in asynchronous processes to retrieve iterators for batches and items.
SynchronousProcessor
Used in synchronous processes to retrieve a iterator for items.

Each of this classes is linked to the corresponding Type of Data by a Qualifier where the String value is the value set in the reference list item:

@Dependent
@ComponentProvider.Qualifier("JSON")
public class JSONDataProcessor extends DataProcessor<JSONArray> {

The @Dependent annotation is included to ensure that each EDLRequest process uses a unique instance of the class.

DataProcessor

This class is used to convert the received raw data to the type used for batches of items. In the example it is used the JSONArray. The abstract class defines this type as generic T. It has to be overridden with the required type on the class declaration.

public class JSONDataProcessor extends DataProcessor<JSONArray> {

The abstract class only has a method to implement getDataFromRaw() which has to perform the conversion of the rawData field to the type defined in the class. This method can be overridden by EDLProcess implementations. The first lines of the method should check if this implementation exists by checking if the processDataProcessor field is initialized. In the JSONDataProcessor example it the rawData is not null it tries to parse it to JSONArray by several generic implementations and if it is unable a NotImplementedException is thrown. This exception means that there is a problem in the received data or that it is required to implement a ProcessDataProcessor for the failing EDL Process to properly parse the rawData. Note that when the process is executed by the Process button of EDL Request window the rawData field is loaded from the Storage Location and its type could be different than the original type: File for Attachment Storage Location and String for Database.

  public JSONArray getDataFromRaw() {
    if (processDataProcessor != null) {
      return processDataProcessor.getDataFromRaw();
    } else if (rawData != null) {
      // skipped code
    }
    log.error("Unable to load data from raw");
    log.error("rawData instanceof {}", rawData.getClass());
    log.error("Content of rawData {}", rawData);
    throw new NotImplementedException();
  }

Optional Implementations

The DataProcessor class includes the implementation of the Database and Attachment storage locations. The saveData converts the rawData to a File or to a String when the Storage Location is set to Database or Attachment. If the Type of Data that is implemented receives the rawData in a format that DataProcessor is not able to manage a NotImplementedException is thrown. To solve this issue some methods can be implemented by ProcessDataProcessor class by each EDL Process or more generically by the Type of Data overriding 2 methods writeRawDataToFile() and getRawDataAsString()

writeRawDataToFile() it has to write in the attFile File parameter the rawData. It should call the processDataProcessor field implementation if exists and the method is implemented and later the corresponding implemention.

  @Override
  protected void writeRawDataToFile(File attFile) {
    if (processDataProcessor != null && processDataProcessor.isWriteRawDataToFileImplemented()) {
      processDataProcessor.writeRawDataToFile(attFile);
    }
    // Type of Data Implementation
  }

getRawDataAsString() it has to parse the rawData to String in a format that later can be parsed back to the required type by the getDataFromRaw() method. It should also call processDataProcessor when available.

  @Override
  protected String getRawDataAsString() {
    if (processDataProcessor != null && processDataProcessor.isGetRawDataAsStringImplemented()) {
      return processDataProcessor.getRawDataAsString();
    }
    // Type of Data Implementation
  }

AsynchronousProcessor

When the Type of Data is used by Asynchronous processes this class has to be implemented. As in the DataProcessor implementation the class definition has to declare the type of the Items and the Batches. In the example JSONObject for items and JSONArray for batches.

@ComponentProvider.Qualifier("JSON")
public class JSONAsynchProcessor extends AsynchronousProcessor<JSONObject, JSONArray> {

The class requires to implement 3 different methods: getDataBatcher(), getBatchFromList() and getItemIterator().

The getDataBatcher() has to return a Iterator<String> where each value is the String representation of a batch of items as it is stored in the database on each Request Line of the request. The maximum number of items included in the batch should be the Record Size defined for the EDL Process. This value is available in the recordSize field. The data of the EDL Request is retrieved from the dataProcessor field.

In the case of the JSON example the dataProcessor.getDataFromRaw() returns a JSONArray. This JSONArray is then used to initialize a JSONBatcherIterator instance that it is in charge of iterating over the array and returning an String with the required number of JSONObject items. JSONBatcherIterator is a Iterator implementation included in the same java class.

  @Override
  protected Iterator<String> getDataBatcher() {
    JSONArray data = dataProcessor.getDataFromRaw();
    return new JSONBatcherIterator(data, recordSize);
  }

The getBatchFromList() has to parse a List of items to a String representation of a batch. This method is used to generate new request lines or to update the processed request line when a error happens.

In the case of the JSON example the List is converted to a JSONArray which is returned as a String:

  @Override
  protected String getBatchFromList(List<JSONObject> items) {
    JSONArray errorArray = new JSONArray();
    for (JSONObject item : items) {
      errorArray.put(item);
    }
    return errorArray.toString();
  }

The getItemIterator() has to load the String batch from the request line and return a Iterator of items. The Request Line is available in the requestLine field.

In the case of the JSON example. The line data is converted to JSONArray and this array is used to initialize a JSONObjectIterator that iterates the array returning JSONObject items. JSONObjectIterator is a Iterator implementation included in the same java class.

  @Override
  protected Iterator<JSONObject> getItemIterator() throws OBException {
    String batch = requestLine.getLinedata();
    JSONArray itemArray;
    try {
      itemArray = new JSONArray(batch);
      return new JSONObjectIterator(itemArray);
    } catch (JSONException e) {
      throw new OBException(OBMessageUtils.messageBD("obedl_json_asynch_batch_error"), e);
    }
  }

SynchronousProcessor

When the Type of Data is used by Synchronous processes this class has to be implemented. As in the DataProcessor implementation the class definition has to declare the type of the Items and the Batches. In the example JSONObject for items and JSONArray for batches.

@ComponentProvider.Qualifier("JSON")
public class JSONSynchProcessor extends SynchronousProcessor<JSONObject, JSONArray> {

The class required to implement a single method getItemIterator(). As in the Asynchronous implementation this method has to load the data in the batch type and return a Iterator of items. The data is loaded from the dataProcessor field by executing dataProcessor.getDataFromRaw().

In the case of the JSON example. The data is loaded as a JSONArray and this array is used to initialize a JSONIterator that iterates the array returning JSONObject items. JSONIterator is a Iterator implementation included in the same java class.

  @Override
  protected Iterator<JSONObject> getItemIterator() {
    JSONArray data = dataProcessor.getDataFromRaw();
    return new JSONIterator(data);
  }

Retrieved from "http://wiki.openbravo.com/wiki/Module:External_Data_Integration/Developers_Guide/How_To_Implement_A_EDL_Type_Of_Data"

This page has been accessed 69 times. This page was last modified on 19 September 2016, at 16:42. Content is available under Creative Commons Attribution-ShareAlike 2.5 Spain License.