Module:External Data Integration/Developers Guide/How To Implement A EDL Type Of Data
Contents |
Introduction
This document describes the required steps to implement a new EDL Type of Data. The Type of Data of a process determines the type of data and its structure to be processed by the EDL Process. It determines the type of each Item and of each Batch of Items. This how-to takes as example the JSON type of data included in the module.
Type of Data definition
The definition in the Application Dictionary consists on adding a new item to the EDL Types of Data List Reference.
Java Implementation
To implement a new Type of Data it is required to extend 3 different abstract classes:
- DataProcessor
- It manages the raw data as it is expected to be received.
- AsynchronousProcessor
- Used in asynchronous processes to retrieve iterators for batches and items.
- SynchronousProcessor
- Used in synchronous processes to retrieve a iterator for items.
Each of this classes is linked to the corresponding Type of Data by a Qualifier where the String value is the value set in the reference list item:
@Dependent @ComponentProvider.Qualifier("JSON") public class JSONDataProcessor extends DataProcessor<JSONArray> {
The @Dependent annotation is included to ensure that each EDLRequest process uses a unique instance of the class.
DataProcessor
This class is used to convert the received raw data to the type used for batches of items. In the example it is used the JSONArray. The abstract class defines this type as generic T. It has to be overridden with the required type on the class declaration.
public class JSONDataProcessor extends DataProcessor<JSONArray> {
The abstract class only has a method to implement getDataFromRaw() which has to perform the conversion of the rawData field to the type defined in the class. This method can be overridden by EDLProcess implementations. The first lines of the method should check if this implementation exists by checking if the processDataProcessor field is initialized. In the JSONDataProcessor example it the rawData is not null it tries to parse it to JSONArray by several generic implementations and if it is unable a NotImplementedException is thrown. This exception means that there is a problem in the received data or that it is required to implement a ProcessDataProcessor for the failing EDL Process to properly parse the rawData. Note that when the process is executed by the Process button of EDL Request window the rawData field is loaded from the Storage Location and its type could be different than the original type: File for Attachment Storage Location and String for Database.
public JSONArray getDataFromRaw() { if (processDataProcessor != null) { return processDataProcessor.getDataFromRaw(); } else { JSONArray data = convertRawData(rawData); if (data == null) { throw new NotImplementedException(); } return data; } }
Optional Implementations
The DataProcessor class includes the implementation of the Database and Attachment storage locations. The saveData converts the rawData to a File or to a String when the Storage Location is set to Database or Attachment. If the Type of Data that is implemented receives the rawData in a format that DataProcessor is not able to manage a NotImplementedException is thrown. To solve this issue some methods can be implemented by ProcessDataProcessor class by each EDL Process or more generically by the Type of Data overriding 2 methods writeRawDataToFile() and getRawDataAsString()
writeRawDataToFile() it has to write in the attFile File parameter the rawData. It should call the processDataProcessor field implementation if exists and the method is implemented and later the corresponding implemention.
@Override protected void writeRawDataToFile(File attFile) { if (processDataProcessor != null && processDataProcessor.isWriteRawDataToFileImplemented()) { processDataProcessor.writeRawDataToFile(attFile); } // Type of Data Implementation }
getRawDataAsString() it has to parse the rawData to String in a format that later can be parsed back to the required type by the getDataFromRaw() method. It should also call processDataProcessor when available.
@Override protected String getRawDataAsString() { if (processDataProcessor != null && processDataProcessor.isGetRawDataAsStringImplemented()) { return processDataProcessor.getRawDataAsString(); } // Type of Data Implementation }
Setting the EDL Request Search Key
It is possible to define a Search Key for the EDL requests generated by the EDL process. This can be useful when looking for a particular set of requests in the EDL Request window. Currently, there exists two different approaches to set the Search Key:
a) Overriding the loadContextData() method of the DataProcessor
@Override protected void loadContextData() { super.loadContextData(); try { contextData.put(ProcessRequest.REQUEST_SEARCHKEY, "MySearchKey"); } catch (JSONException ignore) { } }
b) Defining a ProcessDataProcessor which overrides the getRequestSearchKey() method. This method should return the value of the Search Key. In this case, it will also be necessary to implement the getDataFromRaw() method.
@ComponentProvider.Qualifier("MyQualifier") public class MyProcessDataProcessor extends ProcessDataProcessor<JSONArray> { @Override public JSONArray getDataFromRaw() { // Custom Implementation } @Override public String getRequestSearchKey() { return "MySearchKey"; } }
AsynchronousProcessor
When the Type of Data is used by Asynchronous processes this class has to be implemented. As in the DataProcessor implementation the class definition has to declare the type of the Items and the Batches. In the example JSONObject for items and JSONArray for batches.
@ComponentProvider.Qualifier("JSON") public class JSONAsynchProcessor extends AsynchronousProcessor<JSONObject, JSONArray> {
The class requires to implement 3 different methods: getDataBatcher(), getBatchFromList() and getItemIterator().
The getDataBatcher() has to return a Iterator<String> where each value is the String representation of a batch of items as it is stored in the database on each Request Line of the request. The maximum number of items included in the batch should be the Record Size defined for the EDL Process. This value is available in the recordSize field. The data of the EDL Request is retrieved from the dataProcessor field.
In the case of the JSON example the dataProcessor.getDataFromRaw() returns a JSONArray. This JSONArray is then used to initialize a JSONBatcherIterator instance that it is in charge of iterating over the array and returning an String with the required number of JSONObject items. JSONBatcherIterator is a Iterator implementation included in the same java class.
@Override protected Iterator<String> getDataBatcher() { JSONArray data = dataProcessor.getDataFromRaw(); return new JSONBatcherIterator(data, recordSize); }
The getBatchFromList() has to parse a List of items to a String representation of a batch. This method is used to generate new request lines or to update the processed request line when a error happens.
In the case of the JSON example the List is converted to a JSONArray which is returned as a String:
@Override protected String getBatchFromList(List<JSONObject> items) { JSONArray errorArray = new JSONArray(); for (JSONObject item : items) { errorArray.put(item); } return errorArray.toString(); }
The getItemIterator() has to load the String batch from the request line and return a Iterator of items. The Request Line is available in the requestLine field.
In the case of the JSON example. The line data is converted to JSONArray and this array is used to initialize a JSONObjectIterator that iterates the array returning JSONObject items. JSONObjectIterator is a Iterator implementation included in the same java class.
@Override protected Iterator<JSONObject> getItemIterator() throws OBException { String batch = requestLine.getLinedata(); JSONArray itemArray; try { itemArray = new JSONArray(batch); return new JSONObjectIterator(itemArray); } catch (JSONException e) { throw new OBException(OBMessageUtils.messageBD("obedl_json_asynch_batch_error"), e); } }
SynchronousProcessor
When the Type of Data is used by Synchronous processes this class has to be implemented. As in the DataProcessor implementation the class definition has to declare the type of the Items and the Batches. In the example JSONObject for items and JSONArray for batches.
@ComponentProvider.Qualifier("JSON") public class JSONSynchProcessor extends SynchronousProcessor<JSONObject, JSONArray> {
The class required to implement a single method getItemIterator(). As in the Asynchronous implementation this method has to load the data in the batch type and return a Iterator of items. The data is loaded from the dataProcessor field by executing dataProcessor.getDataFromRaw().
In the case of the JSON example. The data is loaded as a JSONArray and this array is used to initialize a JSONIterator that iterates the array returning JSONObject items. JSONIterator is a Iterator implementation included in the same java class.
@Override protected Iterator<JSONObject> getItemIterator() { JSONArray data = dataProcessor.getDataFromRaw(); return new JSONIterator(data); }