Module:External Data Integration/Developers Guide/How To Implement A EDL Process

Contents 1 Introduction 2 EDL Process definition 2.1 Synchronous process 2.2 Asynchronous process 3 EDL Process implementation 4 EDL Process execution 4.1 Creating EDL Requests Transactionally

Introduction

In this document is described the required steps to implement a new EDL Process. A EDL Process is a custom development used to extract or load data to other systems.

EDL Process definition

The EDL Process is defined in the EDL Process window as System Administrator. These are some common fields that need to be defined:

Module

The EDL Process must belong to an extension module. Make sure you have created one and it is in development.

Name

The name of the process is used as identifier in the EDL Configuration window to select the processes.

Search Key

The search key or value is used to retrieve the java class implementing the process.

Type of Data

Based on how is received or processed the data. Determines the type of each processed item and the type of a batch of items. The module includes a basic JSON implmementation. It is possible to develop new types to support different types like xmls, csvs,...

Has Output

If the process generates data that has to be processed by a Output processor this flag needs to be checked. When this flag is checked it is required to add at least one Output Process in the EDL Configuration window.

Has Input

In case the process starts by receiving some data that needs to be processed.

Storage Location

Determines how is stored the received raw data. Note that if No Stored is selected it might not be possible to reprocess a request in case of error.

Synchronous

Determines how the process is executed. Depending on this flag additional configurations are needed

Enable Output Retry

Enables to ability to configure the retry options on the EDL Configuration window.

Reprocess Policy

Determines if, and how, EDL requests from this process will be reprocessed. Available options are:

• No Reprocess: The Process button will not be shown
• Errors only: Only the EDL request (or EDL request lines for async processes) that have failed will be reprocessed.
• All requests can be reprocessed: Default behaviour, all requests can be reprocessed, regardless of its status.

Synchronous process

Synchronous processes are executed in the same thread where the request is added. It is useful when it is needed to have the response of the process immediately.

Additional fields to define on synchronous processes:

Has Direct Response

If this flag is checked it is required to call the addRequest() method with a writer where the response is writer. This is used when the request is launched from a web service. So it is possible to write the response in the same response writer of the original WebService call.

Asynchronous process

Asynchronous processes are scheduled and processed in background using Import Entries. It is useful to process large amounts of data that could take several minutes / hours to process. In this type the process the data is split in request lines of item batches.

Additional fields to define on synchronous processes:

Record size

Determines the number of items included on each batch or request line.

Commit items

Determines the number of items to process before doing a commit. This number should be equal or lower than the record size.

EDL Process implementation

The process is developed by implementing the org.openbravo.externaldata.integration.process.ItemProcessor class.

The link to the process is done by a Qualifier:

  @ComponentProvider.Qualifier("MyProcessSearhKey")
  public class MyProcessItemProcessor extends ItemProcessor {

The main logic of the process is implemented in the processItem method. The T type is defined by the Type of Data. All the items of a request are processed using the same instance of ItemProcessor so it is possible to use class fields to store information related to the request. For example it is possible to define a StringBuilder that it is updated by each item to generate the output string of the response.

protected abstract void processItem(T item) throws OBException;

Asynchronous executions are automatically commit based on the commit items configuration. This means that in case a item fails and a rollback is done several successfully processed items might also be rolledback. In case it is required to process them again the reprocessNotCommitedItemsOnException has to be implemented.

  /**
   * Asynchronous execution process a number of items configured in the EDL Process before
   * committing them. In case of exception a rollback is done. This method determines if the
   * processes that have not been committed need to be processed again or not.
   * 
   * @return TRUE if the items have to be reprocessed.
   */
  protected abstract boolean reprocessNotCommitedItemsOnException();

If direct response is required it has to be written in the directResponse class field on each processItem() method execution.

protected StringBuilder directResponse = new StringBuilder();

If the process has output the writeResponse, writeErrorResponse and the outputExtraActions methods may be overridden.

• The writeResponse has to append to the writer the response generated during the request execution.
• The outputExtraActions it is received as parameter the OutputDataProcessor instance where the output process is executed so it is possible to read for example the response of a web service call where the response has been sent.
• The writeErrorResponse method can be implemented to directly store into the database the output content when the execution of the EDL request fails. This can be useful for example to save the content generated by the item processor when the error happened. This information is saved in the EDL request in the same way the OutputDataProcessor does with the content provided by the writeResponse method.

  /**
   * Method called from the {@link OutputDataProcessor} when there is something more to do with the
   * output, for example reading the response of a webservice request.
   * 
   * @param outputDataProcessor
   *          The data processor that is being processed.
   * @param output
   *          The configuration that is being processed.
   */
  public void outputExtraActions(OutputDataProcessor outputDataProcessor, OBEDLConfigOutput output) {
    // Not implemented by default
  }
  
  /**
   * Method called from the {@link OutputDataProcessor} to write the response.
   * 
   * @param writer
   *          the writer configured and opened. It must remain opened when this method has finished.
   * @param configOutput
   *          configuration of the output processing. Null is the
   */
  public void writeResponse(Writer writer, OBEDLConfigOutput configOutput) throws IOException {
    // Not implemented by default
  };
 
  /**
   * Writes the process output information, if desired, when the processing of an EDL request fails.
   * This information will be saved into the database as the {@link OBEDLOutputContent} linked to
   * the failed EDL request (synchronous process) or failed EDL request line (asynchronous process).
   * If nothing is written into the writer, then nothing will be saved into the database.
   *
   * @see #saveErrorOutput(OBEDLRequest, OBEDLRequestLine)
   *
   * @param writer
   *          the writer configured and opened. It must remain opened when this method has finished.
   */
  protected void writeErrorResponse(Writer writer) throws IOException {
    // Not implemented by default
  }

EDL Process execution

When a new process is defined it is also required to develop the needed processes to launch the requests. To create and process a new EDL Request it is required to use the ProcessRequest class. There are different addRequest() methods depending on the type of process to be executed.

The addRequest can be called from different type of processes: EventHandlers, processes (menu or window), exposed web services or background processes.

In case it is required to override the Execution Mode it is available the addRequestForceExecutionMode() method. This method includes a parameter executionMode to force the Execution Mode. When SYNCHRONOUS is used the request is processed in Synchronous Mode and if ASYNCHRONOUS the request is processed in Asynchronous Mode regardless of the EDL Process configuration. Lastly if PROCESS_DEFAULT is used the Process configured mode is used. It is also available a new addRequest() method with the same parameter executionMode. Note that not all the EDL Processes are developed to be executed in both execution modes.

For example in a WebService with direct response:

  w = response.getWriter();
  response.setContentType("application/json;charset=UTF-8");
  response.setHeader("Content-Type", "application/json;charset=UTF-8");
  ...
  if ("POST".equals(method) || "PUT".equals(method)) {
    String processId = "required edl process id";
 
    ProcessRequest<?, ?> processor = ProcessRequest.getProcessRequestInstance();
    processor.addRequest(processId, content, w);

Creating EDL Requests Transactionally

By default, when calling the addRequest method of an instance retrieved with ProcessRequest.getProcessRequestInstance() the current DAL transaction is commited. This means that this method does not allow us to work in transactional way if we invoke it in the middle of a process that expects the commit to happen when it finishes.

So if we want to create EDL requests in a transactional way, there is a different method that we should use to retrieve the ProcessRequest instance:ProcessRequest.getTransactionalProcessRequestInstance().

  
  ...
    ProcessRequest<?, ?> transactionalProcessor = ProcessRequest.getTransactionalProcessRequestInstance();
    processor.addRequest(processId, content, w);

So by creating the EDL request in this way, it implies that the EDL request will be created in the database when the current transaction is eventually commited. And in case the current transaction is rolled back the EDL request will not be created in the database.