View source | Discuss this page | Page history | Printable version   

Module:External Data Integration/Developers Guide/How To Implement A EDL Process

Contents

Introduction

In this document is described the required steps to implement a new EDL Process. A EDL Process is a custom development used to extract or load data to other systems.

EDL Process definition

The EDL Process is defined in the EDL Process window as System Administrator. These are some common fields that need to be defined:

Module
The EDL Process must belong to an extension module. Make sure you have created one and it is in development.
Name
The name of the process is used as identifier in the EDL Configuration window to select the processes.
Search Key
The search key or value is used to retrieve the java class implementing the process.
Type of Data
Based on how is received or processed the data. Determines the type of each processed item and the type of a batch of items. The module includes a basic JSON implmementation. It is possible to develop new types to support different types like xmls, csvs,...
Has Output
If the process generates data that has to be processed by a Output processor this flag needs to be checked. When this flag is checked it is required to add at least one Output Process in the EDL Configuration window.
Has Input
In case the process starts by receiving some data that needs to be processed.
Storage Location
Determines how is stored the received raw data. Note that if No Stored is selected it might not be possible to reprocess a request in case of error.
Synchronous
Determines how the process is executed. Depending on this flag additional configurations are needed
Enable Output Retry
Enables to ability to configure the retry options on the EDL Configuration window.
Bulbgraph.png   The Output Retry has to be enabled with caution to avoid performance issues in other processes that are run in parallel. Main reasons for these performance issues:
  • The database connection is not closed until all attempts are done. Make sure that the EDL Process is not creating any lock that could affect other processes.
  • The thread is not released until all the attempts are done. This is specially troublesome in asynchronous processes as they are executed on a thread of the Import Entry Processor, so any process using Import Entries might be affected.
Reprocess Policy
Determines if, and how, EDL requests from this process will be reprocessed. Available options are:

Synchronous process

Synchronous processes are executed in the same thread where the request is added. It is useful when it is needed to have the response of the process immediately.

Additional fields to define on synchronous processes:

Has Direct Response
If this flag is checked it is required to call the addRequest() method with a writer where the response is writer. This is used when the request is launched from a web service. So it is possible to write the response in the same response writer of the original WebService call.

Asynchronous process

Asynchronous processes are scheduled and processed in background using Import Entries. It is useful to process large amounts of data that could take several minutes / hours to process. In this type the process the data is split in request lines of item batches.

Additional fields to define on synchronous processes:

Record size
Determines the number of items included on each batch or request line.
Commit items
Determines the number of items to process before doing a commit. This number should be equal or lower than the record size.

EDL Process implementation

The process is developed by implementing the org.openbravo.externaldata.integration.process.ItemProcessor class.

The link to the process is done by a Qualifier:

  @ComponentProvider.Qualifier("MyProcessSearhKey")
  public class MyProcessItemProcessor extends ItemProcessor {


The main logic of the process is implemented in the processItem method. The T type is defined by the Type of Data. All the items of a request are processed using the same instance of ItemProcessor so it is possible to use class fields to store information related to the request. For example it is possible to define a StringBuilder that it is updated by each item to generate the output string of the response.

protected abstract void processItem(T item) throws OBException;


Asynchronous executions are automatically commit based on the commit items configuration. This means that in case a item fails and a rollback is done several successfully processed items might also be rolledback. In case it is required to process them again the reprocessNotCommitedItemsOnException has to be implemented.

  /**
   * Asynchronous execution process a number of items configured in the EDL Process before
   * committing them. In case of exception a rollback is done. This method determines if the
   * processes that have not been committed need to be processed again or not.
   * 
   * @return TRUE if the items have to be reprocessed.
   */
  protected abstract boolean reprocessNotCommitedItemsOnException();


If direct response is required it has to be written in the directResponse class field on each processItem() method execution.

protected StringBuilder directResponse = new StringBuilder();

If the process has output the writeResponse, writeErrorResponse and the outputExtraActions methods may be overridden.

  /**
   * Method called from the {@link OutputDataProcessor} when there is something more to do with the
   * output, for example reading the response of a webservice request.
   * 
   * @param outputDataProcessor
   *          The data processor that is being processed.
   * @param output
   *          The configuration that is being processed.
   */
  public void outputExtraActions(OutputDataProcessor outputDataProcessor, OBEDLConfigOutput output) {
    // Not implemented by default
  }
  
  /**
   * Method called from the {@link OutputDataProcessor} to write the response.
   * 
   * @param writer
   *          the writer configured and opened. It must remain opened when this method has finished.
   * @param configOutput
   *          configuration of the output processing. Null is the
   */
  public void writeResponse(Writer writer, OBEDLConfigOutput configOutput) throws IOException {
    // Not implemented by default
  };
 
  /**
   * Writes the process output information, if desired, when the processing of an EDL request fails.
   * This information will be saved into the database as the {@link OBEDLOutputContent} linked to
   * the failed EDL request (synchronous process) or failed EDL request line (asynchronous process).
   * If nothing is written into the writer, then nothing will be saved into the database.
   *
   * @see #saveErrorOutput(OBEDLRequest, OBEDLRequestLine)
   *
   * @param writer
   *          the writer configured and opened. It must remain opened when this method has finished.
   */
  protected void writeErrorResponse(Writer writer) throws IOException {
    // Not implemented by default
  }

EDL Process execution

When a new process is defined it is also required to develop the needed processes to launch the requests. To create and process a new EDL Request it is required to use the ProcessRequest class. There are different addRequest() methods depending on the type of process to be executed.

The addRequest can be called from different type of processes: EventHandlers, processes (menu or window), exposed web services or background processes.

In case it is required to override the Execution Mode it is available the addRequestForceExecutionMode() method. This method includes a parameter executionMode to force the Execution Mode. When SYNCHRONOUS is used the request is processed in Synchronous Mode and if ASYNCHRONOUS the request is processed in Asynchronous Mode regardless of the EDL Process configuration. Lastly if PROCESS_DEFAULT is used the Process configured mode is used. It is also available a new addRequest() method with the same parameter executionMode. Note that not all the EDL Processes are developed to be executed in both execution modes.

Bulbgraph.png   Ability to force Asynchronous or Synchronous execution modes available since version 2.0.200

For example in a WebService with direct response:

  w = response.getWriter();
  response.setContentType("application/json;charset=UTF-8");
  response.setHeader("Content-Type", "application/json;charset=UTF-8");
  ...
  if ("POST".equals(method) || "PUT".equals(method)) {
    String processId = "required edl process id";
 
    ProcessRequest<?, ?> processor = ProcessRequest.getProcessRequestInstance();
    processor.addRequest(processId, content, w);

Creating EDL Requests Transactionally

Bulbgraph.png   Ability of creating EDL requests transactionally for asynchronous processes is available since version 2.0.1700. For synchronous processes it is completely supported since version 2.0.1800.

By default, when calling the addRequest method of an instance retrieved with ProcessRequest.getProcessRequestInstance() the current DAL transaction is commited. This means that this method does not allow us to work in transactional way if we invoke it in the middle of a process that expects the commit to happen when it finishes.

So if we want to create EDL requests in a transactional way, there is a different method that we should use to retrieve the ProcessRequest instance:ProcessRequest.getTransactionalProcessRequestInstance().

  
  ...
    ProcessRequest<?, ?> transactionalProcessor = ProcessRequest.getTransactionalProcessRequestInstance();
    processor.addRequest(processId, content, w);

So by creating the EDL request in this way, it implies that the EDL request will be created in the database when the current transaction is eventually commited. And in case the current transaction is rolled back the EDL request will not be created in the database.

Retrieved from "http://wiki.openbravo.com/wiki/Module:External_Data_Integration/Developers_Guide/How_To_Implement_A_EDL_Process"

This page has been accessed 4,364 times. This page was last modified on 30 June 2022, at 16:28. Content is available under Creative Commons Attribution-ShareAlike 2.5 Spain License.