ERP/2.60/Platform Team Development Plan
 | This document is still a work in progress. It may contain inaccuracies or errors. |
This page discusses the development plans for the Platform team for the Openbravo 2.60 release. It gives an overview of the main topics and provides a first timing esitmate.
The main theme for the Openbravo 2.60 release is the re-design of the UI to support new UI concepts as outlined in the Openbravo ERP UI Redesign. This in combination with using Data Access Layer functionality.
User Interface Improvements: Select UI technology
To improve the user interface in 2.60 one fundamental choice needs to be made: which UI technology will we be using. This part of the development plan covers the steps needed to select a new UI technology.
Project Steps:
- Specify a long-list of possible UI-technologies
- Define criteria by which to make a short-list of UI-technologies, possible criteria:
- open source project which does not pose any licencing restrictions, active user base and development team, active forums, reactive developers
- regular releases of new versions
- work on firefox and IE
- have a mature and complete component set: field components, form, layout, grid components, excel, image/chart, etc.
- skinnable
- customizable by developers
- customizable by end-users
- personalizable
- internationalizable
- fit in a view-generation approach
- testable by automatic test procedures/test cases
- bandwidth requirements, size of a page, caching
- Integration with server side logic, support for conversations (a flow which takes multiple views before committing all information)
- Select short-list, 2-3 UI-technologies
- Define/design a proof of concept, based on the current design of Rob, part of the new UI design should be selected. The proof-of-concept should be developed on top of Openbravo 2.50 using the Data Access Layer
- Execute proof-of-concept for each of the short-listed UI-technologies
- Present results and make selection
A side goal of the proof of concept is that it can be used as input for the future UI-server architecture and identify the development tasks which need to be done.
Timing:
- Selection of short-list UI technology: takes a week or so
- Define Proof-of-concept content: 1-2 weeks throughput time
- Proof-of-concept execution: around 3 weeks throughput time (if team members do not do other work)
Formalizing the new UI design: Functional Design
The new UI design needs to be formalized in a functional design. The focus of this functional design is on how to model the new UI, it should cover the following topics:
- Analyze the new UI design and identify main components (views, grids)
- Determine changes/additions to Application Dictionary to model the new UI layer
- Identify and describe UI related functionality (save UI preferences, save bookmark, etc.)
The proposal in this document is to not change the current UI (windows/tabs, the WAD generation), but instead develop a completely new UI on top of the data access layer and using the Application Dictionary. The functional design should follow this base philosophy.
Timing: throughput time around 2-3 weeks
Technical Design of new UI-layer development
The technical design combines the results of the proof-of-concept and the functional design into one combined design. This design is made to finalize development decisions on how to incorporate the new UI layer into Openbravo:
- define the architecture (UI-tier, mid-tier) and the components of that archictecture
- define the UI components and the mapping from the (new) Application Dictionary components to UI components
- describes how views can be generated on the basis of the Application Dictionary
Timing: throughput time 2-3 weeks
Development of new UI-Layer
To be determined. This can then be done from mid-end may and further.
Implement business logic using the Data Access Layer in the mid-tier
The next step in the development of the data access layer is to implement business logic using the Data Access Layer concepts. There are two types of current business logic which can be moved to the java layer:
- stored procedures
- triggers
The proposal here is to first move stored procedures to the java layer. As this development plan assumes that the current UI remains untouched a choice can be made to not remove stored procedures but to add their java implementation to Openbravo. The current UI can then still use the current stored procedures. While the new UI makes use of the new java-stored procedures.
Regarding triggers it is slightly more complex. The triggers need to remain in the system to support the current UI. There are two possibilities:
- for 2.60 do not implement triggers in Java
- implement triggers in Java but maintain them for the current UI and make it possible to automatically disable triggers when Hibernate connects to the database.
The second option is feasible and only requires a small change in the trigger code.
From a project planning perspective it can be best to start this effort with one stored procedure (for example create invoice from shipments). The experience with this implementation should then be used to develop generic components relevant for business logic. In addition a focused developers guide should be written. The guide can be used for additional implementations (by people outside of the platform team). So for the Platform team planning the assumption is made that only one or a few stored procedures are re-implemented by this team.
Improve Application Dictionary
During the 2.50 phase a number of areas were identified on which the Application Dictionary should be improved:
- Definition of References: The current reference definition is a combination of domain definitions (like type) and UI characteristics. The different dimensions (data-type, UI) need to identified and separated into their correct entity (column, reference, field).
- Default sql types: we should review what the sql types we use for different column types (nvarchar versus varchar, etc.)
- References: should be extendable and customizable (through modules)
- Definition of Column and Field: also here there are characteristics related to UI which are modeled on Column level, these concepts should move to the Field.
- Button definition: a button requires a column in the database/table, this needs to be solved
The actual changes first have to be designed and then developed. The exact timing can not be defined at this point.
Middle-Tier
The current Openbravo middle-tier consists of servlets which directly call specific java classes. This approach has the following drawbacks:
- it is difficult to extend/customize the inner-parts of Openbravo as customizers need to change the current source which would get overwritten with a next update
- configuration of components, exception handling and logging are done in a way which is less standard for java development
- testing is more difficult because it is difficult to build a test context and it is difficult to test the logic layer (as it is tightly integrated with the ui-servlet-layer)
- it does not support state-full web dialogs and conversations, see below for a short description.
Conversations: stateful web applications
Traditional servlet web application have three scopes: application, session and request. This means that data can either be stored globally (for all users - application), for a user (e.g. the login is stored in the session), or when a page is opened or clicked (request). There is one scope which is missing in this list: conversation or dialog scope. The conversation scope allows the application to keep data over multiple page views of a user. Having conversation scope has the following great technical benefits:
- data is kept isolated in-memory between page views/requests, without this data being visible to other page requests
- the software can do commits at the database at specific points in time. So not every page request (post) will result in a commit
These technical benefits translate in the following functional benefits:
- it is possible to move from one page to another and back and return in your original state: for example, one enters a search value in a grid, gets the search result and clicks on a record to edit it, after editing and returning to the search view, the search view opens with the original result search set.
- it is much easier to create wizard type flows, so a multi-page flow which builds up an in-memory object set before committing it to the database.
- it is easy to implement copy-record functionality
- it is possible to create a complete business object in the ui (a business partner with locations, banks, etc.) by going through multiple views and save the complete structure in one step at the end
Without an explicit conversation scope it is very difficult to implement these type of functionalities.
Customizing business logic
In the 2.60 release we will be moving business logic to the middle-tier. The business logic is implemented in java classes which are instantiated by a servlet or by other classes. In the current Openbravo approach the instantiation (the creation of a component/class) occurs with the new operator:
MyBusinessLogic procedure = new MyBusinessLogic();
In this case the caller/user of MyBusinessLogic is tightly coupled to the implementation of the business logic, nl. the MyBusinessLogic class. Now assume that a customer/partner wants to make a slight change to the business logic. A first idea would be to implement this by creating a subclass of MyBusinessLogic which has this small change. However, then the real issue arises, it is difficult for the developer to actually let Openbravo make use of his custom subclass: CustomBusinessLogic as Openbravo hard-coded instantiates one specific object (MyBusinessLogic) and not the custom version. To overcome this disadvantage there are two approaches:
- create our own object factory which creates the correct object on request (has been implemented in 2.50), so then the Openbravo code has to do something like this:
MyBusinessLogic procedure = OBFactory.getInstance().create(MyBusinessLogic.class);
The OBFactory can then at runtime be configured to return an instance of CustomBusinessLogic, this is valid as this class extends MyBusinessLogic.
- use dependency injection/factory containers such as Seam/Spring. These container frameworks offer the same capabilities, the actual implementation class for a component is defined in configuration files.
In general it makes sense to make use of standard dependency injection/factory containers instead of creating our own factory implementation.
Other reasons for using a standard middle-tier framework
- Our development methods will de-facto industry-standards and will be easier to explain
- It will be easier for other java developers to extend/customize Openbravo.
- Testing is done more according to industry practices, frameworks also provide testing functionality which can be used
- Session/Transaction handling is done more according to industry practices
- Integration with UI technology will in general be easier
Middle-Tier solutions
In general one can say that there are two main solutions for the middle-tier:
Both solutions are stable, used by many companies and have a very vibrant user community.
A summary which touches on the main difference between the two frameworks: Spring is an integration framework for many popular Java frameworks whether they are web frameworks (Struts2, JSF, Spring MVC), persistence (Hibernate, iBatis) and so on.
Seam on the other hand focuses heavily on the JEE 5 technologies (EJB3, JPA, JSF) and provides tight coupling to JBoss technologies (Drools, jBPM, RichFaces).
After a short analysis and discussion with several people, here is a list of pros of each of the two frameworks. The pros of one framework are the cons of the other.
The pros of Seam:
- Simple, promotes a clear development approach towards configurations and handling of beans.
- Contains functionality important for applications in the core product:
- templating of emails
- pdf integration
- work flow (jbpm)
- rules engine (drools)
- Supports conversations in the core product.
- Easier to extend: the core of Seam can be replaced/extended by custom implementations.
- Better transaction handling (fits better to webapps in my view than Spring's).
- Seam team is very active in jsr standard boards.
The pros of Spring:
- Very extensive framework supporting many different configurations and implementation choices.
- Has strong support/attention for aspect oriented programming.
- Integration with dynamic languages like groovy, jruby
Background links related to seam/spring: http://linkedinspin.ning.com/profiles/blogs/1601474:BlogPost:28386 http://www.coderanch.com/t/61238/Application-Frameworks/Seam-vs-Spring http://www.andygibson.net/articles/seam_spring_comparison/html/index.html http://ptrthomas.wordpress.com/2006/05/28/spring-vs-hibernate-and-why-i-dont-care-about-sun-standards/ [TO BE DONE: comparison]
Define the technical database model in the Application Dictionary
The technical database model is now defined in xml files which are used by the DBSourceManager to create and update the database schema. This information (the technical database model) can also be maintained in the Application Dictionary. This has the following advantages:
- makes it possible to create new database fields/tables/triggers/stored procedures directly from the Openbravo application without going through the database
- less change of mismatches between Application Dictionary definitions and the database
- makes the Application Dictionary cleaner from a design perspective: the domain model and technical database model are defined together but still separate (as they serve different needs)
Next to defining the technical database model in the application dictionary also the logic which creates and updates the database model should be integrated into the current Openbravo application. It will probably make sense to re-use the current logic and for example generate the xml used by the DBSourceManager from the technical database definition in the Application Dictionary.
From a project perspective this task needs to be first designed (functional/detailed design), implemented and tested. The design phase will take approx. 1-2 weeks. The implementation phase can be estimated later.
Make Openbravo Reporting easier to change/edit by end-users
The main goal of a new integration with a (new) reporting tool is to make it easier to let end-users create and change reports in Openbravo without needing to contact consultants. Currently we support Jasper Reports, the idea is to research if we can support other open source reporting tools as well. Other options which should be researched:
- integrate with OpenOffice
- research extensions to the UI (Excel-like UI's)
The following steps can be done as part of this task:
- quick scan of current open source reporting tools, select 2-3, check UI-extensions as a solution.
- analyze integration points with these open source reporting tools/ui-extensions. The reporting tool/ui should 'talk' to Openbravo on metadata/application dictionary level so not on database level. This means that we probably need to develop data source drivers which work on the Data Access Layer/Metadata level.
- analyze (for the open source reporting tools/ui) how designed reports can be uploaded to and then run/generated from within Openbravo.
- design common functionality which is reusable across different open source reporting tools
- analyze how much development work integrating with each open source reporting tool requires
Translated Business Objects and the Data Access Layer
In Openbravo 2.50 there is no explicit support for translatable business objects. Translatable business objects are business objects for which the data itself can be translated into a specific language, an example is the Country which has a translated CountryTrl equivalent. There are two situations were it makes sense to show/use translated data:
- in listboxes used in the UI
- in reports
To support these cases the Data Access Layer should offer an api which returns the translated version of a business object or a set of business objects.
The total development effort to accomplish the above is fairly limited and can probably be done as part of the overall UI re-development.
Re-design security model
The Data Access Layer currently makes use of the window_access information to determine to which tables a user has access (read/write and derived read).
In the 2.60 release the access and authorization definition and use must be made much more detailed, granular and flexible. For example:
- It should be possible to define access by table (entity), column and individual object
- The access definition should be role based and can differ for different client and organizations.
- Access can be defined for different privileges: read, write and usage
While designing the new security model attention should be paid to the question if certain access checks are done in specific layers of the system (for example read access in the UI layer).
This development task requires a thorough design phase before implementation can start.
Improve Job system
In Openbravo 2.60 it should be possible to directly define a job from a (process) window. This makes it possible to create a job using parameters defined by the user.
Other aspects of the 2.60 Job system:
- should be possible to flexibly schedule a job
- it should be possible to define output destinations for the output of a job, for example the output should be send as email to a specified email address or saved in a location on a file or ftp server.
- jobs should be run under the user defining the job
- a user defining a job should be able to maintain its parameters and to disable/enable a job
Specific attention should be paid to support of date fields which are part of the job parameter sets, for example sometimes it makes sense to automatically move the date after a job has run.
This task requires a functional design to be done before the development can start.
It should be decided if the required changes are done for the new UI, the current UI or for both.
Improve Test-driven Development Practices
In the 2.60 release Openbravo should further extend its test-driven development practices. More specifically time and resources should be spent on the following tasks:
- follow a test-driven approach in all new developments done by the platform team and other scrum teams
- define a good test dataset which is stable and usable for tests of different parts of the system
- run tests automatically as part of the central build process
- check test coverage by incorporating test coverage tools in the central build process
- create a test data set for performance testing
- define and execute performance tests on a weekly/periodical basis
Optimistic Locking
Database locking is important when two or more users open the same data, then change it and save the data to the database. In Openbravo 2.50, the save action of the last user will overwrite all changes of previous users. To prevent this the concept of optimistic locking should be implemented in Openbravo. Optimistic locking is a standard feature of Hibernate. It requires that every table gets an additional column to keep a version number.
Conversations (described above) will result in longer periods over which users work with in-memory data. This means that in a new UI implementation (supporting conversations) it is even more important to use optimistic locking.
The proposal here is to only support optimistic locking in the new UI and not change the old UI.
Improve Import Client to handle very large datasets
The current Import Client procedure reads all data in-memory before committing it in one transaction to the database. This means that the import client function is usefull for setting up initial clients (as the dataset is comparably small, like the demo data). However, if the copy client function needs to be used for backup and restore functions then the current logic will not work. It needs to be decided if export/import indeed needs to support backup scenarios. If so then the following changes should be made:
- export should create multiple XML files based on a configuration parameter (number of objects per XML file)
- import should be able to import file by file, only keeping in memory the last file, all files are handled in one transaction
- the import should probably use a special Hibernate feature: stateless sessions
Modularity Improvements
[TO BE COMPLETED]
Some ideas:
- Add possibility for install scripts when a module installs
- Add module validation, so a module can validate itself after installing
- Wizard to create a module, with default example modules to use as a start
Points from the product backlog
These are topics of the current Platform team product backlog which are not explicitly included in the above sections. We have to decide what fits where in the development plan for 2.60 and under which sections:
- Develop sizing guidelines
- Writing high performance SQL guide
- Implement in DBSourceManager: Speed up filter with normalized search fields (C_IGNORE_ACCENT)
- Measure performance of 2.50
- Prepare a more exhaustive testing to be run before publishing: automated smoke test, modularity features, other...
- Prepare a document to explain how to add new validations
- Add validations to check: synchronize termininology needs to be run, translation needs to be run
- Create an ant task to export demo data. Define who will be in charge of this maintainance
- Solve -or improve- the memory leak problem. Aim to run Virtual Appliance on 512MB
- Improve update.database log
- Review onCreateDefault execution rules
- 2nd phase to improve build.quick: address translation, improve performance (Antonio, does update.database differentiate structure and data???):
http://spreadsheets.google.com/ccc?key=p3YLNiADn1kUwiybnXC6cZA&inv=ismael.ciordia%40openbravo.com&t=6639542375519552724 Prepare a mechanism to update demo-data dates to a current date (current demo-data is from 2006). It should be consistent (accounting, duplicated data such as dateordered, ...)
- package a client as a module so that you would be able to download and install the demo data (or the sample data or the accounting data or any other demo that the community produces) directly from the Module Management console
- Improve the logging/architecture approach in the application
- Improve combination of Ant and Openbravo to make it easier to handle
- Review the implementation of "model validations" in development (ant) tasks taking into account the following requirements:
- Validation rules are defined and implemented once and reused when needed