ERP 2.50:Automated Testing

An essential step in the development of any kind of software is the (almost) final set of tests done on the system with all its components properly integrated and interacting with each other. In projects that are model-driven this is particularly important, because here we verify that the system we are building gives to the users the real value they expect.

That's why we have build an automated testing framework that allows us to exercise the application as a whole through the user interface generated from the model.

Quick start

If you are looking for a fast way to start automating your Openbravo developments, please read the Quick start guide.

Documentation

Before automating a test we always document its steps and expected results in Testlink. There you will find also some tests that are not a priority in the automation efforts, so they are executed manually with varying frequencies.

You have to register in order to view the specifications of the tests.

Integration testing framework

Automated tests at Openbravo should be reusable and stable, but not too complex.

With this in mind we have chosen a design that's not so fast to implement as the recorded tests; and that is not so slow as a multilayered mirror of the application. On the other hand, this design is more flexible and harder to break than recording scripts. However, it's not so reusable as the multilayered approach.

It's a compromise between stability and time.

Design

When possible, the framework should try to load dynamically all the information required to execute actions on the application from the same web page the tests are exercising, without requiring to write any code for the interaction.

For the cases when this is not possible, or when we want to expose our own APIs for a better expressiveness, the framework classes should be generated automatically from the application model stored in the database.

And for manual application code that is not fully expressed in the database model, the framework should allow to extend the generated functionality with manual test code.

The definition of test steps and parameters is left entirely to the test writer, but this should be almost the only manual task. And it should be as expressive and easy as possible, with good semantics, few things to memorize and a short learning curve.

As the picture shows, we have identified four different parts that will be present in every test.

Screen definitions: for every screen and every element that composes those screens, the framework API will publish all the possible actions that a user could execute on those elements and screens. For example, a user can click a button and enter text on a textfield; then, those elements will have the corresponding functions that let us simulate the possible user interactions. There are some other kinds of actions possible with screen elements, that are those that give clues to the user of what's happening. For example, checking the text of a message or verifying that an element is visible or not.

Test controllers: with the screen definitions ready, a test controller can be used to execute a sequence of simple actions on screen elements in order to accomplish a task. For example, in order to add a record you would have to navigate to the proper screen, click the "New" button, fill a form, and click the "Save" button.

Parameters: parameters are the data that will be passed to the test controllers. In the example previously presented, the parameter is the information that will be used to fill the form. Then, it would be possible to create a record with different values passing different data as a parameter to the same function.

Test cases: test cases are defined to test a flow of the application that gives value to an user. For example, a user might like to enter a sales order and complete it, in order to generate an invoice and ship the goods. So, this test cases contain the steps that define a test, and those steps will be calls to test controller functions passing to them some parameters that make sense to accomplish the specific test.

In the end, what we will present for the tests writers is a layer in front of the application that allows him to simulate any possible action that a user can perform, and verify that the result of that action is the expected.

Please refer to the Implementation section to see the implementation details of these design ideas.

Tools

The main tool we use to automate ERP integration tests is Selenium, a free software testing framework for web applications.

Ant is used for the build process and command line tests execution. For example, we have an ant task that runs daily to perform a smoke test on pre-integration code. And we can also use it to define the series of preconditions required by a test. Thus we could test every functionality by separate.

We use JUnit as a testing framework to define test cases, test suites, parameters; and to assert that the expected behaviour matches the one shown by the application. With JUnit we can define abstract test cases adding the Test annotation to functions. And we can define the parameters in a separate class using the Parameters annotation. The main benefit of this is that we can reuse the test cases with different parameters for different contexts. And with JUnit assertions we can code in a readable way what is the expected result of the test.

In addition to that, JUnit collects the results of the tests and all system variables present during the execution. It presents the results in a nice formatted way that allows us to inspect them after execution and understand the failed assertions.

Sometimes we also use DbUnit, a JUnit extension for database testing, to insert the dependencies of tests into the database.

We prefer the Eclipse IDE to code our tests because it allows to run and debug junit tests in an easy way. So while doing a new test we can check that selenium is running what we expect and that JUnit is doing all the required assertions.

Finally there are other optional tools that we use to ease the automation.

There is a Quick Start Guide that explains how to start using all these tools to automate and execute tests on Openbravo ERP.