Projects:Retail Automation Quarantine

Contents 1 Introduction 1.1 New tests can break existing tests 2 Tasks 2.1 Flow chart 2.2 Fulfilled tasks 2.3 Sorted tasks todo 2.3.1 Stage 1: 2.3.2 Stage 2: 2.4 Required from the RM team 2.5 Required from the Retail team 3 Caveats 4 Issue 5 Anex

Introduction

The goal of this project is to improve the quality of the results given by the Retail Continuous Integration and Retail Try

To achieve this goal, one of the proposed actions, is to create a Quarantine group of jobs in which new tests are verified to be stable before they are added to the final automation executions

Reasons to have a Quarantine Automation Integration:

• New tests can break existing tests
• New code can create instability in the application. This point is not covered by the Quarantine, although the Quarantine will help analyzing them

There are still some other reasons why the Integration could fail, being the main one, that the changesets added to the final repositories, do not pass the integration with the final repository status. This flaw in our flow, allows different changesets that individually have passed try, to break the application when pushed together.

New tests can break existing tests

There are some reasons why new tests can make existing tests to fail

• Test chaining: to improve the performance of the automation and come closer to a real production environment, the tests that are executed in the same terminal, do reuse the same session and database. There are some problems associated with this:
  • If a test does not leave the terminal with the same UI elements, the next test could fail
  • If a test changes preferences or other sensitive data in the database, other tests, that expect that data to be pristine, will fail
  • As new receipts and other regular application operations are created, later tests could expect pristine data and fail
• Test itself is faulty: Creating new tests requires some understanding of the Test Logic. Reviews and wiki should help this topic
• Application code becomes unstable: This topic is out of the Quarantine automation objectives and must be addressed by the Continuous Integration

Tasks

Flow chart

Fulfilled tasks

• Gather statistics about the results of the tests. Passed, failed

Sorted tasks todo

Stage 1:

• Create a clone of the group of the selenium jobs in Retail Try. ret-tests and ret-modules
• Create 3 files in a common storage. Current approach: file 'tests_in_quarantine' will be the lists of tests that are in quarantine. File 'tests_passed_quarantine' will hold the list of tests that already passed the quarantine. 'tests_pending_quarantine' will hold the tests that failed in the Quarantine. CI and Try will not execute tests listed in 'tests_in_quarantine' or in 'tests_pending_quarantine'.
• Create a script that detects when a new test has been added to the pi-mobile repository. Pick the email of the pusher. Add the test and the pusher to the 'tests_in_quarantine' list
• Create logic to, if a test fails, send an email to the creator of a failing test. Also the test has to be moved to the 'tests_pending_quarantine' file and will no longer be executed until another push to that file is detected
• Gather statistics, as the tests are being executed
• Promote tests after 10 green executions: remove them from the 'tests_in_quarantine' file and added them to the 'tests_passed_quarantine' file
• Send an email to the creator when the test has been promoted

Stage 2:

• Consider the possibility to keep the files in the pi-mobile repository

Required from the RM team

• 10 jobs, 4 pgsql suites + 4 oracle suites + 1 pgsql module + 1 oracle module. Clones of the Retail Try jobs
• an isolated server not affecting CI or Try
• a way to send emails from the pi-mobile code

Required from the Retail team

• create the logic to auto-quarantine new tests
• create the logic to auto-promote quarantined tests
• create the logic to notify the owner of a failing/promoted test

Caveats

• there is a situation in which we cannot verify the stability of the new test: when a test fails. When a test fails, it can cause later tests to fail because the developer didn't create the proper logic in case the test fails. There are at leasst 3 ways to prevent this:
  • properly code review the test
  • make a test fail, execute all the remaining tests and verify that they are not affected
  • create test logic in which the developers have to initialize sensitive data + code review

Issue

https://issues.openbravo.com/view.php?id=0033365

Anex

• Developers guide for Retail Automation