Issue tracker QA status
Contents |
Introduction
Currently (December 2009) Openbravo's Issue Tracker does not have a way of indicating if an issue has been tested or not. As a workaround the development team accorded that every issue in the resolved status is not tested yet, while those that are closed are. This is far from being ideal. So this is a proposal to integrate our QA work with the issue tracker in a more helpful way.
Current workflow
The current workflow has the following statuses:
- New: the initial status once a issue has been reported or a closed one reopened.
- Feedback: the assignee/triage team asks for more information to the reporter.
- Acknowledged: the assignee/triage team confirms that the issue exists.
- Scheduled: the triage team schedules this issue to be fixed in a specific release by a specific developer.
- Resolved: the assignee fixes the issue.
- Closed: the QA team has tested the resolved the issue. Or the assignee/triage team has rejected the issue.
And this is the workflow involving all the different development teams (Triage, Engineering, QA team):
This model has flaws in regards to the QA process:
- There is no a QA status, just a convention that the development team follows.
- We consider that a issue is resolved before any testing has happened. This is an important problem, it conditions the way the development team thinks about testing.
- Although not related exactly to the statuses, we currently don't have a way to assign a QA tester.
Proposal
One option that has been around for a long time is to introduce a new status called tested, between resolved" and closed. This has a fundamental flaw, though: why should the tested status be placed after resolved and not before? You cannot claim to fix an issue if it's not tested. It might be a terminology difference only, but it makes a big difference on the way developers understand the importance of testing.
So this is a proposal to have a new status between scheduled" and resolved, called Ready for testing. This new status represents that a fix has been committed to the Mercurial repository, but that a formal testing has not happened yet as to call it resolved.
But now we have the Continuous Integration factor as well. So need to differentiate between "CI testing" and "Manual testing/verification/review".
So this is the possible workflow that reflects these ideas:
- New: the initial status once a issue has been reported or a closed one reopened.
- Feedback: the assignee/triage team asks for more information to the reporter.
- Acknowledged: the assignee/triage team confirms that the issue exists.
- Scheduled: the triage team schedules this issue to be fixed in a specific release by a specific developer.
- Ready for testing: a Mercurial commit moves an issue from Scheduled to Ready for testing.
- Ready for review: the CI moves an issue from Ready for testing to Ready for review.
- Resolved: the reviewers (right now QA) move the issues from Ready for review to "Resolved".
- Closed: And when a release is out Release Management move the Resolved issues to Closed.
So this is the new development workflow:
Following this flow, the merge from erp/devel/pi to erp/devel/main happens once the issue has gone to the Ready for review status, without waiting for it to go to resolved. The reason is simple: as of now the issue verification consumes a lot of time and this would incredibly slow down the integration.
Open issues
 | The discussion period for these issue is closed. These are the final status names:
New -> Feedback -> Acknowledged -> Scheduled -> Ready for integration -> Ready for QA -> Resolved -> Closed |
There are discussions on what the correct names of the new statuses should be. Proposals:
- First status, after Scheduled:
- Ready for testing.
- Fixed.
- Resolved (DME comment: I propose to leave current status because people got used to it so, in general it is just a semantics which is explained at our Statuses agenda in Mantis).
- Ready for Integration (added by PJU: this seems to be a natural description of what this status is about - a developer has committed the fix in pi and the code is ready to be integrated into main provided that it passes the automated tests)
- Second status, after the previous one:
- Ready for review.
- Auto tested (DME comment: I vote for this one, it shows the process behind it).
- CI tested.
- Verified.
- Ready for QA (added by PJU: I propose this one because it shows the process behind it).
- Third status, after the previous one:
- Verified (DME comment: I vote for this one) (PJU comment: I also vote for this).
References
- Discussion in the openbravo-development mailing list.
- Current workflow in the bug reporting guidelines.