QA Processes/Defects Severity Priority
This document is intended to be a guide for Triage Team when prioritizing reported defects.
On a weekly basis, Triage Team verifies new defects (with status New).
Triaging a bug involves:
- Making sure the defect has enough information for the developers and makes sense
- Making sure the defect is filed in the correct place
- Making sure the defect is not duplicated
- Making sure the defect has sensible "Severity" and "Priority" fields
- Decide which developer has to fix the bug
How to determine the severity of defects
Severity is a technical measure. It is an assessment of the impact of the defect without regard to other remaining work or the current schedule. The only reason severity should change is if exists new information that can be used to re-evaluate the previous assessment.
The classification of impact of a defect is important for following reasons:
- It helps to determine the efficiency of Test Process.
- It helps to decide the priority of the defect, hence improves overall development process by fixing higher priority defects first.
- The bug tracking process can be made more effective if the severity of the defect is clearly defined.
What is defect Severity?
A defect is a product anomaly or flaw, which is variance from desired product specification. The classification of defect based on its impact on operation of product is called Severity.
A defect usually refers to as bug only if it affects operation of system and negatively impacts the user of the system, while defect itself may not have any impact on operation of system. In other terms, all bugs are defects but not all defects are bugs. Since severity classification also includes those anomalies, which does not have any impact on operation of system, it is appropriate to mention as Defect Severity rather than Bug Severity.
Following list consist of some questions that may allow you to decide yourself the measure of severity.
- Does the system stop working after defect occurs?
- Does the system recover from the defect by any means?
- If the defect is recoverable, does the system require external effort to recover from the defect? (i.e. it will not recover on its own)
- Did I check whether the same defect is reflected in all other related sections (or entire system)?
- Can I be able to repeat the defect in some other system having same configuration (O/S, Browsers) as that of the system where I found the defect?
- Can I be able to repeat the defect in other configurations also?
- Does the defect affect all users? (i.e. Only a particular category of users will face the defect)
- Does the defect occurs frequently?
- Are the inputs to make the defect easy to get? (i.e. not special data has to be created)
The number of Yes on the previous questions should give you a good idea about the severity.
For example, if the system stops working, then further testing of the system is not possible and hence severity is high. Also the defect should be generalized as far as possible. I.e. after you find the defect, try to find out that the defect is repeated in all cross-browsers, cross-O/S etc.
Some tips on finding the severity
Decide the impact
Some defects are obvious to decide its severity. For example, "404 HTTP error occurs when navigating to particular screen". Sometimes, a minor defect repeats in all sections or such defect is much more frequent. In those cases, the impact of the defect is higher from the point of view of an user even though it is minor defect. Hence, such defects get higher severity.
Isolate the defect
Isolating the defect helps to find out its depth of impact.
- Analyse the defect with what class of inputs does the defect supports.
- Make sure that the defect occurs only with particular sequence of operation or list out other sequences, which cause the defect.
Defect Classification by severity
The impact of Defect Severity can be classified into four categories: Critical, Major, Minor and Trivial.
Critical defects are those which results in the failure of the complete software system, of a critical subsystem so that no work or testing can be carried out after the occurrence of the defect. It also applies to data loss failures and with processes that leave inconsistent data stored on the database.
Major defects are those which also causes failure of entire or part of system, but there are some processing alternatives which allows further operation of the system. It also applies to the system crashing, or aborting, during normal operation of a non-critical flow.
Minor defects do not result in failure but causes the system to show incorrect, incomplete, or inconsistent results (note that show inconsistent results is way different of generating inconsistent results at database level as explained above). A critical usability issue fits also in this category, as well as if there is a simple workaround.
Trivial defects are small errors that do not prevent or hinder functionality, typos, grammar mistakes, wrong terminology, general usability issues and styling.
Following are examples of type of Defects, which falls under each category.
- It is not possible to use the system on some of the most common flows.
- Login does not work
- Main page does not load
- A null pointer exception on a critical subsystem
- Functionality missed out / Incorrect implementation (major deviation from Requirements).
- Performance Issues (If specified by a Customer).
- Browser incompatibility and Operating systems incompatibility issues (depending on the impact of the error and the statistical usage data).
- Functionality incorrectly implemented (minor deviation from Requirements).
- Performance Issues (if not specified by a Customer).
- Mandatory validations for mandatory fields.
- Images, graphics, text missing/overlapping which hinders functionality.
- Front End / Main Page alignment issues.
- Screen layout issues
- Keyboard shortcuts not working
- Documentation errors
- Page titles missing.
- Pages style different than Front End / Main Page style.
- Default Value missing for the fields required.
- Cursor Set Focus and Tab flow on the Page.
- Missing images or text which does not hinders functionality.
- Spelling Mistakes / Grammatical Mistakes.
- Alt Text for Images.
- GUI image colours etc.
Assigning a priority
What is Defect Priority?
Defect Priority signifies how important and urgently it is to fix this defect. In other words Priority means how high it is in the backlog of tasks. The Defect priority status is set by Management/Triage Team based on the existing functionality, customer requirements and based on Product Roadmap. It also has to be aligned with the Company Objectives. Priority is related to scheduling to resolve the problem. It is a pointer towards the importance of the bug and if High priority is mentioned then the developer has to fix it at the earliest. It also influenced by the technical aspect of the product, reflecting how bad the bug is for the system and also largely related to Business or Marketing aspect.
Defect Classification by priority
Defect Priority is classified into the following categories:
- Immediate: The defect needs to be fixed right now, everything else can wait, the build cannot be released with this defect.
- Urgent: Needs to be fixed before any other high, medium or low defect should be fixed
- High: Should be fixed as early as possible
- Normal: May be fixed after the release / in the next release. This is the priority set as default.
- Low: Fixing can be deferred until all other priority defects are fixed
- None: This priority should not be used. It could be temporarly set in case a defect requires further discussion after triage, particularly in case is was not yet decided to accept the issue as valid.
Nevertheless, defect priority is a subjective decision so it could vary from the above categories and, to determine the proper priority, following few points needs to considered:
- Urgency as Business point of view to fix the defect
- Impact of the defect on functionality
- Visibility of the defect
- Check if resources are available or not to fix this means developer to fix this and Testers to verify the fixes.
- Main constraint in Availability of Time to fix the defect
Definition of priority
The level of business importance assigned to an item, e.g. defect.
Especially when there is a large of number of defects then management of the defect is taken care based on the Defect Priority of the Defect which helps to minimize the product instability.
Setting the priority on a defect
While it is very important for a reporter to set the right Severity, the Priority should be kept as Normal, by letting Triage to set the proper one.
It is possible for a reporter to set a Priority, though. It can be done to help the Development Team on taking the most relevant issues to be fixed first. For example, if the reported defect is a regression on a Stable release currently experienced by several Customer, the fix must be ready before a regular issue.
It is important to note that Priority may change over the time. Triage, on top of setting the priority when receiving a new issue, will periodically review the backlog of open issues to ensure the order of fixing matches current Product Roadmap.
Developers should use Priority to manage their work precedence. This tasks queue is strict. If an Immediate defect is still open, no work on Urgent (nor lower) defects should be started. In case a lower priority defect fix is required, its priority should be raised to meet this criteria.
In case of two or more defects with same priority, the higher severity has to be addressed first.