The 3 Essentials of Replicating Customer Found Defects

Quality Teams test their Business Applications with all their expertise, they use their hard earned business knowledge of their application, the technical expertise that takes to understand it and the ability to translate their functional and technical expertise to view the product from a customer's viewpoint. WOW!! Testing is challenging and fun in its own way.  Once the product is tested to the satisfaction of the Quality Team, the product is shipped or put on production portals for the customers to uptake it. While the Quality team feels a bit relaxed about completing testing and moving onto the next release, the customer usage and their overall experience about the application is the true gauge and indicator of the Quality of the Product.

Customer adoption usually gives lot of feedback about the product, things that work well, things that do not work well and things that could make the product better. Things that work well and things that could make the product better, if many, bring lot of accolades and shape the future direction of the product. Things that do not work well generally translate to Customer Found Defects (CFDs).

Product teams, when learn about Customer Found Defects, first try to replicate them in-house in their System Test or a Production replica environment. Some of the defects are pretty straight forward, they get replicated in the QE env pretty fast. Some CFDs take long time to replicate because they happen only in some combinations. These CFDs are hard to replicate, hard to triage and may be hard to fix. They sometimes cause frustration for the customer experiencing them, support teams for trying to really understand what the issue is and product teams (Engineering, QE) to address it. What should QE teams do to address these kind of issues? What are the best known industry wide practices used to solve these kind of issues.

In my experience, the biggest effort is to try to replicate the CFDs in house. Once the issue is replicated, triaging and fixing it becomes relatively easy. What techniques can be adopted? Let us understand with the help of this below picture.

 The first technique that helps is, to find out if there is any customization involved at the customer's end. This is especially true for ERP and CRM applications. Enterprise applications rarely get implemented without customization. When someone from the customer's end talks to the support team about the problem, they sometimes forget the customization involved. It is always a good practice to first understand the customization involved. This helps the product teams to understand the diff between their QE instance and the customer instance. Understanding the customization also sometimes helps the product teams to categorize the problem as product code related OR custom code related. Custom code is usually implemented by consultants working at the customer's end. 

The second technique is to clearly understand the exact business use case. A clear click path arriving towards the problem almost always helps. The exact business flow sometimes get lost between the noise of escalations, ,multi-page long debug log files , war room discussions, scrum meeting updates and what not.  Having a clear understanding of the clickpath ( for UI based applications) or a recording of the issue almost always nails it. 

The third technique is to understand what data replicates the issue. This is equally hard to triage sometimes. Customer-like data is sometimes hard to create and maintain in internal QE instances. This is where the experience of the QE professional, their business understanding of the product and their triaging of the customer defects helps. Some defects get replicated with heavy volume of data, some get replicated with some rare combinations, some get replicated with data which is specific to certain businesses. 

These 3 techniques usually help with the replication of  CFDs. Good QE teams create similar data with the exact business flow used by the customer to avoid repetition of the CFDs in future releases. Understanding impact of customization can be done by closely working with Dev and support teams. Any additional setup that needs to be done in QE env can be done after this.

Once these steps are taken, adding test cases related to CFDs in the test repository should be done with topmost priority. This makes the product go out with better quality and increases the credibility of the product teams. Happy Testing !!!