The bug we presented in our previous post got a lot of very interesting comments. So, big thanks to all of you who did share your point of view about it

I am not a QA expert; my experience as a consultant is more oriented towards code quality and development lifecycle’s best practices.

So I was curious about more authorized opinions before to start formulating some hypothesis in this second post.

First, let’s remember the bug we have been talking about.

An online merchant web-site offers for sale every day different products of different brands, each one like a virtual shop that you can navigate in order to select articles in your (virtual) cart. Once you validated it and paid for your purchases, you can go back into another shop and proceed to new acquisitions from another brand. In that case, the site will propose you to group them with your previous purchases in a single delivery, and thus avoid to add costs of transport. This is the default process.

In my case, I bought two different items from the same brand, but when I wanted to pay, a message told me that I needed two different deliveries, so I had to do a first payment for the first product and a second one for the other product. Which I did … and was proposed to group the buyings into a unique delivery.

I must confess that I could ‘smell’ the bug and my curiosity was the strongest, so I accepted, to find myself with:

• A generic error page explaining very imprecisely that something had gone wrong, absolutely unhelpful.
• My web session completely unstable, and the inability to reinitiate my cart and to get back to a normal process.

Getting out of the site was completely unsuccessful, I had to close my browser and to log again into the site in order to proceed to my purchases. I hope that the articles I did order will be at the height of the difficulties I did meet to acquire them.

If I try to think from a developper’s perspective: the usual ‘default’ delivery process does not apply and the correct ‘exception’ process has been correctly implemented. But the integration of this new feature with the ‘default’ process has not been done correctly, and that meant a bug that I would call critical, that even a user can find easily just by following the normal process proposed by the web site.

I tried to imagine what could cause such a bug, and this is when I decided to write the previous post and to ask her opinion to the community of QA specialists. Cherchez l’erreur.
Your reactions (well, a synthesis):

• This is not poor testing, this is no testing.
• This is not poor testing but poor design.
• A good tester must be able to go to the other side and think as an user to do his testing. I must say that this opinion was formulated only by spanish testers.
• This is a classical situation where time-to-market pressure, lack of resources, poor planning, etc. leads to a lack of testing.

The primary and main objective of testing is to ensure that a product – a software product in our case – works correctly according to the requirements. Testing is a profession, a specialty needing lot of skills and experience. In our example, we can consider that the result is not what was expected, and that this new feature or its integration has not been tested or was incorrectly tested.

The fact that a unique generic very imprecise page answers any problem let me think that the Dev / QA budget is not in phase with the growth of the web site. I see there some laziness or some lack of resources. An admission of impotence. Error handling is not really a priority.

What? I cannot do a buying and this is not a priority? They don’t want my money?

I wanted to know more and I called a friend of mine, a freelance consultant with a good knowledge of these merchant web sites from the IT point of view, and the difficulties they can meet. Here is what he said me:

• This site get more than 100 000 connections a day, with a very good response time and never knew any major failure or interruption.
• Sales last year were from approximatively from 1 500 different brands.
• Development and QA are done in-house, with a 60 people team. No outsourcing.
• This department realizes its own developments but also uses ERP. Project methodologies: Waterfall and Scrum.
• New subsidiaries have been opened regularly around the world, and a rumor says that there could be a project to go into the Cloud and propose an SaaS infrastructure, so that anybody could use it to have his own virtual shop with his own brand.

A story that told me this friend: anyone buying on this site can ask to be refunded, once the products are delivered, without any justification. No need for a defectuous product or a wrong size. You don’t want it anymore: just return it and ask for a refund. Now, it happened that the refunds were suspended during several weeks, because of a bug. The management simply decided to stop refunding users while he developed again the application (or implemented an ERP with the same feature). Can you imagine how users / buyers were upset and the bad image for this site?

Not doing of its users a priority is not the choice of single developer or of a lazy tester: this is a choice from the management. And I know only one priority above users, for the management: budgets.

Which is not the problem in that case. This web site has been the first on his market of private sales, both a pioneer and a leader, with an impressive growth. I am talking of a startup (well, with seven years of existence) conquering the world, not of a century-old company on a declining market.

So, I believe that what we have here is an IT which main priorities are to address the business growth, to be ready anytime a new office opens in a new country, to manage logistics for 1 500 different providers and thousands and thousands of customers, to avoid any interruption or poor performance, … And that in order to complete these objectives, they have added every year new systems, new ERPs, new features into the existing applications, customizations for each new reglementation of new countries, etc.

There comes a time when the accumulation of technical debt is so high that:

• A whole module – like the one of refunding – cannot work anymore. Just stop to pay for several weeks, without even trying to find a workaroud because it will cost more than implementing a new application or an ERP.
• The number of bugs is so high that nobody ever try to manage them and a unique generic error page is implemented – and let the user manage its own errors with that.
• And because the user is no more a priority, and because any bug is ‘managed’, why bother to do appropriate testing?

So, as a Quality consultant, my recommendations to this site would be to:

• Do a systematic analysis of the quality of the code, for the whole portfolio of applications.
• Verify which systems are heavy to maintain, occasioning delays and poor time-to-market, with a lot of bugs found in QA.
• Cross these project’s data with the code quality data, in order to decide which parts ofr the portfolio need a refactoring.

I would also verify which methodology is used on which project. For example, a Waterfall project lifecycle could mean that testing is done too late, when the number of bugs accumulated is already too important, and the effort to correct them is too high.
And if the rumor is true that this web site wants to go into the Cloud, refactoring its porfolio of applications would be mandatory.

I already met this kind of situations with some Telco companies. They had to face this same tremendous growth and their ITs have gone through the same path, and are now stuck with systems of very poor quality, lot of bugs, … except that now this market in not any more growing and IT budgets are shrinking and no refactoring is possible.

And this is when the management decides to put back the user at the main place, because when there is no more growth and no more users to win, you begin to cherish the ones you have and not losing them becomes a priority.

Cherchez l’erreur… before it is too late.