Vicente Merino asked me in the last post about Complexity and QA effort : « How to estimate the effort when you do not have code? » And more specifically « Is it possible to decide at the beginning of the project, if it will be a project important enough to require an independent QA team and formalize a test plan? ».

For example, imagine that you are responsible of the applications in aTelco company. It is therefore your responsibility that:

• customers can log on to the website to view their bill, the number of points, acquire new services, a new cellular, etc.
• employees can connect to the same site but also to other applications in order to verify a customer account, a potential default of payment, etc.
• commercial applications can sell and financial applications can charge.

One day, the sales director and the chief financial officer say you ‘We will launch a new commercial offer: applications must be ready within two months.’

Obviously, you want to clarify the characteristics of this new offer in order to prepare a project plan and mobilize your teams. Mostly, the answers are vague: you understand that the changes required are significant and affect many applications. It is not simply the creation of a new offer but the modification of some sales mechanisms, promotion, delivery, payment, etc.. and data structures for these treatments.

In this example, it is not possible to expect that the specifications are sufficiently formalized to start the project, it must start as soon as possible and you will work with users in iterative cycles to create specifications and develop gradually as they become clearer. An Agile methodology will be most effective in this kind of situation.

We have also seen in this post ‘Use cases – Working seamlessly together’ how Continuous Integration allows developers to test the code as soon as it is developed. With the estimation of the complexity of the different developments for the different applications, it will be possible to decide a phase of Quality Gate, at the earliest in the project, allowing for a team to achieve a QA campaign on the most critical modules.

In this example, the project has already started, which does not exactly answer the question asked by Vicente, but I just wanted to show that it is possible to determine as soon as possible if the services of such a team will be required.

Consider now the case where the evaluation of this effort, and indeed of the whole project, must be done before it starts. This time, the sales director is talking about a completely new product and the CFO asks you what it would cost to develop new applications for completely different customers, and whether it would be possible to use existing financial applications or if you will need to create new ones.

How to evaluate the development effort and QA, calculate man/days and a draft schedule? In such a case, we will proceed by analogy.

Even if it is a new project for which we do not have specifications, we have already made applications for such business cases, and we can try to assess the size and complexity of the project based on points for which we have data reference. Gustavo Terrera, QA professional with a very good blog devoted to these topics: testingbaires.com, provided me a list with the following elements:

• Number of applications, number of new modules or modules to change, number of features per module: we have sufficient business knowledge to design the chain of the different applications needed for this new offer and identify the functional modules needed.
• Criticality of each module, criticality of each feature within each module, to develop / to test, user profiles. Once the map of these modules is established, we can try to estimate the complexity of each of them, again reasoning by analogy with similar applications that we know well.
• Number of databases, number of batch treatments, number of interfaces, number of browsers for which perform different tests, technologies implemented, development languages, performance testing, security testing, … We are now interested in the technical dimension, in order to evaluate the testing effort. We can ask the QA team which should have more references to make this assessment.

With these data, we can now evaluate the workload needed to define and perform tests and for the realization of the documentation.

This estimation by analogy is not necessarily very accurate, but we can now design one or more scenarios and justify them with data based on our knowledge of similar business cases.

These two different examples should normally cover most situations:

• A critical business need most often will lead to a project and, especially if it is urgent, the project will start with a minimum analysis of costs and without a precise evaluation of the development and QA efforts. Some methodologies or processes (Agile, CI), however, allow us to specify them as soon as possible in the lifecycle of the project.
• When the project is really big or new or pose a risk to the company, it is necessary to calculate the return on investment. You will then need to proceed to a more rigorous analysis of the potential costs, which will be possible if your functional knowledge and experience is close to this project. The disadvantage lies in the existence or collection of these data, in sufficient numbers to be able to create a repository that allows an assessment.

There is still a case, probably not as usual, but that can occur. This time, the sales manager and the finance director explain you that they are studying the possibility of going to a new market: selling insurances by phone to all customers, travel insurance, health insurance, accident insurance, etc …

This time, we no longer have references that allow us to estimate, by analogy, the cost of achieving such a system, nor have the necessary business knowledge. What to do?

I asked to Capers Jones, certainly the person most expert in cost estimation (and Function Points) who sent me a document available on its website Namcook.com, regarding the cost estimate BEFORE that the functional specifications are available, using a (patented) method based on a questionnaire about data normally available before the start of the project, such as:

1. Local average team salary and burden rates.
2. Planned start date for the project.
3. Desired delivery date for the project.
4. Development methodologies that will be used: Agile, RUP, TSP, etc.
5. CMMI level of the development group.
6. Programming language(s) that will be used: C#, C++, Java, SQL, etc.
7. Nature of the project: new, enhancement, etc.
8. Scope of the project: subprogram, program, departmental system, etc.
9. Class of the project: internal use, open-source, commercial, etc.
10. Type of the project: embedded, web application, client-server, etc.
11. Problem complexity ranging from very low to very high.
12. Code complexity ranging from very low to very high.
13. Data complexity ranging from very low to very high.

The answers to these questions form a ‘pattern’ which is then compared to a repository of more than 13,000 projects, with the idea that projects with identical patterns should normally have similar size and effort.

You can benefit from a repository with the data that you are missing and a method to estimate the size of the project and the level of costs and risks. You will find on Namcook web site all the informations about this method Software Risk Master.

Last point: the three examples we have seen can be combined in many complementary approaches.

And you, what methods do you use to estimate the effort for development and QA at the start of a project?

This post is also available in Leer este articulo en castellano and Lire cet article en français.