In the first post of this series ‘How to choose a code analysis tool?’, we mentioned four criteria frequently highlighted even though their importance varies depending on what you want to do.
Number of technologies
Do not choose a tool only because he can also analyze PL1 or Powerbuilder if these technologies are less than 10% of your information system.
You do not develop new applications with these languages, you expect that existing applications are migrated or die. And you know that the investment a software editor will have on these technologies is proportional to its sales: a PL1 analyzer will never offer the same level of functionality as a J2EE analyzer. Do not put them on the same level.
Number of metrics
Most use cases – Continuous Integration / Improvement, Quality Gate Management outsourcing – are based on a reduced set of 10 to 20 metrics. Identify the critical metrics for the use cases that you want to implement. Then identify the metrics that seem interesting or useful. The rest is ‘nice to have’.
Ability to create new metrics
Why 90% of customers are satisfied with out-of-the-box metrics? Because they correspond to market standards. Avoid playing the sorcerer’s apprentice by creating your own standards: it comes with a cost of development, maintenance, implementation and support each time you upgrade the tool.
Links between components
Of the three technologies most often encountered – J2EE, Cobol, SAP – only the first is multi-tiers. If you use J2EE frameworks, metrics based on the links between components may be useful. Remember, however, that they represent a small proportion of the total rules, and that their use induces a cost, including the validation of false-positive.
Now let’s see three criteria that I consider really important.
Multiple Quality Models
The set of metrics and their organization on different axis is your Quality Model, the measurement grid that you will use to evaluate the quality of your applications. The more flexibility you get in building models, the easier and more efficient will be your evaluation.
How many lines of codes has a very large Cobol program? How much points of complexity (CC) to decide it is very complicated? Batch programs are larger and more complex than TP (transactional) programs and you will wish to assign different thresholds on these metrics. Otherwise, a project team in charge of the maintenance of a batch application will be disadvantaged in comparison with a team that manages an application TP.
Some other examples:
- A database increases with the accumulation of data and over time, some bad practices of SQL programmation will begin to take effect, detrimental to application performance. Do not expect to be in the red at first with the default thresholds: to raise their level of criticality allows a pro-active management of risks of non-performance.
- You know how expensive it is to upgrade versions of SAP? You know the number of days required to identify the components that will stop working because they use syntaxes not supported by the new version or because they have access to the core? Or you have a C++ application designed to control cash machines of different brands with different OS.
In such examples, the portability of the code becomes the most critical factor in your model quality when, by default, it is usually the least important. That means a specific model for these specific cases.
There is no unique Quality Model: it is like trying to measure apples and potatoes. The ability to define and choose a Quality Model suitable for a given application is a more important criterion than those listed above.
Customization of the dashboard
Metric results calculated on the source code of your applications appears in a dashboard that allows you to control and monitor the quality. The ability to customize the dashboard, based on use cases, technology or even a simple application is a factor of success of your Quality project.
Imagine you are asked to perform an audit of a critical application and to present the results to the architects and the project team. At the last moment before entering the room, you are told that the CIO will be present. He immediately begins to read his mails on his Blackberry without giving you any attention. What do you do to interest him into your presentation?
Simple. Show him how much this application costs him.
A Java architect or a quality manager knows what the is a metric like Cyclomatic Complexity or LCOM4. So I can show them a dashboard with some metrics (LOC, CC, percentage of comments or duplicated code, …), go to the list of the most serious defects and conclude with a qualitative and quantitative evaluation.
For an IT manager, or an Outsourcing manager, I will start with a financial evaluation. I want to show the technical debt at the top of the first page of the dashboard, then go on functional axis such as maintainability or reliability of the application, a list of indicators to build an SLA and finish with an action plan.
Imagine being able to build any dashboard by drag and drop of different widgets, saying to the user at your side: ‘You want this?’, ‘This diagram first?’, ‘What are the indicators of interest for you?’
This is an important criterion.
Intégration with other tools
As you already know, the sooner a defect is identified, the less expensive to solve it. And the sooner you can identify a defect is when the programmer just did it. As a manager said me once: “If I have 30 developers on an application, I will not wait until the end of the week to track their bugs.” At 5 days per week per programmer, this is 150 days of defects. Too late.
Ideally, you want:
- To be able to manage the components of an application in a version control or configuration repository.
- To trigger a build (new version) of the application every time a developer creates a new version of a component in the repository. This way, you can check that the application compiles correctly and automatically trigger a source code analysis to identify new defects ASAP.
- To have automatic alerts on the occurrence of certain defects or a dashboard that allows you to easily review new defects and request a correction.
- Automatically update the list of tasks with each developer to make corrections.
For more details, see also ‘Continuous Improvement‘.
Cost
Obviously, it will be an important factor in your choice of tools and we would need an entire post to discuss it. I would just notice the emergence of new cost models.
Everyone knows the traditional model based on a per user license which is characterized by an initial cost more or less important – depending on the number of licenses, any additional modules and other factors such as the end of the quarter or the end of the year when a software editor is more easily disposed to some favors.
The Open Source world has brought new practices with a license by server and not by user and a business model focused on services: you no longer pay maintenance, but a higher level of support. What’s the difference? You can stop paying without losing the benefit of new updates or support from the community.
Finally, one sees more and more SaaS solutions, with a cost by usage or by subscription.
I ran some simulations and I encourage you to do the same. First conclusions would need to be refined, but:
- The ‘license’ model is more expensive than the ‘subscription’ model (Open Source / Saas) on the early years; the ‘subscription’ model can – possibly – become more expensive after a long period (8 to 10 years minimum).
- The success of the ‘subscription’ model is highly dependent on renewal rate.
A difference of plus or minus 5% in the renewal rate of subscriptions has an huge impact on the profitability and value of an SaaS or Open Source editor. You understand why they better had to give you a good support?
In conclusion, base your selection criteria on an accurate identification of your needs, focused on your application portfolio and the use cases you want to implement.
I will try to illustrate some of these cases in the future. In the meantime, feel free to share your comments.
This post is also available in Leer este articulo en castellano and Lire cet article en français.