I am always surprised when an article claims to compare code analysis tools or proposes criteria to choose such a tool. There is no definitive answer to this question, other than ‘it depends’.
If you want to buy a car, you will of course make your choice based on some criteria depending on the usage you plan for your vehicle. A coupe or convertible sports car are certainly attractive objects but impractical when it comes to transport your large family, do your shopping at Carrefour or go the beach with your surfing board.
Same when it comes to the selection of a tool to analyze the quality of your applications.
So this post is not intended to define the most important eight criteria of choice of such software, but to list four criteria that I meet a lot and (next time) three other ones that I don’t see so much. Yes, I know, does’nt make 8. In fact, the last criterion, the cost of software, will be treated separately.
Number of technologies
It is almost always the first criterion of the selection grid. I do not understand why a tool capable of covering 20 languages would be higher in the list than one who does just 5 analysis when both of them can analyze the code of your applications. Today, 50% of applications are J2EE-type and 25% Legacy (Cobol Mainframe / SAP). It might vary more or less, depending on the country: some continents are more Microsoft than others, with an higher rate of C++ and .Net. It also varies according to sectors: banks and insurance companies would stop working if Cobol disappears.
Software publishers are well aware of that, so they will focus their efforts on the technologies most used by their customers and do not really invest in Powerbuilder or Delphi.
Unless you are among the largest and the oldest companies in your industry, and you have accumulated layers and layers of technologies over the years, or you have merged with other companies, or you have often changed development strategies, it is likely that you have more than 75% of your application portfolio on 2 or 3 different technologies. And most likely J2EE / Cobol / SAP. And certainly the most critical part of them, representing the heart of your business.
Quit of your list softwares that do not cover your applications. For the remaining tools, check in detail – with a pilot for example – the quality of analysis performed. The number of other technologies covered is purely informative, not a real criterion.
Number of metrics
As with the previous point, focus on the quality and value of the proposed metrics rather than their number. It’s nice to have 20 naming rules for all types of objects in your applications, but it is not the most important and not worth a metric that will cover a serious breach with a strong possibility of stopping your application or causing data corruption.
Identify from 10 to 20 metrics on which you:
- Validate (Go/NoGo) the delivery of an application (Quality Gate).
- Check your providers SLAs.
- Implement processes of Continuous Integration / Improvement.
In my experience, the number of metrics will play a role in two specific cases:
- The calculation of technical debt (we will have the opportunity to speak of this later).
- The audit of the quality of an application that you will ‘acquire’ (for maintenance for example) or to identify a particular problem (of performance for example).
But even in this case, it is rare to find exactly the bug in question or a specific defect using a particular metric. In addition, this case corresponds to a situation of consulting and not of software acquisition.
Build a list of best practices necessary for your applications: there are standards on the market and normally most of the tools should integrate them. If not, it might be that the product is probably still a little young for this technology.
Identify then the metrics that are interesting but not essential, depending on your use cases. The rest is ‘nice to have’.
Ability to create new metrics
Warning: Dangerous criterion.
A basic principle: if it is available, you will use it. If it is possible to do it, you will do it. Even if you do not need it.
Do you remember that ‘little chemist’ game you were given at Christmas, as an educational toy designed to awaken your scientific instincts? Who would have guessed that you would eventually play the ‘mad scientist’ in search of the most explosive mixture of differently colored substances.
The same syndrome is sometimes found among some architects or Quality managers who are reinventing the wheel of ISO 9216 with their own metrics. I have often seen such attempts abandoned after 2 or 3 years, simply because the benefit of such customization is lower than the costs of development and maintenance. In the Open Source world, it is possible that your new metrics will be incorporated in the tool by the community. In the proprietary world, forget it. You will need to do your own support of your own model alone, with each software upgrade.
There are few cases where the need for new metrics is justified, specificaly when there are no well-defined standards in the market. This can happen for a few technologies. Otherwise, the software editor did not include the most important rules corresponding to the right good practices. Quit him from the list, or work with him to incorporate these rules in a future release.
In all cases, measure well the cost of development of new metrics and make sure it answers a need. Keep it simple, do not play alchemists.
Links between components
Some people looking for a code analysis tool have a curious attitude: they try to entrap the ones who propose them a solution. “Let’s see if they will find the dynamic calls between the layers of my proprietary framework”. “Will they be able to identify the links between these components?”.
If you have a specific need, if your applications use a specific mechanism, explain it clearly. Many service companies have created their own version of Struts or Hibernate frameworks and their customers are now bound to absolutely no standard architectures. In some Cobol applications, Cobol, programs, JCLs and even CICS transactions are not called directly but through a table of codification of these links. Now, these are not the most common situations and better to be avoided.
Most metrics identify violations of good programming practices. In other words, they spot the incorrect or incomplete syntaxes or incorrect structures of various instructions. Good practices of architecture are not based on a special syntax, but on links between different components in different layers of the application.
I see two main interests in an analysis tool that can detect such links:
- Conduct impact analysis: there we go out of the field of code quality to get into the domain of retro-documentation or assistance with migration.
- Verify compliance with the rules of use of frameworks, proprietary or not.
I think this second point has a genuine interest. For example, when you centralize access to the data layer on some specialized components, or when all the objects in the presentation layer must go through a unique component to access the business logic layer, you like to be able to identify objects that do not respect these rules. Because the next change on one of these layers may not be implemented for these objects.
But these are mostly necessary for 3-tiers architectures, so J2EE and .Net mainly. Also, if your frameworks and architecture best practices are proprietary, you should expect a cost of customization.
These metrics are those that present the greatest number of false-positives. It is (relatively) easy to verify that a particular syntax is respected or not, so metrics corresponding to good programming practices are generally reliable. A metric that checks calls between certain types or classes of components will be much more context-sensitive and may require a systematic validation of the links after each analysis. At an higher cost of implementation and analysis.
Finally, you will probably find a proportion of 1 to 10 between these different kinds of rules: metrics about architecture good practices will not represent more than 10% of all rules proposed by the tool. And all are not critical.
So again, please check the suitability and cost, to your needs. The most complete tool is not necessarily the one that suits you best.
No need to buy a Hummer to go shopping: when you realize the effort to maneuver it and the time wasted trying to park, you may quickly leave it in the garage.
This post is also available in Leer este articulo en castellano and Lire cet article en français.