Last article in our series on creating custom metrics.

We will address different costs regarding the customization of metrics, generally poorly identified or completely ignored in a project of selection of a code analysis tool.

We will conclude by summarizing all the different questions to ask to a customer in order to determine if creating custom metrics is really a criterion for choosing such a tool.

Hidden costs

Implementation costs

A customer will usually ask the question “Is it easy to create new metrics with this tool?” but not “What will it cost me to manage my own metric?”. There are different costs of different types. First, there is a cost for developing metrics. Obviously, software vendors will not tell you, but instead will hide you them with a nice demo of creating a metric on live, and thus let you believe that it is simple and without any effort. But it will always be a simple metric and not a complex rule. And we saw in the third article in this series, dedicated to customizing metrics for different technologies, that these complex rules are the most useful and the most frequent. So contrary to what one often wants you to believe, the creation of a custom metric does not happen with a snap of your fingers, but requires a minimum of work and thinking:

• Are you looking to identify in the code the use of an instruction representing a violation of a good programming practice?
  • And in this case, is it a simple instruction identified by a unique string or rather a compound statement as an ‘IF … THEN … ELSE … ENDIF ‘
• Instead, are you looking to implement an architecture rule, as those regarding frameworks, as we have seen in the third article in this series.

Even for a simple metric based on a single statement, you must think of all its possible expressions. For example, a catch() can be written with a space or not before the parenthesis, and sometimes this space is actually a tabulation, and you can even write this statement on more than one line if you want.

Even for a simple metric, we need a minimum of thinking to define the regular expression, the famous RegExp that will allow to implement it, and that means trial and error and multiple testing. Believe me, it’s only in the demos that it works at first time!

So you can imagine that for a compound statement, it becomes much more complicated. And you have a lot of rules that combine several instructions, such as rules about code structure with nested statements (an IF in an IF in an IF, etc..). Or violations of good practices for performance regarding memory consuming instructions (creating an object, or calling methods for example) encountered in a loop.

And we have only talked so far about the identification of specific code through a RegExp. If you must parse an Abstract Syntax Tree, it will be more complicated. If you have scheduled to develop a rule of architecture, it will be more complicated. In most cases, it is always more complicated.

The development does not stop always with the identification of instructions, it may be necessary to program the calculation of the results of the analysis. Or to program the display of the metric in the dashboard with some mandatory data as, for example, the percentage of violations encountered among all the instructions, the number of empty catch() divided by all existing catch().

Finally, all tools will necessarily categorize a metric on an axis of gravity – from minor to critical or blocking. Some will also require you to specify additional parameters, for example the place of the rule in a qualimetric model, depending on whether it impacts maintainability, or performance, or reliability, etc.

What I want to emphasize is that creating custom metrics must be addressed as a real project with:

• A design phase: what are the metrics you want to implement? Are they defined in a document? What are the different cases, the different possible syntaxis, the corresponding RegExp, etc.
• A development phase: writing the RegExp or programming the research in an Syntax Tree, and a minimum of unit testing.
• A QA phase, with real tests, covering all possible cases that you have identified previously, combining multiple instances of the same rule in the same program, the verification of the results – counting violations, identifying false-positives – and their display in the dashboard.

Maintenance costs

The cost of developing custom rules can therefore quickly reach several man-days or more. But there are other subsequent costs.

The first, frequent enough and always forgotten: what will happen when you install a new version of the code analysis software? You made a development, it is necessarily implemented somewhere in a file, a directory, a repository, a workspace, a database, etc.

For example: the RegExp is defined in an XML file in a specific directory, as well as a table in the database with a stored procedure to calculate some results to display. Well, what will happen if the software is upgraded? Will the new version replace your xml files? Do you need to update an xml file installed by default by the software and / or to specify its location? Should you import and reapply the stored procedure? Should you import your own qualimetric model defining your own rules in order to display these metrics in the new version of the dashboard?

Another type of maintenance cost: the software has changed and your development no longer works. Yes, the way metrics are managed can, and will certainly change over time in future versions. You will need to change your ‘package’ to adapt to this new version. Costs of evolutions.

Note that you can also find bugs. For example, you must specify a number or an identifier for each of your metrics, and bad luck, the new version of the software implements new metrics that use these identifiers. Your rules do not appear anymore in the dashboard and you must find out why, and then adapt your own development in order to solve these conflicting IDs. Costs of corrections.

This means that you will have to manage your development in versions. I have seen customers very unhappy, who reported regressions on a new version, because some of their metrics disappeared after the installation of the latest release, when in fact, after the upgrade, they just reinstalled an older version of their rules in which these metrics do not exist. And believe me, it takes a lot of time to investigate and identify the origin of the so-called “regression” (because you cannot imagine someone has done such an error).

Finally, in order to avoid this, it is best to test the new version of the software with your own rules. This means some documentation of some testing procedures (in addition to the documentation of the software upgrade procedure, and version management of your ‘package’ of rules, etc..). So basically, I have included in the same category of maintenance costs all kinds of upgrade costs, corrections and changes, testing, documentation, etc. But it is important to identify them from the start as you can see they are not really small.

The questions

As you probably have understood, the creation of custom metrics is justified only if there is a real and important need. How easy it is to create them should not be the main criterion for the selection of a code analysis tool. At best, it is a plus.

If you want to buy a bike, and you can afford a bike with a titanium frame, a carbon hand-made fork, 15 speeds and tubeless wheels, great. But if you intend to use it to go shopping, it will not be the most practical, and the price will far exceed the benefits. Unless your intention is to make you a gift, of course.

Unfortunately, this is often the case, and therefore it becomes difficult to reason someone who wants to buy himself a gift. It is our job as a consultant to verify the actual existence of a need and to measure the hidden costs. So the questions:

• Do you have specific quality rules and good practices that you need to automate? If not, I do not really see what can justify the need to create new metrics.

Often, the answer is ‘No, but …’ followed by an attempt to justify his little caprice with not very accurate arguments. In this case, I will recall that 90% of users of a code analysis tool use existing rules, and 90% of the 10% who develop their own metrics will abandon them after two years.

• What kind of metrics?

I had a customer who insisted in programming Halstead metrics. These metrics count the number of operands and / or operators, as a measure of complexity. Sometimes people want the SEI maintainability indexes (combination of metrics such as the number of lines of code, cyclomatic complexity, the rate of comments …).

These metrics are obsolete, which is why most of the code analysis tools do not offer them anymore. Why invest in the creation of such metrics while existing ones are most relevant? Why make more complex the use of a tool and the interpretation of results and the lecture of a dashboard with inaccurate measurements?

There are also those who are not satisfied with the categories ‘Critical’ or ‘Blockers’, and want to create a special category just for them, something like ‘super-extra-blockers’. I call this the Spinal Tap syndrome: This one goes to eleven.

• If so, what technologies?

As seen in the previous post, it is in the domain of SAP applications that you will encounter quite often a real need. In the world of new technologies like J2EE, NET, etc. there are already enough rules and no need to create new ones, except in the particular case of proprietary frameworks.

• What part of the application portfolio this technology does represent?

You will always find customers who want metrics for unusual or old languages: PL1, Pacbase, Informix, Fortran, etc. They even often will have a list of rules used internally for manual controls. It is therefore a real need, but applications developed in these languages ​​rarely represent more than 10% of the application portfolio, and generally are old and proven and change not so often, etc. There might be some benefits sure, but are they worth the costs?

• Do you have a list of rules or any documentation of your rules?

If this is not the case, I doubt that these rules are implemented.

If you have a real need for custom metrics, if you must really choose a code analysis tool on this criterion of creating and managing custom metrics, it is very important to accurately identify:

• What are the different steps involved in creating a metric.
• What the code analysis tool can do automatically.
• What it does not: it’s up to you to do so.
• What are the different actions to be taken in case of an upgrade of version.

Prepare a test (some kind of a pilot) for the 3 or 4 softwares you have chosen to short-list, with a few cases such as:

• Development of a simple metric of the indicator type. For example, you wish to be able to enter a number like the project cost, or the number of men/days it took, or the number of people in the project team, and divide it by the number of blocking, critical or major defects. Something like that.
• Development of a simple metric for a programming best practice. For example, identify all instructions write() and writeln() – let’s imagine they are banned in favor of a proprietary system of messages management. Plan some test files with very twisted instructions with spaces or line breaks.
• Development of a complex metric: identify the same previous instructions write() and writeln() in a loop. Again, imagine some very complex tests with several instructions inside loops within other loops on several pages, for example.
• Version upgrade: ask each software vendor to package the files (or any other object) resulting from the development of the previous three rules, with an installation or upgrade documented procedure. Make a test of version upgrade (or re-install the current version) with this procedure in order to measure the complexity of the operation.

If you have a real need to create custom metrics, do a thorough test and identify hidden costs. Do not go for a simple demo, do a pilot, see if the the software editor is willing to work with you on this. You may also ask to a Quality consultant to assist you on this.

If you want to buy a racing bike, won’t you try it? Or will you just ask how easy it is to change change the wheel ?