A quality consultant asks me if it is “Possible to create your own metrics with Sonar?” and “Is it easy to introduce new rules?”.

I know he is working for a customer who wants to choose a code analysis tool. And this is the typical criteria that you will systematically found in the specifications that are elaborated when it comes to choosing such a tool.

It has become a universally accepted rule, even to the point that some software vendors have made of this a selling point and include in their pre-sales demos the creation of a new rule. This is not critical but people like it. Can you imagine a car salesman opening the hood of a running vehicle to adjust the carburator or to add fuel? It would be spectacular, but is it really useful?

I already said that this criteria was dangerous in this post: 8 choice criteria for a code analysis tool. But I had not really developed the reasons why I think that such a criterion is not so much important and even can be a mistake.

The question of this consultant (and friend) is an opportunity to present some arguments in favor of this thesis, but especially:

• Help those who want to make a choice of a code analysis tool to understand how to use this criteria.
• Help Quality consultants to ask the questions needed for a better definition and use of this criteria in the specifications, depending on the client, its context, its technologies, goals, etc.

In short, if we have to use this criteria, let’s try to do it intelligently and knowingly.

I will first introduce the different types of ‘custom’ metrics and then we’ll see how to use them, for what technologies, and at which cost.

The different types of metrics

There are all kinds of metrics, some simple, some complex and some impossible. A preliminary note for the purists of Quality and terminology: for the sake of simplicity, I will gather under the same word of ‘metric’ different notions of ‘rules’, ‘indicators’, ‘diagnostics’, etc.. So thank you to not criticize the vocabulary used in this post. This is my blog, not the Dictionary of the French Academy.

Simple metrics

Let’s suppose you want to measure the number of critical or blocking errors compared to the size of the code. An application has 50,000 lines of code and 500 critical violations: the value of this metric, in this example, will be one critical violations per 100 lines of code.

In this case, we should not speak of a metric but of an indicator because this result does not come directly from the analysis of the code, but from crossing two values derived from the analysis. In fact, you do not need a tool to be able to create this kind of metrics, but only to let you access to the results of the analysis, usually through a query in the database where results are stored, or using a Web service as with Sonar.

Best programming practices

These metrics measure (usually) the use of syntaxes that violate good programming practices. In all programmation languages, some instructions are prohibited because they pose a risk to the corrrect behavior of the application, its performance, the maintainability and readability of the code, etc. The goal here is to identify these bad syntaxex and to count the number of occurrences.

This is usually the metric that corresponds to the criteria of ‘creating new rules’, and the case most often seen in a demo. It is relatively simple as the goal is to find a text string corresponding to the prohibited instruction: for instance, a Java ‘catch (Throwable)’ which is an example of incorrect error handling. Or a ‘BREAK’ in ABAP that will effectively brutally interrupt the program or its equivalent ‘STOP .. RUN’ in Cobol. A programmer can use these instructions in order to debug code, and then forget to remove them, with the risk that this instruction is activated in a production environment. I don’t know of ‘Blocker’ more ‘blocking’ than this one.

The search for a text string is not very complicated, but we will have to consider all cases:

• ‘catch(Throwable xxx)’
• ‘catch (Throwable xxx)’, with a space before the parenthesis, or sereral spaces, or a tabulation
• or even: ‘catch
  (Throwable xxx)’ with a line break before the parenthesis.

Developping such metrics requires the use of regular expressions – the famous RegExp – that I will not detail here as I’ve never been an expert of them (too old to have learned that in school).

I just want to point that even a simple metric based on a single statement or string will require a minimum of thinking, not to mention of design in order to identify all possible cases, and a good or advanced knowledge of RegExp syntax, and also enough tests to ensure that all cases are covered.

Most if not all code analysis tools allow the creation of such metrics. More or less easily, of course, but the ease “to introduce a new rule” is not in fact an important criteria because the difficulty lies in the development and testing regular expressions, as this is what will take more time.

My advice is simply to identify the tools that have very complex or even dangerous ways to create new rules:

• Use of XML files to define custom rules and manage their regular expressions, these files necessarily located in a specific directory (usually not mentioned in the documentation).
• Non-automated management of metric’s ID. Each rule must have a unique ID and you have to manage yourself your own index of rules, and you can be sure that you will encounter conflicts, especially if there are several people to manage these rules and a range of specific IDs is not standardized.
• Programmation of treatments in the database to count the number of violations encountered.
• Non-automated management of the display of the metrics: some tools group metrics by category, by weight, or by different factors, and you need to specify where you want to display the metric in your dashboard, for which category or which group of metrics.

To be continued in the next post : we will talk about complex or impossible metrics.

See you soon.

 

This post is also available in Leer este articulo en castellano and Lire cet article en français.