Incrementally measure code coverage

For new app or repos with a close to ideal level code coverage, the populor code coverage solution on coverage metrics threshold check would be efficient. However, to maintain a legacy or low coverage level repo, it is not eonough to just check coverage percentage on metrics. This post described an idea to check coverage json diff with istanbul-diff on node.js repos.

Problem

Usually in Jenkins Pipeline or SAAS DevOps infrastructure, the code coverage check is implemented with Cobertura or cloud service Coverage.

As described in previous posts, here are samples of Coverage service and on-premise Cobertura.

The coverage check is implemented with metrics and thresholds, in other wors, the score of code coverage on current baseline. This won't be a problem when the repo has an ideal coverage leve.

For example, if the threshold is set to 95% on lines, functions and branches thress metrics, when the change breaks the threshold, the coverage check will fail.

On an legacy repo, this would potentially be a problem with a low coverage level. For an example, if the repo has 45% overall lines coverage. On one of the feature branch, the code change lower down some source code coverage by accidently introduced a wrong condition in Jest. But the feature branch aslo introduced a batch of new source and keep 100% on these new added source filed. Therefore, it is possible to see an increase in Total Coverage. And due to a lower level of Cobertura threshold on existing code, this cannot be discovered by the coverage check at all. The feature branch can be merged to master branch with successful coverage endorsement.

Above is a real case in coverage overall check with one of my projects.

Solution

Since the project mentioned above is a node.js front-end app, the coverage measurement is implemented with Jest coverage. Underneath the jest framework, istanbul is the code coverage lib. This triggered me to seek a way to compare the coverage result files from the source branch to target branch.

The solution could rely on JsonDiff lib to compare the coverage between two branches and fail when there is any nodes on source tree has decrease on coverage unless the leave nodes (file-line, function, branch path) are removed from source branch.

Here the term leave node depends on which coverage metrics are selected. It could be one or more from lines, functions and branches. The three coverage metrics are supported by istanbul.

The first condition can be satisfied by applying an npm lib istabul-diff. Which is based on jsondiffpath lib to compare the increments between source coverage summary and target (existing) one.
The second condition would be resolved with traditional way -- Artifactory. On Jenkins Pipeline, a goovy closure will be defined to push coverage-summary JSON to artifactory if current BUILD passes and it is on master branch.

So the artifactory specific PATH will only keep a latest copy of master branch coverage result (in JSON format).

When Pipeline determines the build is on a feature branch, it will automatically download the master coverage summary from Artifactory and apply istanbul-diff to find if there is any loss on coverage but will accept all the positive (incremental) coverage.
To utilize istanbul-diff tool, istanbul reporter json-summary is required. By default Jest would apply parameter ["json", "lcov", "text", "clover"] (refer to Jest Doc)

So the package.json could be updated as:
{
...
"jest": {
"coverageReporters": [
"json",
"lcov",
"text",
"clover",
"json-summary"
]
}
}

Change Log

Jun 09, 2019: Initial and roughly tested with sample node.js repo.

+U Maxout!

Incrementally measure code coverage

Problem

Solution

More topics

Change Log