Tag Archives : how well is it tested?

How well is it tested: Puppet (part 2 of 2)

In the last post, I showed analysis results of Puppet’s test duplication along with other interesting findings. The relationships discovered seem to suggest that test duplication is not due to chance alone, and that there is a complex relationship in terms of amount of tests in the suites that is not entirely linear. The results in the post also […]

Test duplication vs mutation score (log-scaled and reversed)

How well is it tested: Puppet (part 1 of 2)

Puppet is a “Server automation framework and application” that it utilized by people worldwide and even Fortune 100 companies (according to Puppet’s website) to manage part of their devops work. It is also one of the key projects that “powers Github.” Puppet is written in Ruby and unit-tested using the BDD framework Rspec. Given Puppet’s popularity and utility, […]

How well is it tested: Composer

Composer is a very popular dependency manager that many PHP projects use. With the newly introduced support for PHP by Mutator, I set off to rate the quality of Composer. For the analysis, I utilized the Composer snapshot of version 1.0-dev, from Aug 24 2015. Recall that the mutation score of a test suite is a quality metric […]

95% confidence interval for Composer's test quality

Mutant's test quality

How well is it tested: Mutant

This time, Mutator analyzes another mutation testing tool for Ruby called Mutant. Mutant is an alternative to mutation analysis for Ruby projects. It is fast because it applies mutations on the abstract syntax tree. However, Mutant has several disadvantages. First, since it goes under the Ruby hood, it only supports Ruby exclusively and, second, it only support […]

How well is it tested: AI4R

This time we apply Mutator to Artificial Intelligence for Ruby (AI4R) to analyze how well it has been tested.   Basics Mutator analyzed AI4R version 1.13 which has 30 test suites and 35 core source files. With the exception of one outlier (a failing test suite, see below), all test suites were selected for analysis. Since practically all […]

How well is it tested: PITest

One of the coolest things about Mutator is that since it is cross-language, it can analyze any Java and Ruby project — even other so-called mutation analyzers! We recently analyzed a tool called PITest that claims to do mutation analysis for Java. What we found about the quality of its tests will surprise you. Before we begin, recall […]