Incremental Continuous Integration - Testing the differences

The speed of a test execution and the amount of tests can make the continuous integration quite slow. I personally think that a test that takes 5 seconds isn't too bad. However if you got a thousand tests that takes 5 seconds running in sequence it will amount to way over an hour. Not very short feedback loop and it is way under the 10 minutes that are recommended as a maximum. I know of projects that have got a lot of web tests that takes a few hours to run all together. Distributing the tests on different machines surely reduces the time but it still takes a lot of time. So I understand why some people think that web tests suck even if I think they are great.

In the world of backup one wouldn't make a complete backup every time but the first time and then periodically like every week or month. Then every day between those periods the files that have changed are backed up. This is called an incremental backup and it means that only a few files are backed up most of the time. Otherwise the amounts of data stored would be too great. The best backup algorithms can handle changes on parts of large files as well. This means that only parts of the files are backed up every day. This makes a good balance between storage space and speed of restore.

Why isn't the same idea used on continuous integration. Why are we executing all the tests every time? I think that all tests should be executed every night as part of the nightly build. However integrations hooks on commits should not run all the tests as it possibly makes feedback loops way too long. To fix this problem one should only run the tests that are likely to break meaning an incremental continuous integration.

The algorithm for doing the incremental continuous integration isn’t very difficult. On check-ins one should only run tests that have changed or been added. One should also track all the application code a test touches when it runs. When application code is changed one can look up which tests that executed that application code and only execute those tests in the build. In some applications there are also some resources like configuration files that have great impact on the application. When these resources changes one might have to execute all tests depending on the configuration file.

Examples:

An application with 900 tests that each runs in 4 seconds. The initial nightly build takes at least 3600 seconds to execute if the tests are executed sequentially. That is an hour. This is no problem for a nightly build as it’s done way before people get back to work the next morning.

If one checks in a new test that takes 4 seconds the complete build will still take more than an hour. I think that is way to long. In an incremental approach it should run in 4 seconds. The same is the case if a test is changed.

Let’s look at a subset of the tests and classes where the Xs describes which classes a given test executes:

		Test A		Test B		Test C		Test D
Class 1		X				X
Class 2		X
Class 3		X		X				X
Class 4						X		X

So if there is a change in Class 4 only Test 3 and Test 4 should be executed as the other tests doesn't touch the code in Class 4. This gives us only 8 seconds of test execution instead an hour.

This means 12 seconds in total when adding 4 second for the new test and 8 seconds for the tests that executes the class that had changed. This is quite short compared to the hour a complete build would take.

The idea for incremental continuous integration would have to be supported in the continuous integration tools or the build tools. I think test coverage reports can be handy for identifying which tests to run. Another solution would be to use AOP for the same reason. However it shouldn’t be very difficult to do. I hope that build tools and continuous integration tools will support this idea in the future.