Conditional Swift testing

The problem

Some tests need to be skipped when executing on CI (continuous integration like travis.org).

What tests? In my case, it's performance tests.

Why? because I can't trust all CI results and sometimes, randomly, some tests will fail without good reason, like this one:

error: testPerformance : failed: The relative standard deviation of the measurements is 11.061% which is higher than the max allowed of 10.000%.

This is due to actual, temporary machine load that affects results of performance measurements. Happens.


Active Compilation Conditions
a.k.a. conditional compilation flag.

"Active Compilation Conditions" is a new build setting for passing conditional compilation flags to the Swift compiler.

note for Objective-C programmers: it's -DMYFLAG from OTHER_FLAGS


// do stuff

I named it CI and added to my Travis build script. Since I'm using Swift Package Manager it's called like this

$ swift test -Xswiftc -DCI

Now, wherever I check for CI I may distinguish if a test is run on continuous integration server.

#if !CI
extension FooTests {
    func testPerformance() {
         // ...

finally, enough adjust list of tests for Swift Package Manager to not try run tests not intended for CI

// ...
#if !CI
    tests += [
        ("testPerformance", testPerformance),
// ...


Sidenote about #if

Swift #if ... #endif directive behaves differently than the one known from C, and it has implications. See comments to this tweet