Results on 37signals’ open-source apps

To test rails-jsdom on real apps, we moved the system tests of three open-source Rails apps from 37signals to interaction tests wherever possible: Campfire, Fizzy and Writebook. The work is on the interaction-tests branch of each fork: once-campfire, fizzy and writebook. The upstream apps still use their system tests.

App System tests on main Moved to interaction tests Still system tests Moved tests as system tests Moved tests as interaction tests Speedup
Campfire 30 28 3 (1 is new, see below) 317.7s 19.3s 16.5x faster (94% less time)
Fizzy 47 18 29 21.5s 9.6s 2.2x faster (55% less time)
Writebook 3 3 0 6.8s 3.7s 1.8x faster (45% less time)

The times compare only the tests that moved: each moved test as a system test on the app’s main branch, against the interaction test that replaced it. The tests that are still system tests are not in either time.

Campfire has 3 system tests left, not 2, because one moved test saved screenshots. Its checks moved to an interaction test, and its screenshots stay in a new system test.

How we measured

Why the gains differ

Bugs found

Changes the apps needed

Most tests moved with the same steps. Some needed small changes in the app’s test code, not in the app:

Tests that are still system tests

Each remaining system test needs something that jsdom does not do. The exception is one Fizzy test that we have not tried yet.

Campfire (3):

Fizzy (29):