Saturday, May 17, 2014

test-studio – Unit Testing for nodejs How it Should be

It’s been a while! Freakin’ awesome to see you again.

We’ve been working on some cool stuff! The next evolution of tribe is coming and it’s going to change how we think about developing applications on the HTML/JS/CSS stack. But that’s for later.

What’s test-studio?

We were going to prepare a video showing the cool stuff it can do but quickly realised that was going to take a while, and we’re busting to show you. So, here’s a nice pretty picture instead.

 test-studio

Looks kinda like a web based front end for nodejs unit tests, complete with sweet debugging.

Why Would I Want to Use it?

It

  • can run tests individually, in groups or whole suites,
  • integrates sweetly with node-inspector and can step the debugger in to individual tests,
  • keeps your tests in sync automatically as you work,
  • captures console output and error details for each test,
  • installs in seconds using this npm package.

Basically the sorts of things you would expect from a good test client.

It currently only supports mocha, but we will add support for other test frameworks given demand. There are also a few other useful things we’ve added.

Other Useful Things?

To show you some of the goodies we’ve added, we’ll use some examples. The tests below are all focused around the following simple module.

    var i = 0;

    module.exports = {
        next: function (increment) {
            return i += (increment || 1);
        }
    }

Pretty basic stuff, increment a variable by the argument if supplied, or by 1 if not, and return the result.

sinon and chai Integration

We’ve integrated sinon and chai and exposed them as psuedo-globals so you don’t need to require them in every file.

    suite('counter', function () {
        test('counter increments by one if no argument is supplied', function () {
            var counter = require('./counter');
            expect(counter.next()).to.equal(1);
            expect(counter.next()).to.equal(2);
        });
    });

The chai assert function is also available. We’ll show you an example of sinon in a sec. You can, of course, still use whatever libraries you like.

Refreshing Modules

A common issue with testing nodejs modules is that they are cached and return the same instance for each request. When we add another test for our counter, we don’t want to depend on the first test we wrote being executed first. Enter require.refresh.

    test('counter increments by the supplied argument', function () {
        require.refresh('./counter');
        var counter = require('./counter');
        expect(counter.next(2)).to.equal(2);
        expect(counter.next(2)).to.equal(4);
    });

We would also add the refresh to the first test, or use a test setup method to refresh it for each test.

Stubbing Modules

Substituting test stubs for dependencies is fundamental to unit testing. test-studio gives you a simple way of achieving this.

    test('modules can be easily stubbed', function () {
        require.stub('./counter', {
            next: sinon.stub().returns(42)
        });

        var counter = require('./counter');
        expect(counter.next()).to.equal(42);
        expect(counter.next.callCount).to.equal(1);
    });

You can even stub out native node modules like fs and http. Stubbed modules are refreshed at the beginning of each test and last for the lifetime of each test.

Suite Hierarchies

test-studio breaks down your suite names at each dot character and arranges them in a hierarchy, even across separate files.

We haven’t quite finished our tree view for the UI, but it will allow much better organisation of tests and the ability to focus on individual branches of your test suite as you develop.

Known Issues

The current release of test-studio is a very “alpha” release and has not undergone extensive use in many real world applications. We’ve run it against existing suites of tests that use mocha, like mocha itself, express and connect, with an almost perfect pass rate for express and connect.

Occasionally, things can get a little bit in knots when you are running asynchronous tests that throw exceptions (i.e. fail) with the debugger attached. The “Restart Process” button restarts the process in which the tests actually execute and should restore operation to normal. Failing this, restart test-studio.

To see the complete list of issues and planned enhancements, head over to the github issues page for test-studio. Feel free to log new issues with the test-studio label.

Why Did We Create test-studio?

In a nutshell, running and debugging unit tests from the command line sucks.

Having used qunit for unit testing in the browser for some time, we became used to at least a little bit of UI sweetness. An improved UI even just for browser testing was on the radar, and when we saw the state of affairs with nodejs unit testing, it seemed like a perfect application for our new platform, tribe. Much more on this soon!

What’s Coming

We’re always working on test-studio for our own projects and there is some sweet stuff in the pipeline.

Vastly Improved UI

As well as the hierarchical view of tests we mentioned before, we’re working on improving the UI to make working with your tests as easy as possible.

Client Side (Browser) Testing

Get the same experience and debugging power with your client tests as well.

A full functional testing framework is on the horizon with a simple, extensible domain specific language for programmatically driving and making assertions about your UI.

Assertion Counting

Exception based assertions are by far the easiest to implement, but they suffer from a few problems when compared with assertion counting.

  • Execution stops when an exception is raised, meaning the rest of the assertions in a test are not verified.
  • When testing asynchronous code, tests pass if your assertions are never reached. Assertion counting addresses this by expecting at least one assertion by default.
  • Assertion counting gives you better reporting, showing the number of assertions as well as the type and result of each assertion. This can lead to improved flexibility around how you structure your tests.

Cloud9 Integration

We’ve only dabbled in the world of developing in the cloud, but the idea of a cloud hosted IDE for nodejs with all the functionality of a desktop IDE, including integrated testing and debugging, makes us froth at the mouth.

Dependency Graphs

We have a release scheduled for January 2132 that gives you a visual mapping of the dependency tree of your modules. But seriously, this is on the radar.

Other Various Goodies

Lots of other minor improvements and functionality enhancements are on our to do list, and we’re humbly listening to you, our users. If there’s something that would make your development life easier that we might be able to add, or you just want to say hi, give us a holler on twitter @tribejs.

Until Next Time…

This is as much for you as it is for making our own development experience better. Try it out, let us know what we need to take it to the next level.