Thursday, May 02, 2013

knockout.composite v0.2 – Introducing Tribe

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference.

We’d love to spend time giving you the juice on the freakin’ amazing new features we’ve been working on, but for now, you’ll have to make do with just a glimpse…

What is Tribe?

Tribe is a platform for building rich, highly interactive applications targeting desktop and mobile devices. It is comprised of a number of components:

  • Composite – The composite JavaScript framework you know and love, completely rewritten and supercharged with new features.
  • MessageHub – A SignalR based message gateway that seamlessly broadcasts messages to any device and publishes commands to internal services via message buses like NServiceBus.
  • Mobile – A set of skinnable panes and utilities for creating high performance, professional quality mobile device apps in minutes
  • Desktop – A set of reusable panes for common requirements such as dialog, grid, graph, tab panel, expander and more.
  • PubSub – A powerful, lightweight publish / subscribe engine with built-in lifetime management.

 

Composite – Concepts Crystallised

Tribe.Composite takes the concepts explored in knockout.composite, formalises the best concepts and adds new, much needed features

  • Memory management – memory management is fully automated. Subscribe to messages or document events in your models, when the pane is removed from the DOM, everything is cleaned up for you without needing a line of code.
  • Extensible rendering pipeline – each step of the rendering process is now expressed in a separate, loosely coupled unit. Panes have an extensible, formally defined life cycle from initial resource loading through to disposal.
  • Extensible resource loading strategies – when and how panes and other resources are loaded is now completely customisable. Out of the box, you can load panes completely on demand, completely preloaded or by component. We’ll even provide a mechanism for loading sets of panes as AMD components.
  • Hierarchical page structure – Composite tracks the creation and destruction of panes and maintains a hierarchy of “nodes"
  • Transitions – transitions are now a first class citizen of Composite. Transition any node, pane or element in or out using powerful, hardware accelerated CSS transitions.
  • Sagas – we’re working on creating a neat abstraction over user processes, vaguely modelled around the concept of message bus “sagas”. We’re still working on the best abstraction, but it’s looking sweet with simple syntax and semantics, and automatic state persistence.
  • Much improved everything else – API, testing, testability, extensibility. Everything is 100% better than v0.1!

 

MessageHub – Seamless, Secure Message Distribution

This is the really exciting stuff. MessageHub is a gateway built on Microsoft’s SignalR technology, giving you seamless, scalable and secure channel based communications between any connected device.

MessageHub also allows you to use simple conventions to map client messages to internal message classes and seamlessly authorise and send messages to internal services. We’ll give you an NServiceBus adapter out of the box, and building adapters for other buses is trivial.

MessageHub has built in message record and replay semantics, with both client and server side persistence. Let’s talk later about powerful event sourcing with built in client and server side fault tolerance.

 

End to End, Integrated Mobile and Desktop Platform

Our aim is to make building integrated systems with real time information sharing easy – not just apps. There are a bazillion frameworks out there for building “apps”, but this is just one piece of the puzzle.

Mobile devices are changing everything. The explosion of cheap devices has opened a world of opportunity for capturing data like never before, but the mobile device form factor is not appropriate for every task. Does anyone seriously ever use a word processor on their mobile phone?

The HTML / JS / CSS technology stack has evolved to the point where “traditional” desktop apps built with proprietary technologies offer very few advantages. There is little you can do with a desktop app that you can’t do with a web app. Combine this with the ability to maintain a single code base that targets browsers, iOS, Android and virtually any other device, it’s unbeatable.

Hybrid mobile apps have a bit of a bad reputation on the performance front, mostly thanks to Facebook’s botched effort at a hybrid app. We are going to bring you a native looking app that is virtually indistinguishable from a native app, even on old hardware. There are already plenty of examples out there, like Touralot (by knockout’s very own Steve Sanderson). Seriously, learning or acquiring skills in Objective C, Java, as well as the HTML/JS/CSS stack is a massive and entirely unnecessary investment.

Tribe is built from the ground up with scalability and performance in mind. 100 users or 1,000,000 users with consistently responsive user experience. We’ll give you some guidance on choosing server side technologies that will support this sort of scale.

But I Ramble…

Less talk, more create.

Details, samples, documentation, all coming soon. We can’t wait to show you.

If you haven’t seen knockout.composite v0.1, check it out here.

Monday, January 21, 2013

PackScript – Better. Because…

*UPDATE* Check out http://packscript.com/ for a complete reference and examples.

What is this PackScript Jiggery?

PackScript is a powerful open source (MIT license) resource build tool designed specifically for the combining and minification JavaScript, HTML and CSS files in modern, single page web applications. It was built from real need, specifically to support knockout.composite, and is focused around taking all of the pain out of managing all of the resources for your web app. You can find it on github.

PackScript is the next generation of build tools for the web and handles every aspect of building and optimising your web resources. Say goodbye to painful optimisation and to managing code for your debug, test and production environments, say hello to the optimum debugging experience in your browser. PackScript leaves you to do what you do best – code.

Next Generation Build? What makes PackScript better than everything else out there?

First of all, PackScript is much more than a minification tool. It actually delegates the task of compressing resources to separate tools. Microsoft AjaxMin is used by default, but adding support for others is simple. YUICompressor, Closure Compiler, uglifyjs, jsMin and minify are other tools that fall into this category.

So PackScript handles “bundling” together a bunch of files and compresses them. There are a number of existing tools out there that do this…

Server-side Solutions

A significant number of these tools rely on a server side component to dynamically combine and minify your resources. ASP.NET bundling, Carabiner, SquishIt and bundlejs are all examples of this kind of build tool. Generally, they do the job quite well, but they have some drawbacks.

The most obvious issue is that these solutions require a server side component, adding a dependency, tying you to a platform and consuming your valuable CPU cycles. Modern, single page web applications perform rendering tasks on the client, relying on server side processes purely for process and data access. Adding a dependency like this just adds unnecessary complexity.

PackScript produces static resources, as you code and as part of your build process, that can be deployed to any high-performance web server, anywhere, even to a global CDN.

The other not so obvious issue is one of debugging. These tools generally present your debugger with one behemoth file; a less than desirable experience. PackScript allows your files to be packaged in a way that splits your files back up and gives you the debugging experience you need. Even if you are using a server side rendering framework, you can still benefit from PackScript.

Client-side Solutions

Client side tools are not really combining and minifying tools, they optimise the loading of multiple files asynchronously. headjs and labjs are examples of these tools. PackScript will happily coexist with these sorts of tools if you really want to optimise your loading times.

At some point I intend to create an extension that will split your resources into an number of equal size chunks, load them with one of the above tools and piece them together on the client. I’m expecting this to have pretty significant impact on load times!

JavaScript Frameworks

If you’re using a framework like CommonJS or Dojo, they often come with their own set of resource management tools. Use them!

Having said this, PackScript can still offer you some benefits. Read on, and check out the features that PackScript offers.

RequireJS / AMD

RequireJS is arguably the best resource management build tool out there (until now…), popular enough and similar enough to PackScript that it deserves it’s own section.

RequireJS essentially allows you to declaratively define dependencies in your code. Dependencies can consist of individual JavaScript files or you can define named modules that can themselves have dependencies. Modules implement the Asynchronous Module Definition API. RequireJS will ensure that all dependencies (including nested dependencies) have been fulfilled.

AMD is a good standard that is gaining popularity. It takes a lot of the pain out of managing complex dependency trees, and RequireJS minifies your resources along the way. We think PackScript is better for many situations, and can actually coexist with an AMD approach.

Two main problems that RequireJS has are that it’s intrusive and that it introduces a degree of friction.

Intrusive

By intrusive, I mean that you have to write code in a specific way to satisfy RequireJS; modules must be defined and dependencies must be declared using the AMD API. This is not all bad, enforcing a consistent approach to this sort of stuff is not a bad idea, but it introduces a direct dependency on a third party product and unnecessarily ties you to an API.

PackScript manages your dependencies externally to your code and builds static, optimised files at build time. No code needs to be written to manage resources that will ever make it into your running application.

Friction

Friction is anything that consumes your time, most often repetitive and error prone tasks – a “force resisting relative movement”. As an example, creating an optimised set of resources with RequireJS involves creating a configuration file and executing the optimiser against that new file. We also need to make sure the build process includes creation of this resource set, so update the build configuration for each environment.

This sort of friction can (and should) usually be eliminated with some custom solutions – create an executable that scans our project folder for build profiles that conform to a specific convention and execute the optimiser against them with a common set of arguments.

PackScript is focused around eliminating friction. It builds your application in the background as you work and is designed to be easily integrated with any automated build process. In most cases, you can just add your resources to the project and move on. If not, add a simple configuration file and PackScript does the rest, even as you code.

Flexibility and Extensibility

PackScript also offers a great deal more flexibility and extensibility than RequireJS.

Don’t like the API? Change it.

Doing the same configuration over and over? Abstract it out.

Optimising the buggery out of your resources? Load multiple modules from one combined script at key points in the application lifecycle, or in the background after loading a set of bootstrapper resources.

Got a stack of icons that are loaded individually? Combine them into a single file and generate a CSS map for them. Add an icon to your project and it’s there. Frictionless.

PackScript works with any static resource and is easily extensible with either JavaScript or .NET. Combine this with the declarative dependency management of knockout.composite, and you get the best of it all.

What Are You Waiting For?

Head over to the feature overview for PackScript, check out an example, have a look at the core unit tests and integration tests, download it from github and have a play! It’s worth noting there are a number of issues on github.

What do you think? Have I missed anything? Is there another solution that’s not mentioned here?

Thursday, January 17, 2013

PackScript – An Example - Freedom From Script Tags!

*UPDATE* Check out http://packscript.com/ for a complete reference and examples.

We’re guessing you got here from the last post. If not, it’s probably worth reading for a quick intro, but hey, you’re smart, you’ll probably pick it up as we go along.

This example demonstrates how to use configuration files to define script and stylesheet outputs in both debug and production forms, create reusable functions and use templates.

As promised, we’re going to show you how we can work on our code in separate files and be able to edit, create, move etc. these files on the fly and have full debugging experience without needing to ever manage script or stylesheet references.

Chrome is currently the only browser that supports this technique using the sourceURL tag. FireBug claims to, but I haven’t been successful. You do use Chrome for web development, right?

The Sample

We’re going to build a silly little text animation thing. Head on over to the github repository and have a quick squiz at what’s there… Oh. Couldn’t be bothered? OK, well, it looks something like this.

PackScriptSample-files

At least that’s what it looks like in VS2012.

The Build folder is where our combined files are going to end up. They’re not included in the solution, otherwise search operations in Visual Studio return duplicate results. It will get generated as part of your build process, anyway!

The Tools folder contains a copy of the PackScript assemblies.

The Source folder isn’t really that interesting either. It’s got jQuery, some CSS, a controller file that handles a button click and a simple animate function. Enough to demonstrate what we’re trying to get at.

The html files contain a simple page from which to trigger our animation. They contain duplicate markup, the solution to which lies with our friend knockout.composite.

The Interesting Bits

Firstly, let’s have a look at the PackScript config file, pack.js.

pack({
    to: 'Build/site.js',
    include: 'Source/*.js',
    prioritise: 'jquery.js',
    template: 'debug'
});

pack({
    to: 'Build/site.min.js',
    include: 'Source/*.js',
    prioritise: 'jquery.js',
    minify: true
});

pack({
    to: 'Build/site.css',
    include: 'Source/*.css'
});

A little bit going on here, but pretty easy to follow.

First up, build site.js from all the JavaScript in the Source folder, put jQuery first and apply some template called ‘debug’ to them. Didn’t I see some debug template thingy before?

The production version, site.min.js, is pretty similar, except rather than applying the template, minify it all. We could have minified the CSS too, but you get the idea.

The Template

So it looks like there are two parts to the template. Since you’ve read the first part, you can already guess what each file is for. debug.template.js:

window.eval("<%= prepareContent(content, pathRelativeToConfig) %>");

Looks suspiciously like a very short template. What’s this prepareContent stuff?

this.prepareContent = function (content, path) {
    return content
        .replace(/\r/g, "")                 // exclude windows linefeeds
        .replace(/\\/g, "\\\\")             // double escape
        .replace(/\n/g, "\\n")              // replace literal newlines with control characters
        .replace(/\"/g, "\\\"")             // escape double quotes
        + "\\n\/\/@ sourceURL="             // append sourceURL tag
        + path.toString().replace(/\\/g, '/');
}

Ahh. Cool, we can put reusable functions into these “configuration” files, like debug.pack.js.

Note we attach our prepareContent function to the “this” object. PackScript restricts the scope of the execution of each file to help avoid conflicts, so to make something available globally, we need to explicitly attach it to the global object. Having said this, when you write them, organise these sorts of functions into pseudo-namespaces. Do it. Thank me later.

So what is actually going on here? The template is wrapping the script content in a call to window.eval. This gives us the chance to inject some content, in this case, a neat little tag that Chrome will interpret. There’s some other stuff in there to make it play nicely with an eval statement, pretty straightforward stuff.

The other magic variables used in the template are passed to the template engine by PackScript. Check out the github readme for an up to date list of the built in variables.

The Result?

You can see the “debug” version here and the “production” version here. Open DevTools for the debug version and you can see our scripts nicely in the source list.

PackScriptSample-debugger

The lack of an ‘S’ on Source is a tad conspicuous (DevTools bug?), but other than that, everything looks as expected. This might seem trivial with only three files (one of which you should probably load from a CDN anyway), but when you have hundreds of source files, it is sweetness to see them arrayed out in neat organisation.

Let’s add a file without adding a script reference.

PackScriptSample-debugger2

Cool. Looks like it works. Breakpoints even play the game. The production version works pretty much as you would expect.

So there you have it! Nicely separated code that you can mess with to your heart’s content, debug and production versions, combined CSS, and all without ever having to touch a script tag again*. Feels good.

* - OK, maybe occasionally.

Just To Make It Perfectly Clear…

None of this sample is necessarily “best practice”. It is presented purely to demonstrate some of the power of PackScript. knockout.composite does use a similar technique for this issue, but has a relatively sophisticated resource management system that takes all sorts of pain away.

PackScript is fully functional and under active development. I already have a list of planned enhancements as long as my arm (did someone say “source maps”?), so dip your toes in, give it a try, we think you’ll love it!

In the not-too-distant future, we will cobble together some sort of reference site that shows how to get some serious power out of knockout.composite and PackScript and how it fits very nicely with a SOA flavoured system.

In the meantime, we will be putting together some more posts on how it’s built (JavaScript core running in a Windows console), extensibility and some other advanced stuff.

What would you like to hear about most?

Tuesday, January 15, 2013

PackScript – Next Generation Build for the Web

*UPDATE* Check out http://packscript.com/ for a complete reference and examples.

What?

PackScript is a powerful open source (MIT license) resource build tool designed specifically for the combining and minification JavaScript, HTML and CSS files in modern, single page web applications. It was built from real need, specifically to support knockout.composite, and is focused around taking all of the pain out of managing all of the resources for your web app. You can find it on github.

If you’re pretty familiar with these sorts of tools, you might want to have a look at the post on a few reasons why we believe PackScript is better than all current alternatives.

The Story

Too much of my life as a web developer has been spent managing the multitude of JavaScript, HTML and CSS for whatever current project. Keeping track of where that JavaScript function is. Making sure script and link tags reference the right files from multitudes of scripts and stylesheets. Or worse… Keeping all of the JavaScript for a project in one behemoth file in a team environment.

Now remember we need to have different sets of files for each of our development environments. Facilitate debugging in the dev environment, minify but include some debugging or tracing stuff in the test environment, supercharge everything in production. Ok, that’s not so bad.

But wait… Our mobile app uses some (but not all) of the stuff from our desktop site, but needs some custom code to do all that “responsive” bollocks.

To top all this off, our JavaScript codebase is getting pretty big. We could combine it all into one massive file and load it up front, but it would be even better if we could load largish chunks of it as the user hits various parts of the app.

My head hurts. No more!

Powerful and Flexible Configuration API

PackScript is written primarily in JavaScript and runs in an instance of the Google V8 engine, courtesy of the excellent Noesis.Javascript library. It exposes a powerful and flexible JavaScript API for specifying input and output files and other options.

pack({
    to: 'site.min.js',
    include: '*.js',
    recursive: true,
    minify: true
});

PackScript scans the target folder recursively and, by default, picks up any file that ends with ‘pack.js’ and treats it as a configuration file. These files are executed in an instance of a V8 JavaScript engine.

The fact that the configuration is live JavaScript and not just static text allows us to do some pretty cool things.

Convention Over Configuration

PackScript allows you to define powerful conventions about the location and naming of your files. Tell your devs where to put and what to call your script files and forget about those script tags and AJAX calls.

GlobalExcludes = ['*.debug.js', 'admin.js'];
pack({
    to: 'site.js',
    include: '*.js',
    exclude: GlobalExcludes
});

The GlobalExcludes variable can be defined in a configuration file in another location and used throughout your project. You can also create reusable functions and extend the API to your liking by creating custom “transforms”; each property of the object passed to the pack method is a transform, like ‘include’ and ‘minify’.

We will have a reference site for knockout.composite and PackScript released soon that will show you some much more complex examples.

Painless, Seamless Debugging

PackScript is not fussy about your IDE and provides a memory-resident mode where file changes are detected and the appropriate outputs updated automatically. Keep your code neatly split up in separate files, edit and restructure them, refresh your browser and each file is there in your debugger, no messing with script references, all loaded using a single HTTP connection. Isn’t this how it should work?

The example in the next post will show you how you can keep your code in separate files and retain full debugging support (at least in Chrome) without needing to maintain script references. A production-ready script is produced at the same time.

Templates

PackScript gives you powerful and extensible templating for your content files. Out of the box, it provides support for underscore.js templates, but you can easily plug in support for other engines like handlebars, mustache or jsRender. More on this later.

By default, the target folder is scanned recursively for files with an extension of ‘.template.*’. A template for wrapping script files in HTML script tags might be called wrapScript.template.html and might look something like:

<script type="text/javascript">
<%=content%>
</script>

These templates can then be referenced in your pack configurations using the filename without extension.

pack({
    to: 'scripts.htm',
    include: '*.js',
    template: 'wrapScript'
});
Templates can be applied to any file type. A much more useful application of templates is explored in the next post.
Minification

Minification is provided out of the box courtesy of the Microsoft AjaxMin library and can be applied to any or all of your script and stylesheet files by setting the minify option to true. AjaxMin gives quite good compression for zero effort and exposes a .NET interface, so it was a good choice for the first supported minifier. Support for Closure Compiler and others is coming, and plugging in your own is easy.

Prioritisation

Simple prioritisation is offered by passing a file name or array of file names to the prioritise option. The files passed will be combined first in the order specified. This is enough to satisfy most scenarios, but will be expanded to allow for wildcards, etc.

Extensibility

PackScript offers a great deal of extensibility. Reusable functions will cater for many situations, but custom transforms can easily be created for simple APIs.

APIs implemented in .NET can also be exposed within JavaScript, again thanks to the awesome Noesis.Javascript library. The file access, minification and logging functions are implemented in this way. A simple marker interface is provided for this, more in a future post! If you’re interested in learning more, leave a comment or drop me a tweet.

What’s Coming?

Some extensions coming will include

  • SASS integration,
  • source map integration,
  • build information (build configuration, version, etc),
  • automatic checkout of files from TFS (though it’s arguable that it would be a better option to switch to git than to use it),
  • combining images and icons into a single file with CSS map generation,
  • HTML minification.
A Cool Trick

The console PackScript application also serves as a JavaScript console where you can interrogate configuration, test functions, etc. This feature is provided by a small wrapper for the Noesis.Javascript library - Noesis.Javascript.Extensions. The wrapper also adds support for dynamic invocation and return types to the JavaScript engine.

> Pack.options
{
  "configurationFileFilter": "*pack.config.js",
  "packFileFilter": "*pack.js",
  "templateFileExtension": ".template.*",
  "logLevel": "debug"
}
>

Until Next Time…

I hope this gives you a reasonable idea of the capabilities of PackScript. It’s worth noting at this point that it is a very early release of the product, and while it is well tested, there may be some reliability issues, and many more features on the way.

You can see a more detailed list of configuration options at the github repository. The unit tests and integration tests are also good sources of information.

Head on over to the next post for a neat practical application!

Saturday, September 15, 2012

knockout.composite: Parting Thoughts and Future Directions

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference.

This is the final part of a multi-part series. Check out some of the earlier posts if you haven’t already.

Part 1: Introducing knockout.composite
Part 2: The Basics
Part 3: Getting Around
Part 4: Putting it All Together
Part 5: Unit Testing: Model Citizens 
Part 6: Integration Testing: Playing Nicely Together
Part 7: Functional Testing: Automated UI Tests, Anywhere, Any Time
Part 8: Pack Your App and Make it Fly
Part 9: Parting Thoughts and Future Directions

 

Finally! We’re at the end of the series! Some parting thoughts before we call it a day…

First and foremost, knockout.composite is a version 0.1 product. It is being used in a production application with great success and stability, but it is not without it’s shortcomings.

The current version is a gradually evolved product and the need for a significant rewrite increases with each added feature. Some of the separate modules are quite tightly coupled; this could be significantly reduced with a better design. Subtle temporal couplings in panes can also be quite difficult to debug.

All this aside, we think even the current version knockout.composite offers some brilliant features and makes building high performance single page web apps easier than ever before. The next version will offer improvements in every area with increased flexibility and extensibility. If you’re interested in contributing, send me a tweet and we’ll take it from there.

*UPDATE*: v0.2 is coming, and it’s part of a much bigger vision we want you to see – get a glimpse here.

What’s Coming?

Devices, Devices, Devices

As mentioned in part 1, we think knockout.composite has great potential within the hybrid web / native mobile application space. Our focus over the next few months is to fine tune the whole experience for mobile devices and develop a set of themes for each operating system. Think single, highly testable codebase for every device, with the bonus of decent performance.

SOA

The composite application pattern is ideal for service oriented architectures. A autonomous business service can easily “own” application components and have them stitched in to applications simply and easily. Seamlessly integrating client and server side message buses makes developing client applications for message based architectures a breeze.

We’ll be releasing libraries to enable this seamless integration between knockout.composite and a range of messaging platforms, like NServiceBus, Azure Service Bus, etc.

Control Libraries

We’ve already got a stack of reusable controls, we intend to release these under an open source license for use in your application. CSS drives the look and feel, so customising each is straightforward.

Amongst Other Things…

Like state machines for navigation, extensible support for validation libraries, better memory management and debugging support, more out of the box transitions, more flexibility and extensibility. The list goes on.

Stay Tuned!

In the meantime, drop me a line and let me know what you think, get involved and help make the web a better place to develop.

knockout.composite: Pack Your App and Make it Fly

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference. PackScript also completely supersedes resPack – more details at http://packscript.com/.

This is part 8 of a multi-part post. It’s assumed that you have a basic knowledge of the data binding and observability features of knockout.js and have read the example presented in part 4.

Part 1: Introducing knockout.composite
Part 2: The Basics
Part 3: Getting Around
Part 4: Putting it All Together
Part 5: Unit Testing: Model Citizens 
Part 6: Integration Testing: Playing Nicely Together
Part 7: Functional Testing: Automated UI Tests, Anywhere, Any Time
Part 8: Pack Your App and Make it Fly
Part 9: Parting Thoughts and Future Directions

 

Throughout this series of posts, we have completely ignored performance. As far as rendering goes, the framework does not add significant overhead. How you use knockout and how you structure your HTML and CSS have a far bigger impact on performance.

What about loading resources? You may have noticed in the examples that we’re generating quite a large number of HTTP requests, and well… they’re pretty slow in the grand scheme of things.

The examples are also fairly trivial, what happens when we have 30 panes on a single screen? 3 requests per pane (html / js / css) gives a total of 90 HTTP requests, just to display one screen!

Enter resPack

resPack is a small .NET console application that combines and minifies JavaScript and CSS files based on simple XML configuration files. It

  • provides simple templating capabilities that allow us to inject things like the source file path (hello, sourceURL tag),
  • has a “watch” mode where it will execute immediately when files change, giving us the ability to keep our code in separate files but still have a simple save-refresh debugging experience,
  • executes recursively, so we can be working in a separate library, save, refresh our app that depends on it and all is kept up to date,
  • uses the Microsoft AjaxMin library for minification. It’s quite efficient and causes minimal side effects.

Let’s look at some example configuration files.

Basic Options
<resPack>
  <output filename="../Content/Panes.css">
    <include files="*.css" recursive="true" />
    <exclude files="Common\*.css" />
  </output>
</resPack>

I don’t think I need to explain this.

Minifying

This is currently only supported for JavaScript files, but the AjaxMin library does support CSS, so this feature will be added soon.

<output filename="../Content/Panes.js" minify="true">
  <include files="*.js" recursive="true" />
</output>

An option on the output turns minification on. This will generate an additional file with the extension .min.js, in this example, Panes.min.js.

A Template for Templates
<output filename="../Content/Panes.htm">
  <include files="*.htm" recursive="true" />
  <template>
&lt;script type="text/template" id="Panes-{2}">{0}&lt;/script>
  </template>
</output>

This combines all of our HTML files for our panes and wraps them each in a script tag. The {0} token substitutes the actual file contents and {2} substitutes a formatted version of the path suitable for HTML attributes.

A Special Case for Models

For our models, we need to tell knockout.composite the path to the model and add in the sourceURL tag to enable debugging support in Chrome.

<output filename="../Content/Panes.js" minify="true">
  <include files="*.js" recursive="true" />
  <template>
{0}
ko.composite.resources.assignModelPath('/{3}');
//@ sourceURL=/Panes/{1}
  </template>
</output>


Loading it All Up

Now we’ve got our combined and minified files, how do we load them up? For CSS, we don’t need to do anything special, just add the usual link element in your header. We can also do this for our models, but we lose debugging support. This is perfectly acceptable for production.

knockout.composite gives us a simple initialise helper function that will load up these resources and begin the rendering process.

ko.composite.initialise(null, { scripts: ['/Content/Debug.js'], templates: ['/Content/Site.htm'] });

This ensures our models are loaded up with full debugging support and our templates are loaded and rendered to the header of the page. To achieve full debugging support, the framework splits JavaScript files based on the sourceURL tag and executes each script in a separate eval statement.

At this stage, Chrome is the only browser that provides full debugging support. Firebug apparently supports the sourceURL tag, but it doesn’t work in this case and I haven’t had the time to determine why. For other browsers, it makes debugging significantly easier turning this option off.

ko.composite.options.debug.splitScripts = false;

This will leave all of our script in a single file, allowing us to use the debugger’s search function to locate specific sections of code and place breakpoints.

Being Selective

Loading up your entire app up front works great for small to medium applications, but as they start to grow, file sizes may start to cause unacceptable load times. We can selectively leave out panes from our resPack configuration files. If a resource isn’t loaded on startup, it will be dynamically loaded when required.

The next version of knockout.composite will provide a mechanism for specifying different load strategies, such as a basic subset up front, by folder when requested, and even in the background after your app has fully loaded.

Make it Fly

As all of our application resources are static text files, there are some things we can do to really boost load times.

Gzip Compression

Most web servers can be configured to compress static text files on first request and transmit the gzipped version. Since our files are plain text, this is a significant saving. IIS requires special configuration to force it to compress on first request. Stackoverflow has the answers…

Deploying to a CDN

A Content Delivery Network replicates your files on a number of servers around the world that are strategically located to minimise the number of hops required for users in various places around the world.

Be aware that this introduces a number of complexities around deployment and versioning. It also requires your service layer to support cross domain requests, though JSONP makes this relatively simple.

Conclusion…

The tool is available in the Tools folder in the knockout.composite github repository. I have not yet released the source, but will do so soon, tweet me if you’re interested. The code is in need of an overhaul and I’d like to be able to specify configuration in JSON format as well as XML.

Next up, finally, a conclusion. Of sorts. For now.

knockout.composite: Functional Testing: Automated UI Tests, Anywhere, Any Time

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference.

This is part 7 of a multi-part post. It’s assumed that you have a basic knowledge of the data binding and observability features of knockout.js and have read parts 4 through 6.

Part 1: Introducing knockout.composite
Part 2: The Basics
Part 3: Getting Around
Part 4: Putting it All Together
Part 5: Unit Testing: Model Citizens 
Part 6: Integration Testing: Playing Nicely Together
Part 7: Functional Testing: Automated UI Tests, Anywhere, Any Time
Part 8: Pack Your App and Make it Fly
Part 9: Parting Thoughts and Future Directions

 

The JavaScript / HTML / CSS stack also makes it quite easy to perform automated UI testing – simply open your deployed app in an iframe element and we can then manipulate the UI with JavaScript, all driven by your favourite unit testing framework. This works very well in practice. Check out the yoursports.net functional test suite here and here. Click on each test to see the series of steps that was performed.

knockout.composite gives us an additional layer of useful stuff to plug into. Using the rendered and paneRendered events that are published on the event bus, we can ensure that our application is ready for manipulation and assertions.

A significant challenge to overcome is dealing with the asynchronous nature of applications. Trying to write this stuff in a procedural style with JavaScript ends up being a deeply nested mess of callbacks and deferred objects.

Instead, creating an API and specifying tests as a set of steps that are queued up and executed serially is a much simpler approach, and also allows reusing these sets of steps. Here is an example from the yoursports.net test suite.

var logon = [
    click(tabButton('Log On')),
    waitForVisible('.logon'),
    setValue('.logon input:eq(0)', 'demo'),
    setValue('.logon input:eq(1)', 'qweqwe'),
    click('.logon button:contains("Log On")'),
    waitForPane('/Team/details')
];

var logoff = [
    click(tabButton('Account')),
    waitForVisible('.account'),
    click('.account a:contains("Log off")'),
    waitForNotVisible('.contentHeader a:contains("Enter Scoresheet")')
];

functionalTest('Logon / Logoff', [
    logon,
    textEqual('.heading:eq(0) span', 'Team: Hit and Run', 'Correct team loaded'),
    logoff
]);

We can see that the API consists of a bunch of functions that correspond to either actions or assertions, and an entry point function, functionalTest.

Under the Covers

The functionalTest function performs all necessary setup for the test, primarily creating the iframe and waiting for our application to finish rendering. It accepts a test name and an array of steps. Each step can be either another array of steps or a function that returns another function to be queued for execution later.

If you’re not familiar with JavaScript, that might sound a bit confusing, but it is one of the powerful features of the language. For an in depth discussion of function currying in JavaScript, check out this Crockford article.

The functionalTest function also ensures that qunit if started once all steps have completed or an uncaught exception occurs.

Step by Step

So each individual step consists of a function that returns a function. Let’s have a look at an example.

function setValue(selector, value) {
    return function () {
        singleExists(selector)();
        ok(true, "Setting value of '" + selector + "' to " + value);
        $$(selector).val(value).change();
    };
};

This example sets the value of an element with a specified selector. The initial setValue call “loads up” our internal function with the necessary arguments and returns the internal function to the test framework for execution in series.

The singleExists function is another “step” function that asserts that only one and only one element exists with the specified selector. The $$ variable gives us access to the jQuery instance inside our iframe.

We can also return a jQuery deferred object from each step and the test framework will wait until the deferred is either resolved or rejected. We can use this to wait for specific events like rendering completion and elements becoming visible, etc. The implementation is somewhat beyond the scope of this article.

Builds, Environments and Browsers

The yoursports.net tests are run against a deployed service layer and a predefined set of data, so we are testing the integration of the entire system, from UI through to the data store. They can easily be deployed with the application and run as part of a build, targeting any environment and run from multiple browsers.

Part of the roadmap is for a simple test agent that can execute these tests in multiple browsers, taking screenshots at specified points in time, gather timings and provide an interface for reporting results.

Conclusion

I have to say at this point that this is still in an experimental stage. While it does work well, it can be somewhat painful to debug. Subtle temporal couplings can make things incredibly confusing! This will improve drastically in the next version of knockout.composite.

I have not yet made the source code for this available on github, but will do so in the near future. If you’re particularly keen, drop me a tweet.

Next time, the last piece, at least for now. Phew!

Thursday, September 13, 2012

knockout.composite: Integration Testing: Playing Nicely Together

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference.

This is part 6 of a multi-part post. It’s assumed that you have a basic knowledge of the data binding and observability features of knockout.js and have read parts 4 and 5.

Part 1: Introducing knockout.composite
Part 2: The Basics
Part 3: Getting Around
Part 4: Putting it All Together
Part 5: Unit Testing: Model Citizens 
Part 6: Integration Testing: Playing Nicely Together
Part 7: Functional Testing: Automated UI Tests, Anywhere, Any Time
Part 8: Pack Your App and Make it Fly
Part 9: Parting Thoughts and Future Directions

 

The JavaScript / HTML / CSS technology stack has some serious advantages over traditional desktop technology stacks when it comes to testing, particularly the fact we can run our tests in any browser, from anywhere, in any environment (dev / test / prod / etc) and with no extra software. Additionally, the highly integrated nature of JavaScript and HTML make it trivial to perform assertions against rendered markup.

So what exactly do I mean by integration testing?

knockout.composite allows us to render a single pane into any element. This may consist of a number of child panes, so we can test the interactions between these panes. We can also make assertions about the rendered UI, testing the integration between models and views. Let’s have a look at some examples.

Some infrastructure…

Before we start rendering panes and making assertions against results, we need a mechanism for rendering the pane and waiting for the render operation to complete before performing any assertions. Unfortunately, due to some calls to setTimeout, we lose the ability to run our tests synchronously, so we need to make use of qunit’s support for asynchronous tests.

We’ll use the addPane utility function to render the pane and then wait for the childrenRendered message to be published before we start making assertions.

function render(options, tests) {
    var pane = ko.composite.utils.addPane($('#qunit-fixture')[0], options);
    pubsub.subscribe('childrenRendered', function () {
        tests();
        start();
    });
    return pane;
};

You can see the tests in action here.

A simple test

We’re going to perform some integration tests against the webmail sample we built in part 4. We can use the same configuration file and index.html as we did for unit testing. Let’s see how it works.

asyncTest("Folders pane renders the correct number of buttons", function() {
    render('folders', function() {
        equal($('.folders li').length, 4);
    });
});

Looks pretty simple. Render the pane. Make sure there are four list items.

Interacting with the UI

Since we’ve got our pane rendered and sitting there in our browser, why not poke it a bit?

asyncTest("Clicking a folder name sets the selected class", function () {
    render('folders', function () {
        ok($('.folders li:contains("Sent")')
            .click()
            .hasClass('selected'));
    });
});
Accessing the model

The pane object exposes a model property that contains the underlying model. Use this if you want to make assertions against the model after rendering the pane.

asyncTest("Clicking a folder name sets the selectedFolder property", function () {
    var pane = render('folders', function () {
        $('.folders li:contains("Sent")').click();
        equal(pane.model.selectedFolder(), 'Sent');
    });
});

Accessing the model of child panes can be done by using the knockout function ko.dataFor.

Doing More

That’s all I’m going to show for now, but this is just a small taste of what can be tested with this approach. See the tests in action here.

Next Up!

Next post, we’re going to take a quick look at some complete, automated, end to end functional testing for your application.

Wednesday, September 12, 2012

knockout.composite: Unit Testing: Model Citizens

*UPDATE* Tribe is here! Check out http://tribejs.com/ for guides and API reference.

This is part 5 of a multi-part post. It’s assumed that you have a basic knowledge of the data binding and observability features of knockout.js and have read part 4.

Part 1: Introducing knockout.composite
Part 2: The Basics
Part 3: Getting Around
Part 4: Putting it All Together
Part 5: Unit Testing: Model Citizens 
Part 6: Integration Testing: Playing Nicely Together
Part 7: Functional Testing: Automated UI Tests, Anywhere, Any Time
Part 8: Pack Your App and Make it Fly
Part 9: Parting Thoughts and Future Directions

 

knockout.composite gives you the ability to easily unit test your individual models, test the integration between panes and perform full functional testing of your application, all from within a browser.

This post covers the first of these – unit testing your models. We’ll be using qunit (the JavaScript unit testing framework used by jQuery), the excellent sinon.js mocking library and mockjax for stubbing out ajax calls.

Unit Testing Your Models

For this example, we’re going to write some tests against models from the webmail sample we wrote in part 4. You can see the tests in action and the complete source for the tests on github.

The index.html file contains the necessary script and CSS references to fire up qunit and knockout.composite.

As we’re testing panes from the webmail sample, we need to let knockout.composite know where to load panes from. We’ll also tell it to load things synchronously. This simplifies our tests considerably. We’ve put this in a separate configuration file in the Resources folder.

ko.composite.options.basePath = '../6. Webmail/4. The Final Pieces/';
ko.composite.options.synchronous = true;
Creating models

So how do we load up our model and create an instance of it? The ko.composite.resources namespace contains a function called loadModel that loads the model for us, then we can just create an instance of it. Remember, model constructors take three arguments – pubsub, data, pane.

ko.composite.resources.loadModel('folders');
var model = new ko.composite.models['folders'].constructor(pubsub, data, pane);

We’ve separated that out into a helper function called createModel to simplify our code a bit.

A simple test

This is a simple test against the model for the folders pane. It asserts that the value of an observable changes when the selectFolder function is called.

test("selecting a folder sets selectedFolder observable", function () {
    var pubsub = { publish: function () { } };
    var model = createModel('folders', pubsub);
    model.selectFolder('Sent');
    equal(model.selectedFolder(), 'Sent');
});

Pretty simple stuff. Create the model, call the selectFolder function and assert that the value of the selectedFolder observable has been set. The pubsub variable is created because we know that the selectFolder function calls the pubsub.publish function. If this function doesn’t exist, an exception will occur.

Making sure messages are published

A sinon spy is perfect for this. This time, instead of an empty function, we’ll attach a spy to the publish function. The spy gives us access to information about calls made and arguments passed.

test("selecting a folder publishes folderSelected method", function () {
    var pubsub = { publish: sinon.spy() };
    var model = createModel('folders', pubsub);
    model.selectFolder('Sent');

    ok(pubsub.publish.calledOnce);
    equal(pubsub.publish.firstCall.args[0], 'folderSelected');
    equal(pubsub.publish.firstCall.args[1], 'Sent');
});

Our spy has replaced the publish method, so we can assert that the selectFolder function also publishes the folderSelected message with appropriate arguments.

Testing message subscribers

This time, we’re going to test the layout model and it’s message subscription functions. To invoke these, we can create a real instance of pubsub, pass it to the model constructor and call the publish method. We’ve also mocked out the pane.navigate function with a spy.

test("model navigates when folderSelected message is published", function () {
    var pubsub = new PubSub({ forceSync: true });
    var pane = { navigate: sinon.spy() };
    var model = createModel('layout', pubsub, null, pane);
        
    pubsub.publish('folderSelected', 'Sent');
    ok(pane.navigate.calledOnce);
    equal(pane.navigate.firstCall.args[0], 'mails');
    deepEqual(pane.navigate.firstCall.args[1], { folder: 'Sent' });
});
Mocking AJAX calls

Last little trick – stubbing out your AJAX requests with mockjax. This is not actually part of knockout.composite, but is useful enough to warrant a mention here. This test exercises the initialise function of the mails pane.

test("initialise loads specified folder and sets data observable", function () {
    $.mockjax({
        url: '../data/folder/Sent',
        responseText: JSON.stringify({ test: 'test' }),
        responseTime: 0
    });

    var model = createModel('mails', null, { folder: 'Sent' });
    model.initialise();
    deepEqual(model.data(), { test: 'test' });
});

Again, pretty simple stuff. Set up the AJAX call, create the model, call stuff, test stuff.

Check the running tests out here.

Until Next Time…

Integration testing up next. See you then!