Universal modules (browser, Node.js): imports and universal tests

[2011-10-20] dev, javascript, jsmodules, jslang

Update 2011-11-19. This post is now superseded by “Bridging the module gap between Node.js and browsers”.

This post explains how to write modules that are universal – they run on browsers and Node.js. A previous post showed a simple way of doing so, this post presents a more sophisticated solution that also handles modules importing other modules. Additionally, we use the unit test framework Jasmine to write tests for those modules that are equally universal.

The universal module pattern

Requirements:

Find a module format that works on browsers and Node.js.
Browsers: allow one to use existing script loaders such as Require.js.
Node.js: let universal modules load Node.js modules, let Node.js modules load universal modules. Then you can use a universal module from normal Node code, without even knowing about universal modules.

There are two kinds of module systems in the JavaScript world: Node.js has a synchronous module system: You ask for a module and get it back immediately. That’s why can you use code like this:

    var format = require("util").format;

In a browser, when you ask for a module, it usually means that it has to be loaded over the internet. While that happens, your the browser goes on to do other things and calls your code back once the module code has arrived. That’s why browser module systems are asynchronous, you have to work with callbacks. AMD (Asynchronous Module Definition) [1] is a standard for specifying asynchronous modules. Universal modules support the following subset of its syntax – a global function define() is used to define a module:

    define(dependencies?, factory);

The optional parameter dependencies is an array of names of modules that the module imports. After the imported modules have been loaded, they are passed to the function factory as parameters. factory then returns the module contents, usually an object. Example of an AMD:

    define([ "./other_module" ],
        function (other_module) {
            return {
                myfunction: function() {
                    other_module.func(...);
                    ...
                }
            };
        }
    );

In browsers, you support AMDs by loading a script that defines a global function define(). To support Node.js as well, universal modules add a prefix to an AMD that enables their use on that platform:

    "use strict";

    (typeof define === "function"
        ? {define:define}
        : require("um_node")._(require, exports)).
    define([ "./other_module" ],
        function (other_module) {
            return {
                myfunction: function() {
                    other_module.func(...);
                    ...
                }
            };
        }
    );

Explanations:

    "use strict";

Enable ECMAScript 5 strict mode [4] for the complete file. Strict mode helps with writing better code, because it eliminates a few quirks and performs more checks.

    (typeof define === "function"
        ? {define:define}
        : require("um_node")._(require, exports)).

If a global function define() already exists, use it, otherwise load a definition from the Node.js module um_node (“universal modules for Node.js”). A more detailed explanation of the above three lines is given in Sect. “Explaining the universal module pattern”, below.

    define([ "./other_module" ],
        function (other_module) {

Import a module called ./other_module. After it has been loaded it is handed to the function in the second line as the parameter other_module.

    return {
        myfunction: function() {
            other_module.func(...);
            ...
        }
    };

This is the actual module. Caveat: In a module method, you cannot use this to refer to sibling methods, because such a method might not be invoked as a method, but as a function:

    var f = mymodule.myfunction;
    f();

Then it can’t reach its siblings via this. A work-around is to assign the module object to a variable before returning it:

    var module = {
        func1: function() {
            ...
        },
        func2: function() {
            module.func1();
        }
    };
    return module;

Module names. The legal names of universal modules are a subset of Node’s. The file extension ".js" is always omitted. There are three kinds of names:

Global module names: a single identifier, without dots or slashes. Example: "bar".
Such names can refer to built in “core” modules. If not, Node searches directories relative to the current module. For example, if the current module is /home/joe/js/foo.js and imports the module "bar", then Node searches for the following files:
```
    /home/joe/js/node_modules/bar.js
    /home/joe/node_modules/bar.js
    /home/node_modules/bar.js
    /node_modules/bar.js
```
The browser AMD loader that comes with the universal_modules project lets you map global module names to paths (as searching the file system in the above manner makes less sense in a browser). That allows one to use a single name for a module that has a browser-specific and a Node-specific implementation. See below for details.
Relative module names: "./" or "../" (repeated one or more times), followed by a sequence of identifiers separated by slashes. Examples: "./bar", "./bar/baz", "../bar", "../../bar".
Their names interpreted as paths relative to the path of the current module. Example: resolving relative module names.
```
    /home/joe/js/foo.js – path of current module
    
    ./bar     → /home/joe/js/bar.js
    ./bar/baz → /home/joe/js/bar/baz.js
    ../bar    → /home/joe/bar.js
    ../../bar → /home/bar.js
```
Absolute module names: a slash followed by one or more identifiers separated by slashes. Examples: "/bar", "/bar/baz".
The given name is used as an absolute path to find the JavaScript file.

Accessing globals. On browsers, you can access the global object (with all global variables in it) via the variable window or via this – if you are at the top level. On Node.js, you are never at the top level and window does not make sense. Therefore, it introduces the variable global to do so. um_browser.js creates this variables on browsers, if you use another AMD library, you will have to create it yourself:

    <script>
        if (!window.global) {
            window.global = window;
        }
    </script>

It would be nice to achieve the above task without window, but there is no easy way to do so.

Installation. Download the universal modules project. Copy the directory node_modules/ to your project.

Using universal modules

Given a directory with the following file structure:

    ├── client.html
    ├── mymodule.js
    ├── node_modules/
    │   ├── um_browser.js
    │   └── um_node.js
    └── server.js

The directory node_modules/ contains the universal module support. mymodule.js is a universal module with the following content:

    "use strict";

    (typeof define === "function"
        ? {define:define}
        : require("um_node")._(require, exports)).
    define(function () {
            return {
                twice: function(value) {
                    return value + value;
                }
            };
        }
    );

How can we use that module in a browser and on Node.js?

Browser. client.html uses that module from a browser:

    <!doctype html>
    <html>
        <head>
            <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
            <title>My web page</title>
1:          <script type="text/javascript" src="node_modules/um_browser.js"></script>
2:          <script>
2:              um_browser.globalNames.foo = "../path/to/foo.js";
2:              // alternative: um_browser.globalNames = function (moduleName) { ... };
2:          </script>
        </head>
        <body>
3:          <script>
3:              define([ "./mymodule" ],
3:                  function(mymodule) {
3:                      alert("Twice foo: "+mymodule.twice("foo"));
3:                  }
3:              );
3:          </script>
        </body>
    </html>

The core ingredients are:

Load um_browser.js so that you can define and import universal modules.
Set up global names (optional): You use um_browser.global_names to resolve global module names.
- The result of a resolution is either a path to a script to be loaded or an object (which is the module).
- The resolution can be performed by an object (mapping module names to paths or objects) or a function (taking a module name and returning a path or an object).
Define a module that imports mymodule.

Node.js. server.js uses that module from Node.js.

    var twice = require("./mymodule").twice;

    console.log("Twice foo: "+twice("foo"));

Note that mymodule is used like any other Node.js module. Alternatively, you could have written server.js as a universal module.

server.js is run from a command line via

    node server.js

Explaining the universal module pattern

Given an AMD.

    define([ "./other_module" ],
        function (other_module) {
            return {
                ...
            };
        }
    );

To fulfill the previously mentioned requirements, we have to make sure that browsers and Node.js understand the above AMD. On browsers, we simply assume that a global function define() has been defined, e.g. via a library. On Node.js, the goal is that a universal module can be loaded as either an AMD (via other AMDs) or a normal Node module.

Using an AMD on Node.js. On Node.js, we need to connect the AMD to the native module system. Possible solutions:

Declare a local variable.
```
    var define = (typeof define === "function"
        ? define
        : require("um_node")._(require, exports));
    define(...);
```
In browsers, there already is a define() function. Therefore, we check whether it exists. If not, we create a Node.js version of define(). To do so, we use the module um_node (“universal modules for Node.js”, see below for details on its contents). Each Node.js module needs a custom definition of define(), because that function must use the module-specific require() function and exports object. For require(), it is obvious why it is module-specific: If you are in a module A and load another module B via a module-relative path, then require() must know A’s absolute path to compute B’s absolute path. Node.js provides modules with require() functions that have this knowledge. Note that newer versions of Node.js allow one to access both require and exports via the module-specific variable module, but we also want to support older versions (where there is only module.require).
Main problem with this approach: Hoisting [5] can prevent this from working – it means that the above code is usually interpreted as
```
    var define;
    define = (typeof define === "function"
        ? define
        : require("um_node")._(require, exports));
    define(...);
```
Then typeof define will always be "undefined".
Define a global function.
```
    if (typeof define !== "function") {
        global.define = require("um_node")._(require, exports);
    }
    define(...);
```
The main drawback of this approach is that using a global variable this way is especially inelegant in a module system like Node’s which is all about local state. Constantly giving it a new value is even worse.
Use an Immediately Invoked Function Expression (IIFE). For example:
```
    (function (define) {
        define(...);
    }(typeof define === "function"
        ? define
        : require("um_node")._(require, exports));
```
The above is almost the same as defining a local variable, above. But this time, we create a new scope for that variable to live in. This is a very clean solution. However, it is a bracket around the AMD. It would be nicer if we could do with just a prefix.
Turn define() into a method:
```
    (typeof define === "function"
        ? {define:define}
        : require("um_node")._(require, exports)).
    define(...);
```
Granted, this changes the nature of define() (from a function to a method), but it is still just a prefix. If the global function define() already exists, we create a temporary object with that function as a single method. Otherwise, we let module um_node create such an object, via its _() function.

AMD support for browsers

Two popular AMD-compatible script loaders are:

The universal modules project comes with a no-frills AMD-compatible script loader (um_browser.js). This section explains the challenges in writing one for the browser.

The basic way of loading a script is inspired by Nicholas C. Zakas: You insert a script tag into the DOM to trigger the asynchronous loading of a script.

    function loadScript(url, callback) {
        var script = document.createElement("script")
        script.type = "text/javascript";

        if (script.readyState) { // IE
            script.onreadystatechange = function () {
                if (script.readyState === "loaded" ||
                        script.readyState === "complete") {
                    script.onreadystatechange = null;
                    callback();
                }
            };
        } else { // Others
            script.onload = function(){
                callback();
            };
        }

        script.src = url;
        document.getElementsByTagName("head")[0].appendChild(script);
    }

Problem: you cannot easily pass arguments to a script you load. But you need to do so to resolve relative module paths, as modules don’t know their own path. Only the script that loads them does. This assumes that it all starts in an HTML file whose path is "" (“here”, against which relative paths can be resolved).
Solution: Store the unevaluated module data somewhere when the script is executed. Next, process that data via the onload handler of the script tag (which is guaranteed to run directly after the script has been loaded). That handler is set by the loader, so one can pass parameters on to it.
Problem: The very first define() must execute the module right away, it cannot store it away for later processing, because there is no onload handler and thus no later processing.
Solution: Count how many modules are currently being loaded. If that count is zero then the module can be evaluated at once.

AMD support for Node.js

The file um_node.js in the universal modules project contains the following code:

    exports._ = function(clientRequire, clientExports) {
        return {
            define: function (importNames, moduleBody) {
                if (arguments.length === 1 && typeof importNames === "function") {
                    moduleBody = importNames;
                    importNames = [];
                }
            
                // Step 1
                var imports = importNames.map(function (x) { return clientRequire(x) });
                
                // Step 2
                var data = moduleBody.apply(null, imports);
                
                // Step 3
                if (data) {
                    Object.keys(data).forEach(function(key) {
                        clientExports[key] = data[key];
                    });
                }
            }
        };
    }

The function that provides the support is called _.

Its input are the values of require and exports from the module that invokes it (the “client module”). You need those values to perform imports and exports for the client. Why imports are needed is not immediately obvious – it is because relative module names must be resolved with the client as the base.
Its output is an object with a define() method that works for the client module.

That method performs three steps:

Step 1: Convert the names of the imported modules to objects, via clientRequire.
Step 2: Call the body of the module that is to be defined, hand in the module objects.
Step 3: If the AMD exports anything, then copy the exports to the clientExports object.

Universal unit tests

We use Jasmine to write unit tests for a universal module that are equally universal. Read [6] as an introduction if you are not familiar with it. The key insight is that a Jasmine test spec becomes universal if you implement it as a universal module. The module to be tested then becomes an import.

The spec: universal_modules/demo/spec/strset.spec.js

    "use strict";

    (typeof define === "function" ? {define:define} : require("um_node")._(require, exports)).
    define([ "../strset" ], // the universal to module to be tested
        function (strset) {
            describe('strset', function () {
                it('creates sets via the constructor', function () {
                    var sset = new strset.StringSet("a");
                    expect(sset.contains("a")).toBeTruthy();
                    expect(sset.contains("b")).toBeFalsy();
                });
                ...
            });
        }
    );

The above code tests the universal module universal_modules/demo/strset.js and can be run directly via jasmine-node:

    $ jasmine-node strset.spec.js 
    Started
    ...

    Finished in 0.001 seconds
    1 test, 5 assertions, 0 failures

Running the spec in a browser: universal_modules/demo/spec/strset.runner.html
The pertinent lines are:

    <!-- The code to be tested and the tests -->
    <script type="text/javascript" src="../../node_modules/um_browser.js"></script>
    <script type="text/javascript" src="strset.spec.js"></script>

You only need to load two scripts: First, um_browser.js to ensure that universal modules work. Second, strset.spec.js to load the code to be tested and to run the tests.