Making transpiled ES modules more spec-compliant

[2017-01-20] dev, javascript, esnext, babel, jsmodules
(Ad, please don’t block)

A proposed “spec mode” for Babel makes transpiled ES modules more spec-compliant. That’s a crucial step in preparing for native ES modules. You’ll also learn how ES modules and CommonJS modules will interoperate on Node.js and how far along ES module support is on browsers and Node.js.

Update 2017-05-08: follow-up blog post:

Transpiling ES modules to CommonJS via Babel  

At the moment, the main way to use ES modules on Node.js and browsers is to transpile them to CommonJS modules via Babel.

The benefit of this approach is that integration with the CommonJS ecosystem, including npm modules, is seamless.

On the flip side, the code that Babel currently generates does not comply with the ECMAScript specification. That is a problem, because code that works with Babel now, won’t work as native modules.

That’s why Diogo Franco has created a pull request that adds a so-called “spec mode” to transform-es2015-modules-commonjs. Modules transpiled in this mode conform as closely to the spec as is possible without using ES6 proxies. The downside is that the only way to access normal (untranspiled) CommonJS modules is via a default import.

Spec mode is switched on like this:

{
    "presets": [
        ["es2015", { "modules": false }]
    ],
    "plugins": [
        ["transform-es2015-modules-commonjs", {
            "spec": true
        }]
    ]
}

How does the spec mode work?  

In this section, I explain where current transpilation deviates from ES module semantics and how the spec mode fixes that.

ES module imports are live views of exports  

In an ES module, the imports are live views of the exported values. Babel simulates that in CommonJS in two steps.

Step 1: keep variables and exports in sync. Whenever you update an exported variable foo, Babel currently also updates the corresponding property exports.foo, as you can see in lines A, B and C.

// Input
export let foo = 1;
foo = 2;
function bar() {
  foo++;
}

// Output
Object.defineProperty(exports, "__esModule", {
  value: true
});
var foo = exports.foo = 1; // (A)
exports.foo = foo = 2; // (B)
function bar() {
  exports.foo = foo += 1; // (C)
}

The marker property __esModule lets importing modules know that this is a transpiled ES module (which matters especially for default exports).

Spec mode stays much closer to the specification by implementing each export as a getter (line A) that returns the current value of the exported variable. This looks as follows.

const exports = module.exports = Object.create(null, {
    __esModule: {
        value: true
    },
    [Symbol.toStringTag]: {
        value: 'Module'
    },
    foo: {
        enumerable: true,
        get() { return foo; }
    },
});
Object.freeze(exports);

let foo = 1;
foo = 2;
function bar() {
  foo++;
}

Each property in the second argument of Object.create() is defined via a property descriptor. For example, __esModule is a non-enumerable data property whose value is true and foo is an enumerable getter.

If you use spec mode without transpiling let (as I’m doing in this blog post), exports will also handle the temporal dead zone correctly (you can’t access an export before its declaration was executed).

The object stored in exports is an approximation of an ECMAScript module record, which holds an ES module plus its metadata.

Step 2: The transpiled non-spec-mode Babel code always refers to imports via the imported module. It never stores their values in variables. That way, the live connection is never broken. For example:

// Input
import {foo} from 'bar';
console.log(foo);

// Output
var _bar = require('bar');
console.log(_bar.foo);

Spec mode handles this step the same way.

In ES modules, imported namespace objects are immutable and have no prototype  

Without spec mode, the transpiled Babel code lets you change properties of imported namespace objects and add properties to them. Additionally, namespace objects still have Object.prototype as their prototype when they shouldn’t have one.

// otherModule.js
export function foo() {}

// main.js
import * as otherModule from './otherModule.js';

otherModule.foo = 123; // should not be allowed
otherModule.bar = 'abc'; // should not be allowed

// proto should be null
const proto = Object.getPrototypeOf(otherModule);
console.log(proto === Object.prototype); // true

As previously shown, spec mode fixes this by freezing exports and by creating this object via Object.create().

In ES modules, you can’t access and change exports  

Without spec mode, Babel lets you add things to exports and work around ES module exporting:

export function foo() {} // OK
exports.bar = function () {}; // should not be allowed

As previously shown, spec mode prevents this by freezing exports.

In ES modules, you can only import what has been exported  

In non-spec mode, Babel allows you to do the following:

// someModule.js
export function foo() {}

// main.js
import {bar} from './someModule.js'; // should be error
console.log(bar); // undefined

In spec mode, Babel checks during importing that all imports have corresponding exports and throws an error if they don’t.

An ES module can only default-import CommonJS modules  

The way it looks now, ES modules in Node.js will only let you default-import CommonJS modules:

import {mkdirSync} from 'fs'; // no
mkdirSync('abc');

import fs from 'fs'; // yes
fs.mkdirSync('abc');

import * as fs from 'fs';
fs.mkdirSync('abc'); // no
fs.default.mkdirSync('abc'); // yes

That is unfortunate, because it often does not reflect what is really going on – whenever a CommonJS module simulates named exports via an object. As a result, turning such a module into an ES module means that import statements have to be changed.

However, it can’t be helped (at least initially), due to how much the semantics of both kinds of module differ. Two examples:

  • The previous subsection mentioned a check for declarative imports – they must all exist in the modules one imports from. This check must be performed before the body of the module is executed. You can’t do that with CommonJS modules, which is why you can only default-import them.

  • module.exports may not be an object; it could be null, undefined, a primitive value, etc. A default import makes it easier to deal with these cases.

In non-spec mode, all imports shown in the previous code fragment work. Spec mode enforces the default import by wrapping imported CommonJS modules in module records, via the function specRequireInterop():

function specRequireInterop(obj) {
    if (obj && obj.__esModule) {
        // obj was transpiled from an ES module
        return obj;
    } else {
        // obj is a normal CommonJS module,
        // wrap it in a module record
        var newObj = Object.create(null, {
            default: {
                value: obj,
                enumerable: true
            },
            __esModule: {
                value: true
            },
            [Symbol.toStringTag]: {
                value: 'Module'
            },
        });
        return Object.freeze(newObj);
    }
}

ES module specifiers are URLs  

ES modules treat all module specifiers as URLs (much like the src attribute in script elements). That leads to a variety of issues: ending module specifiers with .js may become common, the % character leads to URL-decoding, etc.

Import path resolution in Node.js  

Importing modules statically (import statements) or dynamically (import() operator) resolves module specifiers roughly the same as require() (source):

import './foo';
// looks at
//   ./foo.js
//   ./foo/package.json
//   ./foo/index.js
//   etc.

import '/bar';
// looks at
//   /bar.js
//   /bar/package.json
//   /bar/index.js
//   etc.

import 'baz';
// looks at:
//   ./node_modules/baz.js
//   ./node_modules/baz/package.json
//   ./node_modules/baz/index.js
// and parent node_modules:
//   ../node_modules/baz.js
//   ../node_modules/baz/package.json
//   ../node_modules/baz/index.js
//   etc.

import 'abc/123';
// looks at:
//   ./node_modules/abc/123.js
//   ./node_modules/abc/123/package.json
//   ./node_modules/abc/123/index.js
// and ancestor node_modules:
//   ../node_modules/abc/123.js
//   ../node_modules/abc/123/package.json
//   ../node_modules/abc/123/index.js
//   etc.

The following non-local dependencies will not be supported by ES modules on Node.js:

  • $NODE_PATH
  • $HOME/.node_modules
  • $HOME/.node_libraries
  • $PREFIX/lib/node

require.extensions won’t be supported, either.

As far as URL protocols go, Node.js will support at least file:. Browsers support all protocols, including data:.

Import path resolution in browsers  

In browsers, the resolution of module specifiers will probably continue to work as they do when you use CommonJS modules via Browserify and webpack:

  • You will install native ES modules via npm.
  • A module bundler will transpile modules. At the very least it will convert Node.js-style specifiers ('baz') to URLs ('./node_modules/baz/index.js'). It may additionally combine multiple ES modules into either a single ES module or a custom format.
  • As an alternative to transpiling modules statically, it is conceivable that you’ll be able to customize a module loader in a manner similar how RequireJS does it: mapping 'baz' to './node_modules/baz/index.js', etc.

If we are already transpiling module specifiers, it’d be nice if we could also have variables in them. The main use case being going from:

import '../../../util/tool.js';

to:

import '$ROOT/util/tool.js';

Open issue: distinguishing ES modules from other JavaScript files  

With ES modules, there are now two kinds of files in JavaScript:

  • Modules: can declaratively import modules, live in a module-local scope, etc.
  • Scripts: cannot declaratively modules, live in global scope, etc.

For more information consult Sect. “Browsers: asynchronous modules versus synchronous scripts” in “Exploring ES6”.

Both files are used differently and their grammars differ. However, as of now, there is overlap: some files can be either scripts or modules. For example:

console.log(this === undefined);

In order to execute this file correctly, you need to know whether it is a script (in which case it logs false) or a module (in which case it logs true).

Browsers  

In browsers, it is always clear whether a file is a script or a module. It depends on how one refers to it:

  • Script: <script src="foo.js">
  • Module: <script src="foo.js" type=module>
  • Module: import 'foo.js';

Node.js  

In Node.js, the plan is to allow declarative imports of CommonJS files. Then one has to decide whether a given file is an ES module or not.

Two approaches for doing so were rejected by the Node.js community:

  • Marking ES modules via "use module";
  • Using metadata in package.json to specify which modules are ESM, as outlined in “In Defense of .js”.

Two other approaches are currently being discussed:

  • Ensure that ES modules and scripts have non-overlapping grammars (details).
  • Give modules the dedicated file name extension .mjs (details).

The former approach is currently being favored, but I prefer the latter approach, because detection would not require “looking into” files. The downside is that JavaScript tools (editors etc.) would need to be made aware of the new file name extension. But they also need to be updated to handle ES modules properly.

How long until we have native ES modules?  

Native ES modules in browsers  

Native ES modules in Node.js  

James M. Snell recently tweeted where Node.js is w.r.t. supporting ES modules:

  • Prerequisite: Several issues need to be sorted out before Node.js can support ES modules – mainly: async vs. sync loading, timing, and the ability to support CommonJS modules.
  • Once these obstacles are removed, the JavaScript engines that Node.js is based on (esp. V8) need to implement the spec changes.
  • Given the time needed for the spec changes and the implementation, support for ES modules in Node.js will, at the earliest, be ready in early 2018. Experimental previews may happen before then, but nothing that is officially supported.

Interoperability looks as follows:

  • ES module (ESM):
    • Import declaratively via import: ESM and CommonJS (default export only)
    • Import programmatically via import(): ESM and CommonJS (module record property default)
    • Import programmatically via require(): ESM (module record) and CommonJS
  • CommonJS module:
    • Import programmatically via import(): ESM and CommonJS (module record property default)
    • Import programmatically via require(): ESM (module record) and CommonJS

Further reading  

Sources of this blog post  

More information on ES modules  

Acknowledgements: Thanks to Bradley Farias and Diogo Franco for reviewing this blog post.