Publishing and consuming ECMAScript modules via packages – the big picture

[2022-01-12] dev, javascript, esm

Updates:

2022-07-23: Documented how to specify a license for a package in package.json
2022-07-22: Complete rewrite of section “Packages: JavaScript’s units for software distribution”.
2022-01-17: Added material on bare specifiers with subpaths that have filename extensions.

The ecosystem around delivering ECMAScript modules via packages is slowly maturing. This blog post explains how the various pieces fit together:

Packages – JavaScript’s units for software distribution
The three kinds of ECMAScript module specifiers
Providing and using packages via module specifiers in Node.js, Deno and web browsers

Required knowledge: I’m assuming that you are loosely familiar with the syntax of ECMAScript modules. If you are not, you can read chapter “modules” in “JavaScript for impatient programmers”.

Packages: JavaScript’s units for software distribution

In the JavaScripte ecosystem, a package is a way of organizing software projects: It is a directory with a standardized layout. A package can contain all kinds of files - for example:

A web application written in JavaScript, to be deployed on a server
JavaScript libraries (for Node.js, for browsers, for all JavaScript platforms, etc.)
Libraries for programming languages other than JavaScript: TypeScript, Rust, etc.
Unit tests (e.g. for the libraries in the package)
Node.js-based shell scripts – e.g., development tools such as compilers, test runners, and documentation generators
Many other kinds of artifacts

A package can depend on other packages (which are called its dependencies):

Libraries needed by the package’s JavaScript code
Shell scripts used during development
Etc.

The dependencies of a package are installed inside that package (we’ll see how soon).

One common distinction between packages is:

Published packages can be installed by us:
- Global installation: We can install them globally so that their shell scripts become available at the command line.
- Local installation: We can install them as dependencies into our own packages.
Unpublished packages never become dependencies of other packages, but do have dependencies themselves. Examples include web applications that are deployed to servers.

The next subsection explains how packages can be published.

Publishing packages: package registries, package managers, package names

The main way of publishing a package is to upload it to a package registry – an online software repository. The de facto standard is the npm registry but it is not the only option. For example, companies can host their own internal registries.

A package manager is a command line tool that downloads packages from a registry (or other sources) and installs them as shell scripts and/or as dependencies. The most popular package manager is called npm and comes bundled with Node.js. Its name originally stood for “Node Package Manager”. Later, when npm and the npm registry were used not only for Node.js packages, that meaning was changed to “npm is not a package manager” (source). There are other popular package managers such as yarn and pnpm. All of these package managers use the npm registry by default.

Each package in the npm registry has a name. There are two kinds of names:

Global names are unique across the whole registry. These are two examples:
```
minimatch
mocha
```
Scoped names consist of two parts: A scope and a name. Scopes are globally unique, names are unique per scope. These are two examples:
```
@babel/core
@rauschma/iterable
```
The scope starts with an @ symbol and is separated from the name with a slash.

The file system layout of a package

Once a package my-package is fully installed, it almost always looks like this:

my-package/
  package.json
  node_modules/
  [More files]

What are the purposes of these file system entries?

package.json is a file every package must have:
- It contains metadata describing the package (its name, its version, its author, etc.).
- It lists the dependencies of the package: other packages that it needs, such as libraries and tools. Per dependency, we record:
  - A range of version numbers. Not specifying a specific version allows for upgrades and for code sharing between dependencies.
  - By default, dependencies come from the npm registry. But we can also specify other sources: a local directory, a GZIP file, a URL pointing to a GZIP file, a registry other than npm’s, a git repository, etc.
node_modules/ is a directory into which the dependencies of the package are installed. Each dependency also has a node_modules folder with its dependencies, etc. The result is a tree of dependencies.

Some packages also have the file package-lock.json that sits next to package.json: It records the exact versions of the dependencies that were installed and is kept up to date if we add more dependencies via npm.

`package.json`

This is a starter package.json that can be created via npm:

{
  "name": "my-package",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

What are the purposes of these properties?

Some properties are required for public packages (published on the npm registry):
- name specifies the name of this package.
- version is used for version management and follows semantic versioning with three dot-separated numbers:
  - The major version is incremented when incompatible API changes are made.
  - The minor version is incremented when functionality is added in a backward compatible manner.
  - The patch version is incremented when small changes are made that don’t really change the functionality.
Other properties for public packages are optional:
- description, keywords, author are optional and make it easier to find packages.
- license clarifies how this package can be used. It makes sense to provide this value if the package is public in any way. “Choose an open source license” can help with making this choice.
main is a property for packages with library code. It specifies the module that “is” the package (explained later in this chapter).
scripts is a property for setting up abbreviations for development-time shell commands. These can be executed via npm run. For example, the script test can be executed via npm run test.

Other useful properties:

dependencies lists the dependencies of a package. Its format is explained soon.
devDependencies are dependencies that are only needed during development.
The following setting means that all files with the name extension .js are interpreted as ECMAScript modules. Unless we are dealing with legacy code, it makes sense to add it:
```
"type": "module"
```
bin lists modules within the package that are installed as shell scripts. Its format is explained soon.
license specifies a license for the package. Its format is explained soon.
Normally, the properties name and version are required and npm warns us if they are missing. However, we can change that via the following setting:
```
"private": true
```
That prevents the package from accidentally being published and allows us to omit name and version.

For more information on package.json, see the npm documentation.

Property `"dependencies"` of `package.json`

This is what the dependencies in a package.json file look like:

"dependencies": {
  "minimatch": "^5.1.0",
  "mocha": "^10.0.0"
}

The properties record both the names of packages and constraints for their versions.

Versions themselves follow the semantic versioning standard. They are up to three numbers (the second and third number are optional and zero by default) separated by dots:

Major version: This number changes when a packages changes in incompatible ways.
Minor version: This number changes when functionality is added in a backward compatible manner.
Patch version: This number changes when backward compatible bug fixes are made.

Node’s version ranges are explained in the semver repository. Examples include:

A specific version without any extra characters means that the installed version must match the version exactly:
```
"pkg1": "2.0.1",
```
major.minor.x or major.x means that the components that are numbers must match, the components that are x or omitted can have any values:
```
"pkg2": "2.x",
"pkg3": "3.3.x",
```
* matches any version:
```
"pkg4": "*",
```
>=version means that the installed version must be version or higher:
```
"pkg5": ">=1.0.2",
```
<=version means that the installed version must be version or lower:
```
"pkg6": "<=2.3.4",
```
version1-version2 is the same as >=version1 <=version2:
```
"pkg7": "1.0.0 - 2.9999.9999",
```
^version (as used in the previous example) is a caret range and means that the installed version can be version or higher but must not introduce breaking changes. That is, the major version must be the same:
```
"pkg8": "^4.17.21",
```

Property `"bin"` of `package.json`

This is how we can tell npm to install modules as shell scripts:

"bin": {
  "my-shell-script": "./src/shell/my-shell-script.mjs",
  "another-script": "./src/shell/another-script.mjs"
}

If we install a package with this "bin" value globally, Node.js ensures that the commands my-shell-script and another-script become available at the command line.

If we install the package locally, we can use the two commands in package scripts or via the npx command.

A string is also allowed as the value of "bin":

{
  "name": "my-package",
  "bin": "./src/main.mjs"
}

This is an abbreviation for:

{
  "name": "my-package",
  "bin": {
    "my-package": "./src/main.mjs"
  }
}

Property `"license"` of `package.json`

The value of property "license" is always a string with a SPDX license ID. For example, the following value denies others the right to use a package under any terms (which is useful if a package is unpublished):

"license": "UNLICENSED"

The SPDX website lists all available license IDs. If you find it difficult to pick one, the website “Choose an open source license” can help – for example, this is the advice if you “want it simple and permissive”:

The MIT License is short and to the point. It lets people do almost anything they want with your project, like making and distributing closed source versions.

Babel, .NET, and Rails use the MIT License.

You can use that license like this:

"license": "MIT"

Archiving and installing packages

Packages in the npm registry are often archived in two different ways:

For development, they are stored in a git repository.
To make them installable via npm, they are uploaded to the npm registry.

Either way, the package is archived without its dependencies – which we have to install before we can use it.

If a package is stored in a git repository:

We normally want the same dependency tree to be used every time we install the package.
- That’s why package-lock.json is usually included.
We can regenerate artifacts from other artifacts – for example, compile TypeScript files to JavaScript files.

If a package is published to the npm registry:

It should be flexible with its dependencies so that upgrading dependencies and sharing packages in a dependency tree becomes possible.
- That’s why package-lock.json is never uploaded to the npm registry.
It often contains generated artifacts - for example, JavaScript files compiled from TypeScript files are included so that people who only use JavaScript don’t have to install a TypeScript compiler.

Dev dependencies (property devDependencies in package.json) are only installed during development but not when we install the package from the npm registry.

Note that unpublished packages in git repositories are handled similarly to published packages during development.

Installing an existing package from git

To install an existing package pkg from git, we clone its repository and:

cd pkg/
npm install

Then the following steps are performed:

node_modules is created and the dependencies are installed. Installing a dependency also means downloading that dependency and installing its dependencies (etc.).
Sometimes additional setup steps are performed. Which ones those are can be configured via package.json.

If the root package doesn’t have a package-lock.json file, it is created during installation (as mentioned, dependencies don’t have this file).

In a dependency tree, the same dependency may exist multiple times, possibly in different versions. There a ways to minimize duplication, but that is beyond the scope of this blog post.

Reinstalling a package

This is a (slightly crude) way of fixing issues in a dependency tree:

cd pkg/
rm -rf node_modules/
rm package-lock.json
npm install

Note that that may result in different, newer, packages being installed. We can avoid that by not deleting package-lock.json.

Creating a new package and installing dependencies

There are many tools and technique for setting up new packages. This is one simple way:

mkdir my-package
cd my-package/
npm init --yes

Afterward, the directory looks like this:

my-package/
  package.json

This package.json has the starter content that we have already seen.

Installing dependencies

Right now, my-package doesn’t have any dependencies. Let’s say we want to use the library lodash-es. This is how we install it into our package:

npm install lodash-es

This command performs the following steps:

The package is downloaded into my-package/node_modules/lodash-es.
Its dependencies are also installed. Then the dependencies of its dependencies. Etc.

A new property is added to package.json:

"dependencies": {
  "lodash-es": "^4.17.21"
}

package-lock.json is updated with the exact version that was installed.

Referring to ECMAScript modules via specifiers

Code in other ECMAScript modules is accessed via import statements (line A and line B):

// Static import
import {namedExport} from 'https://example.com/some-module.js'; // (A)
console.log(namedExport);

// Dynamic import
import('https://example.com/some-module.js') // (B)
.then((moduleNamespace) => {
  console.log(moduleNamespace.namedExport);
});

Both static imports and dynamic imports use module specifiers to refer to modules:

The string after from in line A.
The string argument in line B.

There are three kinds of module specifiers:

Absolute specifiers are full URLs – for example:
```
'https://www.unpkg.com/browse/yargs@17.3.1/browser.mjs'
'file:///opt/nodejs/config.mjs'
```
Absolute specifiers are mostly used to access libraries that are directly hosted on the web.
Relative specifiers are relative URLs (starting with '/', './' or '../') – for example:
```
'./sibling-module.js'
'../module-in-parent-dir.mjs'
'../../dir/other-module.js'
```
Every module has a URL whose protocol depends on its location (file:, https:, etc.). If it uses a relative specifier, JavaScript turns that specifier into a full URL by resolving it against the module’s URL.

Relative specifiers are mostly used to access other modules within the same code base.
Bare specifiers are paths (without protocol and domain) that start with neither slashes nor dots. They begin with the names of packages. Those names can optionally be followed by subpaths:
```
'some-package'
'some-package/sync'
'some-package/util/files/path-tools.js'
```
Bare specifiers can also refer to packages with scoped names:
```
'@some-scope/scoped-name'
'@some-scope/scoped-name/async'
'@some-scope/scoped-name/dir/some-module.mjs'
```
Each bare specifier refers to exactly one module inside a package; if it has no subpath, it refers to the designated “main” module of its package. A bare specifier is never used directly but always resolved – translated to an absolute specifier. How resolution works depends on the platform. We’ll learn more soon.

Filename extensions in module specifiers

Absolute specifiers and relative specifiers always have filename extensions – usually .js or .mjs.
There are three styles of bare specifiers:
- Style 1: no subpath
- Style 2: a subpath without a filename extension. In this case, the subpath works like a modifier for the package name:
```
'my-parser/sync'
'my-parser/async'

'assertions'
'assertions/strict'
```
- Style 3: a subpath with a filename extension. In this case, the package is seen as a collection of modules and the subpath points to one of them:
```
'large-package/misc/util.js'
'large-package/main/parsing.js'
'large-package/main/printing.js'
```

Caveat of style 3 bare specifiers: How the filename extension is interpreted depends on the dependency and may differ from the importing package. For example, the importing package may use .mjs for ESM modules and .js for CommonJS modules, while the ESM modules exported by the dependency may have bare paths with the filename extension .js.

Using class `URL` to explore how module specifiers work

Module specifiers are based on URLs, which are a subset of URIs. RFC 3986, the standard for URIs, distinguishes two kinds of URI-references:

A URI starts with a scheme followed by a colon separator.
All other URI references are relative references.

Class URL is available on most JavaScript platforms and can be instantiated in two ways:

new URL(uri)

uri must be a URI. It specifies the URI of the new instance.
new URL(uriRef, baseUri)

baseUri must be a URI. If uriRef is a relative reference, it is resolved against baseUri and the result becomes the URI of the new instance.

If uriRef is a URI, it completely replaces baseUri as the data on which the instance is based.

Here we can see the class in action:

// If there is only one argument, it must be a proper URI
assert.equal(
  new URL('https://example.com/public/page.html').toString(),
  'https://example.com/public/page.html'
);
assert.throws(
  () => new URL('../book/toc.html'),
  /^TypeError \[ERR_INVALID_URL\]: Invalid URL$/
);

// Resolve a relative reference against a base URI 
assert.equal(
  new URL(
    '../book/toc.html',
    'https://example.com/public/page.html'
  ).toString(),
  'https://example.com/book/toc.html'
);

Acknowledgement: The idea of using URL in this manner and the functions isAbsoluteSpecifier() and isBareSpecifier() come from Guy Bedford.

Relative module specifiers

URL allows us to test how relative module specifiers are resolved against the baseUrl of an importing module:

// URL of importing module
const baseUrl = 'https://example.com/public/dir/a.js';
assert.equal(
  new URL('./b.js', baseUrl).toString(),
  'https://example.com/public/dir/b.js'
);
assert.equal(
  new URL('../c.mjs', baseUrl).toString(),
  'https://example.com/public/c.mjs'
);
assert.equal(
  new URL('../../private/d.js', baseUrl).toString(),
  'https://example.com/private/d.js'
);

Absolute module specifiers

Due to new URL() throwing an exception if a string isn’t a valid URI, we can use it to determine if a module specifier is absolute:

function isAbsoluteSpecifier(specifier) {
  try {
    new URL(specifier)
    return true;
  }
  catch {
    return false;
  }
}

assert.equal(
  isAbsoluteSpecifier('./other-module.js'),
  false
);
assert.equal(
  isAbsoluteSpecifier('bare-specifier'),
  false
);
assert.equal(
  isAbsoluteSpecifier('file:///opt/nodejs/config.mjs'),
  true
);
assert.equal(
  isAbsoluteSpecifier('https://www.unpkg.com/browse/yargs@17.3.1/browser.mjs'),
  true
);

Bare module specifiers

We use isAbsoluteSpecifier() to determine if a specifier is bare:

function isBareSpecifier(specifier) {
  if (
    specifier.startsWith('/')
    || specifier.startsWith('./')
    || specifier.startsWith('../')
  ) {
    return false;
  }
  return !isAbsoluteSpecifier(specifier);
}

assert.equal(
  isBareSpecifier('bare-specifier'), true
);
assert.equal(
  isBareSpecifier('fs/promises'), true
);
assert.equal(
  isBareSpecifier('@big-co/lib/tools/strings.js'), true
);

assert.equal(
  isBareSpecifier('./other-module.js'), false
);
assert.equal(
  isBareSpecifier('file:///opt/nodejs/config.mjs'), false
);
assert.equal(
  isBareSpecifier('node:assert/strict'), false
);

Module specifiers in Node.js

Let’s see how module specifiers work in Node.js. Especially bare specifiers are handled differently than in browsers.

Resolving module specifiers in Node.js

The Node.js resolution algorithm works as follows:

Parameters:
- URL of importing module
- Module specifier
Result: Resolved URL for module specifier

This is the algorithm:

If a specifier is absolute, resolution is already finished. Three protocols are most common:
- file: for local files
- https: for remote files
- node: for built-in modules (discussed later)
If a specifier is relative, it is resolved against the URL of the importing module.
If a specifier is bare:
- If it starts with '#', it is resolved by looking it up among the package imports (which are explained later) and resolving the result.
- Otherwise, it is a bare specifier that has one of these formats (the subpath is optional):
  - «package»/sub/path
  - @«scope»/«scoped-package»/sub/path
  The resolution algorithm traverses the current directory and its ancestors until it finds a directory node_modules that has a subdirectory matching the beginning of the bare specifier, i.e. either:
  - node_modules/«package»/
  - node_modules/@«scope»/«scoped-package»/
  That directory is the directory of the package. By default, the (potentially empty) subpath after the package ID is interpreted as relative to the package directory. The default can be overridden via package exports which are explained next.

The result of the resolution algorithm must point to a file. That explains why absolute specifiers and relative specifiers always have filename extensions. Bare specifiers mostly don’t because they are abbreviations that are looked up in package exports.

Module files usually have these filename extensions:

If a file has the name extension .mjs, it is always an ES module.
A file that has the name extension .js is an ES module if the closest package.json has this entry:
- "type": "module"

If Node.js executes code provided via stdin, --eval or --print, we use the following command-line option so that it is interpreted as an ES module:

--input-type=module

Package exports: controlling what other packages see

In this subsection, we are working with a package that has the following file layout:

my-lib/
  dist/
    src/
      main.js
      util/
        errors.js
      internal/
        internal-module.js
    test/

Package exports are specified via property "exports" in package.json and support two important features:

Hiding the internals of a package:
- Without property "exports", every module in package my-lib can be accessed via a relative path after the package name – e.g.:
```
'my-lib/dist/src/internal/internal-module.js'
```
- Once the property exists, only specifiers listed in it can be used. Everything else is hidden from the outside.
Nicer module specifiers: Package export let us define bare specifier subpaths for modules that are shorter and/or have better names.

Recall the three styles of bare specifiers:

Style 1: bare specifiers without subpaths
Style 2: bare specifiers with extension-less subpaths
Style 3: bare specifiers with subpaths with extensions

Package exports help us with all three styles

Style 1: configuring which file represents (the bare specifier for) the package

package.json:

{
  "main": "./dist/src/main.js",
  "exports": {
    ".": "./dist/src/main.js"
  }
}

We only provide "main" for backward-compatibility (with older bundlers and Node.js 12 and older). Otherwise, the entry for "." is enough.

With these package exports, we can now import from my-lib as follows.

import {someFunction} from 'my-lib';

This imports someFunction() from this file:

my-lib/dist/src/main.js

Style 2: mapping extension-less subpaths to module files

package.json:

{
  "exports": {
    "./util/errors": "./dist/src/util/errors.js"
  }
}

We are mapping the specifier subpath 'util/errors' to a module file. That enables the following import:

import {UserError} from 'my-lib/util/errors';

Style 2: better subpaths without extensions for a subtree

The previous subsection explained how to create a single mapping for an extension-less subpath. There is also a way to create multiple such mappings via a single entry:

package.json:

{
  "exports": {
    "./lib/*": "./dist/src/*.js"
  }
}

Any file that is a descendant of ./dist/src/ can now be imported without a filename extension:

import {someFunction} from 'my-lib/lib/main';
import {UserError}    from 'my-lib/lib/util/errors';

Note the asterisks in this "exports" entry:

"./lib/*": "./dist/src/*.js"

These are more instructions for how to map subpaths to actual paths than wildcards that match fragments of file paths.

Style 3: mapping subpaths with extensions to module files

package.json:

{
  "exports": {
    "./util/errors.js": "./dist/src/util/errors.js"
  }
}

We are mapping the specifier subpath 'util/errors.js' to a module file. That enables the following import:

import {UserError} from 'my-lib/util/errors.js';

Style 3: better subpaths with extensions for a subtree

package.json:

{
  "exports": {
    "./*": "./dist/src/*"
  }
}

Here, we shorten the module specifiers of the whole subtree under my-package/dist/src:

import {InternalError} from 'my-package/util/errors.js';

Without the exports, the import statement would be:

import {InternalError} from 'my-package/dist/src/util/errors.js';

Note the asterisks in this "exports" entry:

"./*": "./dist/src/*"

These are not filesystem globs but instructions for how to map external module specifiers to internal ones.

Exposing a subtree while hiding parts of it

With the following trick, we expose everything in directory my-package/dist/src/ with the exception of my-package/dist/src/internal/

"exports": {
  "./*": "./dist/src/*",
  "./internal/*": null
}

Note that this trick also works when exporting subtrees without filename extensions.

Conditional package exports

We can also make exports conditional: Then a given path maps to different values depending on the context in which a package is used.

Node.js vs. browsers. For example, we could provide different implementations for Node.js and for browsers:

"exports": {
  ".": {
    "node": "./main-node.js",
    "browser": "./main-browser.js",
    "default": "./main-browser.js"
  }
}

The "default" condition matches when no other key matches and must come last. Having one is recommended whenever we are distinguishing between platforms because it takes care of new and/or unknown platforms.

Development vs. production. Another use case for conditional package exports is switching between “development” and “production” environments:

"exports": {
  ".": {
    "development": "./main-development.js",
    "production": "./main-production.js",
  }
}

In Node.js we can specify an environment like this:

node --conditions development app.mjs

Package imports

Package imports let a package define abbreviations for module specifiers that it can use itself, internally (where package exports define abbreviations for other packages). This is an example:

package.json:

{
  "imports": {
    "#some-pkg": {
      "node": "some-pkg-node-native",
      "default": "./polyfills/some-pkg-polyfill.js"
    }
  },
  "dependencies": {
    "some-pkg-node-native": "^1.2.3"
  }
}

The package import # is conditional (with the same features as conditional package exports):

If the current package is used on Node.js, the module specifier '#some-pkg' refers to package some-pkg-node-native.
Elsewhere, '#some-pkg' refers to the file ./polyfills/some-pkg-polyfill.js inside the current package.

(Only package imports can refer to external packages, package exports can’t do that.)

What are the use cases for package imports?

Referring to different platform-specific implementations modules via the same module specifier (as demonstrated above).
Aliases to modules inside the current package – to avoid relative specifiers (which can get complicated with deeply nested directories).

Be careful when using package imports with a bundler: This feature is relatively new and your bundler may not support it.

`node:` protocol imports

Node.js has many built-in modules such as 'path' and 'fs'. All of them are available as both ES modules and CommonJS modules. One issue with them is that they can be overridden by modules installed in node_modules which is both a security risk (if it happens accidentally) and a problem if Node.js wants to introduce new built-in modules in the future and their names are already taken by npm packages.

We can use the node: protocol to make it clear that we want to import a built-in module. For example, the following two import statements are mostly equivalent (if no npm module is installed that has the name 'fs'):

import * as fs from 'node:fs/promises';
import * as fs from 'fs/promises';

An additional benefit of using the node: protocol is that we immediately see that an imported module is built-in. Given how many built-in modules there are, that helps when reading code.

Due to node: specifiers having a protocol, they are considered absolute. That’s why they are not looked up in node_modules.

Module specifiers in browsers

Filename extensions in browsers

Browsers don’t care about filename extensions, only about content types.

Hence, we can use any filename extension for ECMAScript modules, as long as they are served with a JavaScript content type (text/javascript is recommended).

Using npm packages in browsers

On Node.js, npm packages are downloaded into the node_modules directory and accessed via bare module specifiers. Node.js traverses the file system in order to find packages. We can’t do that in web browsers. Two approaches are common for bringing npm packages to browsers.

Approach 1: Use `node_modules` with bare specifiers and a bundler

A bundler is a build tool. It works roughly as follows:

Given a directory with a web app. We point the bundler to the app’s entry point – the module where execution starts.
It collects everything that module imports (its imports, the imports of the imports, etc.).
It produces a bundle, a single file with all the code. That file can be used from an HTML page.

If an app has multiple entry points, the bundler produces multiple bundles. It’s also possible to tell it to create bundles for parts of the application that are loaded on demand.

When bundling, we can use bare import specifiers in files because bundlers know how to find the corresponding modules in node_modules. Modern bundlers also honor package exports and package imports.

Why bundle?

Loading a single file tends to be faster than loading multiple files – especially if there are many small ones.
Bundlers only include code in the file that is really used (which is especially relevant for libraries). That saves storage space and also speeds up loading.

A downside of bundling is that we need to bundle the whole app every time we want to run it.

Approach 2: Convert npm packages to browser-compatible files

There are package managers for browsers that let us download modules referenced via bare specifiers as single bundled files that can be used in browsers.

As an example, consider the following directory of a web app:

my-web-app/
  assets/
    lodash-es.js
  src/
    main.js

We used a bundler to install the module referenced by lodash-es into a single file. Module main.js can import it like this:

import {pick} from '../assets/lodash-es.js';

To deploy this app, the contents of assets/ and src/ are copied to the production server (in addition to non-JavaScript artifacts).

What are the benefits of this approach compared to using a bundler?

We install the external dependencies once and then can always run our app immediately – no prior bundling required (which can be time-consuming).
Unbundled code is easier to debug.

This approach can be further improved: Import maps are a browser technology that lets us define abbreviations for module specifiers – e.g. 'lodash-es' for '../assets/lodash-es.js'. Import maps are explained later.

Note that with this approach, package exports are not automatically honored. We have to take care that we either use the correct paths into packages and/or set up our import maps correctly.

It’s also possible to use tools such as JSPM Generator that generate import maps automatically. Such tools can take package exports into consideration.

Hybrid approaches

We have seen two approaches for using npm packages in browsers:

Bundling a web app into a single file. Benefits: faster loading, less storage required.
Running a web app with separate modules. Benefits: app runs without bundling, easier debugging.

In other words: Approach (2) is better during development. Approach (1) is better for deploying software to production servers.

And there are indeed build tools that combine both approaches – for example, Vite:

During development, a web app is run via its local development server. Whenever a browser requests a JavaScript file, the dev server first examines the file. If any import has a bare specifier, the server does two things:
- The imported module is looked up in node_modules and bundled into a single JavaScript file. That file is cached, so this step only happens the first time a bare specifier is encountered.
- The JavaScript file is changed so that the import specifier isn’t bare anymore, but refers to the bundle.
The changes to the JavaScript code are minimal and local (only one file is affected). That enables on-demand processing via the dev server. The web app remains a collection of separate modules.
For deployment, Vite compiles the JavaScript code of the app and its dependencies into one file (multiple ones if parts of the app are loaded on demand).

Import maps

An import map is a data structure with key-value entries. This is what an import map looks like if we store it inline – inside an HTML file:

<script type="importmap">
{
  "imports": {
    "date-fns": "/node_modules/date-fns/esm/index.js"
  }
}
</script>

We can also store import maps in external files (the content type must be application/importmap+json):

<script type="importmap" src="import-map.importmap"></script>

Terminology:

The keys of an import map are called specifier keys.
The values are called resolution results.

How do import maps work? Roughly, Whenever JavaScript encounters an import statement or a dynamic import(), it resolves its module specifier: It looks for the first map entry whose specifier key matches the import specifier and replaces the key’s occurrence with the resolution result. We’ll see concrete examples soon. Let’s dig into more details first.

Specifier keys. There are two general categories of specifier keys:

Specifier keys without trailing slashes match import specifiers exactly.
Specifier keys with trailing slashes match prefixes of import specifiers.

In category 1, there are three kinds of specifier keys:

Bare specifier keys with or without subpaths:
```
"lodash-es"
"lodash-es/pick"
```
Relative specifier keys start with '/', './' or '../':
```
"/js/app.mjs"
"../util/string-tools.js"
```

Absolute specifier keys start with protocols:

"https://www.unpkg.com/lodash-es/lodash.js"

Category 2 contains the same kinds of specifier keys – except that they all end with slashes.

Resolution results. Resolution results can also end with slashes or not (they have to mirror what the specifier key looks like). There are only two kinds of non-prefix resolution results (bare specifiers are not allowed):

URLs:

"https://www.unpkg.com/lodash-es/lodash.js"

Relative references (must start with '/', './', '../'):

"/node_modules/lodash-es/lodash.js"
"./src/util/string-tools.mjs"

Prefix resolution results also consist of URLs and relative references but always end with slashes.

Normalization. For import maps, normalization is important:

Specifier keys are immediately normalized:
- Bare specifier keys and absolute specifier keys remain unchanged.
- Relative specifier keys are resolved against the URL of their import map (.html file or .importmap file).
Resolution results are also immediately normalized, in the same manner.
Import specifiers are resolved against the URLs of the importing modules before they are matched against normalized specifier keys.

Normalization makes it possible to match relative specifier keys against relative import specifiers.

Resolving import module specifiers. Resolution is the process of converting a module specifier to a fetchable URL. To resolve an import specifier, JavaScript looks for the first map entry whose specifier key matches the import specifier:

A specifier key without a slash matches an import specifier if both are exactly equal. In that case, resolution returns the corresponding resolution result.
A specifier key with a slash matches any import specifier that starts with it – in other words if the specifier key is a prefix of the import specifier. In that case, resolution replaces the occurrence of the specifier key in the import specifier with the resolution result and returns the outcome.
If there is no matching map entry, the import specifier is returned unchanged.

Note that import maps do not affect <script> elements, only JavaScript imports.

Comparing package exports with import maps

With package exports, we can define:

Subpaths (empty, with filename extensions, without filename extensions) for single modules:

".": "./dist/src/main.js",
"./util/errors": "./dist/src/util/errors.js",
"./util/errors.js": "./dist/src/util/errors.js"

Subpaths with filename extensions for directories with modules:
```
"./*": "./dist/src/*",
```
Subpaths without filename extensions for directories with modules:
```
"./lib/*": "./dist/src/*.js",
```

Import maps have analogs to the first two features, but nothing that is similar to the last feature.

Examples

Example: bare specifier keys without subpaths

{
  "imports": {
    "lodash-es": "/node_modules/lodash-es/lodash.js",
    "date-fns": "/node_modules/date-fns/esm/index.js"
  }
}

This import map enables these imports:

import {pick} from 'lodash-es';
import {addYears} from 'date-fns';

Example: non-prefix bare specifier keys with subpaths

{
   "imports": {
     "lodash-es": "/node_modules/lodash-es/lodash.js",
     "lodash-es/pick": "/node_modules/lodash-es/pick.js",
     "lodash-es/zip.js": "/node_modules/lodash-es/zip.js",
   }
 }

We can now import like this:

import pick from 'lodash-es/pick'; // default import
import zip from 'lodash-es/zip.js'; // default import
import {pick as pick2, zip as zip2} from 'lodash-es';

Example: prefix bare specifier keys with subpaths

{
   "imports": {
     "lodash-es/": "/node_modules/lodash-es/",
   }
 }

We can now import like this:

import pick from 'lodash-es/pick.js'; // access default export
import {pick as pick2} from 'lodash-es/lodash.js';

Example: non-prefix absolute specifier keys

{
  "imports": {
    "https://www.unpkg.com/lodash-es/lodash.js": "/node_modules/lodash-es/lodash.js"
  }
}

We are remapping the import from an external resource to a local file:

import {pick} from 'https://www.unpkg.com/lodash-es/lodash.js';

Example: prefix absolute specifier keys

{
  "imports": {
    "https://www.unpkg.com/lodash-es/": "/node_modules/lodash-es/"
  }
}

The online package contents are remapped to the version we downloaded into node_modules/:

import {pick} from 'https://www.unpkg.com/lodash-es/lodash.js';
import pick from 'https://www.unpkg.com/lodash-es/pick.js';

Example: non-prefix relative specifier keys

To ensure that trees of connected files (think HTML using CSS, image and JavaScript files) are updated together, a common technique is to mention the hash of the file (which is also called its message digest) in the filename. The hash serves as a version number for the file.

An import map enables us to use import specifiers without hashes:

{
  "imports": {
    "/js/app.mjs": "/js/app-28501dff.mjs",
    "/js/lib.mjs": "/js/lib-1c9248c2.mjs",
  }
}

Import map scopes

Scopes lets us override map entries depending on where an import is made from. That looks as follows (example taken from the import maps explainer document):

{
  "imports": {
    "querystringify": "/node_modules/querystringify/index.js"
  },
  "scopes": {
    "/node_modules/socksjs-client/": {
      "querystringify": "/node_modules/socksjs-client/querystringify/index.js"
    }
  }
}

Module specifiers in Deno

Bare specifiers are rarely used in Deno (but import maps are available). Instead, libraries are accessed via URLs with version numbers – for example:

'https://deno.land/std@0.120.0/testing/asserts.ts'

These URLs are abbreviated via the following technique: Per project, there is a file deps.ts that re-exports library exports that are used more than once:

// my-proj/src/deps.ts
export {
  assertEquals,
  assertStringIncludes,
} from 'https://deno.land/std@0.120.0/testing/asserts.ts';

Other files get their imports from deps.ts:

// my-proj/src/main.ts
import {assertEquals} from './deps.ts';

assertEquals('abc', 'abc');

Deno caches absolute imports. Its cache can be persisted, e.g. to ensure that it’s available when a computer is offline. Example in Deno’s manual:

# Download the dependencies.
DENO_DIR=./deno_dir
deno cache src/deps.ts

# Make sure the variable is set for any command which invokes the cache.
DENO_DIR=./deno_dir
deno test src

Publishing and consuming ECMAScript modules via packages – the big picture

Packages: JavaScript’s units for software distribution #

Publishing packages: package registries, package managers, package names #

The file system layout of a package #

package.json #

Property "dependencies" of package.json #

Property "bin" of package.json #

Property "license" of package.json #

Archiving and installing packages #

Installing an existing package from git #

Reinstalling a package #

Creating a new package and installing dependencies #

Installing dependencies #

Referring to ECMAScript modules via specifiers #

Filename extensions in module specifiers #

Using class URL to explore how module specifiers work #

Relative module specifiers #

Absolute module specifiers #

Bare module specifiers #

Module specifiers in Node.js #

Resolving module specifiers in Node.js #

Package exports: controlling what other packages see #

Style 1: configuring which file represents (the bare specifier for) the package #

Style 2: mapping extension-less subpaths to module files #

Style 2: better subpaths without extensions for a subtree #

Style 3: mapping subpaths with extensions to module files #

Style 3: better subpaths with extensions for a subtree #

Exposing a subtree while hiding parts of it #

Conditional package exports #

Package imports #

node: protocol imports #

Module specifiers in browsers #

Filename extensions in browsers #

Using npm packages in browsers #

Approach 1: Use node_modules with bare specifiers and a bundler #

Approach 2: Convert npm packages to browser-compatible files #

Hybrid approaches #

Import maps #

Comparing package exports with import maps #

Examples #

Example: bare specifier keys without subpaths #

Example: non-prefix bare specifier keys with subpaths #

Example: prefix bare specifier keys with subpaths #

Example: non-prefix absolute specifier keys #

Example: prefix absolute specifier keys #

Example: non-prefix relative specifier keys #

Import map scopes #

Module specifiers in Deno #

Further reading and sources of this post #

Packages: JavaScript’s units for software distribution

Publishing packages: package registries, package managers, package names

The file system layout of a package

`package.json`

Property `"dependencies"` of `package.json`

Property `"bin"` of `package.json`

Property `"license"` of `package.json`

Archiving and installing packages

Installing an existing package from git

Reinstalling a package

Creating a new package and installing dependencies

Installing dependencies

Referring to ECMAScript modules via specifiers

Filename extensions in module specifiers

Using class `URL` to explore how module specifiers work

Relative module specifiers

Absolute module specifiers

Bare module specifiers

Module specifiers in Node.js

Resolving module specifiers in Node.js

Package exports: controlling what other packages see

Style 1: configuring which file represents (the bare specifier for) the package

Style 2: mapping extension-less subpaths to module files

Style 2: better subpaths without extensions for a subtree

Style 3: mapping subpaths with extensions to module files

Style 3: better subpaths with extensions for a subtree

Exposing a subtree while hiding parts of it

Conditional package exports

Package imports

`node:` protocol imports

Module specifiers in browsers

Filename extensions in browsers

Using npm packages in browsers

Approach 1: Use `node_modules` with bare specifiers and a bundler

Approach 2: Convert npm packages to browser-compatible files

Hybrid approaches

Import maps

Comparing package exports with import maps

Examples

Example: bare specifier keys without subpaths

Example: non-prefix bare specifier keys with subpaths

Example: prefix bare specifier keys with subpaths

Example: non-prefix absolute specifier keys

Example: prefix absolute specifier keys

Example: non-prefix relative specifier keys

Import map scopes

Module specifiers in Deno

Further reading and sources of this post