Update 2022-01-17: Added material on bare specifiers with subpaths that have filename extensions.
The ecosystem around delivering ECMAScript modules via packages is slowly maturing. This blog post explains how the various pieces fit together:
Required knowledge: I’m assuming that you are loosely familiar with the syntax of ECMAScript modules. If you are not, you can read chapter “modules” in “JavaScript for impatient programmers”.
Table of contents:
In the JavaScript ecosystem, the directories of most projects are laid out according to a standard called package. Packages include:
Some packages are published to the npm software registry (explained soon), others only exist locally – especially web applications.
Most packages contain all kinds of files – both JavaScript code and related artifacts:
In principle, packages can contain no JavaScript at all.
The dominant way of distributing packages is via the npm software registry. Packages can be uploaded to it and downloaded from it. Each package in the registry has a name. There are two kinds of names:
lodash-es
date-fns
@babel/core
The scope starts with an @
symbol and is separated from the name with a slash.The default tool for managing npm packages is the npm package manager. But there are also other popular package managers that support the npm registry.
Let’s assume, we have a package whose name is my-proj
. On disk, this package looks as follows:
my-proj/
package.json
node_modules/
[more files and directories]
What are these contents used for?
package.json
is a JSON file that serves two main purposes:
node_modules/
is a directory into which the dependencies of the package are installed.node_modules
If we download a package from somewhere, it often doesn’t have the directory node_modules/
and we must install its dependencies – e.g. via these shell commands:
cd some-package/
npm install
npm install
creates node_modules/
and downloads the dependencies listed in package.json
into it.
If something inside node_modules/
is broken, a common way of resetting a package is:
cd some-package/
rm -rf node_modules
npm install
There is no risk in doing so because package.json
contains a complete list of what’s inside node_modules/
.
(package.json
is complemented by package-lock.json
but that’s not immediately important and beyond the scope of this blog post.)
package.json
There are many tools and technique for setting up new packages. This is one simple way:
mkdir my-package
cd my-package/
npm init --yes
npm has created the following package.json
for us:
{
"name": "my-package",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC"
}
This file contains various kinds of data:
name
specifies the name of this package.version
is used for version management and follows semantic versioning, with three numbers:
description
, keywords
, author
are optional and make it easier to find packages.license
clarifies how this package can be used. It makes sense to provide this value if the package is public in any way. “Choose an open source license” can help with making this choice.main
: specifies the module that “is” the package (explained later in this chapter).scripts
: are commands that we can execute via npm run
. For example, the script test
can be executed via npm run test
.Tips:
The following setting means that all files with the name extension .js
are interpreted as ECMAScript modules. Unless we are dealing with legacy code, it makes sense to add it:
"type": "module"
Normally, the properties "name"
and "version"
are required and npm warns us if they are missing. However, we can change that via the following setting:
"private": true
That prevents the package from accidentally being published and allows us to omit name and version.
More information on package.json
: see the npm documentation.
Right now, my-package
doesn’t have any dependencies. Let’s say we want to use the library lodash-es
. This is how we install it into our package:
npm install lodash-es
This command does two things:
my-package/node_modules/lodash-es
.package.json
:"dependencies": {
"lodash-es": "^4.17.21"
}
The dependencies record both the names of packages and constraints for their versions. The latter follow the the Node.js semantic versioning format. ^4.17.21
is a caret range and means that, when my-package
is installed, lodash-es
must have version 4.17.21 or a newer (higher) version. The newer version must have the same leftmost non-zero version component (“4” in this case). Roughly, the motivation is that upgrading lodash-es
is OK, as long as there are no breaking changes.
Even though the npm registry was initially only used for Node.js code, it now also hosts many packages that are used in frontend code:
On both Node.js and browsers, we use:
import
statements with module specifiers to access those modules.On all platforms, configuration is needed before module specifiers work. The next section explains the details.
Code in other ECMAScript modules is accessed via import
statements (line A and line B):
// Static import
import {namedExport} from 'https://example.com/some-module.js'; // (A)
console.log(namedExport);
// Dynamic import
import('https://example.com/some-module.js') // (B)
.then((moduleNamespace) => {
console.log(moduleNamespace.namedExport);
});
Both static imports and dynamic imports use module specifiers to refer to modules:
from
in line A.There are three kinds of module specifiers:
Absolute specifiers are full URLs – for example:
'https://www.unpkg.com/browse/yargs@17.3.1/browser.mjs'
'file:///opt/nodejs/config.mjs'
Absolute specifiers are mostly used to access libraries that are directly hosted on the web.
Relative specifiers are relative URLs (starting with '/'
, './'
or '../'
) – for example:
'./sibling-module.js'
'../module-in-parent-dir.mjs'
'../../dir/other-module.js'
Every module has a URL whose protocol depends on its location (file:
, https:
, etc.). If it uses a relative specifier, JavaScript turns that specifier into a full URL by resolving it against the module’s URL.
Relative specifiers are mostly used to access other modules within the same code base.
Bare specifiers are paths (without protocol and domain) that start with neither slashes nor dots. They begin with the names of packages. Those names can optionally be followed by subpaths:
'some-package'
'some-package/sync'
'some-package/util/files/path-tools.js'
Bare specifiers can also refer to packages with scoped names:
'@some-scope/scoped-name'
'@some-scope/scoped-name/async'
'@some-scope/scoped-name/dir/some-module.mjs'
Each bare specifier refers to exactly one module inside a package; if it has no subpath, it refers to the designated “main” module of its package. A bare specifier is never used directly but always resolved – translated to an absolute specifier. How resolution works depends on the platform. We’ll learn more soon.
.js
or .mjs
.'my-parser/sync'
'my-parser/async'
'assertions'
'assertions/strict'
'large-package/misc/util.js'
'large-package/main/parsing.js'
'large-package/main/printing.js'
Caveat of style 3 bare specifiers: How the filename extension is interpreted depends on the dependency and may differ from the importing package. For example, the importing package may use .mjs
for ESM modules and .js
for CommonJS modules, while the ESM modules exported by the dependency may have bare paths with the filename extension .js
.
URL
to explore how module specifiers work Module specifiers are based on URLs, which are a subset of URIs. RFC 3986, the standard for URIs, distinguishes two kinds of URI-references:
Class URL
is available on most JavaScript platforms and can be instantiated in two ways:
new URL(uri)
uri
must be a URI. It specifies the URI of the new instance.
new URL(uriRef, baseUri)
baseUri
must be a URI. If uriRef
is a relative reference, it is resolved against baseUri
and the result becomes the URI of the new instance.
If uriRef
is a URI, it completely replaces baseUri
as the data on which the instance is based.
Here we can see the class in action:
// If there is only one argument, it must be a proper URI
assert.equal(
new URL('https://example.com/public/page.html').toString(),
'https://example.com/public/page.html'
);
assert.throws(
() => new URL('../book/toc.html'),
/^TypeError \[ERR_INVALID_URL\]: Invalid URL$/
);
// Resolve a relative reference against a base URI
assert.equal(
new URL(
'../book/toc.html',
'https://example.com/public/page.html'
).toString(),
'https://example.com/book/toc.html'
);
Acknowledgement: The idea of using URL
in this manner and the functions isAbsoluteSpecifier()
and isBareSpecifier()
come from Guy Bedford.
URL
allows us to test how relative module specifiers are resolved against the baseUrl
of an importing module:
// URL of importing module
const baseUrl = 'https://example.com/public/dir/a.js';
assert.equal(
new URL('./b.js', baseUrl).toString(),
'https://example.com/public/dir/b.js'
);
assert.equal(
new URL('../c.mjs', baseUrl).toString(),
'https://example.com/public/c.mjs'
);
assert.equal(
new URL('../../private/d.js', baseUrl).toString(),
'https://example.com/private/d.js'
);
Due to new URL()
throwing an exception if a string isn’t a valid URI, we can use it to determine if a module specifier is absolute:
function isAbsoluteSpecifier(specifier) {
try {
new URL(specifier)
return true;
}
catch {
return false;
}
}
assert.equal(
isAbsoluteSpecifier('./other-module.js'),
false
);
assert.equal(
isAbsoluteSpecifier('bare-specifier'),
false
);
assert.equal(
isAbsoluteSpecifier('file:///opt/nodejs/config.mjs'),
true
);
assert.equal(
isAbsoluteSpecifier('https://www.unpkg.com/browse/yargs@17.3.1/browser.mjs'),
true
);
We use isAbsoluteSpecifier()
to determine if a specifier is bare:
function isBareSpecifier(specifier) {
if (
specifier.startsWith('/')
|| specifier.startsWith('./')
|| specifier.startsWith('../')
) {
return false;
}
return !isAbsoluteSpecifier(specifier);
}
assert.equal(
isBareSpecifier('bare-specifier'), true
);
assert.equal(
isBareSpecifier('fs/promises'), true
);
assert.equal(
isBareSpecifier('@big-co/lib/tools/strings.js'), true
);
assert.equal(
isBareSpecifier('./other-module.js'), false
);
assert.equal(
isBareSpecifier('file:///opt/nodejs/config.mjs'), false
);
assert.equal(
isBareSpecifier('node:assert/strict'), false
);
Let’s see how module specifiers work in Node.js. Especially bare specifiers are handled differently than in browsers.
The Node.js resolution algorithm works as follows:
This is the algorithm:
If a specifier is absolute, resolution is already finished. Three protocols are most common:
file:
for local fileshttps:
for remote filesnode:
for built-in modules (discussed later)If a specifier is relative, it is resolved against the URL of the importing module.
If a specifier is bare:
If it starts with '#'
, it is resolved by looking it up among the package imports (which are explained later) and resolving the result.
Otherwise, it is a bare specifier that has one of these formats (the subpath is optional):
«package»/sub/path
@«scope»/«scoped-package»/sub/path
The resolution algorithm traverses the current directory and its ancestors until it finds a directory node_modules
that has a subdirectory matching the beginning of the bare specifier, i.e. either:
node_modules/«package»/
node_modules/@«scope»/«scoped-package»/
That directory is the directory of the package. By default, the (potentially empty) subpath after the package ID is interpreted as relative to the package directory. The default can be overridden via package exports which are explained next.
The result of the resolution algorithm must point to a file. That explains why absolute specifiers and relative specifiers always have filename extensions. Bare specifiers mostly don’t because they are abbreviations that are looked up in package exports.
Module files usually have these filename extensions:
.mjs
, it is always an ES module..js
is an ES module if the closest package.json
has this entry:
"type": "module"
If Node.js executes code provided via stdin, --eval
or --print
, we use the following command-line option so that it is interpreted as an ES module:
--input-type=module
In this subsection, we are working with a package that has the following file layout:
my-lib/
dist/
src/
main.js
util/
errors.js
internal/
internal-module.js
test/
Package exports are specified via property "exports"
in package.json
and support two important features:
"exports"
, every module in package my-lib
can be accessed via a relative path after the package name – e.g.:'my-lib/dist/src/internal/internal-module.js'
Recall the three styles of bare specifiers:
Package exports help us with all three styles
package.json
:
{
"main": "./dist/src/main.js",
"exports": {
".": "./dist/src/main.js"
}
}
We only provide "main"
for backward-compatibility (with older bundlers and Node.js 12 and older). Otherwise, the entry for "."
is enough.
With these package exports, we can now import from my-lib
as follows.
import {someFunction} from 'my-lib';
This imports someFunction()
from this file:
my-lib/dist/src/main.js
package.json
:
{
"exports": {
"./util/errors": "./dist/src/util/errors.js"
}
}
We are mapping the specifier subpath 'util/errors'
to a module file. That enables the following import:
import {UserError} from 'my-lib/util/errors';
The previous subsection explained how to create a single mapping for an extension-less subpath. There is also a way to create multiple such mappings via a single entry:
package.json
:
{
"exports": {
"./lib/*": "./dist/src/*.js"
}
}
Any file that is a descendant of ./dist/src/
can now be imported without a filename extension:
import {someFunction} from 'my-lib/lib/main';
import {UserError} from 'my-lib/lib/util/errors';
Note the asterisks in this "exports"
entry:
"./lib/*": "./dist/src/*.js"
These are more instructions for how to map subpaths to actual paths than wildcards that match fragments of file paths.
package.json
:
{
"exports": {
"./util/errors.js": "./dist/src/util/errors.js"
}
}
We are mapping the specifier subpath 'util/errors.js'
to a module file. That enables the following import:
import {UserError} from 'my-lib/util/errors.js';
package.json
:
{
"exports": {
"./*": "./dist/src/*"
},
"typesVersions": {
"*": {
"*": ["dist/src/*"]
}
}
}
Here, we shorten the module specifiers of the whole subtree under my-package/dist/src
:
import {InternalError} from 'my-package/util/errors.js';
Without the exports, the import statement would be:
import {InternalError} from 'my-package/dist/src/util/errors.js';
Note the asterisks in this "exports"
entry:
"./*": "./dist/src/*"
These are not filesystem globs but instructions for how to map external module specifiers to internal ones.
With the following trick, we expose everything in directory my-package/dist/src/
with the exception of my-package/dist/src/internal/
"exports": {
"./*": "./dist/src/*",
"./internal/*": null
}
Note that this trick also works when exporting subtrees without filename extensions.
We can also make exports conditional: Then a given path maps to different values depending on the context in which a package is used.
Node.js vs. browsers. For example, we could provide different implementations for Node.js and for browsers:
"exports": {
".": {
"node": "./main-node.js",
"browser": "./main-browser.js",
"default": "./main-browser.js"
}
}
The "default"
condition matches when no other key matches and must come last. Having one is recommended whenever we are distinguishing between platforms because it takes care of new and/or unknown platforms.
Development vs. production. Another use case for conditional package exports is switching between “development” and “production” environments:
"exports": {
".": {
"development": "./main-development.js",
"production": "./main-production.js",
}
}
In Node.js we can specify an environment like this:
node --conditions development app.mjs
Package imports let a package define abbreviations for module specifiers that it can use itself, internally (where package exports define abbreviations for other packages). This is an example:
package.json
:
{
"imports": {
"#some-pkg": {
"node": "some-pkg-node-native",
"default": "./polyfills/some-pkg-polyfill.js"
}
},
"dependencies": {
"some-pkg-node-native": "^1.2.3"
}
}
The package import #
is conditional (with the same features as conditional package exports):
If the current package is used on Node.js, the module specifier '#some-pkg'
refers to package some-pkg-node-native
.
Elsewhere, '#some-pkg'
refers to the file ./polyfills/some-pkg-polyfill.js
inside the current package.
(Only package imports can refer to external packages, package exports can’t do that.)
What are the use cases for package imports?
Be careful when using package imports with a bundler: This feature is relatively new and your bundler may not support it.
node:
protocol imports Node.js has many built-in modules such as 'path'
and 'fs'
. All of them are available as both ES modules and CommonJS modules. One issue with them is that they can be overridden by modules installed in node_modules
which is both a security risk (if it happens accidentally) and a problem if Node.js wants to introduce new built-in modules in the future and their names are already taken by npm packages.
We can use the node:
protocol to make it clear that we want to import a built-in module. For example, the following two import statements are mostly equivalent (if no npm module is installed that has the name 'fs'
):
import * as fs from 'node:fs/promises';
import * as fs from 'fs/promises';
An additional benefit of using the node:
protocol is that we immediately see that an imported module is built-in. Given how many built-in modules there are, that helps when reading code.
Due to node:
specifiers having a protocol, they are considered absolute. That’s why they are not looked up in node_modules
.
Bare specifiers are rarely used in Deno. Instead, libraries are accessed via URLs with version numbers – for example:
'https://deno.land/std@0.120.0/testing/asserts.ts'
These URLs are abbreviated via the following technique: Per project, there is a file deps.ts
that re-exports library exports that are used more than once:
// my-proj/src/deps.ts
export {
assertEquals,
assertStringIncludes,
} from 'https://deno.land/std@0.120.0/testing/asserts.ts';
Other files get their imports from deps.ts
:
// my-proj/src/main.ts
import {assertEquals} from './deps.ts';
assertEquals('abc', 'abc');
Deno caches absolute imports. Its cache can be persisted, e.g. to ensure that it’s available when a computer is offline. Example in Deno’s manual:
# Download the dependencies.
DENO_DIR=./deno_dir
deno cache src/deps.ts
# Make sure the variable is set for any command which invokes the cache.
DENO_DIR=./deno_dir
deno test src
Browsers don’t care about filename extensions, only about content types.
Hence, we can use any filename extension for ECMAScript modules, as long as they are served with a JavaScript content type (text/javascript
is recommended).
On Node.js, npm packages are downloaded into the node_modules
directory and accessed via bare module specifiers. Node.js traverses the file system in order to find packages. We can’t do that in web browsers. Two approaches are common for bringing npm packages to browsers.
node_modules
with bare specifiers and a bundler A bundler is a build tool. It works roughly as follows:
If an app has multiple entry points, the bundler produces multiple bundles. It’s also possible to tell it to create bundles for parts of the application that are loaded on demand.
When bundling, we can use bare import specifiers in files because bundlers know how to find the corresponding modules in node_modules
. Modern bundlers also honor package exports and package imports.
Why bundle?
A downside of bundling is that we need to bundle the whole app every time we want to run it.
There are package managers for browsers that let us download modules referenced via bare specifiers as single bundled files that can be used in browsers.
As an example, consider the following directory of a web app:
my-web-app/
assets/
lodash-es.js
src/
main.js
We used a bundler to install the module referenced by lodash-es
into a single file. Module main.js
can import it like this:
import {pick} from '../assets/lodash-es.js';
To deploy this app, the contents of assets/
and src/
are copied to the production server (in addition to non-JavaScript artifacts).
What are the benefits of this approach compared to using a bundler?
This approach can be further improved: Import maps are a browser technology that lets us define abbreviations for module specifiers – e.g. 'lodash-es'
for '../assets/lodash-es.js'
. Import maps are explained later.
Note that with this approach, package exports are not automatically honored. We have to take care that we either use the correct paths into packages and/or set up our import maps correctly.
It’s also possible to use tools such as JSPM Generator that generate import maps automatically. Such tools can take package exports into consideration.
We have seen two approaches for using npm packages in browsers:
In other words: Approach (2) is better during development. Approach (1) is better for deploying software to production servers.
And there are indeed build tools that combine both approaches – for example, Vite:
During development, a web app is run via its local development server. Whenever a browser requests a JavaScript file, the dev server first examines the file. If any import has a bare specifier, the server does two things:
The imported module is looked up in node_modules
and bundled into a single JavaScript file. That file is cached, so this step only happens the first time a bare specifier is encountered.
The JavaScript file is changed so that the import specifier isn’t bare anymore, but refers to the bundle.
The changes to the JavaScript code are minimal and local (only one file is affected). That enables on-demand processing via the dev server. The web app remains a collection of separate modules.
For deployment, Vite compiles the JavaScript code of the app and its dependencies into one file (multiple ones if parts of the app are loaded on demand).
An import map is a data structure with key-value entries. This is what an import map looks like if we store it inline – inside an HTML file:
<script type="importmap">
{
"imports": {
"date-fns": "/node_modules/date-fns/esm/index.js"
}
}
</script>
We can also store import maps in external files (the content type must be application/importmap+json
):
<script type="importmap" src="import-map.importmap"></script>
Terminology:
How do import maps work? Roughly, Whenever JavaScript encounters an import statement or a dynamic import()
, it resolves its module specifier: It looks for the first map entry whose specifier key matches the import specifier and replaces the key’s occurrence with the resolution result. We’ll see concrete examples soon. Let’s dig into more details first.
Specifier keys. There are two general categories of specifier keys:
In category 1, there are three kinds of specifier keys:
"lodash-es"
"lodash-es/pick"
'/'
, './'
or '../'
:"/js/app.mjs"
"../util/string-tools.js"
"https://www.unpkg.com/lodash-es/lodash.js"
Category 2 contains the same kinds of specifier keys – except that they all end with slashes.
Resolution results. Resolution results can also end with slashes or not (they have to mirror what the specifier key looks like). There are only two kinds of non-prefix resolution results (bare specifiers are not allowed):
"https://www.unpkg.com/lodash-es/lodash.js"
'/'
, './'
, '../'
):"/node_modules/lodash-es/lodash.js"
"./src/util/string-tools.mjs"
Prefix resolution results also consist of URLs and relative references but always end with slashes.
Normalization. For import maps, normalization is important:
.html
file or .importmap
file).Normalization makes it possible to match relative specifier keys against relative import specifiers.
Resolving import module specifiers. Resolution is the process of converting a module specifier to a fetchable URL. To resolve an import specifier, JavaScript looks for the first map entry whose specifier key matches the import specifier:
A specifier key without a slash matches an import specifier if both are exactly equal. In that case, resolution returns the corresponding resolution result.
A specifier key with a slash matches any import specifier that starts with it – in other words if the specifier key is a prefix of the import specifier. In that case, resolution replaces the occurrence of the specifier key in the import specifier with the resolution result and returns the outcome.
If there is no matching map entry, the import specifier is returned unchanged.
Note that import maps do not affect <script>
elements, only JavaScript imports.
With package exports, we can define:
".": "./dist/src/main.js",
"./util/errors": "./dist/src/util/errors.js",
"./util/errors.js": "./dist/src/util/errors.js"
"./*": "./dist/src/*",
"./lib/*": "./dist/src/*.js",
Import maps have analogs to the first two features, but nothing that is similar to the last feature.
{
"imports": {
"lodash-es": "/node_modules/lodash-es/lodash.js",
"date-fns": "/node_modules/date-fns/esm/index.js"
}
}
This import map enables these imports:
import {pick} from 'lodash-es';
import {addYears} from 'date-fns';
{
"imports": {
"lodash-es": "/node_modules/lodash-es/lodash.js",
"lodash-es/pick": "/node_modules/lodash-es/pick.js",
"lodash-es/zip.js": "/node_modules/lodash-es/zip.js",
}
}
We can now import like this:
import pick from 'lodash-es/pick'; // default import
import zip from 'lodash-es/zip.js'; // default import
import {pick as pick2, zip as zip2} from 'lodash-es';
{
"imports": {
"lodash-es/": "/node_modules/lodash-es/",
}
}
We can now import like this:
import pick from 'lodash-es/pick.js'; // access default export
import {pick as pick2} from 'lodash-es/lodash.js';
{
"imports": {
"https://www.unpkg.com/lodash-es/lodash.js": "/node_modules/lodash-es/lodash.js"
}
}
We are remapping the import from an external resource to a local file:
import {pick} from 'https://www.unpkg.com/lodash-es/lodash.js';
{
"imports": {
"https://www.unpkg.com/lodash-es/": "/node_modules/lodash-es/"
}
}
The online package contents are remapped to the version we downloaded into node_modules/
:
import {pick} from 'https://www.unpkg.com/lodash-es/lodash.js';
import pick from 'https://www.unpkg.com/lodash-es/pick.js';
To ensure that trees of connected files (think HTML using CSS, image and JavaScript files) are updated together, a common technique is to mention the hash of the file (which is also called its message digest) in the filename. The hash serves as a version number for the file.
An import map enables us to use import specifiers without hashes:
{
"imports": {
"/js/app.mjs": "/js/app-28501dff.mjs",
"/js/lib.mjs": "/js/lib-1c9248c2.mjs",
}
}
Scopes lets us override map entries depending on where an import is made from. That looks as follows (example taken from the import maps explainer document):
{
"imports": {
"querystringify": "/node_modules/querystringify/index.js"
},
"scopes": {
"/node_modules/socksjs-client/": {
"querystringify": "/node_modules/socksjs-client/querystringify/index.js"
}
}
}
Further reading:
Sources of this blog post:
Acknowledgements:
The following people provided important input for this blog post:
I’m very grateful that these people reviewed this blog post: