Creating ESM-based shell scripts for Unix and Windows with Node.js

In this blog post, we learn how to implement shell scripts via Node.js ESM modules. There are two common ways of doing so:

• We can write a stand-alone script and install it ourselves.
• We can put our script in an npm package and use a package manager to install it. That also gives us the option to publish the package to the npm registry so that others can install it, too.

Required knowledge  #

You should be loosely familiar with the following two topics:

• ECMAScript Modules, as explained in chapter “modules” in “JavaScript for impatient programmers”.
• npm packages, as explained in the blog post “Publishing and consuming ECMAScript modules via packages – the big picture”.

What’s next in this blog post  #

Windows doesn’t really support standalone shell scripts written in JavaScript. Therefore, we’ll first look into how to write standalone scripts with filename extensions for Unix. That knowledge will help us with creating packages that contain shell scripts. Later, we’ll learn:

• A trick for writing standalone shell scripts on Windows.
• A trick for writing standalone shell scripts without filename extensions on Unix.

Installing shell scripts via packages is the topic of another blog post.

Node.js ESM modules as standalone shell scripts on Unix  #

Let’s turn an ESM module into a Unix shell script that we can run without it being inside a package. In principle, we can choose between two filename extensions for ESM modules:

• .mjs files are always interpreted as ESM modules.
• .js files are only interpreted as ESM modules if the closest package.json has the following entry:

  "type": "module"
  

However, since we want to create a standalone script, we can’t rely on package.json being there. Therefore, we have to use the filename extension .mjs (we’ll get to workarounds later).

The following file has the name hello.mjs:

import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);

We can already run this file:

node hello.mjs

Node.js shell scripts on Unix  #

We need to do two things so that we can run hello.mjs like this:

./hello.mjs

These things are:

• Adding a hashbang line at the beginning of hello.mjs
• Making hello.mjs executable

Hashbangs on Unix  #

In a Unix shell script, the first line is a hashbang – metadata that tells the shell how to execute the file. For example, this is the most common hashbang for Node.js scripts:

#!/usr/bin/env node

This line has the name “hashbang” because it starts with a hash symbol and an exclamation mark. It is also often called “shebang”.

If a line starts with a hash, it is a comment in most Unix shells (sh, bash, zsh, etc.). Therefore, the hashbang is ignored by those shells. Node.js also ignores it, but only if it is the first line.

Why don’t we use this hashbang?

#!/usr/bin/node

Not all Unixes install the Node.js binary at that path. How about this path then?

#!node

Alas, not all Unixes allow relative paths. That’s why we refer to env via an absolute path and use it to run node for us.

For more information on Unix hashbangs, see “Node.js shebang” by Alex Ewerlöf.

Passing arguments to the Node.js binary  #

What if we want to pass arguments such as command line options to the Node.js binary?

One solution that works on many Unixes is to use option -S for env which prevents it from interpreting all of its arguments as a single name of a binary:

#!/usr/bin/env -S node --disable-proto=throw

On macOS, the previous command works even without -S; on Linux it usually doesn’t.

Hashbang pitfall: creating hashbangs on Windows  #

If we use a text editor on Windows to create an ESM module that should run as a script on either Unix or Windows, we have to add a hashbang. If we do that, the first line will end with the Windows line terminator \r\n:

#!/usr/bin/env node\r\n

Running a file with such a hashbang on Unix produces the following error:

env: node\r: No such file or directory

That is, env thinks the name of the executable is node\r. There are two ways to fix this.

First, some editors automatically check which line terminators are already used in a file and keep using them. For example, Visual Studio Code, shows the current line terminator (it calls it “end of line sequence”) in the status bar at the bottom right:

• LF (line feed) for the Unix line terminator \n
• CRLF (carriage return, line feed) for the Windows line terminator \r\n

We can switch pick a line terminator by clicking on that status information.

Second, we can create a minimal file my-script.mjs with only Unix line terminators that we never edit on Windows:

#!/usr/bin/env node
import './main.mjs';

Making files executable on Unix  #

In order to become a shell script, hello.mjs must also be executable (a permission of files), in addition to having a hashbang:

chmod u+x hello.mjs

Note that we made the file executable (x) for the user who created it (u), not for everyone.

Running hello.mjs directly  #

hello.mjs is now executable and looks like this:

#!/usr/bin/env node

import * as os from 'node:os';

const {username} = os.userInfo();
console.log(`Hello ${username}!`);

We can therefore run it like this:

./hello.mjs

Alas, there is no way to tell node to interpret a file with an arbitrary extension as an ESM module. That’s why we have to use the extension .mjs. Workarounds are possible but complicated, as we’ll see later.

Creating an npm package with shell scripts  #

In this section we create an npm package with shell scripts. We then examine how we can install such a package so that its scripts become available at the command line of your system (Unix or Windows).

The finished package is available here:

• On GitHub as rauschma/demo-shell-scripts
• On npm as @rauschma/demo-shell-scripts

Setting up the package’s directory  #

These commands work on both Unix and Windows:

mkdir demo-shell-scripts
cd demo-shell-scripts
npm init --yes

Now there are the following files:

demo-shell-scripts/
  package.json

package.json for unpublished packages  #

One option is to create a package and not publish it to the npm registry. We can still install such a package on our system (as explained later). In that case, our package.json looks as follows:

{
  "private": true,
  "license": "UNLICENSED"
}

Explanations:

• Making the package private means that no name or version is needed and that it can’t be accidentally published.
• "UNLICENSED" denies others the right to use the package under any terms.

package.json for published packages  #

If we want to publish our package to the npm registry, our package.json looks like this:

{
  "name": "@rauschma/demo-shell-scripts",
  "version": "1.0.0",
  "license": "MIT"
}

For your own packages, you need to replace the value of "name" with a package name that works for you:

• Either a globally unique name. Such a name should only be used for important packages because we don’t want to prevent others from using the name otherwise.

• Or a scoped name: To publish a package, you need an npm account (how to get one is explained later). The name of your account can be used as a scope for package names. For example, if your account name is jane, you can use the following package name:

  "name": "@jane/demo-shell-scripts"
  

Adding dependencies  #

Next, we install a dependency that we want to use in one of our scripts – package lodash-es (the ESM version of Lodash):

npm install lodash-es

This command:

• Creates the directory node_modules.
• Installs package lodash-es into it.
• Adds the following property to package.json:

  "dependencies": {
    "lodash-es": "^4.17.21"
  }
  

• Creates the file package-lock.json.

If we only use a package during development, we can add it to "devDependencies" instead of to "dependencies" and npm will only install it if we run npm install inside our package’s directory, but not if we install it as a dependency. A unit testing library is a typical dev dependency.

These are two ways in which we can install a dev dependency:

• Via npm install some-package.
• We can use npm install some-package --save-dev and then manually move the entry for some-package from "dependencies" to "devDependencies".

The second way means that we can easily postpone the decision whether a package is a dependency or a dev dependency.

Adding content to the package  #

Let’s add a readme file and two modules homedir.mjs and versions.mjs that are shell scripts:

demo-shell-scripts/
  package.json
  package-lock.json
  README.md
  src/
    homedir.mjs
    versions.mjs

We have to tell npm about the two shell scripts so that it can install them for us. That’s what property "bin" in package.json is for:

"bin": {
  "homedir": "./src/homedir.mjs",
  "versions": "./src/versions.mjs"
}

If we install this package, two shell scripts with the names homedir and versions will become available.

You may prefer the filename extension .js for the shell scripts. Then, instead of the previous property, you have to add the following two properties to package.json:

"type": "module",
"bin": {
  "homedir": "./src/homedir.js",
  "versions": "./src/versions.js"
}

The first property tells Node.js that it should interpret .js files as ESM modules (and not as CommonJS modules – which is the default).

This is what homedir.mjs looks like:

#!/usr/bin/env node
import {homedir} from 'node:os';

console.log('Homedir: ' + homedir());

This module starts with the aforementioned hashbang which is required if we want to use it on Unix. It imports function homedir() from the built-in module node:os, calls it and logs the result to the console (i.e., standard output).

Note that homedir.mjs does not have to be executable; npm ensure executability of "bin" scripts when it installs them (we’ll see how soon).

versions.mjs has the following content:

#!/usr/bin/env node

import {pick} from 'lodash-es';

console.log(
  pick(process.versions, ['node', 'v8', 'unicode'])
);

We import function pick() from Lodash and use it to display three properties of the object process.versions.

Running the shell scripts without installing them  #

We can run, e.g., homedir.mjs like this:

cd demo-shell-scripts/
node src/homedir.mjs

How npm installs shell scripts  #

Installation on Unix  #

A script such as homedir.mjs does not need to be executable on Unix because npm installs it via an executable symbolic link:

• If we install the package globally, the link is added to a directory that’s listed in $PATH.
• If we install the package locally (as a dependency), the link is added to node_modules/.bin/

Installation on Windows  #

To install homedir.mjs on Windows, npm creates three files:

• homedir.bat is a Command shell script that uses node to execute homedir.mjs.
• homedir.ps1 does the same for PowerShell.
• homedir does the same for Cygwin, MinGW, and MSYS.

npm adds these files to a directory:

• If we install the package globally, the files are added to a directory that’s listed in %Path%.
• If we install the package locally (as a dependency), the files are added to node_modules/.bin/

Publishing the example package to the npm registry  #

Let’s publish package @rauschma/demo-shell-scripts (which we have created previously) to npm. Before we use npm publish to upload the package, we should check that everything is configured properly.

Which files are published? Which files are ignored?  #

The following mechanisms are used to exclude and include files when publishing:

• The files listed in the top-level file .gitignore are excluded.

  • We can override .gitignore with the file .npmignore, which has the same format.

• The package.json property "files" contains an Array with the names of files that are included. That means we have a choice of listing either the files we want to exclude (in .npmignore) or the files we want to include.

• Some files and directories are excluded by default – e.g.:

  • node_modules
  • .*.swp
  • ._*
  • .DS_Store
  • .git
  • .gitignore
  • .npmignore
  • .npmrc
  • npm-debug.log

  Except for these defaults, dot files (files whose names start with dots) are included.

• The following files are never excluded:

  • package.json
  • README.md and its variants
  • CHANGELOG and its variants
  • LICENSE, LICENCE

The npm documentation has more details on what’s included and whats excluded when publishing.

Checking if a package is properly configured  #

There are several things we can check before we upload a package.

Checking which files will be uploaded  #

A dry run of npm install runs the command without uploading anything:

npm publish --dry-run

This displays which files would be uploaded and several statistics about the package.

We can also create an archive of the package as it would exist on the npm registry:

npm pack

This command creates the file rauschma-demo-shell-scripts-1.0.0.tgz in the current directory.

Installing the package globally – without uploading it  #

We can use either of the following two commands to install our package globally without publishing it to the npm registry:

npm link
npm install . -g

To see if that worked, we can open a new shell and check if the two commands are available. We can also list all globally installed packages:

npm ls -g

Installing the package locally (as a depencency) – without uploading it  #

To install our package as a dependency, we have to execute the following commands (while we are in directory demo-shell-scripts):

cd ..
mkdir sibling-directory
cd sibling-directory
npm init --yes
npm install ../demo-shell-scripts

We can now run, e.g., homedir with either one of the following two commands:

npx homedir
./node_modules/.bin/homedir

npm publish: uploading packages to the npm registry  #

Before we can upload our package, we need to create an npm user account. The npm documentation describes how to do that.

Then we can finally publish our package:

npm publish --access public

We have to specify public access because the defaults are:

• public for unscoped packages

• restricted for scoped packages. This setting makes a package private – which is a paid npm feature used mostly by companies and different from "private":true in package.json. Quoting npm: “With npm private packages, you can use the npm registry to host code that is only visible to you and chosen collaborators, allowing you to manage and use private code alongside public code in your projects.”

Option --access only has an effect the first time we publish. Afterward, we can omit it and need to use npm access to change the access level.

We can change the default for the initial npm publish via publishConfig.access in package.json:

"publishConfig": {
  "access": "public"
}

A new version is required for every upload  #

Once we have uploaded a package with a specific version, we can’t use that version again, we have to increase either of the three components of the version:

major.minor.patch

• We increase major if we made breaking changes.
• We increase minor if we made backward-compatible changes.
• We increase patch if we made small fixes that don’t really change the API.

Automatically performing tasks every time before publishing  #

There may be steps that we want to perform every time before we upload a package – e.g.:

• Running unit tests
• Compiling TypeScript code to JavaScript code

That can be done automatically via the package.json property `"scripts". That property can look like this:

"scripts": {
  "build": "tsc",
  "test": "mocha --ui qunit",
  "dry": "npm publish --dry-run",
  "prepublishOnly": "npm run test && npm run build"
}

mocha is a unit testing library. tsc is the TypeScript compiler.

The following package scripts are run before npm publish:

• "prepare" is run:
  • Before npm pack
  • Before npm publish
  • After a local npm install without arguments
• "prepublishOnly" is run only before npm publish.

Standalone Node.js shell scripts with arbitrary extensions on Unix  #

Unix: arbitrary filename extension via a custom executable  #

The Node.js binary node uses the filename extension to detect which kind of module a file is. There currently is no command line option to override that. And the default is CommonJS, which is not what we want.

However, we can create our own executable for running Node.js and, e.g., call it node-esm. Then we can rename our previous standalone script hello.mjs to hello (without any extension) if we change the first line to:

#!/usr/bin/env node-esm

Previously, the argument of env was node.

This is an implementation of node-esm proposed by Andrea Giammarchi:

#!/usr/bin/env sh
input_file=$1
shift
exec node --input-type=module - $@ < $input_file

This executable sends the content of a script to node via standard input. The command line option --input-type=module tells Node.js that the text it receives is an ESM module.

We also use the following Unix shell features:

• $1 contains the the first argument passed to node-esm – the path of the script.
• We delete argument $0 (the path of node-esm) via shift and pass on the remaining arguments to node via $@.
• exec replaces the current process with the one in which node runs. That ensures that the script exits with the same code as node.
• The hyphen (-) separates Node’s arguments from the script’s arguments.

Before we can use node-esm, we have to make sure that it is executable and can be found via the $PATH. How to do that is explained later.

Unix: arbitrary filename extension via a shell prolog  #

We have seen that we can’t specify the module type for a file, only for standard input. Therefore, we can write a Unix shell script hello that uses Node.js to run itself as an ESM module (based on work by sambal.org):

#!/bin/sh
':' // ; cat "$0" | node --input-type=module - $@ ; exit $?

import * as os from 'node:os';

const {username} = os.userInfo();
console.log(`Hello ${username}!`);

Most of the shell features that we are using here are described at the beginning of this blog post. $? contains the exit code of the last shell command that was executed. That enables hello to exit with the same code as node.

The key trick used by this script is that the second line is both Unix shell script code and JavaScript code:

• As shell script code, it runs the quoted command ':' which does nothing beyond expanding its arguments and performing redirections. Its only argument is the path //. Then it pipes the contents of the current file to the node binary.

• As JavaScript code, it is the string ':' (which is interpreted as an expression statement and does nothing), followed by a comment.

An additional benefit of hiding the shell code from JavaScript is that JavaScript editors won’t be confused when it comes to processing and displaying the syntax.

Standalone Node.js shell scripts on Windows  #

Windows: configuring the filename extension .mjs  #

One option for creating standalone Node.js shell scripts on Windows is to the filename extension .mjs and configure it so that files that have it are run via node. Alas that only works for the Command shell, not for PowerShell.

Another downside is that we can’t pass arguments to a script that way:

>more args.mjs
console.log(process.argv);

>.\args.mjs one two
[
  'C:\\Program Files\\nodejs\\node.exe',
  'C:\\Users\\jane\\args.mjs'
]

>node args.mjs one two
[
  'C:\\Program Files\\nodejs\\node.exe',
  'C:\\Users\\jane\\args.mjs',
  'one',
  'two'
]

How do we configure Windows so that the Command shell directly runs files such as args.mjs?

File associations specify which app a file is opened with when we enter its name in a shell. If we associate the filename extension .mjs with the Node.js binary, we can run ESM modules in shells. One way to do that is via the Settings app, as explained in “How to Change File Associations in Windows” by Tim Fisher.

If we additionally add .MJS to the variable %PATHEXT%, we can even omit the filename extension when referring to an ESM module. This environment variable can be changed permanently via the Settings app – search for “variables”.

Windows Command shell: Node.js scripts via a shell prolog  #

On Windows, we are facing the challenge that there is no mechanism like hashbangs. Therefore, we have to use a workaround that is similar to the one we used for extensionless files on Unix: We create a script that runs the JavaScript code inside itself via Node.js.

Command shell scripts have the filename extension .bat. We can run a script named script.bat via either script.bat or script.

This is what hello.mjs looks like if we turn it into a Command shell script hello.bat:

:: /*
@echo off
more +5 %~f0 | node --input-type=module - %*
exit /b %errorlevel%
*/

import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);

Running this code as a file via node would require two features that don’t exist:

• Using a command line option to override extension-less files being interpreted as ESM modules by default.
• Skipping lines at the beginning of a file.

Therefore, we have no choice but to pipe the file’s content into node. We also use the following command shell features:

• %~f0 contains the full path of the current script, including its filename extension. In contrast, %0 contains the command that was used to invoke the script. Therefore, the former shell variable enables us to invoke the script via either hello or hello.bat.
• %* contains the command’s arguments – which we pass on to node.
• %errorlevel% contains the exit code of the last command that was executed. We use that value to exit with the same code that was specified by node.

Windows PowerShell: Node.js scripts via a shell prolog  #

We can use a trick similar to the one used in the previous section and turn hello.mjs into a PowerShell script hello.ps1 as follows:

Get-Content $PSCommandPath | Select-Object -Skip 3 | node --input-type=module - $args
exit $LastExitCode
<#
import * as os from 'node:os';
const {username} = os.userInfo();
console.log(`Hello ${username}!`);
// #>

We can run this script via either:

.\hello.ps1
.\hello

However, before we can do that, we need to set an execution policy that allows us to run PowerShell scripts (more information on execution policies):

• The default policies on Windows clients is Restricted and doesn’t let us run any scripts.
• The policy RemoteSigned lets us run unsigned local scripts. Downloaded scripts must be signed. This is the default on Windows servers.

The following command lets us run local scripts:

Set-ExecutionPolicy -Scope CurrentUser RemoteSigned

Creating native binaries for Linux, macOS, and Windows  #

The npm package pkg turns a Node.js package into a native binary that even runs on systems where Node.js isn’t installed. It supports the following platforms: Linux, macOS, and Windows.

Shell paths: making sure shells find scripts  #

In most shells, we can type in a filename without directly referring to a file and they search several directories for a file with that name and run it. Those directories are usually listed in a special shell variable:

• In most Unix shells, we access it via $PATH.
• In the Windows Command shell, we access it via %Path%.
• In PowerShell, we access it via $Env:PATH.

We need the PATH variable for two purposes:

• If we want to install our custom Node.js executable node-esm.
• If we want to run a standalone shell script without directly referring to its file.

Unix: $PATH  #

Most Unix shells have the variable $PATH that lists all paths where a shell looks for executables when we type in a command. Its value may look like this:

$ echo $PATH
/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin

The following command works on most shells (source) and changes the $PATH until we leave the current shell:

export PATH="$PATH:$HOME/bin"

The quotes are needed in case one of the two shell variables contains spaces.

Permanently changing the $PATH  #

On Unix, how the $PATH is configured depends on the shell. You can find out which shell you are running via:

echo $0

MacOS uses Zsh where the best place to permanently configure $PATH is the startup script $HOME/.zprofile – like this:

path+=('/Library/TeX/texbin')
export PATH

Changing the PATH variable on Windows (Command shell, PowerShell)  #

On Windows, the default environment variables of the Command shell and PowerShell can be configured (permanently) via the Settings app – search for “variables”.