Simple monorepos via npm workspaces and TypeScript project references

[2021-07-21] dev, typescript, esm, nodejs

A monorepo is a single repository that is used to manage multiple projects. In this blog post, we’ll explore how to set up a simple monorepo for two npm packages. All we need is already built into npm and TypeScript.

What is a monorepo and why is it useful?

Whenever we have to develop multiple interdependent npm packages in parallel, we have two options:

We can keep the packages in separate repositories and publish them to npm separately.
We can keep all packages in a single repository and publish them to npm from there.

The benefit of (2) is that it’s easier to keep the packages in sync: We can install and build all packages at the same time. And, in Visual Studio Code, we can jump between packages while editing.

My use case for a monorepo: static site generation

“Monorepo” sounds fancy, but my use case for it is actually relatively simple. I am currently working on a minimal static site generator that is called Stoa. It comes in two parts:

The npm package @rauschma/stoa contains the tool and is published to the npm registry.
The npm package @rauschma/demo-blog is just an (unpublished) directory and contains:
- The blog itself is a directory with Markdown files.
- The look of the site is defined via TypeScript (JSX and Preact).
- A main module invokes the command line interface of Stoa and passes it configuration data (incl. the JSX views for rendering pages).

My first failed attempt: local path installations

I started developing via so-called local path installations:

cd demo-blog/
npm install ../stoa/

Afterward, demo-blog/package.json has the following dependency:

{
  "name": "@rauschma/demo-blog",
  "dependencies": {
    "stoa": "file:../stoa",
    ···
  },
  ···
}

This approach has several upsides:

It’s simple: There is nothing extra to configure or install.
It’s easier than using npm link, which involves two steps and leads to global changes.
In demo-blog/node_modules there is a symbolic link (symlink) to stoa/, which means that, as Stoa is developed, we’ll see the changes from demo-blog/. (Caveat: yarn does not use symlinks, it copies the dependency’s files over.)

But it also has significant downsides:

The way demo-blog/package.json is set up now, it can’t use the version of Stoa in the npm registry.
Installing and building must be done separately for each directory (vs. once for a monorepo).
Due to the symlink, packages that are dependencies of both stoa and demo-blog are not de-duplicated. That is fatal for some packages – for example, we can’t use hooks in React and Preact if the render function and the JSX components come from different packages.

A better solution: npm workspaces and TypeScript project references

After my failed attempt with local path installations, I set up a monorepo for stoa and demo-blog.

Producing ESM modules via TypeScript

In a previous blog post, I explained how to produce ESM modules via TypeScript. That’s also what I have configured for both packages in the monorepo. It has the following file system layout:

stoa-packages/
  stoa/
    package.json
    tsconfig.json
    ts/
      gen/
      client/
      test/
    dist/
  demo-blog/
    package.json
    tsconfig.json
    ts/
      gen/
      client/
    dist/

stoa/package.json looks like this:

{
  "name": "@rauschma/stoa",
  "type": "module",
  "exports": {
    "./gen/*": "./dist/gen/*.js",
    "./client/*": "./dist/client/*.js"
  },
  "typesVersions": {
    "*": {
      "gen/*": [
        "dist/gen/*"
      ],
      "client/*": [
        "dist/client/*"
      ]
    }
  },
  "dependencies": {
    ···
  }
}

"type" tells Node.js to interpret .js files as ESM modules (not CommonJS modules).

"exports" configures the JavaScript level. It means that, e.g.:

File stoa-packages/stoa/dist/gen/util/regexp-tools.js
can be imported via '@rauschma/stoa/gen/util/regexp-tools'.

In other words, this setting achieves two things:

We don’t have to mention directory 'dist' in module specifiers.
We don’t have to mention the filename extension '.js' in module specifiers.

"typesVersions" makes sure that TypeScript finds the type definitions (.d.ts files) that it needs.

This is what’s in demo-blog/package.json:

{
  "name": "@rauschma/demo-blog",
  "type": "module",
  "dependencies": {
    "@rauschma/stoa": "*",
    ···
  },
  "scripts": {
    "all": "node ./dist/gen/main.js all"
  }
}

The command npm run all is defined via "scripts" and starts generation via the JavaScript version of demo-blog/ts/gen/main.ts. The latter file contains:

import { cli } from '@rauschma/stoa/gen/core/cli';

const projectDirPath = url.fileURLToPath(
  new url.URL('../../', import.meta.url));

cli({
  projectDirPath,
  ···
});

So far, we are still not in monorepo territory: Each of the two packages stoa and demo-blog exists in its own (mostly separate) directory.

npm workspaces

A workspace is what npm calls a monorepo: A directory with subdirectories that are npm packages. We turn stoa-packages/ into a workspace by adding a package.json to it:

stoa-packages/
  package.json
  node_modules/
    @rauschma/
      stoa -> ../../stoa
      demo-blog -> ../../demo-blog
  stoa/
  demo-blog/

stoa-packages/package.json looks like this:

{
  "name": "stoa-packages",
  "workspaces": [
    "stoa",
    "demo-blog"
  ]
}

Unfortunately, npm overloads the term “workspaces”: The packages in an npm workspace are also called workspaces.

Now we can do:

cd stoa-packages/
npm install

Then this happens:

All dependencies of stoa and demo-blog are installed into stoa-packages/node_modules.
stoa-packages/node_modules also contains symbolic links to stoa-packages/stoa/ and stoa-packages/demo-blog/.

stoa and demo-blog do not have their own node_modules directory. However, when they import modules, Node.js looks for them in the next node_modules higher up in the file tree. The symlink in node_modules enables demo-blog to import from @rauschma/stoa.

What have we achieved?

Duplicate packages are not an issue anymore because all package dependencies are installed into the same node_modules.
demo-blog can import stoa as if the former were a standalone directory and the latter were a published package.
We can use a single command to install all dependencies.
We can run npm commands in multiple workspaces (details).
demo-blog automatically sees all changes we make in stoa.

TypeScript project references

We still need to compile each of the two packages separately via TypeScript. We can fix that via project references, which are the TypeScript name for a monorepo. We need to create three files:

stoa-packages/tsconfig.json
stoa-packages/stoa/tsconfig.ref.json
stoa-packages/demo-blog/tsconfig.ref.json

The file system layout now looks like this:

stoa-packages/
  tsconfig.json
  stoa/
    tsconfig.json
    tsconfig.ref.json
    ts/
      gen/
      client/
      test/
    dist/
  demo-blog/
    tsconfig.json
    tsconfig.ref.json
    ts/
      gen/
      client/
    dist/

This is stoa-packages/tsconfig.json:

{
  "files": [],
  "references": [
    {
      "path": "./stoa/tsconfig.ref.json"
    },
    {
      "path": "./demo-blog/tsconfig.ref.json"
    },
  ],
}

The normal stoa-packages/stoa/tsconfig.json (which we need in standalone mode) contains:

{
  "compilerOptions": {
    "rootDir": "ts",
    "outDir": "dist",
    "target": "es2021",
    "lib": [
      "es2021", "DOM"
    ],
    "module": "ES2020",
    "moduleResolution": "Node",
    "strict": true,
    "noImplicitOverride": true,
    // Needed for CommonJS modules: markdown-it, fs-extra
    "allowSyntheticDefaultImports": true,
    //
    "jsx": "react-jsx",
    "jsxImportSource": "preact",
    //
    "sourceMap": true,
    "declaration": true,
    "declarationMap": true, // enables importers to jump to source
  }
}

This tsconfig.json has a sibling tsconfig.ref.json that is required due to the project reference in stoa-packages/tsconfig.json:

{
  "extends": "./tsconfig.json",
  "include": ["ts/**/*"],
  "compilerOptions": {
    "composite": true,
  },
}

Let’s examine the properties:

"extends" lets us add the properties to the standalone tsconfig.json that we need to make project references work. Alas, we can’t add them to tsconfig.json itself because then it wouldn’t work in standalone mode anymore.
"include" is required for project references.
compilerOptions.composite must be true for project references.

What have we achieved? We can now use single commands to clean, build, watch (etc.) all packages.

For example, we can add these scripts to stoa-packages/package.json:

{
  ···
  "scripts": {
    "clean": "tsc --build --clean",
    "build": "tsc --build",
    "watch": "tsc --build --watch"
  },
  ···
}

Another benefit is that we can click (Mac: cmd-click, Windows: ctrl-click) on something that demo-blog imported from stoa and Visual Studio Code will jump to the original source code – and not to the .d.ts file (details).

Tip: What to do when Visual Studio Code doesn’t see changes in another package

Sometimes, we make a change in one package and Visual Studio Code doesn’t see that change in another package that depends on it. There are two things we can do when that happens:

We can execute the command “TypeScript: Restart TS Server” (details).
Opening the relevant .d.ts file also usually helps.

One step remains: publishing

I have not shown you how to publish stoa-packages/stoa to npm and how to turn stoa-packages/demo-blog into a downloadable archive, but that’s relatively easy to achieve.

Conclusion

We have seen how we can set up a very simple monorepo by only using what’s already built into npm and TypeScript. That makes it much easier to develop multiple packages in parallel.

I managed to preserve the ability to compile package demo-blog on its own. I haven’t seen that in the other TypeScript project references setups that I’ve come across.

Three wishes

I am very happy with this setup, but still have three wishes related to TypeScript:

If I refactor in Visual Studio Code, the changes only affect a single package. It would be nice if all packages were changed.
With an npm workspace, I don’t have to change a package when I add it to a workspace. Alas, that’s not true for TypeScript project references where I must add a tsconfig.
I wish I didn’t have to add "typesVersions" to a package.json to make "exports" work with TypeScript. I’m hoping that TypeScript will be able to derive this information from "exports" in the future.

Simple monorepos via npm workspaces and TypeScript project references

What is a monorepo and why is it useful? #

My use case for a monorepo: static site generation #

My first failed attempt: local path installations #

A better solution: npm workspaces and TypeScript project references #

Producing ESM modules via TypeScript #

npm workspaces #

TypeScript project references #

Tip: What to do when Visual Studio Code doesn’t see changes in another package #

One step remains: publishing #

Conclusion #

Three wishes #

Further reading #