ShadowRealms – an ECMAScript proposal for a better eval()

[2022-04-04] dev, javascript, es proposal
(Ad, please don’t block)
  • Update 2022-04-05:
    • Rewrote section “The global data in ShadowRealms is a subset of the platform’s global data”.
    • New section “Uncaught errors thrown in evaluated code”.
    • A longer explanation in section “A simple library for transferring objects across realms”
    • Added Rick Waldron to the list of authors/champions.

This blog post describes the ECMAScript proposal “ShadowRealm API” by Dave Herman, Caridy Patiño, Mark S. Miller, Leo Balter, and Rick Waldron.

Class ShadowRealm provides a new way of evaluating code at runtime – think eval() but better:

  • Each instance has its own global JavaScript scope.
  • Code is evaluated in that scope. If it changes global data, that only affects the ShadowRealm, but not the real global data.

Realms  

A realm is an instance of the JavaScript platform: its global environment with all built-in data set up correctly.

For example, each iframe has an associated realm:

<body>
  <iframe>
  </iframe>
  <script>
    const win = frames[0].window;
    console.assert(win.globalThis !== globalThis); // (A)
    console.assert(win.Array !== Array); // (B)
  </script>
</body>

The global object of the document is different from the global object of the iframe (line A). Global variables such as Array are different, too (line B).

The ShadowRealm API  

Quoting the proposal:

The primary goal of this proposal is to provide a proper mechanism to control the execution of a program, providing a new global object, a new set of intrinsics, no access to objects cross-realms, a separate module graph and synchronous communication between both realms.

ShadowRealms execute code with the same JavaScript heap as the surrounding context where the ShadowRealm is created. Code runs synchronously in the same thread.

ShadowRealm is a class with the following type signature:

declare class ShadowRealm {
  constructor();
  evaluate(sourceText: string): PrimitiveValueOrCallable;
  importValue(specifier: string, bindingName: string): Promise<PrimitiveValueOrCallable>;
}

Each instance of ShadowRealm has its own realm. Two methods allow us to evaluate code inside that realm:

  • .evaluate() synchronously evaluates a string sourceText inside a ShadowRealm. This method is loosely similar to eval().
  • .importValue(): asynchronously imports inside a ShadowRealm and returns the result via a Promise. That means we get non-blocking evaluation (for third-party scripts etc.).

The realm in which we instantiate ShadowRealm is called the incubator realm. The instance is called the child realm.

shadowRealm.evaluate()  

Method .evaluate() has the following type signature:

evaluate(sourceText: string): PrimitiveValueOrCallable;

.evaluate() works much like eval():

const sr = new ShadowRealm();
console.assert(
  sr.evaluate(`'ab' + 'cd'`) === 'abcd'
);

In contrast to eval(), the code is evaluated inside the realm of .evaluate():

globalThis.realm = 'incubator realm';

const sr = new ShadowRealm();
sr.evaluate(`globalThis.realm = 'child realm'`);
console.assert(
  sr.evaluate(`globalThis.realm`) === 'child realm'
);

If .evaluate() returns a function, that function is wrapped so that invoking it from the outside, runs it inside the ShadowRealm:

globalThis.realm = 'incubator realm';

const sr = new ShadowRealm();
sr.evaluate(`globalThis.realm = 'child realm'`);

const wrappedFunc = sr.evaluate(`() => globalThis.realm`);
console.assert(wrappedFunc() === 'child realm');

Whenever a value is passed to or from a ShadowRealm, it must be primitive or callable. If not, an exception is thrown:

> new ShadowRealm().evaluate('[]')
TypeError: value passing between realms must be callable or primitive

More on passing values between realms later.

shadowRealm.importValue()  

Method .importValue() has the following type signature:

importValue(specifier: string, bindingName: string): Promise<PrimitiveValueOrCallable>;

Inside its ShadowRealm, it imports the named import whose name is bindingName from a module whose specifier is specifier and returns its value asynchronously, via a Promise. As with .evaluate() functions are wrapped so that invoking them from outside the ShadowRealm runs them inside the ShadowRealm:

// main.js
const sr = new ShadowRealm();
const wrappedSum = await sr.importValue('./my-module.js', 'sum');
console.assert(wrappedSum('hi', ' ', 'folks', '!') === 'hi folks!');

// my-module.js
export function sum(...values) {
  return values.reduce((prev, value) => prev + value);
}

For now, the API requires the parameter bindingName. In the future, omitting it may return (a Promise for) a module namespace object.

Therefore, if we only want to load a module without importing anything (e.g. a polyfill that changes global data), we have to use a workaround – for example:

// main.js
const sr = new ShadowRealm();
await sr.importValue('./my-module.js', 'default');

// my-module.js
export default true;
// ...

As with .evaluate(), values passed to or from ShadowRealms (incl. arguments and results of cross-realm function calls) must be primitive or callable. More on that soon.

Uncaught errors thrown in evaluated code  

Syntax errors in the code lead to a SyntaxError being thrown:

> new ShadowRealm().evaluate('someFunc(')
SyntaxError: Unexpected end of script

An uncaught error inside a ShadowRealm is delivered via a TypeError:

> new ShadowRealm().evaluate(`throw new RangeError('The message')`)
TypeError: Error encountered during evaluation

Alas, at the moment we don’t get either name, message or stack trace of the original error. Maybe that will change in the future.

The global data in ShadowRealms is a subset of the platform’s global data  

ShadowRealms contain the following global data:

  • The global data of ECMAScript

  • A subset of the platform-specific global data (such as document in browsers or process on Node.js). One key requirement is that all platform-specific properties of the global object must be configurable (which is one of the attributes of properties) so that they can be deleted. That gives us the option to hide those features from the code that we evaluate.

How Content Security Policy (CSP) affects ShadowRealms  

  • Not allowing unsafe-eval for a page prevents synchronous evaluation via .evaluate() in ShadowRealms.
  • Directives such as default-src affect what modules can be loaded via .importValue().

Passing values between realms  

There are several ways in which values can be passed between an incubator realm and a child realm:

  • Sending values to a ShadowRealm:
    • Passing arguments to a function that comes from the ShadowRealm.
  • Receiving values from a ShadowRealm:
    • The results of .evaluate() and .importValue()
    • The result of invoking a function that comes from the ShadowRealm.

Whenever a value crosses realms, the internal specification operation GetWrappedValue() is used to wrap it:

  • A primitive value is returned without any changes.

  • A callable object is wrapped in a wrapped function. More on wrapped functions later.

  • Any other kind of object causes an exception:

    > const sr = new ShadowRealm();

> sr.evaluate('globalThis')
TypeError: value passing between realms must be callable or primitive
> sr.evaluate('({prop: 123})')
TypeError: value passing between realms must be callable or primitive
> sr.evaluate('Object.prototype')
TypeError: value passing between realms must be callable or primitive

> typeof sr.evaluate('() => {}') // OK
'function'
> typeof sr.evaluate('123') // OK
'number'

Non-callable objects can’t cross realms  

Why aren’t non-callable object allowed to cross realms? The goal is complete separation between realms: Code executed in a ShadowRealm should never be able to access its incubator realm.

Objects have prototype chains. When transferring a prototype chain to another realm, we have two options:

  • We could only copy the first object in the prototype chain and let the chain continue in another realm. Alas, almost all prototype chains give us access to Function which lets us execute arbitrary code in the source realm (see code below).

  • We could copy the complete prototype chain. However, there is no simple way to transfer objects such as Array.prototype and Object.prototype.

Almost all objects have the standard property .constructor that refers to the class of the object:

> ({}).constructor === Object
true
> [].constructor === Array
true

Via this property, we can get access to globalThis:

function handleObject(someObject) {
  // Bad: the constructor of an object lets us create more instances
  const constructor = someObject.constructor;

  // Worse: the constructor of a constructor is `Function`
  // which lets us execute arbitrary code in the realm of `someObject`
  const _Function = constructor.constructor;

  const _globalThis = (new _Function('return globalThis'))();
  // We now have access to `globalThis` and can change global data
}

The ShadowRealms repository has an issue where this topic is discussed in more detail. The code above is inspired by a blog post by Joseph Griego.

We’ll see later how we can still transfer objects between realms (spoiler: it might be added as a feature in the future but can also be implemented via a library.)

Functions are wrapped when they cross realms  

Whenever a function crosses realms, it is wrapped:

  • The wrappee is a function from a foreign realm.
  • A so-called wrapped function wraps the wrappee. The wrapper protects the wrappee from the local realm and the local realm from the wrappee.

This is how a wrapped function keeps realms separate:

  • Function call arguments enter the wrappee’s realm and are wrapped for that realm (via GetWrappedValue()). The implicit argument this is treated the same way.

  • Values returned by the wrappee enter the wrapper’s realm and are wrapped for that realm.

  • The wrappee is executed in a new scope whose parent scope is the birth scope of the wrappee. As a consequence, the global scope is the global scope of the wrappee’s realm.

  • Only one feature of the wrappee is exposed: The wrapper forwards function calls. However:
    • The wrapper can’t be new-invoked.
    • Neither the wrappee’s properties nor its prototype can be accessed from the wrapper’s realm.

Note that wrapped functions are never unwrapped: If a wrapper is passed back into the wrappee’s realm, it is simply wrapped again. For details, see this issue.

The following code demonstrates what happens if a sloppy-mode function from a ShadowRealm is wrapped and called: Its this is the globalThis of the ShadowRealm, not the globalThis of the incubator realm:

const sr = new ShadowRealm();
// .evaluate() executes code in sloppy mode
const sloppyFunc = sr.evaluate(`
  (function () {
    switch (this) {
      case undefined:
        return 'this is undefined';
      case globalThis:
        return 'this is globalThis';
      default:
        throw new Error();
    }
  })
`);
console.assert(sloppyFunc() === 'this is globalThis');

This is what happens if the wrappee is a strict-mode function:

const sr = new ShadowRealm();
const strictFunc = sr.evaluate(`
  (function () {
    'use strict';
    switch (this) {
      case undefined:
        return 'this is undefined';
      case globalThis:
        return 'this is globalThis';
      default:
        throw new Error();
    }
  })
`);
console.assert(strictFunc() === 'this is undefined');

How does .evaluate() evaluate its code?  

Pitfall: no static imports  

In code parsed by .evaluate(), we can’t use static imports. Why? Due to top-level await, static imports can be asynchronous – and .evaluate() is supposed to work synchronously.

A workaround is to import dynamically via import():

// Not allowed:
import someFunc from 'some-module';
someFunc();

// Workaround 1:
import('some-module')
  .then(({someFunc}) => {
    someFunc();
  });

// Workaround 2:
async function main() {
  const {someFunc} = await import('some-module');
  someFunc();
}

The code is parsed as a sequence of statements  

The grammar rule used by .evaluate() is Script – the same as it is for non-module <script> elements. That means, it consists of a sequence of statements. The following subsections explore what that means for the code.

Completion: the result of evaluating a statement  

.evaluate() returns a value. How does it do that given that its input is a statement?

In normal JavaScript code, statements don’t evaluate to values. However, the language specification defines so-called completions for statements. Those are used by .evaluate(), eval(), browser consoles, REPLs, etc. We can use eval() to explore how completions work.

The completion of an expression statement is the value of the expression:

> eval(`'abc';`)
'abc'
> eval(`1 + 3;`)
4

The completion of a code block is the completion of the last statement in that block:

> eval(`{'first'; 'second';}`)
'second'

The completion of an if statement is the completion of either the then-branch or the else-branch – depending on the condition:

> eval(`if (true) 'then'; else 'else';`)
'then'
> eval(`if (false) 'then'; else 'else';`)
'else'
> eval(`if (true) {'then'} else {'else'}`)
'then'
> eval(`if (false) {'then'} else {'else'}`)
'else'

Syntax pitfall: function declarations vs. function expressions  

There is a syntactic ambiguity between function declarations and function expressions.

Ordinary function definitions are parsed as function declarations. That is, .evaluate() does not consider the following code to be an anonymous function expression:

> new ShadowRealm().evaluate(`function () {}`)
SyntaxError: Function statements must have a name.

The completion of a function declaration is undefined:

> new ShadowRealm().evaluate(`function myFunc() {}`)
undefined

If we want .evaluate() to return a function, we must ensure that the function definition is parsed as an expression, which we can do by wrapping it in parentheses:

> typeof new ShadowRealm().evaluate(`(function () {})`)
'function'

Note that arrow functions don’t have this issue – they are always parsed as expressions:

> typeof new ShadowRealm().evaluate(`() => {}`)
'function'

Syntax pitfall: code blocks vs. object literals  

Another syntactic ambiguity is between code blocks and object literals. Normally, .evaluate() interprets curly braces as code blocks:

> new ShadowRealm().evaluate(`{label: 'statement'}`)
'statement'

If we want to create an object, we neeed to put parentheses around the curly braces:

> new ShadowRealm().evaluate(`({label: 'statement'})`)
{ label: 'statement' }

Scoping: sloppy mode vs. strict mode  

By default, .evaluate() uses sloppy (non-strict) mode and the so-called var scope is the global scope. That means that var declarations and function declarations create properties of the global object:

const sr = new ShadowRealm();
console.assert(
  sr.evaluate(`var varDecl = 1; 'varDecl' in globalThis`) === true
);
console.assert(
  sr.evaluate(`function funcDecl() {} 'funcDecl' in globalThis`) === true
);

In contrast, a new lexical environment is created for each invocation of .evaluate(). That environment is used by let and const declarations:

const sr = new ShadowRealm();

sr.evaluate(`let letDecl = 1`);
console.assert(
  sr.evaluate(`let letDecl = 1; 'letDecl' in globalThis`) === false
);

// Global lexically scoped declarations are not remembered
// between invocations of .evaluate()
console.assert(
  sr.evaluate('typeof letDecl') === 'undefined'
);

In strict mode, the lexical environment (that is created freshly for each invocation) is also used as the var environment. Therefore, var declarations and function declarations don’t create properties of the global object:

const sr = new ShadowRealm();
console.assert(
  sr.evaluate(`'use strict'; var varDecl = 1; 'varDecl' in globalThis`) === false
);
console.assert(
  sr.evaluate(`'use strict'; function funcDecl() {} 'funcDecl' in globalThis`) === false
);

How does .importValue() evaluate its code?  

The code used by .importValue() is an ECMAScript module and we only import from the code. Therefore, none of the pitfalls of .evaluate() exist:

  • The module has its own scope whose parent scope is the global scope of the ShadowRealm.
  • Modules are in strict mode by default.

Getting objects into and out of ShadowRealms  

There currently is no built-in support for transferring objects across realms. It might be added in the future. One possibility (but this is just my guess) is for it to work like transferrable objects (which is used for moving objects between from and to web workers).

However, transferring objects can also be implemented via a library. Caridy Patiño’s IRealm is a complete experimental solution. It is based on membranes.

A simple library for transferring objects across realms  

Let’s explore what a very simply solution could look like. (Warning: don’t use this code in production. Among other things, it does not protect against code execution via Function.)

The approach is as follows:

  • Each value in a foreign realm is local-wrapped before it is transferred:
    • A primitive value isn’t changed.
    • An object (which may not be callable and therefore transferrable) is wrapped in a function (which is always transferrable). That function applies any ECMAScript Proxy trap with the name trapName and the arguments args to the wrapped object.
    • The arguments passed to the wrapped object come from another realm and must be foreign-wrapped.
    • The result of applying the trap will be transferred to another realm and must be local-wrapped.
  • After crossing over, the value is foreign-wrapped:
    • A primitive value isn’t changed.
    • A callable object (non-callable objects can’t be transferred) is wrapped in a Proxy whose handler feeds all trapped operations to the callable object.
    • The arguments of the trapped operations will be sent to another realm and must be local-wrapped.
    • The result of invoking the callable object comes from another realm and must be foreign-wrapped.

One issue remains: When we evaluate code, we must post-process the result. It must first be local-wrapped in the ShadowRealm and then be foreign-wrapped in the incubator realm. Foreign-wrapping could be done by wrapping the ShadowRealm method .evaluate(). But how do we local-wrap the evaluation result inside the ShadowRealm?

As it turns out, an elegant solution is to transfer function globalThis.eval() of the ShadowRealm:

  • We first local-wrap it inside the ShadowRealm.
  • After we receive it in the incubator realm, we foreign-wrap it.

If we call the double-wrapped eval(), the argument is evaluated inside the ShadowRealm. The result will be properly wrapped (twice), as will be any data that we extract from it (by accessing properties etc.).

// main.js
import {createShadowEval} from './membrane-lib.js';

const sr = new ShadowRealm();
const shadowEval = await createShadowEval(sr);

const obj = shadowEval(`
  ({
    brand: 'Ford',
    owner: {
      first: 'Jane',
      last: 'Doe',
    },
  })
`);

console.log(obj.brand); // 'Ford'
console.log(obj.owner.first); // 'Jane'

// membrane-lib.js
function isObject(value) {
  return (
    (value !== null && typeof value === 'object')
    || typeof value === 'function'
  );
}

/**
 * We can’t transfer a (non-callable) object across realms.
 * But we can transfer a function that applies proxy-trapped operations
 * to the object.
 */
export function wrapLocalValue(value) {
  if (isObject(value)) {
    return (trapName, ...args) => {
      const wrappedArgs = args.map(arg => wrapForeignValue(arg));
      const result = Reflect[trapName](value, ...wrappedArgs);
      return wrapLocalValue(result);
    };
  } else {
    return value;
  }
}

/**
 * To use a wrapped object from another realm in the current realm,
 * we create a Proxy that feeds the operations it traps to the function.
 */
export function wrapForeignValue(value) {
  if (!isObject(value)) {
    return value;
  }

  // All handler methods follow the same pattern.
  // To avoid repetitive code, we create it via a loop.
  const handler = {};
  for (const trapName of Object.getOwnPropertyNames(Reflect)) {
    handler[trapName] = (_target, ...args) => {
      const wrappedArgs = args.map(arg => wrapLocalValue(arg));
      const result = value(trapName, ...wrappedArgs);
      return wrapForeignValue(result);
    };
  }

  const target = function () {};
  return new Proxy(target, handler);
}

export async function createShadowEval(shadowRealm) {
  const _getLocallyWrappedEval = await shadowRealm.importValue(import.meta.url, '_getLocallyWrappedEval');
  return wrapForeignValue(_getLocallyWrappedEval());
}

export async function createShadowImport(shadowRealm) {
  const _getLocallyWrappedImport = await shadowRealm.importValue(import.meta.url, '_getLocallyWrappedImport');
  return wrapForeignValue(_getLocallyWrappedImport());  
}

export function _getLocallyWrappedEval() {
  return wrapLocalValue(globalThis.eval);
}

export function _getLocallyWrappedImport() {
  return wrapLocalValue((moduleSpecifier) => import(moduleSpecifier));
}

For more information on JavaScript Proxies, see the chapter on them in “Deep JavaScript”.

Missing features of this library:

  • It doesn’t preserve identity: If two values are equal in one realm, they should also be equal after they are transferred to another realm. That could be implemented by caching the results of wrapping in a WeakMap.

  • It doesn’t prevent code executed in a ShadowRealm from accessing the eval() and Function of the incubator realm. That can be achieved by pre-populating the cache WeakMap with entries that map the incubator eval() and Function to functions that throw exceptions.

Use cases for ShadowRealms  

What can ShadowRealms be used for?

  • Web apps such as IDEs or paint apps can run third-party code such as plugins or filters.

  • Programming environments can run user code in ShadowRealms.

  • Servers can run third-party code in ShadowRealms.

  • Test runners can run tests in ShadowRealms so that the incubator realm isn’t affected and each suite can start in a fresh realm (which helps with reproducibility).

  • For web scraping (extracting data from web pages) and web app testing, web apps can be run in ShadowRealms.

The next two sections explain the last two items in more detail.

Running tests in ShadowRealms  

In this section we examine a very simple proof of concept of how we could run tests in ShadowRealms.

The test library collects tests that are specified via test() and lets us run them via runTests().

// test-lib.js
const testDescs = [];

export function test(description, callback) {
  testDescs.push({description, callback});
}

export function runTests() {
  const testResults = [];
  for (const testDesc of testDescs) {
    try {
      testDesc.callback();
      testResults.push(`${testDesc.description}: OK\n`);
    } catch (err) {
      testResults.push(`${testDesc.description}: ${err}\n`);
    }
  }
  return testResults.join('');
}

Test code uses the library to specify tests:

// my-test.js
import {test} from './test-lib.js';
import * as assert from './assertions.js';

test('succeeds', () => {
  assert.equal(3, 3);
});

test('fails', () => {
  assert.equal(1, 3);
});

export default true;

In the next example, we dynamically load module my-test.js to collect the tests and then run them. Alas, there is currently no way to load a module without importing anything.

That’s why there is a default export in the last line of the previous example. We use the ShadowRealm method .importValue() to import that default export.

// test-runner.js
async function runTestModule(moduleSpecifier) {
  const sr = new ShadowRealm();
  await sr.importValue(moduleSpecifier, 'default');
  const runTests = await sr.importValue('./test-lib.js', 'runTests');
  const result = runTests();
  console.log(result);
}
await runTestModule('./my-test.js');

Running web apps in ShadowRealms  

The library jsdom creates an encapsulated browser environment which can be used to test web apps, extract data out of HTML, etc. It currently uses the Node.js module vm and could probably be updated to instead use ShadowRealms (with the benefit of the latter being cross-platform).

Comparing ShadowRealms with other ways of evaluating code  

eval() and Function  

ShadowRealms are similar to eval() and Function but improve on them: We can create new realms and evaluate code in them. That protects the realm that triggers the evaluation from the actions performed by the code.

Web Workers  

Web Workers are an even stronger isolation mechanism than ShadowRealms. Code in them runs in separate processes and communication is asynchronous. Therefore, ShadowRealms are a good choice whenever we want something more lightweight. We get the convenience of synchronous evaluation and have more freedom w.r.t. setting up global data.

iframes  

As we have seen, each iframe has its own realm. We can synchronously execute code inside it:

<body>
  <iframe>
  </iframe>
  <script>
    globalThis.realm = 'incubator';
    const iframeRealm = frames[0].window;
    iframeRealm.globalThis.realm = 'child';
    console.log(iframeRealm.eval('globalThis.realm')); // 'child'
  </script>
</body>

Compared to ShadowRealms, there are the following downsides:

  • We can only use iframes in browsers.
  • We need to add an iframe to the DOM in order to initialize it.
    • Afterwards, we can detach the iframe, but then import() won’t work anymore.
  • Each iframe realm contains the complete DOM, which makes some customizations impossible.
  • By default, objects can cross realms which means extra work is required to ensure safe code evaluation.

Module vm on Node.js  

Node’s vm module is similar to the ShadowRealm API but has more features: caching JavaScript engine data for faster startup, intercepting import(), setting up the global object externally (via vm.createContext()), etc.

Upcoming complementary proposals  

Compartments  

The proposal for compartments is currently at stage 1. It could build on ShadowRealms and allows the us to further control evaluation:

  • We can intercept static and dynamic imports.
  • We can configure the locale of the child realm.
  • Etc.

Module blocks proposal  

The proposal for module blocks is currently at stage 2 enables us to nest modules: We can create module blocks inside modules and each one of them defines a module.

Module blocks can be used in many locations where we currently use module specifiers. That makes them convenient for shadowRealm.importValue():

module insideCode {
 export { runTests } from 'test-framework';
 import './my-tests.js';
}

const sr = new ShadowRealm();
const runTests = await sr.importValue(insideCode, 'runTests');

Implementations  

Acknowledgements  

The following people provided important feedback to this blog post:

  • Caridy Patiño
  • Leo Balter
  • Allen Wirfs-Brock

Further reading  

Sources of this blog post:

Related topics: