Implementing a command line with eval in JavaScript

[2012-07-16] eval, dev, javascript, jslang

This blog post explores JavaScript’s eval function by implementing the foundation for an interactive command line. As a bonus, you’ll get to work with ECMAScript.next’s generators (which can already be tried out on current Firefox versions).

Writing an evaluator

Let’s say you want to implement an interactive command line for JavaScript (such as [1]). On one hand, you would need to get the graphical user interface right: The user inputs JavaScript code, the command line evaluates the code and displays the result. On the other hand, you would have to implement the evaluation. That’s what we will take on here. It is more complex that it initially seems and teaches us a lot about eval. For starters, let’s write a constructor Evaluator:

    function Evaluator() {
    }
    Evaluator.prototype.evaluate = function (str) {
        return JSON.stringify(eval(str));
    };

To use the evaluator, we create an instance and send JavaScript code to it:

    > var e = new Evaluator();

    > e.evaluate("Math.pow(2, 53)")
    '9007199254740992'

    > e.evaluate("3 * 7")
    '21'

    > e.evaluate("'foo'+'bar'")
    '"foobar"'

JSON.stringify is used so that the evaluation results can be shown to the user and look like the input. Without stringify, things look as follows:

    > console.log(123)  // OK
    123

    > console.log("abc")  // not OK
    abc

With stringify, everything looks OK:

    > console.log(JSON.stringify(123))
    123

    > console.log(JSON.stringify("abc"))
    "abc"

Note that undefined is not valid JSON, but stringify converts it to undefined (the value, not the string), which is fine for our purposes. What we have implemented so far works for basic things, but still has several problems. Let’s tackle them one at a time.

Problem: declarations

You can evaluate variable and function declarations, but they are forgotten immediately afterwards:

    > e.evaluate("var x = 12;")
    undefined
    > e.evaluate("x")
    ReferenceError: x is not defined

How do we fix this? The following code is a solution:

    function Evaluator() {
        this.env = {};
    }
    Evaluator.prototype.evaluate = function (str) {
        str = rewriteDeclarations(str);
        var __environment__ = this.env;  // (1)
        with (__environment__) {  // (2)
            return JSON.stringify(eval(str));
        }
    };
    function rewriteDeclarations(str) {
        // Prefix a newline so that search and replace is simpler
        str = "\n" + str;
    
        str = str.replace(/\nvar\s+(\w+)\s*=/g,
                          "\n__environment__.$1 =");  // (3)
        str = str.replace(/\nfunction\s+(\w+)/g,
                          "\n__environment__.$1 = function");
    
        return str.slice(1); // remove prefixed newline
    }

this.env holds all variable declarations and function declarations in its properties. We make it accessible to the input in two steps.

Step 1 – declare: We assign this.env to __environment__ (1) and rewrite the input so that, among other things, each var declaration assigns to __environment__ (3). That demonstrates one important aspect of eval: it sees all variables in surrounding scopes. That is, if you invoke eval inside your function, you expose all of its internals. The only way to keep those internals secret is to put the eval call in a separate function and call that function.

Step 2 – access: Use a with statement so that the properties of __environment__ appear as variables to the eval-ed code. This is not an ideal solution, more of a compromise: with should be avoided [2] and can’t be used in the advantageous strict mode [3]. But it is a quick solution for us now. A work-around is quite complex [4].

    > var e = new Evaluator();

    > e.evaluate("var x = 123;")
    '123'
    > e.evaluate("x")
    '123'

Minor drawback: Normal var declarations have the result undefined; due to our rewriting we now get the value that is assigned to the variable.

Problem: exceptions

Right now, throwing an exception in evaluate’s input means that the method will throw:

    > e.evaluate("* 3")
    SyntaxError: Unexpected token *

That is obviously unacceptable: In a graphical user interface, we want to report errors back to the user, not (invisibly) throw an exception. Here is one simple way of doing so:

    Evaluator.prototype.evaluate = function (str) {
        try {
            str = rewriteDeclarations(str);
            var __environment__ = this.env;
            with (__environment__) {
                return JSON.stringify(eval(str));
            }
        } catch (e) {
            return e.toString();
        }
    };

There is nothing surprising in this code, we simply use try-catch and report back what happened. More sophisticated solutions will want to do more, e.g. display the exception’s stack trace. The new evaluator in action:

    > var e = new Evaluator();

    > e.evaluate("* 3")
    'SyntaxError: Unexpected token *'

Problem: console.log

How do we handle calls to console.log in the input? Logged messages should be shown to the user, not be sent to the browser’s console. The solution is surprisingly easy:

    function Evaluator(cons) {
        this.env = {};
        this.cons = cons;
    }
    Evaluator.prototype.evaluate = function (str) {
        try {
            str = rewriteDeclarations(str);
            var __environment__ = this.env;
            var console = this.cons;
            with (__environment__) {
                return JSON.stringify(eval(str));
            }
        } catch (e) {
            return e.toString();
        }
    };

The constructor now receives a custom implementation of console and assigns it to this.cons. By assigning that object to a local variable named console (1), we temporarily shadow the global console for eval, there is no need to replace it. Beware that that shadowing affects all of the function, you won’t be able to use the browser’s console anywhere in evaluate. The new evaluator in action:

    > var cons = { log: function (m) { console.log("### "+m) } };
    > var e = new Evaluator(cons);

    > e.evaluate("console.log('hello')")
    ### hello
    undefined

Problem: eval creates bindings inside the function

One scary feature of eval is that it creates variable bindings inside the function that invokes it:

    > (function () { eval("var x=3"); return x }())
    3

Fortunately, the fix is easy: use strict mode.

    > (function () { "use strict"; eval("var x=3"); return x }())
    ReferenceError: x is not defined

You can’t use with in strict mode, so you’ll have to replace it with a work-around [4].

Keeping declarations in an environment

An environment is where JavaScript keeps the parameters and variables of a function. It maps variable names to values and is thus similar to an object. We might be able to avoid rewriting the input and manage declarations via environments. The idea is as follows. eval puts declarations in some environment:

Non-strict mode: the environment of the surrounding function.
Strict mode: a newly created environment.

What if we could reuse that environment for the next invocation of eval, instead of throwing it away? Then eval would properly remember prior declarations. Strict mode gives us no way to access the temporary environment it creates for each invocation. However, in non-strict mode, we might be able to keep the environment of the surrounding function around. The following subsections explore two ways of doing so.

Declarations via nested scopes

If you create a function g inside another function f, then g permanently retains a reference to f’s current environment env_f. Whenever g is called, a new g-specific environment env_g is created. But env_g points to its parent environment env_f. Variables that can’t be found in g’s scope (as managed via env_g), are looked up in f’s scope (via env_f). Thus, env_f is not lost, as long as g exists.

That gives us a strategy for keeping the environment of the function that calls eval around. In the following code that function is called evalHelper and creates a new function that has to be used for the next call of eval. Hence, declarations made in the former function are accessible in the later function.

    function Evaluator() {
        var that = this;
        that.evalHelper = function (str) {
            that.evalHelper = function (str) {
                return eval(str);
            };
            return eval(str);
        };
    }
    Evaluator.prototype.evaluate = function (str) {
        return this.evalHelper(str);
    };

The fatal problem of this implementation is that you cannot nest to arbitrary depth. But, for the above depth of 2, it works perfectly:

    > var e = new Evaluator();

    > e.evaluate("var x = 7;");
    undefined
    > e.evaluate("x * 3")
    21

Declarations via a generator

It would be great if we could “restart” the function that calls eval, re-enter it with its previous environment still in place. ECMAScript.next’s generators [5] let you do that.

Current versions of Firefox already support generators. Here is a demonstration of how they work in these versions (in ECMAScript.next, you will have to write function*, but apart from that, the code is the same):

    function mygen() {
        console.log((yield 0) + " @ 0");
        console.log((yield 1) + " @ 1");
        console.log((yield 2) + " @ 2");
    }

The above is a generator function. Invoke it and it will create a generator object. On that object, you first need to invoke the next() method to start execution. A yield x inside the code pauses execution and returns x to the the previously called generator object method. After the first next(), you can either call next() or send(y). The latter means that the currently paused yield will continue and produce the value y. The former is equivalent to send(undefined). The following interaction shows mygen in use:

    > var g = mygen();

    > g.next()  // can’t use send() the first time
    0

    > g.send("a")  // continue after yield 0, pause again
    a @ 0
    1

    > g.send("b")
    b @ 1
    2

The following is an implementation of Evaluator that calls eval via the generator evalGenerator. Because of that, eval always sees the same environment and remembers declarations.

    function evalGenerator(console) {
        var str = yield;
        while(true) {
            try {
                var result = JSON.stringify(eval(str));
                str = yield result;
            } catch (e) {
                str = yield e.toString();
            }
        }
    }

    function Evaluator(cons) {
        this.evalGen = evalGenerator(cons);
        this.evalGen.next(); // start
    }
    Evaluator.prototype.evaluate = function (str) {
        return this.evalGen.send(str);
    };

The new evaluator works as expected.

    > var e = new Evaluator();

    > e.evaluate("var x = 7;")
    undefined

    > e.evaluate("x * 2")
    "14"

    > e.evaluate("* syntax_error")
    "SyntaxError: missing ; before statement"

The biggest problem with this solution is that it uses the deprecated features non-strict eval together with the new feature generators. There will probably be a way in ECMAScript.next to make this combination work, but it will be a hack and should thus be avoided.

Conclusion

We have used eval to implement a helper type for a command line. While doing so, we learned a few interesting things about eval: Letting it remember declarations between invocations is complicated; it can access all variables in the scopes surrounding its invocation; and in non-strict mode, it can even create new variables inside the invoking function.

The best solution for remembering declarations would be for eval to have an optional parameter for an environment (to be reused), but that is not in the cards. Therefore, the only truly safe solution in pure JavaScript is to use a full-featured JavaScript parser such as esprima to rewrite critical parts of the input code. That is left as an exercise to the reader.