test262 – ensuring that JavaScript implementations comply with the ECMAScript specification

[2011-07-28] dev, javascript, jslang

With so many JavaScript implementations out there, how do you guarantee that all of them stay true to ECMA-262 (the ECMAScript language standard)? The answer is test262, a suite of tests to be run by an implementation. Quote from the test262 website:

What is test262? test262 is a test suite intended to check agreement between JavaScript implementations and the ECMA-262 Specification (currently 5th Edition). The test suite contains thousands of individual tests, each of which tests some specific requirements of the ECMAScript specification.

A press release [2] has more details:

For the first time, Ecma will publish a standardized test suite for the latest version of ECMA-262 edition 5.1, known as Test262. Via this suite, developers will be able to test how closely implementations of ECMAScript follow the specifications in ECMA-262 edition 5.1. The test suite will be published as an Ecma Technical Report and will be made available to developers via the normal Ecma distribution channels and http://test262.ecmascript.org/. The test suite will be released in increments with the first version planned for December 2011 and updates provided later.
The test suite has been created by the Ecma Technical Committee 39, with committee members including Mozilla, Microsoft, Google and many others helping to make the suite a resource that helps developers provide fine-tuned feedback on support for the standard.
Over 10,000 tests cover the specification today. Test suites are an integral part of the process needed to create interoperable cross-platforms standards.

A Microsoft blog article [1] (which pointed me to the press release) has a neat idea for running test262 faster:

You can use Web Workers to run test262 even faster. To demonstrate the promise of Web workers to make the Web faster overall, we’ve published a test drive that runs the tests from the standards body using Web workers.

Structure of the tests. The test infrastructure currently seems a bit complicated: A JSON “table of contents” file points to files such as the following:

{"testsCollection":{"name":"Chapter - 11.11_Binary_Logical_Operators",
"numTests":"32","tests":[{"code":"RVM1...

The string value of code is Base64-encoded data that looks as follows:

    ES5Harness.registerTest( {
    id: "S11.11.1_A1",

    path: "TestCases/11_Expressions/11.11_Binary_Logical_Operators/11.11.1_Logical_AND_Operator/S11.11.1_A1.js",

    assertion: "White Space and Line Terminator between LogicalANDExpression and \"&&\" or between \"&&\" and BitwiseORExpression are allowed",

    description: "Checking by using eval",

    test: function testcase() {
       //CHECK#1
    if ((eval("true\u0009&&\u0009true")) !== true) {
      $ERROR('#1: (true\\u0009&&\\u0009true) === true');
    }