Iterables and iterators in ECMAScript 6

[2015-02-26] esnext, dev, javascript, iteration

This blog post is outdated. Please read chapter “Iterables and iterators” in “Exploring ES6”.

This blog post is part of a series on iteration in ES6:

ECMAScript 6 introduces a new interface for iteration, Iterable. This blog post explains how it works, which language constructs consume data via it (e.g., the new for-of loop) and which sources provide data via it (e.g., arrays).

Iterability

The idea of iterability is as follows.

Data consumers: JavaScript has language constructs that consume data. For example, for-of loops over values and the spread operator (...) inserts values into arrays or function calls.
Data sources: The data consumers could get their values from a variety of sources. For example, you may want to iterate over the elements of an array, the key-value entries in a map or the characters of a string.

It’s not practical for every consumer to support all sources, especially because it should be possible to create new sources and consumers, e.g. via libraries with data structures or with new ways of processing data. Therefore, ES6 introduces the interface Iterable. Data consumers use it, data sources implement it:

Given that JavaScript does not have interfaces, Iterable is more of a convention:

Source: A value is considered iterable if it has a method whose key is the symbol ^[1] Symbol.iterator that returns a so-called iterator. The iterator is an object that returns values via its method next(). We say: it enumerates items, one per method call.
Consumption: Data consumers use the iterator to retrieve the values they are consuming.

Let’s see what consumption looks like for an array arr. First, you create an iterator via the method whose key is Symbol.iterator:

> let arr = ['a', 'b', 'c'];
> let iter = arr[Symbol.iterator]();

Then you call the iterator’s method next() repeatedly to retrieve the items “inside” the array:

> iter.next()
{ value: 'a', done: false }
> iter.next()
{ value: 'b', done: false }
> iter.next()
{ value: 'c', done: false }
> iter.next()
{ value: undefined, done: true }

As you can see, next() returns each item wrapped in an object, as the value of the property value. The boolean property done indicates when the end of the sequence of items has been reached.

Iterable and iterators are part of a so-called protocol (methods plus rules for using them) for iteration. A key characteristic of this protocol is that it is sequential: the iterator returns values one at a time. That means that if an iterable data structure is non-linear (such as a tree), iteration will linearize it.

Iterable data sources

I’ll use the for-of loop (which is explained in more detail later) to iterate over various kinds of iterable data.

Arrays

Arrays (and typed arrays) are iterables over their elements:

for (let x of ['a', 'b']) {
    console.log(x);
}
// Output:
// 'a'
// 'b'

Strings

Strings are iterable, but they enumerate Unicode code points, each of which may comprise one or two JavaScript “characters”:

for (let x of 'a\uD83D\uDC0A') {
    console.log(x);
}
// Output:
// 'a'
// '\uD83D\uDC0A' (crocodile emoji)

Note that you have just seen that primitive values can be iterable, too. A value doesn’t have to be an object in order to be iterable.

Maps

Maps ^[2] are iterables over their entries. Each entry is encoded as a [key, value] pair, an array with two elements. The entries are always enumerated deterministically, in the same order in which they were added to the map.

let map = new Map().set('a', 1).set('b', 2);
for (let pair of map) {
    console.log(pair);
}
// Output:
// ['a', 1]
// ['b', 2]

Note that WeakMaps ^[2:1] are not iterable.

Sets

Sets ^[2:2] are iterables over their elements (which are enumerated in the same order in which they were added to the set).

let set = new Set().add('a').add('b');
for (let x of set) {
    console.log(x);
}
// Output:
// 'a'
// 'b'

Note that WeakSets ^[2:3] are not iterable.

`arguments`

Even though the special variable arguments is more or less obsolete in ECMAScript 6 (due to rest parameters), it is iterable:

function printArgs() {
    for (let x of arguments) {
        console.log(x);
    }
}
printArgs('a', 'b');

// Output:
// 'a'
// 'b'

DOM data structures

Most DOM data structures will eventually be iterable:

for (let node of document.querySelectorAll('···')) {
    ···
}

Note that implementing this functionality is work in progress. But it is relatively easy to do so, because the symbol Symbol.iterator can’t clash with existing property keys ^[1:1].

Iterable computed data

Not all iterable content does have to come from data structures, it could also be computed on the fly. For example, all major ES6 data structures (arrays, typed arrays, maps, sets) have three methods that return iterable objects:

entries() returns an iterable over entries encoded as [key,value] arrays. For arrays, the values are the array elements and the keys are their indices. For sets, each key and value are the same – the set element.
keys() returns an iterable over the keys of the entries.
values() returns an iterable over the values of the entries.

Let’s see what that looks like. entries() gives you a nice way to get both array elements and their indices:

let arr = ['a', 'b', 'c'];
for (let pair of arr.entries()) {
    console.log(pair);
}
// Output:
// [0, 'a']
// [1, 'b']
// [2, 'c']

Plain objects are not iterable

Plain objects (as created by object literals) are not iterable:

for (let x of {}) { // TypeError
    console.log(x);
}

The reasoning is as follows. The following two activities are different:

Examining the structure of a program (reflection)
Iterating over data

It is best to keep these two activities separate. #1 is relevant for all objects, #2 only for data structures. You could make most objects iterable by adding a method [Symbol.iterator]() to Object.prototype, but they would lose this ability in two cases:

If they are created via Object.create(null). Then Object.prototype is not in their prototype chain.
If they are data structures. Then they need iterability for their data. Not only would you not be able to iterate over the properties of, say, arrays (which are also data structures). But you couldn’t ever later add iterability to an existing class, because that would break code that iterates over the properties of their instances.

Therefore, the safest way to make properties iterable is via a tool function. For example, via objectEntries(), whose implementation is shown later (future ECMAScript versions may have something similar built in):

let obj = { first: 'Jane', last: 'Doe' };

for (let [key,value] of objectEntries(obj)) {
    console.log(`${key}: ${value}`);
}

// Output:
// first: Jane
// last: Doe

It is also important to remember that iterating over the properties of an object is mainly interesting if you use objects as maps ^[3]. But we only do that in ES5 because we have no better alternative. In ECMAScript 6, we have Map.

Iterating language constructs

This section lists all built-in ES6 programming constructs that make use of the iteration protocol.

Destructuring via an array pattern

Destructuring ^[4] via array patterns works for any iterable:

let set = new Set().add('a').add('b').add('c');

let [x,y] = set;
    // x='a'; y='b'

let [first, ...rest] = set;
    // first='a'; rest=['b','c'];

The `for-of` loop

for-of is a new loop in ECMAScript 6. One form of it looks like this:

for (let x of iterable) {
    ···
}

This loop iterates over iterable, assigns each of the enumerated items to the iteration variable x and lets you process it in the body. The scope of x is the loop, it doesn’t exist outside it.

Note that the iterability of iterable is required, otherwise for-of can’t loop over a value. That means that non-iterable values must be converted to something iterable. For example, via Array.from(), which turns array-like values and iterables into arrays:

let arrayLike = { length: 2, 0: 'a', 1: 'b' };

for (let x of arrayLike) { // TypeError
    console.log(x);
}

for (let x of Array.from(arrayLike)) { // OK
    console.log(x);
}

I expect for-of to mostly replace Array.prototype.forEach(), because it is more versatile (forEach() only works for array-like values) and will be faster long term (see FAQ at the end).

Iteration variables: `let` declarations vs. `var` declarations

If you let-declare the iteration variable, a fresh binding (slot) will be created for each iteration. That can be seen in the following code snippet where we save the current binding of elem for later, via an arrow function. Afterwards, you can see that the arrow functions don’t share the same binding for elem, they each have a different one.

let arr = [];
for (let elem of [0, 1, 2]) {
    arr.push(() => elem); // save `elem` for later
}
console.log(arr.map(f => f())); // [0, 1, 2]

// `elem` only exists inside the loop:
console.log(elem); // ReferenceError: elem is not defined

It is instructive to see how things are different if you var-declare the iteration variable. Now all arrow functions refer to the same binding of elem.

let arr = [];
for (var elem of [0, 1, 2]) {
    arr.push(() => elem);
}
console.log(arr.map(f => f())); // [2, 2, 2]

// `elem` exists in the surrounding function:
console.log(elem); // 2

Having one binding per iteration is very helpful whenever you create functions via a loop (e.g. to add event listeners).

`let`-declared iteration variables in `for` loops and `for-in` loops

Two more loops get one binding per iteration if you let-declare their iteration variables: for and for-in.

Let’s look at for with a let-declared iteration variable i:

let arr = [];
for (let i=0; i<3; i++) {
    arr.push(() => i);
}
console.log(arr.map(f => f())); // [0, 1, 2]
console.log(i); // ReferenceError: i is not defined

If you var-declare i, you get the traditional behavior.

let arr = [];
for (var i=0; i<3; i++) {
    arr.push(() => i);
}
console.log(arr.map(f => f())); // [3, 3, 3]
console.log(i); // 3

Similarly, for-in with a let-declared iteration variable key leads to one binding per iteration:

let arr = [];
for (let key in ['a', 'b', 'c']) {
    arr.push(() => key);
}
console.log(arr.map(f => f())); // ['0', '1', '2']
console.log(key); // ReferenceError: key is not defined

var-declaring key produces a single binding:

let arr = [];
for (var key in ['a', 'b', 'c']) {
    arr.push(() => key);
}
console.log(arr.map(f => f())); // ['2', '2', '2']
console.log(key); // '2'

Iterating with existing variables, object properties and array elements

So far, we have only seen for-of with a declared iteration variable. But there are several other forms.

You can iterate with an existing variable:

let x;
for (x of ['a', 'b']) {
    console.log(x);
}

You can also iterate with an object property:

let obj = {};
for (obj.prop of ['a', 'b']) {
    console.log(obj.prop);
}

And you can iterate with an array element:

let arr = [];
for (arr[0] of ['a', 'b']) {
    console.log(arr[0]);
}

Iterating with a destructuring pattern

Combining for-of with destructuring is especially useful for iterables over key-value pairs (encoded as arrays). That’s what maps are:

let map = new Map().set(false, 'no').set(true, 'yes');
for (let [k,v] of map) {
    console.log(`key = ${k}, value = ${v}`);
}
// Output:
// key = false, value = no
// key = true, value = yes

Array.prototype.entries() also returns an iterable over key-value pairs:

let arr = ['a', 'b', 'c'];
for (let [k,v] of arr.entries()) {
    console.log(`key = ${k}, value = ${v}`);
}
// Output:
// key = 0, value = a
// key = 1, value = b
// key = 2, value = c

Therefore, entries() gives you a way to treat enumerated items differently, depending on their position:

/** Same as arr.join(', ') */
function toString(arr) {
    let result = '';
    for (let [i,elem] of arr.entries()) {
        if (i > 0) {
            result += ', ';
        }
        result += String(elem);
    }
    return result;
}

This function is used as follows:

> toString(['eeny', 'meeny', 'miny', 'moe'])
'eeny, meeny, miny, moe'

`Array.from()`

Array.from() ^[5] converts iterable and array-like values to arrays. It is also available for typed arrays.

> Array.from(new Map().set(false, 'no').set(true, 'yes'))
[[false,'no'], [true,'yes']]
> Array.from({ length: 2, 0: 'hello', 1: 'world' })
['hello', 'world']

Array.from() works as expected for a subclass of Array (which inherits this class method) – it converts iterables to instances of the subclass.

Spread

The spread operator ^[4:1] inserts the values of an iterable into an array:

> let arr = ['b', 'c'];
> ['a', ...arr, 'd']
['a', 'b', 'c', 'd']

That means that it provides you with a compact way to convert any iterable to an array:

let arr = [...iterable];

The spread operator also turns an iterable into the arguments of a function, method or constructor call:

> Math.max(...[-1, 8, 3])
8

Maps and sets

The constructor of a map turns an iterable over [key,value] pairs into a map:

> let map = new Map([['uno', 'one'], ['dos', 'two']]);
> map.get('uno')
'one'
> map.get('dos')
'two'

The constructor of a set turns an iterable over elements into a set:

> let set = new Set(['red', 'green', 'blue']);
> set.has('red')
true
> set.has('yellow')
false

The constructors of WeakMap and WeakSet work similarly. Furthermore, maps and sets are iterable themselves (WeakMaps and WeakSets aren’t), which means that you can use their constructors to clone them.

Promises

Promise.all() and Promise.race() accept iterables over promises ^[6]:

Promise.all(iterableOverPromises).then(···);
Promise.race(iterableOverPromises).then(···);

`yield*`

yield* ^[7] yields all items enumerated by an iterable.

function* yieldAllValuesOf(iterable) {
    yield* iterable;
}

The most important use case for yield* is to recursively call a generator ^[7:1] (which produces something iterable).

Implementing iterables

The iteration protocol looks as follows.

An object becomes iterable (“implements” the interface Iterable) if it has a method (own or inherited) whose key is Symbol.iterator. That method must return an iterator, an object that enumerates the items “inside” the iterable via its method next().

In TypeScript notation, the interfaces for iterables and iterators look as follows (based on ^[8]):

interface Iterable {
    [System.iterator]() : Iterator;
}
interface IteratorResult {
    value: any;
    done: boolean;
}
interface Iterator {
    next() : IteratorResult;
    return?(value? : any) : IteratorResult;
}

return is an optional methods that we’ll get to later (so is throw(), but it is practically never used for iterators and therefore explained in a follow-up blog post on generators). Let’s first implement a dummy iterable to get a feeling for how iteration works.

let iterable = {
    [Symbol.iterator]() {
        let step = 0;
        let iterator = {
            next() {
                if (step <= 2) {
                    step++;
                }
                switch (step) {
                    case 1:
                        return { value: 'hello', done: false };
                    case 2:
                        return { value: 'world', done: false };
                    default:
                        return { value: undefined, done: true };
                }
            }
        };
        return iterator;
    }
};

Let’s check that iterable is, in fact, iterable:

for (let x of iterable) {
    console.log(x);
}
// Output:
// hello
// world

The code executes three steps, with the counter step ensuring that everything happens in the right order. First we, return the value 'hello', then the value 'world' and then we indicate that the end of the enumerated items has been reached. Each item is wrapped in an object with the properties:

value which holds the actual item and
done which is a boolean flag that indicates whether the end has been reached, yet.

You can omit done if it is false and value if it is undefined. That is, the switch statement could be written as follows.

switch (step) {
    case 1:
        return { value: 'hello' };
    case 2:
        return { value: 'world' };
    default:
        return { done: true };
}

As is explained in the follow-up blog post on generators, there are cases where you want even the last item with done: true to have a value. Otherwise, next() could be simpler and return items directly (without wrapping them in objects). The end of iteration would then be indicated via a special value (e.g., a symbol).

Let’s look at one more implementation of an iterable. The function iterateOver() returns an iterable over the arguments that are passed to it:

function iterateOver(...args) {
    let index = 0;
    let iterable = {
        [Symbol.iterator]() {
            let iterator = {
                next() {
                    if (index < args.length) {
                        return { value: args[index++] };
                    } else {
                        return { done: true };
                    }
                }
            };
            return iterator;
        }
    }
    return iterable;
}

// Using `iterateOver()`:
for (let x of iterateOver('fee', 'fi', 'fo', 'fum')) {
    console.log(x);
}

// Output:
// fee
// fi
// fo
// fum

Iterators that are iterable

The previous function can be simplified if the iterable and the iterator are the same object:

function iterateOver(...args) {
    let index = 0;
    let iterable = {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            if (index < args.length) {
                return { value: args[index++] };
            } else {
                return { done: true };
            }
        },
    };
    return iterable;
}

Even if the original iterable and the iterator are not the same object, it is still occasionally useful if an iterator has the following method (which also makes it an iterable):

[Symbol.iterator]() {
    return this;
}

All built-in ES6 iterators follow this pattern (via a common prototype, see follow-up blog post on generators). For example, the default iterator for arrays:

> let arr = [];
> let iterator = arr[Symbol.iterator]();
> iterator[Symbol.iterator]() === iterator
true

Why is it useful if an iterator is also an iterable? for-of only works for iterables, not for iterators. Because array iterators are iterable, you can continue an iteration in another loop:

let arr = ['a', 'b'];
let iterator = arr[Symbol.iterator]();

for (let x of iterator) {
    console.log(x); // a
    break;
}

// Continue with same iterator:
for (let x of iterator) {
    console.log(x); // b
}

An alternative is to use a method that returns an iterable. For example, the result of Array.prototype.values() iterates the same way as the default iteration. Therefore, the previous code snippet is equivalent to:

let arr = ['a', 'b'];
let iterable = arr.values();
for (let x of iterable) {
    console.log(x); // a
    break;
}
for (let x of iterable) {
    console.log(x); // b
}

But with an iterable, you can’t be sure that it won’t restart iteration if for-of calls the method [Symbol.iterator](). For example, instances of Array are iterables that start at the beginning whenever you call this method.

One use case for continuing an iteration is that you can remove initial items (e.g. a header) before processing the actual content via for-of.

Optional iterator methods: `return()` and `throw()`

Two iterator methods are optional:

return() gives an iterator the opportunity to clean up if an iteration ends prematurely.
throw() is about forwarding a method call to a generator that is iterated over via yield*. It is explained in the follow-up blog post on generators.

Closing iterators via `return()`

As mentioned before, the optional iterator method return() is about letting an iterator clean up if it wasn’t iterated over until the end. It closes an iterator. In for-of loops, premature (or abrupt, in spec language) termination can be caused by:

break
continue (if you continue an outer loop, continue acts like a break)
throw
return

In each of these cases, for-of lets the iterator know that the loop won’t finish. Let’s look at an example, a function readLinesSync that returns an iterable of text lines in a file and would like to close that file no matter what happens:

function readLinesSync(fileName) {
    let file = ···;
    return {
        ···
        next() {
            if (file.isAtEndOfFile()) {
                file.close();
                return { done: true };
            }
            ···
        },
        return() {
            file.close();
            return { done: true };
        },
    };
}

Due to return(), the file will be properly closed in the following loop:

// Only print first line
for (let line of readLinesSync(fileName)) {
    console.log(x);
    break;
}

The return() method must return an object. That is due to how generators handle the return statement and will be explained in the follow-up blog post on generators.

The following constructs close iterators that aren’t completely “drained”:

for-of
yield*
Destructuring
Array.from()
Map(), Set(), WeakMap(), WeakSet()
Promise.all(), Promise.race()

More examples of iterables

In this section, we look at a few more examples of iterables. Most of these iterables are easier to implement via generators. The follow-up blog post on generators shows how.

Tool functions that return iterables

Tool functions and methods that return iterables are just as important as iterable data structures. The following is a tool function for iterating over the own properties of an object.

function objectEntries(obj) {
    let index = 0;

    // In ES6, you can use strings or symbols as property keys,
    // Reflect.ownKeys() retrieves both
    let propKeys = Reflect.ownKeys(obj);

    return {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            if (index < propKeys.length) {
                let key = propKeys[index];
                index++;
                return { value: [key, obj[key]] };
            } else {
                return { done: true };
            }
        }
    };
}

let obj = { first: 'Jane', last: 'Doe' };
for (let [key,value] of objectEntries(obj)) {
    console.log(`${key}: ${value}`);
}

// Output:
// first: Jane
// last: Doe

Combinators for iterables

Combinators ^[9] are functions that combine existing iterables to create new ones.

Let’s start with the combinator function take(n, iterable), which returns an iterable over the first n items of iterable.

function take(n, iterable) {
    let iter = iterable[Symbol.iterator]();
    return {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            if (n > 0) {
                n--;
                return iter.next();
            } else {
                return { done: true };
            }
        }
    };
}
let arr = ['a', 'b', 'c', 'd'];
for (let x of take(2, arr)) {
    console.log(x);
}
// Output:
// a
// b

zip turns n iterables into an iterable of n-tuples (encoded as arrays of length n).

function zip(...iterables) {
    let iterators = iterables.map(i => i[Symbol.iterator]());
    let done = false;
    return {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            if (!done) {
                let items = iterators.map(i => i.next());
                done = items.some(item => item.done);
                if (!done) {
                    return { value: items.map(i => i.value) };
                }
                // Done for the first time: close all iterators
                for (let iterator of iterators) {
                    iterator.return();
                }
            }
            // We are done
            return { done: true };
        }
    }
}

As you can see, the shortest iterable determines the length of the result:

let zipped = zip(['a', 'b', 'c'], ['d', 'e', 'f', 'g']);
for (let x of zipped) {
    console.log(x);
}
// Output:
// ['a', 'd']
// ['b', 'e']
// ['c', 'f']

Infinite iterables

Some iterable may never be done.

function naturalNumbers() {
    let n = 0;
    return {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            return { value: n++ };
        }
    }
}

With an infinite iterable, you must not iterate over “all” of it. For example, by breaking from a for-of loop:

for (let x of naturalNumbers()) {
    if (x > 2) break;
    console.log(x);
}

Or by only accessing the beginning of an infinite iterable:

let [a, b, c] = naturalNumbers();
    // a=0; b=1; c=2;

Or by using a combinator. take() is one possibility:

for (let x of take(3, naturalNumbers())) {
    console.log(x);
}
// Output:
// 0
// 1
// 2

The “length” of the iterable returned by zip() is determined by its shortest input iterable. That means that zip() and naturalNumbers() provide you with the means to number iterables of arbitrary (finite) length:

let zipped = zip(['a', 'b', 'c'], naturalNumbers());
for (let x of zipped) {
    console.log(x);
}
// Output:
// ['a', 0]
// ['b', 1]
// ['c', 2]

Frequently asked question

Isn’t the iteration protocol slow?

You may be worried about the iteration protocol being slow, because a new object is created for each invocation of next(). However, memory management for small objects is fast in modern engines and in the long run, engines can optimize iteration so that no intermediate objects need to be allocated. A thread on es-discuss has more information.

Conclusion

In this blog post we have seen that even with just the foundations of ES6 iteration, you can already do a lot. Generators ^[7:2] build on that foundation and help with, among other things, implementing iterators.

The JavaScript runtime library is still missing tools for working with iterators. Python has the feature-rich module itertools, JavaScript will eventually get a similar module.

Iterables and iterators in ECMAScript 6

Iterability #

Iterable data sources #

Arrays #

Strings #

Maps #

Sets #

arguments #

DOM data structures #

Iterable computed data #

Plain objects are not iterable #

Iterating language constructs #

Destructuring via an array pattern #

The for-of loop #

Iteration variables: let declarations vs. var declarations #

let-declared iteration variables in for loops and for-in loops #

Iterating with existing variables, object properties and array elements #

Iterating with a destructuring pattern #

Array.from() #

Spread #

Maps and sets #

Promises #

yield* #

Implementing iterables #

Iterators that are iterable #

Optional iterator methods: return() and throw() #

Closing iterators via return() #

More examples of iterables #

Tool functions that return iterables #

Combinators for iterables #

Infinite iterables #

Frequently asked question #

Isn’t the iteration protocol slow? #

Conclusion #

Further reading #