Behind the scenes of my latest book on JavaScript

[2018-08-22] book, javascript

This blog post takes you behind the scenes of my latest book, “JavaScript for impatient programmers” (which I’ll occasionally abbreviate as “Impatient JS”). It describes:

How I chose what to write about.
My techniques for explaining topics.
Tools I used for creating ebooks and other artifacts.
How I unit-tested the code shown in the book and in its quizzes.

Previewing “JavaScript for impatient programmers”

Available previews:

Book:
- Read HTML online (complete book, minus a few bonus chapters)
- Download PDF, EPUB, MOBI (50%)
Quizzes: try online (50%)
Exercises: download ZIP file (50%)

Content: teaching JavaScript as efficiently as possible

My idea for “Impatient JS” was: What would a book look like that teaches current JavaScript to programmers, as efficiently as possible?

In this section, I explain how I picked and presented the content.

What’s in the book?

Focusing on current JavaScript has one big advantage: You can tell a simpler, more consistent story, because recent features have eliminated many quirks and simplified many operations. The book does not explain old features that were superseded by newer and better features, but it includes references that cover them.

It is also satisfying that we can finally explore the full range of asynchronous programming in JavaScript: from callbacks to Promises to async functions to async iteration.

To keep the right balance between “too terse” and “too verbose”, I only mention the history of a feature when it helps understand that feature.

Techniques for teaching topics

The following subsections describe techniques I’ve used for teaching topics.

Exercises and quizzes

If possible, readers should not just passively receive information, they should also be able to practice what they have learned. Two convenient formats for that are exercises and quizzes.

The exercises are based on unit tests for the mocha test framework. This has three advantages:
- The computer becomes the teacher and corrects the reader. Not having a human in the loop even makes some things easier: no more feeling self-conscious about making mistakes.
- Readers learn about unit testing.
- A certain degree of error checking is automatic. That is, the risk of either the exercise or its solution being wrong, decreases.
For the quizzes, I found it important to not just slightly confuse readers with the questions, but to also dispel that confusion via explanations afterwards. That’s especially important for beginners, whose knowledge still isn’t that solidified.

Tutorial vs. reference

Two common ways of writing books about languages are:

Tutorials: are read linearly, from beginning to end and teach the language step by step.
Reference manuals: cover a broad range of topics and make it easy to randomly access what you are interested in.

“Impatient JS” combines elements of both. For example:

It covers the highlights of the standard library in a tutorial fashion, complemented by an exhaustive quick reference of the remaining functionality. The idea is that for much of the standard library, it is enough to know what’s there and to look up the details on demand.
The language is introduced step by step, so that chapters can be read consecutively (like a tutorial). But the book also keeps everything related to a given topic in one place (like a reference), via a trick: Some sections and chapters are marked as “advanced”. Readers can initially skip those sections and can become productive as quickly as possible. Once they are more familiar with the language, they can proceed to the advanced content.

Explaining everything twice

One technique I like is to explain everything twice: First, state something (somewhat abstractly) in English, then prove it via code. For example: If you state “the multiplication operator always coerces its operands to numbers”, you can follow up with a REPL interaction:

> '3' * '4'
12

The redundancy helps readers: Exploring different aspects deepens the understanding. And by using different modes of explanation, you increase the chance of reaching your readers.

Supporting offline use

As important as practicing in front of a computer is, it should also be possible to read a computer book without a computer. To that effect, I attempt to guess what readers would want to try out and try it out for them.

For example, if you write “reading an unknown property produces the value undefined”, then readers are probably curious how that works in practice. If you show them, they don’t need a computer to try it out. One way of doing so is via a REPL interaction:

> const obj = {};
> obj.unknownProp
undefined

This principle is related to the previous principle (explaining everything twice). But its goal is different.

Structuring text so that it is easier to digest

These are techniques that I’ve found useful:

Sections: help partition text logically. It’s important to keep a balance between sections that are too small and sections that are too large.
Bullet lists: are easy to scan and great for enumerating items. If the items in a bullet list become too large, you can turn the bullet list into a section and the items into subsections.
Tables: can be considered two-dimensional bullet lists. They present information even more compactly. They are especially well suited for summarizing traits of things. The table at the end of this subsection is taken from the chapter “Callable values”.
Type notations: are yet another way of describing something. For example, you may write “func is a function with a single parameter, a string, that returns a number”. Then you can follow it up with:
```
func(str: string): number
```
Once again, we are explaining the same thing twice: once in English and once in a formal notation. I’m using TypeScript’s type notation, which I explain in a chapter at the end of the book. A recent blog post of mine is an older version of that chapter.

Table: Capabilities of the four kinds of functions.

	Ordinary function	Arrow function	Method	Class
Function call	`✔`	`✔`	`✔`	`✘`
Method call	`✔`	lexical `this`	`✔`	`✘`
Constructor call	`✔`	`✘`	`✘`	`✔`

Diagrams complement text

Some information is easier to convey visually, especially if structure or processes are involved. For example, the diagram below is from a chapter at the end of the book, that gives an overview of web development in general. The diagram depicts a typical webpack workflow.

Artifacts: delivering the content

Several artifacts are used for delivering the content (the book, the exercises and the quizzes). Let’s look at what those artifacts are and how they are produced.

The following diagram gives an overview. Quizzify and Exbuild are tools I wrote for myself (I’m currently considering if and how I could open-source them). Pandoc is an open-source document converter.

The book

The book is written as Markdown, with one Markdown file per chapter. The output formats are:

EPUB and MOBI for various ebook readers.
PDFs in two versions:
- A screen version has links for internal cross-references. For example:
  “Consult the next chapter for more information.”
- While a print version has page numbers. For example:
  “Consult the next chapter (page 93) for more information.”
A website (HTML).

Pandoc

All of the book artifacts are created via Pandoc, combined with filters (plug-ins) written in the programming language Lua. The website is created by a Node.js script that splits and post-processes Pandoc’s one-file HTML output.

For images, I distinguish:

Vector images: The format I use for vector images depends on the output format:
- For HTML and EPUB, I use SVG images.
- For PDF, I use PDF images.
- For MOBI, I use JPG images.
Bitmap images: For bitmap images, I use the same JPG or PNG file, everywhere.

Pandoc filters

Pandoc parses its input into an abstract syntax tree. Filters operate on that syntax tree and can transform it. In addition to structured text, the abstract syntax tree can also contain “raw” HTML and LaTeX that is passed on to the next output stage without any changes.

The Lua filters that I wrote often have two modes:

If Pandoc produces a PDF (via LaTeX), the filters convert some of the Markdown to LaTeX.
If Pandoc produces HTML, they convert some of the Markdown to HTML.

As an example, consider the following input Markdown:

::: tip
## Try taking a break!
For example, go for a walk or do a breathing exercise.
:::

Pandoc does not have built-in support for boxes with content. The idea is to use a Pandoc “fenced div” with a pre-defined CSS class to express a box. In the previous example, ::: plus the pre-defined class tip starts the box; the three colons at the end, finish the box. A filter converts this div into an actual box.

This is the output in HTML mode, after the input was processed by the filter:

<div class="notebox">

![](img/icons/lightbulb-regular){height="24px"} **Try taking a break!**

For example, go for a walk or do a breathing exercise.

</div>

The filter did the following things:

The first line and the last line were converted to HTML (it actually works slightly differently, but this explanation is easier to understand):
The class tip was only used to select the image for the icon. The actual CSS class for boxes is notebox.
The heading was converted into bold text.

For LaTeX, the filter embeds LaTeX. This time, the syntax for embedding is more verbose:

```{=latex}
\begin{mdframed}[style=exampledefault]
```

![](img/icons/lightbulb-regular){height="24px"} **Try taking a break!**

For example, go for a walk or do a breathing exercise.

```{=latex}
\end{mdframed}
```

Quiz: from Markdown to quiz app

Each quiz is defined via a Markdown file. A question looks like this:

## Spelling

1. Javascript
2. JavaScript
3. Java Script

Solution: 2

Quizzify converts the Markdown to a client-side, React-based program. You can check out the result online.

Exercises

Kinds of exercises

Exercises have two formats that I call “test” and “exercise”:

For a “test”, readers need to implement the tested file.
For an “exercise”, readers need to modify the file itself.

The following code is an (abridged) “exercise”:

/* npm t exercises/operators/typeof_exrc.js
Instructions:
- Run this test (it fails).
- Change the second parameter of each assert.equal() so that the test passes
*/

import {strict as assert} from 'assert';

test('typeof', () => {
  assert.equal(typeof null, '???');
  assert.equal(typeof undefined, '???');

  // (Several assertions are omitted)
});

Exercises: authoring vs. deployment

When it comes to exercises, the file structures for authoring and deployment are different. The tool Exbuild transforms the former to the latter when creating the ZIP file with the code. Why are the structures different?

For authoring, you want everything is one place and you want to test the solutions.
For deployment, you want the solutions to be in a separate folder (so that readers can look them up if they get stuck) and all tests to fail until readers solve the exercises.

Therefore, for a “test”, the test is deployed in the exercises/ directory, the tested file is deployed in the solutions/ directory. Optionally, there may also be a _tmpl file with a skeleton of the solution; to be completed by readers.

Input (authoring)	Output (deployment)
`exercises/foo_test.js`	`exercises/foo_test.js`
`exercises/foo_tmpl.js`	`exercises/foo.js`
`exercises/foo.js`	`solutions/foo.js`

An “exercise” always involve a _tmpl file with mistakes that readers have to fix or blanks that readers have to fill in. The non-template file contains the solution that readers should end up with.

Input (authoring)	Output (deployment)
`exercises/bar_exrc_tmpl.js`	`exercises/bar_exrc.js`
`exercises/bar_exrc.js`	`solutions/bar_exrc.js`

Builds: preview vs. full version

Not only are there many different artifacts – each one of them has to be created in two versions:

The full version contains all of the content.
The preview version contains a subset of the content (currently about half the book).

This is handled by Pantools, Quizzify and Exbuild via builds (Pantools is the Node.js app that manages Pandoc). Each build has a name and specifies:

Where to put the produced artifact.
What files (chapters) to include:
- For the book, you directly specify file names.
- For quizzes, you specify the name of a book build. Then quizzify extracts the names of quiz files from the book files (which are mentioned there, but not shown in the published books).
- For exercises, you do the same.

When you create an artifact, you specify which build to use.

Testing code

All three kinds of content contain JavaScript code:

The book contains examples with JavaScript code.
The quiz asks questions about JavaScript code.
The exercises are JavaScript code.

The best way of avoiding typos in code is to run it in unit tests.

Unit-testing the book

For the book, I wrote my own tool, Marktest, that extracts Markdown code blocks and turns them into mocha unit tests.

From code fragments to unit tests

Take, for example, the following Markdown content:

In the following code, the value returned by the `.catch()` callback in line A becomes a fulfillment value:

<!--marktest before:
const retrieveFileName = () => Promise.resolve();
-->
```js
retrieveFileName()
.catch(() => {
  // Something went wrong, use a default value
  return 'Untitled.txt'; // (A)
})
.then(fileName => {
  // ···
});
```

The comment tells Marktest to put the provided line of source code before the text of the code block. Text in half-brackets (such as return) is not shown in the ebooks and only added to the unit test. Thanks to the return, the test created by Marktest is a proper asynchronous test (because it returns a Promise):

test('Line 238, js', () => {
  const retrieveFileName = () => Promise.resolve();
  return retrieveFileName()
  .catch(() => {
    // Something went wrong, use a default value
    return 'Untitled.txt'; // (A)
  })
  .then(fileName => {
    // ···
  });
});

The line number mentioned in the test name helps with debugging when there are errors. I’ve indented the output to make it easier to read. Marktest currently does not indent code, to avoid breaking it.

From REPL interactions to unit tests

REPL interactions are also supported by Marktest. For example:

This is a REPL interaction:

```node-repl
> 3 + 5
8
```

REPL interactions are converted into a series of invocations of assert.deepEqual():

test('Line 70, repl', () => {
  assert.deepEqual(
  3 + 5
  ,
  8
  );
});

Other features of Marktest

Marktest has more features. For example:

Ignoring code blocks.
Giving code blocks IDs and including them elsewhere. This is helpful whenever code is spread out across multiple code blocks.
Moving imports to the top levels of test files.
Writing code blocks to files so that imports work.

Unit-testing quizzes

To understand how unit-testing a quiz works, let’s use the following question (written in Markdown) as an example:

```js
function foo(x=true, y) {
  return [x, y];
}
const result = foo();
```

What happens?

1. `assert.deepEqual(result, ['a', 'b'])`
2. `assert.deepEqual(result, ['a', undefined])`
3. `assert.deepEqual(result, [true, 'a'])`
4. `assert.deepEqual(result, [true, undefined])`

Solution: 4

Unit-testing the correct answer 4 is straightforward – we simply wrap both code an assertion in a test():

test("Parameter default values #4", () => {
  function foo(x=true, y) {
    return [x, y];
  }
  const result = foo();

  assert.deepEqual(result, [true, undefined])
});

For incorrect answers, we cannot simply switch from assert.deepEqual() to assert.notDeepEqual(), because the answer may also be incorrect due to the code throwing an exception. Therefore, we wrap the whole body of the test() in an assert.throws() – which means that we expect the wrapped code to throw an exception. That exception can come from either the code or the failure of assert.deepEqual().

test("Parameter default values #1", () => {
  assert.throws(() => {
    function foo(x=true, y) {
      return [x, y];
    }
    const result = foo();

    assert.deepEqual(result, ['a', 'b'])
  });
});

Unit-testing exercises

Unit-testing the exercises is relatively simple because the file structure which is authored is the same as one where a reader has successfully completed all exercises. Therefore, we just need to run all tests (i.e., all “tests” and all “exercises”).

Is it worth it?

Making sure that the code in the book and the quizzes is testable is more work. That raises the question: is it worth that work?

On the minus side, seeing assert.equal() etc. in the code takes some getting used to. Compare:

// Old style:
console.log(3 + 4); // 7

// New style:
assert.equal(3 + 4, 7);

On the plus side:

Everything is more explicit with assertions.
You can state things that would be difficult to state by other means. Especially assert.throws() is quite powerful in this regard.
Readers learn to read assertions, which is a useful skill for unit-testing.
The biggest benefit is that automatic testing detects many small and large issues in code. And that is very valuable.

Other checks

My tools also perform a few other checks, such as:

Does a mentioned quiz file or exercise file really exist?
Is the npm t command mentioned at the beginning of an exercise correct?

Any feedback?

I’m interested in feedback:

Is everything in the book easy to understand? Even for people who are not yet familiar with JavaScript?
What are teaching techniques and tools that you like?

Behind the scenes of my latest book on JavaScript

Previewing “JavaScript for impatient programmers” #

Content: teaching JavaScript as efficiently as possible #

What’s in the book? #

Techniques for teaching topics #

Exercises and quizzes #

Tutorial vs. reference #

Explaining everything twice #

Supporting offline use #

Structuring text so that it is easier to digest #

Diagrams complement text #

Artifacts: delivering the content #

The book #

Pandoc #

Pandoc filters #

Quiz: from Markdown to quiz app #

Exercises #

Kinds of exercises #

Exercises: authoring vs. deployment #

Builds: preview vs. full version #

Testing code #

Unit-testing the book #

From code fragments to unit tests #

From REPL interactions to unit tests #

Other features of Marktest #

Unit-testing quizzes #

Unit-testing exercises #

Is it worth it? #

Other checks #

Any feedback? #