Adding upstream version 115.7.0esr.upstream/115.7.0esr

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 1036 insertions, 0 deletions

diff --git a/third_party/rust/jsparagus/js-quirks.md b/third_party/rust/jsparagus/js-quirks.md
new file mode 100644
index 0000000000..20c621c92a
--- /dev/null
+++ b/third_party/rust/jsparagus/js-quirks.md
@@ -0,0 +1,1036 @@
+## JS syntactic quirks
+
+> *To make a labyrinth, it takes*
+> *Some good intentions, some mistakes.*
+> —A. E. Stallings, “Daedal”
+
+JavaScript is rather hard to parse. Here is an in-depth accounting of
+its syntactic quirks, with an eye toward actually implementing a parser
+from scratch.
+
+With apologies to the generous people who work on the standard. Thanks
+for doing that—better you than me.
+
+Thanks to [@bakkot](https://github.com/bakkot) and
+[@mathiasbynens](https://github.com/mathiasbynens) for pointing out
+several additional quirks.
+
+Problems are rated in terms of difficulty, from `(*)` = easy to `(***)`
+= hard. We’ll start with the easiest problems.
+
+
+### Dangling else (*)
+
+If you know what this is, you may be excused.
+
+Statements like `if (EXPR) STMT if (EXPR) STMT else STMT`
+are straight-up ambiguous in the JS formal grammar.
+The ambiguity is resolved with
+[a line of specification text](https://tc39.es/ecma262/#sec-if-statement):
+
+> Each `else` for which the choice of associated `if` is ambiguous shall
+> be associated with the nearest possible `if` that would otherwise have
+> no corresponding `else`.
+
+I love this sentence. Something about it cracks me up, I dunno...
+
+In a recursive descent parser, just doing the dumbest possible thing
+correctly implements this rule.
+
+A parser generator has to decide what to do. In Yacc, you can use
+operator precedence for this.
+
+Yacc aside: This should seem a little outrageous at first, as `else` is
+hardly an operator. It helps if you understand what Yacc is doing. In LR
+parsers, this kind of ambiguity in the grammar manifests as a
+shift-reduce conflict. In this case, when we’ve already parsed `if (
+Expression ) Statement if ( Expression ) Statement`
+and are looking at `else`, it’s unclear to Yacc
+whether to reduce the if-statement or shift `else`. Yacc does not offer
+a feature that lets us just say "always shift `else` here"; but there
+*is* a Yacc feature that lets us resolve shift-reduce conflicts in a
+rather odd, indirect way: operator precedence.  We can resolve this
+conflict by making `else` higher-precedence than the preceding symbol
+`)`.
+
+Alternatively, I believe it’s equivalent to add "[lookahead ≠ `else`]"
+at the end of the IfStatement production that doesn’t have an `else`.
+
+
+### Other ambiguities and informal parts of the spec (*)
+
+Not all of the spec is as formal as it seems at first. Most of the stuff
+in this section is easy to deal with, but #4 is special.
+
+1.  The lexical grammar is ambiguous: when looking at the characters `<<=`,
+    there is the question of whether to parse that as one token `<<=`, two
+    tokens (`< <=` or `<< =`), or three (`< < =`).
+
+    Of course every programming language has this, and the fix is one
+    sentence of prose in the spec:
+
+    > The source text is scanned from left to right, repeatedly taking the
+    > longest possible sequence of code points as the next input element.
+
+    This is easy enough for hand-coded lexers, and for systems that are
+    designed to use separate lexical and syntactic grammars.  (Other
+    parser generators may need help to avoid parsing `functionf(){}` as
+    a function.)
+
+2.  The above line of prose does not apply *within* input elements, in
+    components of the lexical grammar. In those cases, the same basic
+    idea ("maximum munch") is specified using lookahead restrictions at
+    the end of productions:
+
+    > *LineTerminatorSequence* ::  
+    > &nbsp; &nbsp; &lt;LF&gt;  
+    > &nbsp; &nbsp; &lt;CR&gt;[lookahead &ne; &lt;LF&gt;]  
+    > &nbsp; &nbsp; &lt;LS&gt;  
+    > &nbsp; &nbsp; &lt;PS&gt;  
+    > &nbsp; &nbsp; &lt;CR&gt;&lt;LF&gt;
+
+    The lookahead restriction prevents a CR LF sequence from being
+    parsed as two adjacent *LineTerminatorSequence*s.
+
+    This technique is used in several places, particularly in
+    [*NotEscapeSequences*](https://tc39.es/ecma262/#prod-NotEscapeSequence).
+
+3.  Annex B.1.4 extends the syntax for regular expressions, making the
+    grammar ambiguous. Again, a line of prose explains how to cope:
+
+    > These changes introduce ambiguities that are broken by the
+    > ordering of grammar productions and by contextual
+    > information. When parsing using the following grammar, each
+    > alternative is considered only if previous production alternatives
+    > do not match.
+
+4.  Annex B.1.2 extends the syntax of string literals to allow legacy
+    octal escape sequences, like `\033`. It says:
+
+    > The syntax and semantics of 11.8.4 is extended as follows except
+    > that this extension is not allowed for strict mode code:
+
+    ...followed by a new definition of *EscapeSequence*.
+
+    So there are two sets of productions for *EscapeSequence*, and an
+    implementation is required to implement both and dynamically switch
+    between them.
+
+    This means that `function f() { "\033"; "use strict"; }` is a
+    SyntaxError, even though the octal escape is scanned before we know
+    we're in strict mode.
+
+For another ambiguity, see "Slashes" below.
+
+
+### Unicode quirks
+
+JavaScript source is Unicode and usually follows Unicode rules for thing
+like identifiers and whitespace, but it has a few special cases: `$`,
+`_`, `U+200C ZERO WIDTH NON-JOINER`, and `U+200D ZERO WIDTH JOINER` are
+legal in identifiers (the latter two only after the first character), and
+`U+FEFF ZERO WIDTH NO-BREAK SPACE` (also known as the byte-order mark) is
+treated as whitespace.
+
+It also allows any code point, including surrogate halves, even though the
+Unicode standard says that unpaired surrogate halves should be treated as
+encoding errors.
+
+
+### Legacy octal literals and escape sequences (*)
+
+This is more funny than difficult.
+
+In a browser, in non-strict code, every sequence of decimal digits (not
+followed by an identifier character) is a *NumericLiteral* token.
+
+If it starts with `0`, with more digits after, then it's a legacy Annex
+B.1.1 literal. If the token contains an `8` or a `9`, it's a decimal
+number. Otherwise, hilariously, it's octal.
+
+```
+js> [067, 068, 069, 070]
+[55, 68, 69, 56]
+```
+
+There are also legacy octal escape sequences in strings, and these have
+their own quirks. `'\07' === '\u{7}'`, but `'\08' !== '\u{8}'` since 8
+is not an octal digit. Instead `'\08' === '\0' + '8'`, because `\0`
+followed by `8` or `9` is a legacy octal escape sequence representing
+the null character. (Not to be confused with `\0` in strict code, not
+followed by a digit, which still represents the null character, but
+doesn't count as octal.)
+
+None of this is hard to implement, but figuring out what the spec says
+is hard.
+
+
+### Strict mode (*)
+
+*(entangled with: lazy compilation)*
+
+A script or function can start with this:
+
+```js
+"use strict";
+```
+
+This enables ["strict mode"](https://tc39.es/ecma262/#sec-strict-mode-of-ecmascript).
+Additionally, all classes and modules are strict mode code.
+
+Strict mode has both parse-time and run-time effects. Parse-time effects
+include:
+
+*   Strict mode affects the lexical grammar: octal integer literals are
+    SyntaxErrors, octal character escapes are SyntaxErrors, and a
+    handful of words like `private` and `interface` are reserved (and
+    thus usually SyntaxErrors) in strict mode.
+
+    Like the situation with slashes, this means it is not possible to
+    implement a complete lexer for JS without also parsing—at least
+    enough to detect class boundaries, "use strict" directives in
+    functions, and function boundaries.
+
+*   It’s a SyntaxError to have bindings named `eval` or `arguments` in
+    strict mode code, or to assign to `eval` or `arguments`.
+
+*   It’s a SyntaxError to have two argument bindings with the same name
+    in a strict function.
+
+    Interestingly, you don’t always know if you’re in strict mode or not
+    when parsing arguments.
+
+    ```js
+    function foo(a, a) {
+        "use strict";
+    }
+    ```
+
+    When the implementation reaches the Use Strict Directive, it must
+    either know that `foo` has two arguments both named `a`, or switch
+    to strict mode, go back, and reparse the function from the
+    beginning.
+
+    Fortunately an Early Error rule prohibits mixing `"use strict"` with
+    more complex parameter lists, like `function foo(x = eval('')) {`.
+
+*   The expression syntax “`delete` *Identifier*” and the abominable
+    *WithStatement* are banned in strict mode.
+
+
+### Conditional keywords (**)
+
+In some programming languages, you could write a lexer that has rules
+like
+
+*   When you see `if`, return `Token::If`.
+
+*   When you see something like `apple` or `arrow` or `target`,
+    return `Token::Identifier`.
+
+Not so in JavaScript. The input `if` matches both the terminal `if` and
+the nonterminal *IdentifierName*, both of which appear in the high-level
+grammar. The same goes for `target`.
+
+This poses a deceptively difficult problem for table-driven parsers.
+Such parsers run on a stream of token-ids, but the question of which
+token-id to use for a word like `if` or `target` is ambiguous. The
+current parser state can't fully resolve the ambiguity: there are cases
+like `class C { get` where the token `get` might match either as a
+keyword (the start of a getter) or as an *IdentifierName* (a method or
+property named `get`) in different grammatical productions.
+
+All keywords are conditional, but some are more conditional than others.
+The rules are inconsistent to a tragicomic extent. Keywords like `if`
+that date back to JavaScript 1.0 are always keywords except when used as
+property names or method names. They can't be variable names. Two
+conditional keywords (`await` and `yield`) are in the *Keyword* list;
+the rest are not. New syntax that happened to be introduced around the
+same time as strict mode was awarded keyword status in strict mode. The
+rules are scattered through the spec. All this interacts with `\u0065`
+Unicode escape sequences somehow. It’s just unbelievably confusing.
+
+(After writing this section, I
+[proposed revisions to the specification](https://github.com/tc39/ecma262/pull/1694)
+to make it a little less confusing.)
+
+*   Thirty-six words are always reserved:
+
+    > `break` `case` `catch` `class` `const` `continue` `debugger`
+    > `default` `delete` `do` `else` `enum` `export` `extends` `false`
+    > `finally` `for` `function` `if` `import` `in` `instanceof` `new`
+    > `null` `return` `super` `switch` `this` `throw` `true` `try`
+    > `typeof` `var` `void` `while` `with`
+
+    These tokens can't be used as names of variables or arguments.
+    They're always considered special *except* when used as property
+    names, method names, or import/export names in modules.
+
+    ```js
+    // property names
+    let obj = {if: 3, function: 4};
+    assert(obj.if == 3);
+
+    // method names
+    class C {
+      if() {}
+      function() {}
+    }
+
+    // imports and exports
+    import {if as my_if} from "modulename";
+    export {my_if as if};
+    ```
+
+*   Two more words, `yield` and `await`, are in the *Keyword* list but
+    do not always act like keywords in practice.
+
+    *   `yield` is a *Keyword*; but it can be used as an identifier,
+        except in generators and strict mode code.
+
+        This means that `yield - 1` is valid both inside and outside
+        generators, with different meanings. Outside a generator, it’s
+        subtraction. Inside, it yields the value **-1**.
+
+        That reminds me of the Groucho Marx line: Outside of a dog, a
+        book is a man’s best friend. Inside of a dog it’s too dark to
+        read.
+
+    *   `await` is like that, but in async functions. Also it’s not a
+        valid identifier in modules.
+
+    Conditional keywords are entangled with slashes: `yield /a/g` is two
+    tokens in a generator but five tokens elsewhere.
+
+*   In strict mode code, `implements`, `interface`, `package`,
+    `private`, `protected`, and `public` are reserved (via Early Errors
+    rules).
+
+    This is reflected in the message and location information for
+    certain syntax errors:
+
+    ```
+    SyntaxError: implements is a reserved identifier:
+    class implements {}
+    ......^
+
+    SyntaxError: implements is a reserved identifier:
+    function implements() { "use strict"; }
+    ....................................^
+    ```
+
+*   `let` is not a *Keyword* or *ReservedWord*. Usually it can be an
+    identifier. It is special at the beginning of a statement or after
+    `for (` or `for await (`.
+
+    ```js
+    var let = [new Date];   // ok: let as identifier
+    let v = let;            // ok: let as keyword, then identifier
+    let let;           // SyntaxError: banned by special early error rule
+    let.length;        // ok: `let .` -> ExpressionStatement
+    let[0].getYear();  // SyntaxError: `let [` -> LexicalDeclaration
+    ```
+
+    In strict mode code, `let` is reserved.
+
+*   `static` is similar. It’s a valid identifier, except in strict
+    mode. It’s only special at the beginning of a *ClassElement*.
+
+    In strict mode code, `static` is reserved.
+
+*   `async` is similar, but trickier. It’s an identifier. It is special
+    only if it’s marking an `async` function, method, or arrow function
+    (the tough case, since you won’t know it’s an arrow function until
+    you see the `=>`, possibly much later).
+
+    ```js
+    function async() {}   // normal function named "async"
+
+    async();        // ok, `async` is an Identifier; function call
+    async() => {};  // ok, `async` is not an Identifier; async arrow function
+    ```
+
+*   `of` is special only in one specific place in `for-of` loop syntax.
+
+    ```js
+    var of = [1, 2, 3];
+    for (of of of) console.log(of);  // logs 1, 2, 3
+    ```
+
+    Amazingly, both of the following are valid JS code:
+
+    ```js
+    for (async of => {};;) {}
+    for (async of []) {}
+    ```
+
+    In the first line, `async` is a keyword and `of` is an identifier;
+    in the second line it's the other way round.
+
+    Even a simplified JS grammar can't be LR(1) as long as it includes
+    the features used here!
+
+*   `get` and `set` are special only in a class or an object literal,
+    and then only if followed by a PropertyName:
+
+    ```js
+    var obj1 = {get: f};      // `get` is an identifier
+    var obj2 = {get x() {}};  // `get` means getter
+
+    class C1 { get = 3; }     // `get` is an identifier
+    class C2 { get x() {} }   // `get` means getter
+    ```
+
+*   `target` is special only in `new.target`.
+
+*   `arguments` and `eval` can't be binding names, and can't be assigned
+    to, in strict mode code.
+
+To complicate matters, there are a few grammatical contexts where both
+*IdentifierName* and *Identifier* match. For example, after `var {`
+there are two possibilities:
+
+```js
+// Longhand properties: BindingProperty -> PropertyName -> IdentifierName
+var { xy: v } = obj;    // ok
+var { if: v } = obj;    // ok, `if` is an IdentifierName
+
+// Shorthand properties: BindingProperty -> SingleNameBinding -> BindingIdentifier -> Identifier
+var { xy } = obj;       // ok
+var { if } = obj;       // SyntaxError: `if` is not an Identifier
+```
+
+
+### Escape sequences in keywords
+
+*(entangled with: conditional keywords, ASI)*
+
+You can use escape sequences to write variable and property names, but
+not keywords (including contextual keywords in contexts where they act
+as keywords).
+
+So `if (foo) {}` and `{ i\u0066: 0 }` are legal but `i\u0066 (foo)` is not.
+
+And you don't necessarily know if you're lexing a contextual keyword
+until the next token: `({ g\u0065t: 0 })` is legal, but
+`({ g\u0065t x(){} })` is not.
+
+And for `let` it's even worse: `l\u0065t` by itself is a legal way to
+reference a variable named `let`, which means that
+
+```js
+let
+x
+```
+declares a variable named `x`, while, thanks to ASI,
+
+```js
+l\u0065t
+x
+```
+is a reference to a variable named `let` followed by a reference to a
+variable named `x`.
+
+
+### Early errors (**)
+
+*(entangled with: lazy parsing, conditional keywords, ASI)*
+
+Some early errors are basically syntactic. Others are not.
+
+This is entangled with lazy compilation: "early errors" often involve a
+retrospective look at an arbitrarily large glob of code we just parsed,
+but in Beast Mode we’re not building an AST. In fact we would like to be
+doing as little bookkeeping as possible.
+
+Even setting that aside, every early error is a special case, and it’s
+just a ton of rules that all have to be implemented by hand.
+
+Here are some examples of Early Error rules—setting aside restrictions
+that are covered adequately elsewhere:
+
+*   Rules about names:
+
+    *   Rules that affect the set of keywords (character sequences that
+        match *IdentifierName* but are not allowed as binding names) based
+        on whether or not we’re in strict mode code, or in a
+        *Module*. Affected identifiers include `arguments`, `eval`, `yield`,
+        `await`, `let`, `implements`, `interface`, `package`, `private`,
+        `protected`, `public`, `static`.
+
+        *   One of these is a strangely worded rule which prohibits using
+            `yield` as a *BindingIdentifier*.  At first blush, this seems
+            like it could be enforced in the grammar, but that approach
+            would make this a valid program, due to ASI:
+
+            ```js
+            let
+            yield 0;
+            ```
+
+            Enforcing the same rule using an Early Error prohibits ASI here.
+            It works by exploiting the detailed inner workings of ASI case
+            1, and arranging for `0` to be "the offending token" rather than
+            `yield`.
+
+    *   Lexical variable names have to be unique within a scope:
+
+        *   Lexical variables (`let` and `const`) can’t be declared more
+            than once in a block, or both lexically declared and
+            declared with `var`.
+
+        *   Lexically declared variables in a function body can’t have the same
+            name as argument bindings.
+
+    *   A lexical variable can’t be named `let`.
+
+    *   Common-sense rules dealing with unicode escape sequences in
+        identifiers.
+
+*   Common-sense rules about regular expression literals. (They have to
+    actually be valid regular expressions, and unrecognized flags are
+    errors.)
+
+*   The number of string parts that a template string can have is
+    limited to 2<sup>32</sup> &minus; 1.
+
+*   Invalid Unicode escape sequences, like `\7` or `\09` or `\u{3bjq`, are
+    banned in non-tagged templates (in tagged templates, they are allowed).
+
+*   The *SuperCall* syntax is allowed only in derived class
+    constructors.
+
+*   `const x;` without an initializer is a Syntax Error.
+
+*   A direct substatement of an `if` statement, loop statement, or
+    `with` statement can’t be a labelled `function`.
+
+*   Early errors are used to hook up cover grammars.
+
+    *   Early errors are also used in one case to avoid having to
+        specify a very large refinement grammar when *ObjectLiteral*
+        almost covers *ObjectAssignmentPattern*:
+        [sorry, too complicated to explain](https://tc39.es/ecma262/#sec-object-initializer-static-semantics-early-errors).
+
+*   Early errors are sometimes used to prevent parsers from needing to
+    backtrack too much.
+
+    *   When parsing `async ( x = await/a/g )`, you don't know until the
+        next token if this is an async arrow or a call to a function named
+        `async`. This means you can't even tokenize properly, because in
+        the former case the thing following `x =` is two divisions and in
+        the latter case it's an *AwaitExpression* of a regular expression.
+        So an Early Error forbids having `await` in parameters at all,
+        allowing parsers to immediately throw an error if they find
+        themselves in this case.
+
+Many strict mode rules are enforced using Early Errors, but others
+affect runtime semantics.
+
+<!--
+*   I think the rules about assignment targets are related to cover
+    grammars. Not sure.
+
+    *   Expressions used in assignment context (i.e. as the operands of `++`
+        and `--`, the left operand of assignment, the loop variable of a
+        `for-in/for-of` loop, or sub-targets in destructuring) must be valid
+        assignment targets.
+
+    *   In destructuring assignment, `[...EXPR] = x` is an error if `EXPR`
+        is an array or object destructuring target.
+
+-->
+
+
+### Boolean parameters (**)
+
+Some nonterminals are parameterized. (Search for “parameterized
+production” in [this spec
+section](https://tc39.es/ecma262/#sec-grammar-notation).)
+
+Implemented naively (e.g. by macro expansion) in a parser generator,
+each parameter could nearly double the size of the parser. Instead, the
+parameters must be tracked at run time somehow.
+
+
+### Lookahead restrictions (**)
+
+*(entangled with: restricted productions)*
+
+TODO (I implemented this by hacking the entire LR algorithm. Most every
+part of it is touched, although in ways that seem almost obvious once
+you understand LR inside and out.)
+
+(Note: It may seem like all of the lookahead restrictions in the spec
+are really just a way of saying “this production takes precedence over
+that one”—for example, that the lookahead restriction on
+*ExpressionStatement* just means that other productions for statements
+and declarations take precedence over it. But that isn't accurate; you
+can't have an *ExpressionStatement* that starts with `{`, even if it
+doesn't parse as a *Block* or any other kind of statement.)
+
+
+### Automatic Semicolon Insertion (**)
+
+*(entangled with: restricted productions, slashes)*
+
+Most semicolons at the end of JS statements and declarations “may be
+omitted from the source text in certain situations”.  This is called
+[Automatic Semicolon
+Insertion](https://tc39.es/ecma262/#sec-automatic-semicolon-insertion),
+or ASI for short.
+
+The specification for this feature is both very-high-level and weirdly
+procedural (“When, as the source text is parsed from left to right, a
+token is encountered...”, as if the specification is telling a story
+about a browser. As far as I know, this is the only place in the spec
+where anything is assumed or implied about the internal implementation
+details of parsing.) But it would be hard to specify ASI any other way.
+
+Wrinkles:
+
+1.  Whitespace is significant (including whitespace inside comments).
+    Most semicolons in the grammar are optional only at the end of a
+    line (or before `}`, or at the end of the program).
+
+2.  The ending semicolon of a `do`-`while` statement is extra optional.
+    You can always omit it.
+
+3.  A few semicolons are never optional, like the semicolons in `for (;;)`.
+
+    This means there’s a semicolon in the grammar that is optionally
+    optional! This one:
+
+    > *LexicalDeclaration* : *LetOrConst* *BindingList* `;`
+
+    It’s usually optional, but not if this is the *LexicalDeclaration*
+    in `for (let i = 0; i < 9; i++)`!
+
+4.  Semicolons are not inserted only as a last resort to avoid
+    SyntaxErrors. That turned out to be too error-prone, so there are
+    also *restricted productions* (see below), where semicolons are more
+    aggressively inferred.
+
+5.  In implementations, ASI interacts with the ambiguity of *slashes*
+    (see below).
+
+A recursive descent parser implements ASI by calling a special method
+every time it needs to parse a semicolon that might be optional.  The
+special method has to peek at the next token and consume it only if it’s
+a semicolon. This would not be so bad if it weren’t for slashes.
+
+In a parser generator, ASI can be implemented using an error recovery
+mechanism.
+
+I think the [error recovery mechanism in
+yacc/Bison](https://www.gnu.org/software/bison/manual/bison.html#Error-Recovery)
+is too imprecise—when an error happens, it discards states from the
+stack searching for a matching error-handling rule. The manual says
+“Error recovery strategies are necessarily guesses.”
+
+But here’s a slightly more careful error recovery mechanism that could
+do the job:
+
+1.  For each production in the ES spec grammar where ASI could happen, e.g.
+
+    ```
+    ImportDeclaration ::= `import` ModuleSpecifier `;`
+                                                    { import_declaration($2); }
+    ```
+
+    add an ASI production, like this:
+
+    ```
+    ImportDeclaration ::= `import` ModuleSpecifier [ERROR]
+                                       { check_asi(); import_declaration($2); }
+    ```
+
+    What does this mean? This production can be matched, like any other
+    production, but it's a fallback. All other productions take
+    precedence.
+
+2.  While generating the parser, treat `[ERROR]` as a terminal
+    symbol. It can be included in start sets and follow sets, lookahead,
+    and so forth.
+
+3.  At run time, when an error happens, synthesize an `[ERROR]` token.
+    Let that bounce through the state machine. It will cause zero or
+    more reductions. Then, it might actually match a production that
+    contains `[ERROR]`, like the ASI production above.
+
+    Otherwise, we’ll get another error—the entry in the parser table for
+    an `[ERROR]` token at this state will be an `error` entry. Then we
+    really have a syntax error.
+
+This solves most of the ASI issues:
+
+* [x] Whitespace sensitivity: That's what `check_asi()` is for. It
+    should signal an error if we're not at the end of a line.
+
+* [x] Special treatment of `do`-`while` loops: Make an error production,
+    but don't `check_asi()`.
+
+* [x] Rule banning ASI in *EmptyStatement* or `for(;;)`:
+    Easy, don't create error productions for those.
+
+    * [x] Banning ASI in `for (let x=1 \n x<9; x++)`: Manually adjust
+        the grammar, copying *LexicalDeclaration* so that there's a
+        *LexicalDeclarationNoASI* production used only by `for`
+        statements. Not a big deal, as it turns out.
+
+* [x] Slashes: Perhaps have `check_asi` reset the lexer to rescan the
+    next token, if it starts with `/`.
+
+* [ ] Restricted productions: Not solved. Read on.
+
+
+### Restricted productions (**)
+
+*(entangled with: ASI, slashes)*
+
+Line breaks aren’t allowed in certain places. For example, the following
+is not a valid program:
+
+    throw                 // SyntaxError
+      new Error();
+
+For another example, this function contains two statements, not one:
+
+    function f(g) {
+      return              // ASI
+        g();
+    }
+
+The indentation is misleading; actually ASI inserts a semicolon at the
+end of the first line: `return; g();`. (This function always returns
+undefined. The second statement is never reached.)
+
+These restrictions apply even to multiline comments, so the function
+
+```js
+function f(g) {
+  return /*
+    */ g();
+}
+```
+contains two statements, just as the previous example did.
+
+I’m not sure why these rules exist, but it’s probably because (back in
+the Netscape days) users complained about the bizarre behavior of
+automatic semicolon insertion, and so some special do-what-I-mean hacks
+were put in.
+
+This is specified with a weird special thing in the grammar:
+
+> *ReturnStatement* : `return` [no *LineTerminator* here] *Expression* `;`
+
+This is called a *restricted production*, and it’s unfortunately
+necessary to go through them one by one, because there are several
+kinds. Note that the particular hack required to parse them in a
+recursive descent parser is a little bit different each time.
+
+*   After `continue`, `break`, or `return`, a line break triggers ASI.
+
+    The relevant productions are all statements, and in each case
+    there’s an alternative production that ends immediately with a
+    semicolon: `continue ;` `break ;` and `return ;`.
+
+    Note that the alternative production is *not* restricted: e.g. a
+    *LineTerminator* can appear between `return` and `;`:
+
+    ```js
+    if (x)
+      return        // ok
+        ;
+    else
+      f();
+    ```
+
+*   After `throw`, a line break is a SyntaxError.
+
+*   After `yield`, a line break terminates the *YieldExpression*.
+
+    Here the alternative production is simply `yield`, not `yield ;`.
+
+*   In a post-increment or post-decrement expression, there can’t be a
+    line break before `++` or `--`.
+
+    The purpose of this rule is subtle. It triggers ASI and thus prevents
+    syntax errors:
+
+    ```js
+    var x = y       // ok: semicolon inserted here
+    ++z;
+    ```
+
+    Without the restricted production, `var x = y ++` would parse
+    successfully, and the “offending token” would be `z`. It would be
+    too late for ASI.
+
+    However, the restriction can of course also *cause* a SyntaxError:
+
+    ```js
+    var x = (y
+             ++);   // SyntaxError
+    ```
+
+As we said, recursive descent parsers can implement these rules with hax.
+
+In a generated parser, there are a few possible ways to implement
+them. Here are three. If you are not interested in ridiculous
+approaches, you can skip the first two.
+
+*   Treat every token as actually a different token when it appears
+    after a line break: `TokenType::LeftParen` and
+    `TokenType::LeftParenAfterLineBreak`.  Of course the parser
+    generator can treat these exactly the same in normal cases, and
+    automatically generate identical table entries (or whatever) except
+    in states where there’s a relevant restricted production.
+
+*   Add a special LineTerminator token. Normally, the lexer skips
+    newlines and never emits this token. However, if the current state
+    has a relevant restricted production, the lexer knows this and emits
+    a LineTerminator for the first line break it sees; and the parser
+    uses that token to trigger an error or transition to another state,
+    as appropriate.
+
+*   When in a state that has a relevant restricted production, change
+    states if there’s a line break before the next token.  That is,
+    split each such state into two: the one we stay in when there’s not
+    a line break, and the one we jump to if there is a line break.
+
+In all cases it’ll be hard to have confidence that the resulting parser
+generator is really sound. (That is, it might not properly reject all
+ambiguous grammars.) I don’t know exactly what property of the few
+special uses in the ES grammar makes them seem benign.
+
+
+### Slashes (**)
+
+*(entangled with: ASI, restricted productions)*
+
+When you see `/` in a JS program, you don’t know if that’s a
+division operator or the start of a regular expression unless you’ve
+been paying attention up to that point.
+
+[The spec:](https://tc39.es/ecma262/#sec-ecmascript-language-lexical-grammar)
+
+> There are several situations where the identification of lexical input
+> elements is sensitive to the syntactic grammar context that is
+> consuming the input elements. This requires multiple goal symbols for
+> the lexical grammar.
+
+You might think the lexer could treat `/` as an operator only if the
+previous token is one that can be the last token of an expression (a set
+that includes literals, identifiers, `this`, `)`, `]`, and `}`).  To see
+that this does not work, consider:
+
+```js
+{} /x/                  // `/` after `}` is regexp
+({} / 2)                // `/` after `}` is division
+
+for (g of /(a)(b)/) {}  // `/` after `of` is regexp
+var of = 6; of / 2      // `/` after `of` is division
+
+throw /x/;              // `/` after `throw` is regexp
+Math.throw / 2;         // `/` after `throw` is division
+
+++/x/.lastIndex;        // `/` after `++` is regexp
+n++ / 2;                // `/` after `++` is division
+```
+
+So how can the spec be implemented?
+
+In a recursive descent parser, you have to tell the lexer which goal
+symbol to use every time you ask for a token. And you have to make sure,
+if you look ahead at a token, but *don’t* consume it, and fall back on
+another path that can accept a *RegularExpressionLiteral* or
+*DivPunctuator*, that you did not initially lex it incorrectly. We have
+assertions for this and it is a bit of a nightmare when we have to touch
+it (which is thankfully rare). Part of the problem is that the token
+you’re peeking ahead at might not be part of the same production at all.
+Thanks to ASI, it might be the start of the next statement, which will
+be parsed in a faraway part of the Parser.
+
+A table-driven parser has it easy here! The lexer can consult the state
+table and see which kind of token can be accepted in the current
+state. This is closer to what the spec actually says.
+
+Two minor things to watch out for:
+
+*   The nonterminal *ClassTail* is used both at the end of
+    *ClassExpression*, which may be followed by `/`; and at the end of
+    *ClassDeclaration*, which may be followed by a
+    *RegularExpressionLiteral* at the start of the next
+    statement. Canonical LR creates separate states for these two uses
+    of *ClassTail*, but the LALR algorithm will unify them, creating
+    some states that have both `/` and *RegularExpressionLiteral* in the
+    follow set.  In these states, determining which terminal is actually
+    allowed requires looking not only at the current state, but at the
+    current stack of states (to see one level of grammatical context).
+
+*   Since this decision depends on the parser state, and automatic
+    semicolon insertion adjusts the parser state, a parser may need to
+    re-scan a token after ASI.
+
+In other kinds of generated parsers, at least the lexical goal symbol
+can be determined automatically.
+
+
+### Lazy compilation and scoping (**)
+
+*(entangled with: arrow functions)*
+
+JS engines *lazily compile* function bodies. During parsing, when the
+engine sees a `function`, it switches to a high-speed parsing mode
+(which I will call “Beast Mode”) that just skims the function and checks
+for syntax errors. Beast Mode does not compile the code. Beast Mode
+doesn’t even create AST nodes. All that will be done later, on demand,
+the first time the function is called.
+
+The point is to get through parsing *fast*, so that the script can start
+running. In browsers, `<script>` compilation usually must finish before
+we can show the user any meaningful content. Any part of it that can be
+deferred must be deferred. (Bonus: many functions are never called, so
+the work is not just deferred but avoided altogether.)
+
+Later, when a function is called, we can still defer compilation of
+nested functions inside it.
+
+So what? Seems easy enough, right? Well...
+
+Local variables in JS are optimized. To generate reasonable code for the
+script or function we are compiling, we need to know which of its local
+variables are used by nested functions, which we are *not* currently
+compiling. That is, we must know which variables *occur free in* each
+nested function. So scoping, which otherwise could be done as a separate
+phase of compilation, must be done during parsing in Beast Mode.
+
+Getting the scoping of arrow functions right while parsing is tricky,
+because it’s often not possible to know when you’re entering a scope
+until later. Consider parsing `(a`. This could be the beginning of an
+arrow function, or not; we might not know until after we reach the
+matching `)`, which could be a long way away.
+
+Annex B.3.3 adds extremely complex rules for scoping for function
+declarations which makes this especially difficult. In
+
+```js
+function f() {
+    let x = 1;
+    return function g() {
+        {
+            function x(){}
+        }
+        {
+            let x;
+        }
+        return x;
+    };
+}
+```
+
+the function `g` does not use the initial `let x`, but in
+
+```js
+function f() {
+    let x = 1;
+    return function g() {
+        {
+            {
+                function x(){}
+            }
+            let x;
+        }
+        return x;
+    };
+}
+```
+it does.
+
+[Here](https://dev.to/rkirsling/tales-from-ecma-s-crypt-annex-b-3-3-56go)
+is a good writeup of what's going on in these examples.
+
+
+### Arrow functions, assignment, destructuring, and cover grammars (\*\*\*)
+
+*(entangled with: lazy compilation)*
+
+Suppose the parser is scanning text from start to end, when it sees a
+statement that starts with something like this:
+
+```js
+let f = (a
+```
+
+The parser doesn’t know yet if `(a ...` is *ArrowParameters* or just a
+*ParenthesizedExpression*. Either is possible. We’ll know when either
+(1) we see something that rules out the other case:
+
+```js
+let f = (a + b           // can't be ArrowParameters
+
+let f = (a, b, ...args   // can't be ParenthesizedExpression
+```
+
+or (2) we reach the matching `)` and see if the next token is `=>`.
+
+Probably (1) is a pain to implement.
+
+To keep the language nominally LR(1), the standard specifies a “cover
+grammar”, a nonterminal named
+*CoverParenthesizedExpressionAndArrowParameterList* that is a superset
+of both *ArrowParameters* and *ParenthesizedExpression*.  The spec
+grammar uses the *Cover* nonterminal in both places, so technically `let
+f = (a + b) => 7;` and `let f = (a, b, ...args);` are both syntactically
+valid *Script*s, according to the formal grammar. But there are Early
+Error rules (that is, plain English text, not part of the formal
+grammar) that say arrow functions’ parameters must match
+*ArrowParameters*, and parenthesized expressions must match
+*ParenthesizedExpression*.
+
+So after the initial parse, the implementation must somehow check that
+the *CoverParenthesizedExpressionAndArrowParameterList* really is valid
+in context. This complicates lazy compilation because in Beast Mode we
+are not even building an AST. It’s not easy to go back and check.
+
+Something similar happens in a few other cases: the spec is written as
+though syntactic rules are applied after the fact:
+
+*   *CoverCallExpressionAndAsyncArrowHead* covers the syntax `async(x)`
+    which looks like a function call and also looks like the beginning
+    of an async arrow function.
+
+*   *ObjectLiteral* is a cover grammar; it covers both actual object
+    literals and the syntax `propertyName = expr`, which is not valid in
+    object literals but allowed in destructuring assignment:
+
+    ```js
+    var obj = {x = 1};  // SyntaxError, by an early error rule
+
+    ({x = 1} = {});   // ok, assigns 1 to x
+    ```
+
+*   *LeftHandSideExpression* is used to the left of `=` in assignment,
+    and as the operand of postfix `++` and `--`. But this is way too lax;
+    most expressions shouldn’t be assigned to:
+
+    ```js
+    1 = 0;  // matches the formal grammar, SyntaxError by an early error rule
+
+    null++;   // same
+
+    ["a", "b", "c"]++;  // same
+    ```
+
+
+## Conclusion
+
+What have we learned today?
+
+*   Do not write a JS parser.
+
+*   JavaScript has some syntactic horrors in it. But hey, you don't
+    make the world's most widely used programming language by avoiding
+    all mistakes. You do it by shipping a serviceable tool, in the right
+    circumstances, for the right users.