JS Test Suite Readme
====================

The JS test suite is a fairly extensive collection of correctness and regression
tests for the Spidermonkey engine. Two harnesses run these tests: the shell test
harness in this directory and the "reftest" harness built into the browser, used
in continuous integration. The browser reftests require additional manifest files; these are
generated automatically by the build phase 'package-tests' using the
'--make-manifests' option to jstests.py.

Running tests
-------------
See
https://developer.mozilla.org/en-US/docs/SpiderMonkey/Running_Automated_JavaScript_Tests


Creating a test
---------------
For general information, see
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Creating_JavaScript_tests/jsreftests


Adding a test
-------------
    Drop your test file in an appropriate directory under js/src/tests/non262/.

        <fineprint> Some names are forbidden. Do not name your test browser.js,
        shell.js, template.js, user.js, js-test-driver-begin.js, or
        js-test-driver-end.js.
        </fineprint>


Test262, ECMAScript conformance tests
-------------------------------------

    js/src/tests/test262/ contains a snapshot of the ECMAScript
    conformance test suite: https://github.com/tc39/test262.

    We periodically update the snapshot, but it's a manual process:

    1.  File a bug, modeled after bug 1560206, to announce your intent to
        update the test suite.

    2.  Run `test262-update.py`.

    3.  Run the tests.

    4.  Look at each failing test and either add an entry to the
        `UNSUPPORTED_FEATURES` list in `test262-update.py`
        or add a `skip script` entry to jstests.list, filing a bug if needed.
        (Every skipped test should be associated with an open bug.)

    5.  Repeat steps 2-4 until there are no failing tests.

    6.  Post patches (as in bug 1560206).

    When implementing a new feature, it is useful to enable tests for a
    previously unsupported feature without going through all the steps
    above. Here's how to do that:

    1.  Delete an entry from `UNSUPPORTED_FEATURES` in `test262-update.py`.

    2.  Run `./test262-update.py --revision $(head -n 1 test262/GIT-INFO | awk '{print $2}')`.

        (This re-runs the script using the revision of test262 we most recently used,
        instead of the latest revision.)

    3.  Run the tests.

    To export tests to the test262 format, for potential contribution to
    test262, see `test262-export.py`.


Adjusting when and how a test runs
----------------------------------
    If the test is a test262 test, do this by editing jstests.list.
    (The procedure below would be a bad idea because the test files are
    autogenerated from the upstream test262 repository.)

    Otherwise, put a comment at the top of the header matching the format:
        // |reftest| <failure-type> -- <comment>

    Where <failure-type> is a standard reftest <failure-type> string, as documented by:
        http://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt

    Example:
        // |reftest| skip-if(!xulRuntime.shell) -- does not always dismiss alert

        <fineprint> Either // or /* */ style comments may be used. The entire
        comment must appear in the first 512 bytes of the file. The control
        string must be in its own comment block. </fineprint>

    When adding such comments to individual files is not feasible (e.g., for
    imported tests), reftest manifest entries can be added to jstests.list
    instead. Combining in-file comments with entries in this manifest file for
    the same files is not supported (the one from the manifest file will be
    used). Only the following two forms are supported:
        <failure-type> include <relative_path>
        <failure-type> script <relative_path>
        include <relative_path>
    The <type> "include" indicates that <failure-type> should apply to all test
    cases within a directory. A statement for a nested directory or script
    overrides one for an enclosing directory. The <type> "include" without a
    <failure-type> recursively loads another jstests.list file for further
    processing.