Annotation-model: Guidelines for Contributing Tests
===================================================

This document describes the method people should use for authoring tests and
integrating them into the repository.  Anyone is welcome to submit new tests to
this collection.  If you do, please create the tests following the guidelines
below.  Then submit them as a pull request so they can be evaluated

Structure
---------

Tests are organized by major section of the Annotation Model specification.  The
folders associated with these are:

* annotations
* bodiesTargets
* collections
* specificResources
  * selectors
  * states

Within these folders, special files ending with the suffix ".test" provide the source
for the test as a set of declarative assertions about the required shape of the conforming
JSON object.  These files are transformed using a test generation tool into ".html" files
that are then accessed by the Web Platform Test framework.

There are a few other folders that provide supporting materials for the tests:

* common - assertionObjects, conditionObjects, and other supporting materials
* definitions - JSON Schema definitions that can be referenced
* scripts - JavaScript that are included by tests
* tools - supporting scripts and files

NOTE: The files in the definitions folder are expected to be JSON Schema
definitions - basically commonly used concepts that are referenced by other JSON
Schema files in the system.  All of these 'definitions' are preloaded by the
system before any other parts of a test are processed.

Test Cases
----------

Each test is expressed as a simple (or complex) requirement in a test file.
For each section of the document, the requirement is represented as a structure
that describes the nature of the test, and then includes or references minimal
JSON Schema that test the assertions implied by the requirement.

The structure of a test case is defined using a [JSON-LD
Context](JSONtest-v1.jsonld).  That context defines the following terms:

|Keyword        | Values          | Meaning
|---------------|-----------------|---------
|name           | string          | The name of this test for display purposes
|description    | string          | A long self-describing paragraph that explains the purpose of the test and the expected input
|ref            | URI             | An optional reference to the portion of the specification to which the test relates
|testType       | `automated`, `manual`, `ref` | The type of test - this informs [WPT](https://github.com/web-platform-tests/wpt) how the test should be controlled and presented
|skipFailures   | list of strings | An optional list of assertionType values that, if present, should have their test skipped if the result would be "unexpected".  Defaults to the empty list.
|assertions     | list of URI, List @@@ATRISK@@@, or AssertionObject | The ordered collection of tests the input should be run against. See [JSON Schema Usage](#jsonSchema) for the structure of the objects.  URI is relative to the top level folder of the test collection if it has a slash; relative to the current directory if it does not. @@@@ATRISK@@@@ Lists can be nested to define groups of sub-tests.  Assertions / groups can be conditionally skipped.  See [Assertion Lists](#assertionLists) for more details.
|content        | URI or object   | An object containing content to be checked against the referenced assertions, or a URI from which to retrieve that content

Each test case has a suffix of `.test` and a shape like:

<pre>
{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
  "name": "Verify annotation conforms to the model",
  "description": "Supply an example annotation that conforms to the basic structure.",
  "ref": "https://www.w3.org/TR/annotation-model/#model",
  "testType": "manual",
  "assertions": [
    "common/has_context.json",
    "common/has_id.json",
    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "Verify annotation has target",
      "assertionType": "must",
      "expectedResult": "valid",
      "errorMessage": "The object was missing a required 'target' property",
      "type": "object",
      "properties": {
        "target": {
          "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "array",
            "anyOf": [
            {
              "type": "string"
            }
            ]
          }
          ],
            "not": {"type": "object"}
        }
      },
      "required": ["target"]
    }
  ]
}
</pre>

External references are used when the "assertion" is a common one that needs to
be checked on many different test cases (e.g., that there is an @context in the
supplied annotation).

NOTE: The title property of an assertionObject can contain markdown.  This can
help improve readability of the rendered assertions and debugging output.

NOTE: The content property does not yet have a defined use.  One potential use would
be to act as a pointer to a URI that can supply annotations from an implementation.
In that case the URI would take a parameter with the test name as a way of telling
the end point what test is running so it can deliver the right content.

### <a id="assertionLists">Assertion Lists</a> ###

The `assertion` list is an ordered list of assertions that will be evaluated
against the submitted content. The list is *required*, and MUST have at least
one entry. Entries in the list have the following types:

* AssertionObject

An in-line Object as defined in the section [Assertion
Objects](#assertionObjects).
* URI

A relative or absolute URI that references a AssertionObject in a .json file.
If the URI is relative but contains no slashes, then it is considered to be
in the current directory.  If the URI is relative, contains slashes, but
**does not start with a slash** then it is considered relative to the top of
the tree of the current test collection (e.g., `annotation-model`).
* List @@@ATRISK@@@

A nested Assertion List.  While nested Assertion Lists are optional, if one
is present it MUST have at least one entry.  Entries are as in this list.
Assertion Lists can be nested to any depth (but don't do that - it would be
too hard to maintain).


<a id="assertionObjects">Assertion Objects</a>
-----------------

In this collection of tests, Assertion Objects can be contained inline in the
`.test` files or contained in external files with the suffix `.json`.  The
vocabularly and structure is as defined in [JSON Schema
v4](http://json-schema.org/documentation.html) augmented with some additional
properties defined in this section.

In general each JSON Schema definition provided in this test suite should be as
minimal as possible.  This promotes clarity and increases the likelihood that
it is correct.  While it is ---possible--- to create JSON Schema files that
enforce many different requirements on a data model, it increases the
complexity and can also reduce the atomicity of tests / sub-tests (because a
    test ends up testing more than one thing).  Please try to avoid creating
complex JSON Schema.  (A notable exception is the situation where multiple
    properties of a structure are interdependent.)

Tools such as [the JSON Schema Creator](http://jsonschema.net/) may be helpful
in creating schema snippets that can be integrated into JSONtest Assertion
Objects.  Remember that the JSON Schema you create should be as permissive as
possible to address the various shapes that a give property might take (e.g., a
    'foo' might be a string or an object that contains sub-properties that express
    the string, or an array of 1 or more objects with those sub-properties).

In addition to the validation keys defined in JSON Schema v4, Schema files in
this collection are also permitted to use the following keywords:

|Keyword        | Values          | Meaning |
|---------------|-----------------|---------|
|onUnexpectedResult   | `failAndContinue`, `failAndSkip`, `failAndAbort`, `passAndContinue`, `passAndSkip`, `passAndAbort` | Action to take when the result is not as expected. Default is `failAndContinue` |
|assertionType  | `must`, `may`, `should` | Informs the system about the severity of a failure. The default is `must` |
|assertionFile | URI      | An external file that contains an assertion SCHEMA.  When this value is supplied, and local properties will override the ones loaded from the external file.
|errorMessage   | string          | A human readable explanation of what it means if the test fails.  |
|expectedResult | `valid`, `invalid`  | Tells the framework whether validating against this schema is expected to succeed or fail.  The default is `valid` |


### Example Assertion Object ###

<pre>
{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Verify annotation has @context",
    "type": "object",
    "expectedResult" : "valid",
    "properties": {
      "@context": {
        "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "array",
          "anyOf": [
          {
            "type": "string"
          }
          ]
        }
        ],
          "not": {"type": "object"}
      }
    },
    "required": ["@context"]
}
</pre>

Note that in the case where a feature is *optional* the JSON Schema MUST be
crafted such that if the attribute is permitted to be missing from the content
(so that the result is `true`), but when the attribute is present in the
content it conforms to any requirements.



<a id="conditionObjects">Condition Objects</a>
-----------------

A Condition Object is a sub-class of an Assertion Object.  It allows the
specification of the evaluation strategy for the assertions referenced by the
object.  An object is a Condition Object IFF it has a `assertions` property. In
this case, there MUST NOT be an `assertionFile` property.


|Keyword        | Values          | Meaning |
|---------------|-----------------|---------|
|compareWith    | `and`, `or` | How should the result of any referenced assertions be compared.  Defaults to `and`.  Note that this implies there is also an assertions property with a nested list of assertions to compare. |
|assertions     | a list of assertions as in a Test Case above. This is required if there is a compareWith property |


An example of a test that would pass if there were an `@context` OR there were an `@id`:

<pre>
{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "A test that has an 'or' clause",
    "description": "A complex test that uses or-ing among a list of assertions",
    "ref": "https://www.w3.org/TR/annotation-model/#model",
    "testType": "manual",
    "assertions": [
    { "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "must have context or id",
      "description": "A more complex example that allows one of many options to pass",
      "assertions": [
      { "title": "Condition Object",
        "description": "A pseudo-test that will get a result from the aggregate of its children",
        "assertionType": "must",
        "expectedResult": "valid",
        "errorMessage": "Error: None of the various options were present",
        "compareWith": "or",
        "assertions": [
          "common/has_context.json",
        "common/has_id.json"
        ]
      }
      ]
    }
    ]
}
</pre>


Command Line Tools
------------------

### Building the Test Files ###

The actual .html test case files are generated using the script
tools/make_tests.py.  This script will search the directory heriarchy looking for
files ending on `.test` and creating `.html` files from them using the template in
the tools folder.  If you want to regenerate the examples too, supply the
`--examples` option to the script.

Note that when submitting tests to the repository, the `.html` versions must be
included.

### Testing the Tests ###

### Driving Tests with Input Files ###

Complex Examples
----------------

This section is a collection of more complex examples to illustrate the
expressive power of the [Assertion List](#assertionLists) structure.  These can
be used as templates for creating actual `.test` files.

### Including and Overriding an Assertion ###

Assertions can be contained in external `.json` files.  It is possible for an
object in an Assertion List to include the external file and override one or
more of its properties:

<pre>
{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "Permit no target property",
    "description": "Ensure there is no 'target' property when there is a 'randomName' property in the Annotation",
    "assertions": [
    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "Verify annotation has randomName",
      "type": "object",
      "properties": {
        "randomName": {
          "type": "string"
        }
      },
      "required": ["randomName"]
    },
    { "assertionFile" : "common/target.json",
      "title" : "Require target to be missing",
      "expectedResult" : "invalid",
      "errorMessage" : "The target MUST not be present when 'randomName' is also present",
    }
    ]
}
</pre>

### Nested Assertion Collections with Skip ###

Assertion Lists can be nested within Assertion Lists.  This feature, combined
with the `onUnexpectedResult` property, makes it possible to skip a collection
of tests when an assertion in the list is not satisfied:

<pre>
{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "If there is no 'target' property, skip some tests",
    "description": "When 'target' is not present, other properties related to 'target' are not required",
    "assertions": [
      "common/context.json",
    [
    { "assertionFile" : "common/target.json",
      "errorMessage" : "Target was not present so skip the rest of this section",
      "onUnexpectedResult" : "failAndSkip"
    },
    "sometest.json",
    "sometest2.json",
    "sometest3.json"
    ]
    ]
} ;
</pre>

### Assertion that finds a specific @context Value ###

Sometimes you want a property to be flexible, but to have one and only one of a
specific value.  This is especially true with, for example, @context in JSON-LD.
One way you might do this is:

<pre>
{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Verify a specific @context",
    "type": "object",
    "expectedResult" : "valid",
    "properties": {
      "@context": {
        "anyOf": [
        {
          "type": "string"
            "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
        },
        {
          "type": "array",
          "minitems": "1",
          "uniqueItems": true,
          "additionalItems": true,
          "items" : [
          { "type": "string",
            "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
          }
          ]
        }
        ],
          "not": {"type": "object"}
      }
    },
    "required": ["@context"]
}

</pre>