summaryrefslogtreecommitdiffstats
path: root/testing/web-platform/tests/annotation-model/CONTRIBUTING.md
diff options
context:
space:
mode:
Diffstat (limited to 'testing/web-platform/tests/annotation-model/CONTRIBUTING.md')
-rw-r--r--testing/web-platform/tests/annotation-model/CONTRIBUTING.md381
1 files changed, 381 insertions, 0 deletions
diff --git a/testing/web-platform/tests/annotation-model/CONTRIBUTING.md b/testing/web-platform/tests/annotation-model/CONTRIBUTING.md
new file mode 100644
index 0000000000..45a338f3a4
--- /dev/null
+++ b/testing/web-platform/tests/annotation-model/CONTRIBUTING.md
@@ -0,0 +1,381 @@
+Annotation-model: Guidelines for Contributing Tests
+===================================================
+
+This document describes the method people should use for authoring tests and
+integrating them into the repository. Anyone is welcome to submit new tests to
+this collection. If you do, please create the tests following the guidelines
+below. Then submit them as a pull request so they can be evaluated
+
+Structure
+---------
+
+Tests are organized by major section of the Annotation Model specification. The
+folders associated with these are:
+
+* annotations
+* bodiesTargets
+* collections
+* specificResources
+ * selectors
+ * states
+
+Within these folders, special files ending with the suffix ".test" provide the source
+for the test as a set of declarative assertions about the required shape of the conforming
+JSON object. These files are transformed using a test generation tool into ".html" files
+that are then accessed by the Web Platform Test framework.
+
+There are a few other folders that provide supporting materials for the tests:
+
+* common - assertionObjects, conditionObjects, and other supporting materials
+* definitions - JSON Schema definitions that can be referenced
+* scripts - JavaScript that are included by tests
+* tools - supporting scripts and files
+
+NOTE: The files in the definitions folder are expected to be JSON Schema
+definitions - basically commonly used concepts that are referenced by other JSON
+Schema files in the system. All of these 'definitions' are preloaded by the
+system before any other parts of a test are processed.
+
+Test Cases
+----------
+
+Each test is expressed as a simple (or complex) requirement in a test file.
+For each section of the document, the requirement is represented as a structure
+that describes the nature of the test, and then includes or references minimal
+JSON Schema that test the assertions implied by the requirement.
+
+The structure of a test case is defined using a [JSON-LD
+Context](JSONtest-v1.jsonld). That context defines the following terms:
+
+|Keyword | Values | Meaning
+|---------------|-----------------|---------
+|name | string | The name of this test for display purposes
+|description | string | A long self-describing paragraph that explains the purpose of the test and the expected input
+|ref | URI | An optional reference to the portion of the specification to which the test relates
+|testType | `automated`, `manual`, `ref` | The type of test - this informs [WPT](https://github.com/web-platform-tests/wpt) how the test should be controlled and presented
+|skipFailures | list of strings | An optional list of assertionType values that, if present, should have their test skipped if the result would be "unexpected". Defaults to the empty list.
+|assertions | list of URI, List @@@ATRISK@@@, or AssertionObject | The ordered collection of tests the input should be run against. See [JSON Schema Usage](#jsonSchema) for the structure of the objects. URI is relative to the top level folder of the test collection if it has a slash; relative to the current directory if it does not. @@@@ATRISK@@@@ Lists can be nested to define groups of sub-tests. Assertions / groups can be conditionally skipped. See [Assertion Lists](#assertionLists) for more details.
+|content | URI or object | An object containing content to be checked against the referenced assertions, or a URI from which to retrieve that content
+
+Each test case has a suffix of `.test` and a shape like:
+
+<pre>
+{
+ "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
+ "name": "Verify annotation conforms to the model",
+ "description": "Supply an example annotation that conforms to the basic structure.",
+ "ref": "https://www.w3.org/TR/annotation-model/#model",
+ "testType": "manual",
+ "assertions": [
+ "common/has_context.json",
+ "common/has_id.json",
+ {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "Verify annotation has target",
+ "assertionType": "must",
+ "expectedResult": "valid",
+ "errorMessage": "The object was missing a required 'target' property",
+ "type": "object",
+ "properties": {
+ "target": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "array",
+ "anyOf": [
+ {
+ "type": "string"
+ }
+ ]
+ }
+ ],
+ "not": {"type": "object"}
+ }
+ },
+ "required": ["target"]
+ }
+ ]
+}
+</pre>
+
+External references are used when the "assertion" is a common one that needs to
+be checked on many different test cases (e.g., that there is an @context in the
+supplied annotation).
+
+NOTE: The title property of an assertionObject can contain markdown. This can
+help improve readability of the rendered assertions and debugging output.
+
+NOTE: The content property does not yet have a defined use. One potential use would
+be to act as a pointer to a URI that can supply annotations from an implementation.
+In that case the URI would take a parameter with the test name as a way of telling
+the end point what test is running so it can deliver the right content.
+
+### <a id="assertionLists">Assertion Lists</a> ###
+
+The `assertion` list is an ordered list of assertions that will be evaluated
+against the submitted content. The list is *required*, and MUST have at least
+one entry. Entries in the list have the following types:
+
+* AssertionObject
+
+An in-line Object as defined in the section [Assertion
+Objects](#assertionObjects).
+* URI
+
+A relative or absolute URI that references a AssertionObject in a .json file.
+If the URI is relative but contains no slashes, then it is considered to be
+in the current directory. If the URI is relative, contains slashes, but
+**does not start with a slash** then it is considered relative to the top of
+the tree of the current test collection (e.g., `annotation-model`).
+* List @@@ATRISK@@@
+
+A nested Assertion List. While nested Assertion Lists are optional, if one
+is present it MUST have at least one entry. Entries are as in this list.
+Assertion Lists can be nested to any depth (but don't do that - it would be
+too hard to maintain).
+
+
+<a id="assertionObjects">Assertion Objects</a>
+-----------------
+
+In this collection of tests, Assertion Objects can be contained inline in the
+`.test` files or contained in external files with the suffix `.json`. The
+vocabularly and structure is as defined in [JSON Schema
+v4](http://json-schema.org/documentation.html) augmented with some additional
+properties defined in this section.
+
+In general each JSON Schema definition provided in this test suite should be as
+minimal as possible. This promotes clarity and increases the likelihood that
+it is correct. While it is ---possible--- to create JSON Schema files that
+enforce many different requirements on a data model, it increases the
+complexity and can also reduce the atomicity of tests / sub-tests (because a
+ test ends up testing more than one thing). Please try to avoid creating
+complex JSON Schema. (A notable exception is the situation where multiple
+ properties of a structure are interdependent.)
+
+Tools such as [the JSON Schema Creator](http://jsonschema.net/) may be helpful
+in creating schema snippets that can be integrated into JSONtest Assertion
+Objects. Remember that the JSON Schema you create should be as permissive as
+possible to address the various shapes that a give property might take (e.g., a
+ 'foo' might be a string or an object that contains sub-properties that express
+ the string, or an array of 1 or more objects with those sub-properties).
+
+In addition to the validation keys defined in JSON Schema v4, Schema files in
+this collection are also permitted to use the following keywords:
+
+|Keyword | Values | Meaning |
+|---------------|-----------------|---------|
+|onUnexpectedResult | `failAndContinue`, `failAndSkip`, `failAndAbort`, `passAndContinue`, `passAndSkip`, `passAndAbort` | Action to take when the result is not as expected. Default is `failAndContinue` |
+|assertionType | `must`, `may`, `should` | Informs the system about the severity of a failure. The default is `must` |
+|assertionFile | URI | An external file that contains an assertion SCHEMA. When this value is supplied, and local properties will override the ones loaded from the external file.
+|errorMessage | string | A human readable explanation of what it means if the test fails. |
+|expectedResult | `valid`, `invalid` | Tells the framework whether validating against this schema is expected to succeed or fail. The default is `valid` |
+
+
+### Example Assertion Object ###
+
+<pre>
+{
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "Verify annotation has @context",
+ "type": "object",
+ "expectedResult" : "valid",
+ "properties": {
+ "@context": {
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "array",
+ "anyOf": [
+ {
+ "type": "string"
+ }
+ ]
+ }
+ ],
+ "not": {"type": "object"}
+ }
+ },
+ "required": ["@context"]
+}
+</pre>
+
+Note that in the case where a feature is *optional* the JSON Schema MUST be
+crafted such that if the attribute is permitted to be missing from the content
+(so that the result is `true`), but when the attribute is present in the
+content it conforms to any requirements.
+
+
+
+<a id="conditionObjects">Condition Objects</a>
+-----------------
+
+A Condition Object is a sub-class of an Assertion Object. It allows the
+specification of the evaluation strategy for the assertions referenced by the
+object. An object is a Condition Object IFF it has a `assertions` property. In
+this case, there MUST NOT be an `assertionFile` property.
+
+
+|Keyword | Values | Meaning |
+|---------------|-----------------|---------|
+|compareWith | `and`, `or` | How should the result of any referenced assertions be compared. Defaults to `and`. Note that this implies there is also an assertions property with a nested list of assertions to compare. |
+|assertions | a list of assertions as in a Test Case above. This is required if there is a compareWith property |
+
+
+An example of a test that would pass if there were an `@context` OR there were an `@id`:
+
+<pre>
+{
+ "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
+ "name": "A test that has an 'or' clause",
+ "description": "A complex test that uses or-ing among a list of assertions",
+ "ref": "https://www.w3.org/TR/annotation-model/#model",
+ "testType": "manual",
+ "assertions": [
+ { "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "must have context or id",
+ "description": "A more complex example that allows one of many options to pass",
+ "assertions": [
+ { "title": "Condition Object",
+ "description": "A pseudo-test that will get a result from the aggregate of its children",
+ "assertionType": "must",
+ "expectedResult": "valid",
+ "errorMessage": "Error: None of the various options were present",
+ "compareWith": "or",
+ "assertions": [
+ "common/has_context.json",
+ "common/has_id.json"
+ ]
+ }
+ ]
+ }
+ ]
+}
+</pre>
+
+
+Command Line Tools
+------------------
+
+### Building the Test Files ###
+
+The actual .html test case files are generated using the script
+tools/make_tests.py. This script will search the directory heriarchy looking for
+files ending on `.test` and creating `.html` files from them using the template in
+the tools folder. If you want to regenerate the examples too, supply the
+`--examples` option to the script.
+
+Note that when submitting tests to the repository, the `.html` versions must be
+included.
+
+### Testing the Tests ###
+
+### Driving Tests with Input Files ###
+
+Complex Examples
+----------------
+
+This section is a collection of more complex examples to illustrate the
+expressive power of the [Assertion List](#assertionLists) structure. These can
+be used as templates for creating actual `.test` files.
+
+### Including and Overriding an Assertion ###
+
+Assertions can be contained in external `.json` files. It is possible for an
+object in an Assertion List to include the external file and override one or
+more of its properties:
+
+<pre>
+{
+ "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
+ "name": "Permit no target property",
+ "description": "Ensure there is no 'target' property when there is a 'randomName' property in the Annotation",
+ "assertions": [
+ {
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "Verify annotation has randomName",
+ "type": "object",
+ "properties": {
+ "randomName": {
+ "type": "string"
+ }
+ },
+ "required": ["randomName"]
+ },
+ { "assertionFile" : "common/target.json",
+ "title" : "Require target to be missing",
+ "expectedResult" : "invalid",
+ "errorMessage" : "The target MUST not be present when 'randomName' is also present",
+ }
+ ]
+}
+</pre>
+
+### Nested Assertion Collections with Skip ###
+
+Assertion Lists can be nested within Assertion Lists. This feature, combined
+with the `onUnexpectedResult` property, makes it possible to skip a collection
+of tests when an assertion in the list is not satisfied:
+
+<pre>
+{
+ "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
+ "name": "If there is no 'target' property, skip some tests",
+ "description": "When 'target' is not present, other properties related to 'target' are not required",
+ "assertions": [
+ "common/context.json",
+ [
+ { "assertionFile" : "common/target.json",
+ "errorMessage" : "Target was not present so skip the rest of this section",
+ "onUnexpectedResult" : "failAndSkip"
+ },
+ "sometest.json",
+ "sometest2.json",
+ "sometest3.json"
+ ]
+ ]
+} ;
+</pre>
+
+### Assertion that finds a specific @context Value ###
+
+Sometimes you want a property to be flexible, but to have one and only one of a
+specific value. This is especially true with, for example, @context in JSON-LD.
+One way you might do this is:
+
+<pre>
+{
+ "$schema": "http://json-schema.org/draft-04/schema#",
+ "title": "Verify a specific @context",
+ "type": "object",
+ "expectedResult" : "valid",
+ "properties": {
+ "@context": {
+ "anyOf": [
+ {
+ "type": "string"
+ "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
+ },
+ {
+ "type": "array",
+ "minitems": "1",
+ "uniqueItems": true,
+ "additionalItems": true,
+ "items" : [
+ { "type": "string",
+ "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
+ }
+ ]
+ }
+ ],
+ "not": {"type": "object"}
+ }
+ },
+ "required": ["@context"]
+}
+
+</pre>