6 files changed, 266 insertions, 0 deletions
diff --git a/docs/develop/_changelog.md b/docs/develop/_changelog.md
new file mode 100644
index 0000000..d24e7ac
--- /dev/null
+++ b/docs/develop/_changelog.md
@@ -0,0 +1,4 @@
+```{include} ../../CHANGELOG.md
+:relative-docs: docs/
+:relative-images:
+```
diff --git a/docs/develop/architecture.md b/docs/develop/architecture.md
new file mode 100644
index 0000000..f00cdae
--- /dev/null
+++ b/docs/develop/architecture.md
@@ -0,0 +1,27 @@
+# The MyST implementation architecture
+
+This page describes implementation details to help you understand the structure
+of the project.
+
+## A Renderer for markdown-it tokens
+
+At a high level, the MyST parser is an extension of th project. Markdown-It-Py
+is a well-structured Python parser for CommonMark text. It also defines an extension
+point to include more syntax in parsed files. The MyST parser uses this extension
+point to define its own syntax options (e.g., for Sphinx roles and directives).
+
+The result of this parser is a markdown-it token stream.
+
+## A docutils renderer
+
+The MyST parser also defines a docutils renderer for the markdown-it token stream.
+This allows us to convert parsed elements of a MyST markdown file into docutils.
+
+## A Sphinx parser
+
+Finally, the MyST parser provides a parser for Sphinx, the documentation generation
+system. This parser does the following:
+
+* Parse markdown files with the markdown-it parser, including MyST specific plugins
+* Convert these files into docutils objects using the MyST docutils renderer
+* Provide these to Sphinx in order to use in building your site.
diff --git a/docs/develop/background.md b/docs/develop/background.md
new file mode 100644
index 0000000..4be19e0
--- /dev/null
+++ b/docs/develop/background.md
@@ -0,0 +1,82 @@
+# Background
+
+These sections discuss high-level questions about the MyST ecosystem, and explain a few decisions made in the project.
+
+## Why did we create MyST markdown?
+
+While markdown is ubiquitous, it is not powerful enough for writing modern,
+fully-featured documentation. Some flavors of markdown support features needed for this,
+but there is no community standard around various syntactic choices for these features.
+
+Sphinx is a documentation generation framework written in Python. It heavily-utilizes
+reStructuredText syntax, which is another markup language for writing documents. In
+particular, Sphinx defines two extension points that are extremely useful:
+**{ref}`in-line roles<sphinx:rst-roles-alt>`** and **{ref}`block-level directives <sphinx:rst-directives>`**.
+
+**This project is an attempt at combining the simplicity and readability of Markdown
+with the power and flexibility of reStructuredText and the Sphinx platform.** It
+starts with the [CommonMark markdown specification][commonmark], and selectively adds a few extra
+syntax pieces to utilize the most powerful parts of reStructuredText.
+
+```{note}
+The CommonMark community has been discussing an "official" extension syntax for many
+years now (for example, see
+[this seven-year-old thread about directives](https://talk.commonmark.org/t/generic-directives-plugins-syntax/444) as well as
+[this more recent converstaion](https://talk.commonmark.org/t/support-for-extension-token/2771),
+and [this comment listing several more threads on this topic](https://talk.commonmark.org/t/extension-terminology-and-rules/1233)).
+
+We have chosen a "roles and directives" syntax that seems reasonable and follows other
+common conventions in Markdown flavors. However, if the CommonMark community ever
+decides on an "official" extension syntax, we will likely utilize this syntax for
+MyST.
+```
+
+## The relationship between MyST, reStructuredText, and Sphinx
+
+MyST markdown provides a markdown equivalent of the reStructuredText syntax,
+meaning that you can do anything in MyST that you can do with reStructuredText.
+
+The Sphinx documentation engine supports a number of different input types. By default,
+Sphinx reads **reStructuredText** (`.rst`) files. Sphinx uses a **parser** to parse input files
+into its own internal document model (which is provided by a core Python project,
+[docutils](https://docutils.sourceforge.io/)).
+
+Developers can *extend Sphinx* to support other kinds of input files. Any content file
+can be read into the Sphinx document structure, provided that somebody writes a
+**parser** for that file. Once a content file has been parsed into Sphinx, it behaves
+nearly the same way as any other content file, regardless of the language in which it
+was written.
+
+The MyST-parser is a Sphinx parser for the MyST markdown language.
+When you use it, Sphinx will know how to parse content files that contain MyST markdown (by default, Sphinx will assume any files ending in `.md` are written in MyST markdown). Once a document has been parsed into Sphinx, it behaves the same way regardless of whether it has been written in rST or MyST markdown.
+
+```
+myst markdown (.md) ------> myst parser ---+
+                                           |
+                                           +-->Sphinx document (docutils)
+                                           |
+reStructuredText (.rst) --> rst parser ----+
+```
+
+For example, here's how you'd write a `toctree` directive in MyST markdown:
+
+````
+```{toctree}
+My page name <page1>
+page2
+```
+````
+
+and here's the same in rST:
+
+```
+.. toctree::
+
+   My page name <page1>
+   page2
+```
+
+They will both behave the same in Sphinx.
+
+
+[commonmark]: https://commonmark.org/
diff --git a/docs/develop/contributing.md b/docs/develop/contributing.md
new file mode 100644
index 0000000..7f57052
--- /dev/null
+++ b/docs/develop/contributing.md
@@ -0,0 +1,85 @@
+# Contributing
+
+[![Github-CI][github-ci]][github-link]
+[![Coverage Status][codecov-badge]][codecov-link]
+[![Documentation Status][rtd-badge]][rtd-link]
+[![Code style: black][black-badge]][black-link]
+
+We welcome all contributions!
+See the [EBP Contributing Guide](https://executablebooks.org/en/latest/contributing.html) for general details, and below for guidance specific to MyST-Parser.
+
+## Install for development
+
+To install `myst-parser` for development, take the following steps:
+
+```bash
+git clone https://github.com/executablebooks/MyST-Parser
+cd MyST-Parser
+git checkout master
+pip install -e .[code_style,testing,rtd]
+```
+
+## Code Style
+
+Code style is tested using [flake8](http://flake8.pycqa.org),
+with the configuration set in `.flake8`,
+and code formatted with [black](https://github.com/ambv/black).
+
+Installing with `myst-parser[code_style]` makes the [pre-commit](https://pre-commit.com/)
+package available, which will ensure this style is met before commits are submitted, by reformatting the code
+and testing for lint errors.
+It can be setup by:
+
+```shell
+>> cd MyST-Parser
+>> pre-commit install
+```
+
+Optionally you can run `black` and `flake8` separately:
+
+```shell
+>> black .
+>> flake8 .
+```
+
+Editors like VS Code also have automatic code reformat utilities, which can adhere to this standard.
+
+All functions and class methods should be annotated with types and include a docstring. The preferred docstring format is outlined in `MyST-Parser/docstring.fmt.mustache` and can be used automatically with the
+[autodocstring](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) VS Code extension.
+
+## Testing
+
+For code tests, myst-parser uses [pytest](https://docs.pytest.org):
+
+```shell
+>> cd MyST-Parser
+>> pytest
+```
+
+You can also use [tox](https://tox.readthedocs.io), to run the tests in multiple isolated environments (see the `tox.ini` file for available test environments):
+
+```shell
+>> cd MyST-Parser
+>> tox
+```
+
+For documentation build tests:
+
+```shell
+>> cd MyST-Parser/docs
+>> make clean
+>> make html-strict
+```
+
+```{seealso}
+{ref}`develop/testing`
+```
+
+[github-ci]: https://github.com/executablebooks/MyST-Parser/workflows/continuous-integration/badge.svg?branch=master
+[github-link]: https://github.com/executablebooks/MyST-Parser
+[codecov-badge]: https://codecov.io/gh/executablebooks/MyST-Parser/branch/master/graph/badge.svg
+[codecov-link]: https://codecov.io/gh/executablebooks/MyST-Parser
+[rtd-badge]: https://readthedocs.org/projects/myst-parser/badge/?version=latest
+[rtd-link]: https://myst-parser.readthedocs.io/en/latest/?badge=latest
+[black-badge]: https://img.shields.io/badge/code%20style-black-000000.svg
+[black-link]: https://github.com/ambv/black
diff --git a/docs/develop/index.md b/docs/develop/index.md
new file mode 100644
index 0000000..f3a3f97
--- /dev/null
+++ b/docs/develop/index.md
@@ -0,0 +1,15 @@
+# Contribute
+
+This section covers documentation relevant to developing and maintaining the MyST
+codebase, and some guidelines for how you can contribute.
+
+```{toctree}
+contributing.md
+architecture.md
+test_infrastructure.md
+```
+
+## Code of Conduct
+
+The MyST-parser project follows the
+[Executable Book Project code of conduct](https://github.com/executablebooks/.github/blob/master/CODE_OF_CONDUCT.md).
diff --git a/docs/develop/test_infrastructure.md b/docs/develop/test_infrastructure.md
new file mode 100644
index 0000000..ebef011
--- /dev/null
+++ b/docs/develop/test_infrastructure.md
@@ -0,0 +1,53 @@
+(develop/testing)=
+
+# Testing Infrastructure
+
+Where possible, additions to the code should be carried out in a
+[test-driven development](https://en.wikipedia.org/wiki/Test-driven_development)
+manner:
+
+> **Write failing tests that the code should pass, then write code to pass the tests**.
+
+The tests are run using [pytest](https://docs.pytest.org)/[GitHub Actions](https://github.com/features/actions) for unit tests, and [readthedocs](https://readthedocs.org/) for documentation build tests.
+
+The tests are ordered in a hierarchical fashion:
+
+1. In `tests/test_commonmark` the [CommonMark](https://github.com/commonmark/CommonMark.git) test set is run to check that the parser is complying with the CommonMark specification.
+2. In `tests/test_renderers` are tests that check that the Markdown AST is being correctly converted to the docutils/sphinx AST. This includes testing that roles and directives are correctly parsed and run.
+3. In `tests/test_sphinx` are tests that check that minimal sphinx project builds are running correctly, to convert MyST markdown files to HTML.
+4. In `.circleci` the package documentation (written in MyST format) is built and tested for build errors/warnings.
+
+## Test tools
+
+[**pytest-regressions**](https://pytest-regressions.readthedocs.io) is a pytest plugin
+that is used in the test suite, to maintain tests that generate lots of data.
+In particular, they are used in the syntax testing to generate tests for AST trees
+which may change in the future due to changes/additions to the data captured by the parser.
+For example, after writing:
+
+```python
+def test_example_dict(data_regression):
+    data_regression.check({
+        "key1": "value1",
+        "key2": "value2",
+        "more": "data...",
+    })
+def test_example_str(file_regression):
+    file_regression.check("a very long string...")
+```
+
+Running the following will initially fail,
+but will also generate a file (per test) of expected output:
+
+```console
+$ pytest -k test_example
+```
+
+Subsequent times the tests are run, the tests output will now be validated against these stored files.
+
+After a change to the syntax parser, all failing tests can then be 'regenerated' with the new
+expected output, by running:
+
+```console
+$ pytest --force-regen
+```