diff options
Diffstat (limited to 'taskcluster/docs/caches.rst')
| -rw-r--r-- | taskcluster/docs/caches.rst | 98 |
1 files changed, 98 insertions, 0 deletions
diff --git a/taskcluster/docs/caches.rst b/taskcluster/docs/caches.rst new file mode 100644 index 0000000000..dcda4102dd --- /dev/null +++ b/taskcluster/docs/caches.rst @@ -0,0 +1,98 @@ +.. taskcluster_caches: + +Caches +====== + +There are various caches used by the in-tree tasks. This page attempts to +document them and their appropriate use. + +Caches are essentially isolated filesystems that are persisted between +tasks. For example, if 2 tasks run on a worker - one after the other - +and both tasks request the same cache, the subsequent task will be +able to see files in the cache that were created by the first task. +It's also worth noting that TaskCluster workers ensure a cache can only +be used by 1 task at a time. If a worker is simultaneously running +multiple tasks requesting the same named cache, the worker will +have multiple caches of the same name on the worker. + +Caches and ``run-task`` +----------------------- + +``run-task`` is our generic task wrapper script. It does common activities +like ensure a version control checkout is present. + +One of the roles of ``run-task`` is to verify and sanitize caches. +It does this by storing state in a cache on its first use. If the recorded +*capabilities* of an existing cache don't match expectations for the +current task, ``run-task`` bails. This ensures that caches are only +reused by tasks with similar execution *profiles*. This prevents +accidental cache use across incompatible tasks. It also allows run-task +to make assumptions about the state of caches, which can help avoid +costly operations. + +In addition, the hash of ``run-task`` is used to derive the cache name. +So any time ``run-task`` changes, a new set of caches are used. This +ensures that any backwards incompatible changes or bug fixes to +``run-task`` result in fresh caches. + +Some caches are reserved for use with run-task. That property will be denoted +below. + +Common Caches +------------- + +Version Control Caches +:::::::::::::::::::::: + +``level-{{level}}-checkouts-{{hash}}`` + This cache holds version control checkouts, each in a subdirectory named + after the repo (e.g., ``gecko``). + + Checkouts should be read-only. If a task needs to create new files from + content of a checkout, this content should be written in a separate + directory/cache (like a workspace). + + This cache name pattern is managed by ``run-task`` and must only be + used with ``run-task``. + +``level-{{level}}-checkouts-sparse-{{hash}}`` + This is like the above but is used when the checkout is sparse (contains + a subset of files). + +``level-{{level}}-checkouts-{{version}}`` (deprecated) + This cache holds version control checkouts, each in a subdirectory named + after the repo (e.g., ``gecko``). + + Checkouts should be read-only. If a task needs to create new files from + content of a checkout, this content should be written in a separate + directory/cache (like a workspace). + + A ``version`` parameter appears in the cache name to allow + backwards-incompatible changes to the cache's behavior. + + The ``hg-store`` contains a `shared store <https://www.mercurial-scm.org/wiki/ShareExtension>` + that is is used by ``hg robustcheckout``. If you are using ``run-task`` you + should set the ``HG_STORE_PATH`` environment variable to point to this + directory. If you are using ``hg robustcheckout``, pass this directory to the + ``--sharebase`` option. + +Workspace Caches +:::::::::::::::: + +``level-{{level}}-*-workspace`` + These caches (of various names typically ending with ``workspace``) + contain state to be shared between task invocations. Use cases are + dependent on the task. + +Other +::::: + +``level-{{level}}-tooltool-cache-{{hash}}`` + Tooltool invocations should use this cache. Tooltool will store files here + indexed by their hash. + + This cache name pattern is reserved for use with ``run-task`` and must only + be used by ``run-task`` + +``tooltool-cache`` (deprecated) + Legacy location for tooltool files. Use the per-level one instead. |