summaryrefslogtreecommitdiffstats
path: root/t/perf/README
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--t/perf/README210
1 files changed, 210 insertions, 0 deletions
diff --git a/t/perf/README b/t/perf/README
new file mode 100644
index 0000000..8f217d7
--- /dev/null
+++ b/t/perf/README
@@ -0,0 +1,210 @@
+Git performance tests
+=====================
+
+This directory holds performance testing scripts for git tools. The
+first part of this document describes the various ways in which you
+can run them.
+
+When fixing the tools or adding enhancements, you are strongly
+encouraged to add tests in this directory to cover what you are
+trying to fix or enhance. The later part of this short document
+describes how your test scripts should be organized.
+
+
+Running Tests
+-------------
+
+The easiest way to run tests is to say "make". This runs all
+the tests on the current git repository.
+
+ === Running 2 tests in this tree ===
+ [...]
+ Test this tree
+ ---------------------------------------------------------
+ 0001.1: rev-list --all 0.54(0.51+0.02)
+ 0001.2: rev-list --all --objects 6.14(5.99+0.11)
+ 7810.1: grep worktree, cheap regex 0.16(0.16+0.35)
+ 7810.2: grep worktree, expensive regex 7.90(29.75+0.37)
+ 7810.3: grep --cached, cheap regex 3.07(3.02+0.25)
+ 7810.4: grep --cached, expensive regex 9.39(30.57+0.24)
+
+Output format is in seconds "Elapsed(User + System)"
+
+You can compare multiple repositories and even git revisions with the
+'run' script:
+
+ $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh
+
+where . stands for the current git tree. The full invocation is
+
+ ./run [<revision|directory>...] [--] [<test-script>...]
+
+A '.' argument is implied if you do not pass any other
+revisions/directories.
+
+You can also manually test this or another git build tree, and then
+call the aggregation script to summarize the results:
+
+ $ ./p0001-rev-list.sh
+ [...]
+ $ ./run /path/to/other/git -- ./p0001-rev-list.sh
+ [...]
+ $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
+
+aggregate.perl has the same invocation as 'run', it just does not run
+anything beforehand.
+
+You can set the following variables (also in your config.mak):
+
+ GIT_PERF_REPEAT_COUNT
+ Number of times a test should be repeated for best-of-N
+ measurements. Defaults to 3.
+
+ GIT_PERF_MAKE_OPTS
+ Options to use when automatically building a git tree for
+ performance testing. E.g., -j6 would be useful. Passed
+ directly to make as "make $GIT_PERF_MAKE_OPTS".
+
+ GIT_PERF_MAKE_COMMAND
+ An arbitrary command that'll be run in place of the make
+ command, if set the GIT_PERF_MAKE_OPTS variable is
+ ignored. Useful in cases where source tree changes might
+ require issuing a different make command to different
+ revisions.
+
+ This can be (ab)used to monkeypatch or otherwise change the
+ tree about to be built. Note that the build directory can be
+ re-used for subsequent runs so the make command might get
+ executed multiple times on the same tree, but don't count on
+ any of that, that's an implementation detail that might change
+ in the future.
+
+ GIT_PERF_REPO
+ GIT_PERF_LARGE_REPO
+ Repositories to copy for the performance tests. The normal
+ repo should be at least git.git size. The large repo should
+ probably be about linux.git size for optimal results.
+ Both default to the git.git you are running from.
+
+ GIT_PERF_EXTRA
+ Boolean to enable additional tests. Most test scripts are
+ written to detect regressions between two versions of Git, and
+ the output will compare timings for individual tests between
+ those versions. Some scripts have additional tests which are not
+ run by default, that show patterns within a single version of
+ Git (e.g., performance of index-pack as the number of threads
+ changes). These can be enabled with GIT_PERF_EXTRA.
+
+ GIT_PERF_USE_SCALAR
+ Boolean indicating whether to register test repo(s) with Scalar
+ before executing tests.
+
+You can also pass the options taken by ordinary git tests; the most
+useful one is:
+
+--root=<directory>::
+ Create "trash" directories used to store all temporary data during
+ testing under <directory>, instead of the t/ directory.
+ Using this option with a RAM-based filesystem (such as tmpfs)
+ can massively speed up the test suite.
+
+
+Naming Tests
+------------
+
+The performance test files are named as:
+
+ pNNNN-commandname-details.sh
+
+where N is a decimal digit. The same conventions for choosing NNNN as
+for normal tests apply.
+
+
+Writing Tests
+-------------
+
+The perf script starts much like a normal test script, except it
+sources perf-lib.sh:
+
+ #!/bin/sh
+ #
+ # Copyright (c) 2005 Junio C Hamano
+ #
+
+ test_description='xxx performance test'
+ . ./perf-lib.sh
+
+After that you will want to use some of the following:
+
+ test_perf_fresh_repo # sets up an empty repository
+ test_perf_default_repo # sets up a "normal" repository
+ test_perf_large_repo # sets up a "large" repository
+
+ test_perf_default_repo sub # ditto, in a subdir "sub"
+
+ test_checkout_worktree # if you need the worktree too
+
+At least one of the first two is required!
+
+You can use test_expect_success as usual. In both test_expect_success
+and in test_perf, running "git" points to the version that is being
+perf-tested. The $MODERN_GIT variable points to the git wrapper for the
+currently checked-out version (i.e., the one that matches the t/perf
+scripts you are running). This is useful if your setup uses commands
+that only work with newer versions of git than what you might want to
+test (but obviously your new commands must still create a state that can
+be used by the older version of git you are testing).
+
+For actual performance tests, use
+
+ test_perf 'descriptive string' '
+ command1 &&
+ command2
+ '
+
+test_perf spawns a subshell, for lack of better options. This means
+that
+
+* you _must_ export all variables that you need in the subshell
+
+* you _must_ flag all variables that you want to persist from the
+ subshell with 'test_export':
+
+ test_perf 'descriptive string' '
+ foo=$(git rev-parse HEAD) &&
+ test_export foo
+ '
+
+ The so-exported variables are automatically marked for export in the
+ shell executing the perf test. For your convenience, test_export is
+ the same as export in the main shell.
+
+ This feature relies on a bit of magic using 'set' and 'source'.
+ While we have tried to make sure that it can cope with embedded
+ whitespace and other special characters, it will not work with
+ multi-line data.
+
+Rather than tracking the performance by run-time as `test_perf` does, you
+may also track output size by using `test_size`. The stdout of the
+function should be a single numeric value, which will be captured and
+shown in the aggregated output. For example:
+
+ test_perf 'time foo' '
+ ./foo >foo.out
+ '
+
+ test_size 'output size'
+ wc -c <foo.out
+ '
+
+might produce output like:
+
+ Test origin HEAD
+ -------------------------------------------------------------
+ 1234.1 time foo 0.37(0.79+0.02) 0.26(0.51+0.02) -29.7%
+ 1234.2 output size 4.3M 3.6M -14.7%
+
+The item being measured (and its units) is up to the test; the context
+and the test title should make it clear to the user whether bigger or
+smaller numbers are better. Unlike test_perf, the test code will only be
+run once, since output sizes tend to be more deterministic than timings.