17 files changed, 698 insertions, 0 deletions

diff --git a/benches/README.md b/benches/README.md
new file mode 100644
index 0000000..b4b8b19
--- /dev/null
+++ b/benches/README.md
@@ -0,0 +1,124 @@
+# Cargo Benchmarking
+
+This directory contains some benchmarks for cargo itself. This uses
+[Criterion] for running benchmarks. It is recommended to read the Criterion
+book to get familiar with how to use it. A basic usage would be:
+
+```sh
+cd benches/benchsuite
+cargo bench
+```
+
+The tests involve downloading the index and benchmarking against some
+real-world and artificial workspaces located in the [`workspaces`](workspaces)
+directory.
+
+**Beware** that the initial download can take a fairly long amount of time (10
+minutes minimum on an extremely fast network) and require significant disk
+space (around 4.5GB). The benchsuite will cache the index and downloaded
+crates in the `target/tmp/bench` directory, so subsequent runs should be
+faster. You can (and probably should) specify individual benchmarks to run to
+narrow it down to a more reasonable set, for example:
+
+```sh
+cargo bench -- resolve_ws/rust
+```
+
+This will only download what's necessary for the rust-lang/rust workspace
+(which is about 330MB) and run the benchmarks against it (which should take
+about a minute). To get a list of all the benchmarks, run:
+
+```sh
+cargo bench -- --list
+```
+
+## Viewing reports
+
+The benchmarks display some basic information on the command-line while they
+run. A more complete HTML report can be found at
+`target/criterion/report/index.html` which contains links to all the
+benchmarks and summaries. Check out the Criterion book for more information on
+the extensive reporting capabilities.
+
+## Comparing implementations
+
+Knowing the raw numbers can be useful, but what you're probably most
+interested in is checking if your changes help or hurt performance. To do
+that, you need to run the benchmarks multiple times.
+
+First, run the benchmarks from the master branch of cargo without any changes.
+To make it easier to compare, Criterion supports naming the baseline so that
+you can iterate on your code and compare against it multiple times.
+
+```sh
+cargo bench -- --save-baseline master
+```
+
+Now you can switch to your branch with your changes. Re-run the benchmarks
+compared against the baseline:
+
+```sh
+cargo bench -- --baseline master
+```
+
+You can repeat the last command as you make changes to re-compare against the
+master baseline.
+
+Without the baseline arguments, it will compare against the last run, which
+can be helpful for comparing incremental changes.
+
+## Capturing workspaces
+
+The [`workspaces`](workspaces) directory contains several workspaces that
+provide a variety of different workspaces intended to provide good exercises
+for benchmarks. Some of these are shadow copies of real-world workspaces. This
+is done with the tool in the [`capture`](capture) directory. The tool will
+copy `Cargo.lock` and all of the `Cargo.toml` files of the workspace members.
+It also adds an empty `lib.rs` so Cargo won't error, and sanitizes the
+`Cargo.toml` to some degree, removing unwanted elements. Finally, it
+compresses everything into a `tgz`.
+
+To run it, do:
+
+```sh
+cd benches/capture
+cargo run -- /path/to/workspace/foo
+```
+
+The resolver benchmarks also support the `CARGO_BENCH_WORKSPACES` environment
+variable, which you can point to a Cargo workspace if you want to try
+different workspaces. For example:
+
+```sh
+CARGO_BENCH_WORKSPACES=/path/to/some/workspace cargo bench
+```
+
+## TODO
+
+This is just a start for establishing a benchmarking suite for Cargo. There's
+a lot that can be added. Some ideas:
+
+* Fix the benchmarks so that the resolver setup doesn't run every iteration.
+* Benchmark [this section of
+  code](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/ops/cargo_compile.rs#L470-L549)
+  which builds the unit graph. The performance there isn't great, and it would
+  be good to keep an eye on it. Unfortunately that would mean doing a bit of
+  work to make `generate_targets` publicly visible, and there is a bunch of
+  setup code that may need to be duplicated.
+* Benchmark the fingerprinting code.
+* Benchmark running the `cargo` executable. Running something like `cargo
+  build` or `cargo check` with everything "Fresh" would be a good end-to-end
+  exercise to measure the overall overhead of Cargo.
+* Benchmark pathological resolver scenarios. There might be some cases where
+  the resolver can spend a significant amount of time. It would be good to
+  identify if these exist, and create benchmarks for them. This may require
+  creating an artificial index, similar to the `resolver-tests`. This should
+  also consider scenarios where the resolver ultimately fails.
+* Benchmark without `Cargo.lock`. I'm not sure if this is particularly
+  valuable, since we are mostly concerned with incremental builds which will
+  always have a lock file.
+* Benchmark just
+  [`resolve::resolve`](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/core/resolver/mod.rs#L122)
+  without anything else. This can help focus on just the resolver.
+
+[Criterion]: https://bheisler.github.io/criterion.rs/book/
diff --git a/benches/benchsuite/Cargo.toml b/benches/benchsuite/Cargo.toml
new file mode 100644
index 0000000..f4bc358
--- /dev/null
+++ b/benches/benchsuite/Cargo.toml
@@ -0,0 +1,28 @@
+[package]
+name = "benchsuite"
+version = "0.1.0"
+edition = "2021"
+license = "MIT OR Apache-2.0"
+homepage = "https://github.com/rust-lang/cargo"
+repository = "https://github.com/rust-lang/cargo"
+documentation = "https://docs.rs/cargo-platform"
+description = "Benchmarking suite for Cargo."
+
+[dependencies]
+cargo = { path = "../.." }
+# Consider removing html_reports in 0.4 and switching to `cargo criterion`.
+criterion = { version = "0.3.5", features = ["html_reports"] }
+flate2 = { version = "1.0.3", default-features = false, features = ["zlib"] }
+tar = { version = "0.4.38", default-features = false }
+url = "2.2.2"
+
+[lib]
+bench = false
+
+[[bench]]
+name = "resolve"
+harness = false
+
+[[bench]]
+name = "workspace_initialization"
+harness = false
diff --git a/benches/benchsuite/benches/resolve.rs b/benches/benchsuite/benches/resolve.rs
new file mode 100644
index 0000000..d03cd62
--- /dev/null
+++ b/benches/benchsuite/benches/resolve.rs
@@ -0,0 +1,146 @@
+use benchsuite::fixtures;
+use cargo::core::compiler::{CompileKind, RustcTargetData};
+use cargo::core::resolver::features::{FeatureOpts, FeatureResolver};
+use cargo::core::resolver::{CliFeatures, ForceAllTargets, HasDevUnits, ResolveBehavior};
+use cargo::core::{PackageIdSpec, Workspace};
+use cargo::ops::WorkspaceResolve;
+use cargo::Config;
+use criterion::{criterion_group, criterion_main, Criterion};
+use std::path::Path;
+
+struct ResolveInfo<'cfg> {
+    ws: Workspace<'cfg>,
+    requested_kinds: [CompileKind; 1],
+    target_data: RustcTargetData<'cfg>,
+    cli_features: CliFeatures,
+    specs: Vec<PackageIdSpec>,
+    has_dev_units: HasDevUnits,
+    force_all_targets: ForceAllTargets,
+    ws_resolve: WorkspaceResolve<'cfg>,
+}
+
+/// Helper for resolving a workspace. This will run the resolver once to
+/// download everything, and returns all the data structures that are used
+/// during resolution.
+fn do_resolve<'cfg>(config: &'cfg Config, ws_root: &Path) -> ResolveInfo<'cfg> {
+    let requested_kinds = [CompileKind::Host];
+    let ws = Workspace::new(&ws_root.join("Cargo.toml"), config).unwrap();
+    let target_data = RustcTargetData::new(&ws, &requested_kinds).unwrap();
+    let cli_features = CliFeatures::from_command_line(&[], false, true).unwrap();
+    let pkgs = cargo::ops::Packages::Default;
+    let specs = pkgs.to_package_id_specs(&ws).unwrap();
+    let has_dev_units = HasDevUnits::Yes;
+    let force_all_targets = ForceAllTargets::No;
+    // Do an initial run to download anything necessary so that it does
+    // not confuse criterion's warmup.
+    let ws_resolve = cargo::ops::resolve_ws_with_opts(
+        &ws,
+        &target_data,
+        &requested_kinds,
+        &cli_features,
+        &specs,
+        has_dev_units,
+        force_all_targets,
+    )
+    .unwrap();
+    ResolveInfo {
+        ws,
+        requested_kinds,
+        target_data,
+        cli_features,
+        specs,
+        has_dev_units,
+        force_all_targets,
+        ws_resolve,
+    }
+}
+
+/// Benchmark of the full `resolve_ws_with_opts` which runs the resolver
+/// twice, the feature resolver, and more. This is a major component of a
+/// regular cargo build.
+fn resolve_ws(c: &mut Criterion) {
+    let fixtures = fixtures!();
+    let mut group = c.benchmark_group("resolve_ws");
+    for (ws_name, ws_root) in fixtures.workspaces() {
+        let config = fixtures.make_config(&ws_root);
+        // The resolver info is initialized only once in a lazy fashion. This
+        // allows criterion to skip this workspace if the user passes a filter
+        // on the command-line (like `cargo bench -- resolve_ws/tikv`).
+        //
+        // Due to the way criterion works, it tends to only run the inner
+        // iterator once, and we don't want to call `do_resolve` in every
+        // "step", since that would just be some useless work.
+        let mut lazy_info = None;
+        group.bench_function(&ws_name, |b| {
+            let ResolveInfo {
+                ws,
+                requested_kinds,
+                target_data,
+                cli_features,
+                specs,
+                has_dev_units,
+                force_all_targets,
+                ..
+            } = lazy_info.get_or_insert_with(|| do_resolve(&config, &ws_root));
+            b.iter(|| {
+                cargo::ops::resolve_ws_with_opts(
+                    ws,
+                    target_data,
+                    requested_kinds,
+                    cli_features,
+                    specs,
+                    *has_dev_units,
+                    *force_all_targets,
+                )
+                .unwrap();
+            })
+        });
+    }
+    group.finish();
+}
+
+/// Benchmark of the feature resolver.
+fn feature_resolver(c: &mut Criterion) {
+    let fixtures = fixtures!();
+    let mut group = c.benchmark_group("feature_resolver");
+    for (ws_name, ws_root) in fixtures.workspaces() {
+        let config = fixtures.make_config(&ws_root);
+        let mut lazy_info = None;
+        group.bench_function(&ws_name, |b| {
+            let ResolveInfo {
+                ws,
+                requested_kinds,
+                target_data,
+                cli_features,
+                specs,
+                has_dev_units,
+                ws_resolve,
+                ..
+            } = lazy_info.get_or_insert_with(|| do_resolve(&config, &ws_root));
+            b.iter(|| {
+                let feature_opts = FeatureOpts::new_behavior(ResolveBehavior::V2, *has_dev_units);
+                FeatureResolver::resolve(
+                    ws,
+                    target_data,
+                    &ws_resolve.targeted_resolve,
+                    &ws_resolve.pkg_set,
+                    cli_features,
+                    specs,
+                    requested_kinds,
+                    feature_opts,
+                )
+                .unwrap();
+            })
+        });
+    }
+    group.finish();
+}
+
+// Criterion complains about the measurement time being too small, but the
+// measurement time doesn't seem important to me, what is more important is
+// the number of iterations which defaults to 100, which seems like a
+// reasonable default. Otherwise, the measurement time would need to be
+// changed per workspace. We wouldn't want to spend 60s on every workspace,
+// that would take too long and isn't necessary for the smaller workspaces.
+criterion_group!(benches, resolve_ws, feature_resolver);
+criterion_main!(benches);
diff --git a/benches/benchsuite/benches/workspace_initialization.rs b/benches/benchsuite/benches/workspace_initialization.rs
new file mode 100644
index 0000000..af68efe
--- /dev/null
+++ b/benches/benchsuite/benches/workspace_initialization.rs
@@ -0,0 +1,27 @@
+use benchsuite::fixtures;
+use cargo::core::Workspace;
+use criterion::{criterion_group, criterion_main, Criterion};
+
+fn workspace_initialization(c: &mut Criterion) {
+    let fixtures = fixtures!();
+    let mut group = c.benchmark_group("workspace_initialization");
+    for (ws_name, ws_root) in fixtures.workspaces() {
+        let config = fixtures.make_config(&ws_root);
+        // The resolver info is initialized only once in a lazy fashion. This
+        // allows criterion to skip this workspace if the user passes a filter
+        // on the command-line (like `cargo bench -- workspace_initialization/tikv`).
+        group.bench_function(ws_name, |b| {
+            b.iter(|| Workspace::new(&ws_root.join("Cargo.toml"), &config).unwrap())
+        });
+    }
+    group.finish();
+}
+
+// Criterion complains about the measurement time being too small, but the
+// measurement time doesn't seem important to me, what is more important is
+// the number of iterations which defaults to 100, which seems like a
+// reasonable default. Otherwise, the measurement time would need to be
+// changed per workspace. We wouldn't want to spend 60s on every workspace,
+// that would take too long and isn't necessary for the smaller workspaces.
+criterion_group!(benches, workspace_initialization);
+criterion_main!(benches);
diff --git a/benches/benchsuite/src/lib.rs b/benches/benchsuite/src/lib.rs
new file mode 100644
index 0000000..42e0f51
--- /dev/null
+++ b/benches/benchsuite/src/lib.rs
@@ -0,0 +1,197 @@
+use cargo::Config;
+use std::fs;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+use url::Url;
+
+#[macro_export]
+macro_rules! fixtures {
+    () => {
+        $crate::Fixtures::new(env!("CARGO_TARGET_TMPDIR"))
+    };
+}
+
+// This is an arbitrary commit that existed when I started. This helps
+// ensure consistent results. It can be updated if needed, but that can
+// make it harder to compare results with older versions of cargo.
+const CRATES_IO_COMMIT: &str = "85f7bfd61ea4fee08ec68c468762e886b2aebec6";
+
+pub struct Fixtures {
+    cargo_target_tmpdir: PathBuf,
+}
+
+impl Fixtures {
+    pub fn new(cargo_target_tmpdir: &str) -> Self {
+        let bench = Self {
+            cargo_target_tmpdir: PathBuf::from(cargo_target_tmpdir),
+        };
+        bench.create_home();
+        bench.create_target_dir();
+        bench.clone_index();
+        bench.unpack_workspaces();
+        bench
+    }
+
+    fn root(&self) -> PathBuf {
+        self.cargo_target_tmpdir.join("bench")
+    }
+
+    fn target_dir(&self) -> PathBuf {
+        let mut p = self.root();
+        p.push("target");
+        p
+    }
+
+    fn cargo_home(&self) -> PathBuf {
+        let mut p = self.root();
+        p.push("chome");
+        p
+    }
+
+    fn index(&self) -> PathBuf {
+        let mut p = self.root();
+        p.push("index");
+        p
+    }
+
+    fn workspaces_path(&self) -> PathBuf {
+        let mut p = self.root();
+        p.push("workspaces");
+        p
+    }
+
+    fn registry_url(&self) -> Url {
+        Url::from_file_path(self.index()).unwrap()
+    }
+
+    fn create_home(&self) {
+        let home = self.cargo_home();
+        if !home.exists() {
+            fs::create_dir_all(&home).unwrap();
+        }
+        fs::write(
+            home.join("config.toml"),
+            format!(
+                r#"
+                [source.crates-io]
+                replace-with = 'local-snapshot'
+
+                [source.local-snapshot]
+                registry = '{}'
+            "#,
+                self.registry_url()
+            ),
+        )
+        .unwrap();
+    }
+
+    fn create_target_dir(&self) {
+        // This is necessary to ensure the .rustc_info.json file is written.
+        // Otherwise it won't be written, and it is very expensive to create.
+        if !self.target_dir().exists() {
+            fs::create_dir_all(self.target_dir()).unwrap();
+        }
+    }
+
+    /// This clones crates.io at a specific point in time into tmp/index.
+    fn clone_index(&self) {
+        let index = self.index();
+        let maybe_git = |command: &str| {
+            let status = Command::new("git")
+                .current_dir(&index)
+                .args(command.split_whitespace().collect::<Vec<_>>())
+                .status()
+                .expect("git should be installed");
+            status.success()
+        };
+        let git = |command: &str| {
+            if !maybe_git(command) {
+                panic!("failed to run git command: {}", command);
+            }
+        };
+        if index.exists() {
+            if maybe_git(&format!(
+                "rev-parse -q --verify {}^{{commit}}",
+                CRATES_IO_COMMIT
+            )) {
+                // Already fetched.
+                return;
+            }
+        } else {
+            fs::create_dir_all(&index).unwrap();
+            git("init --bare");
+            git("remote add origin https://github.com/rust-lang/crates.io-index");
+        }
+        git(&format!("fetch origin {}", CRATES_IO_COMMIT));
+        git("branch -f master FETCH_HEAD");
+    }
+
+    /// This unpacks the compressed workspace skeletons into tmp/workspaces.
+    fn unpack_workspaces(&self) {
+        let ws_dir = Path::new(env!("CARGO_MANIFEST_DIR"))
+            .parent()
+            .unwrap()
+            .join("workspaces");
+        let archives = fs::read_dir(ws_dir)
+            .unwrap()
+            .map(|e| e.unwrap().path())
+            .filter(|p| p.extension() == Some(std::ffi::OsStr::new("tgz")));
+        for archive in archives {
+            let name = archive.file_stem().unwrap();
+            let f = fs::File::open(&archive).unwrap();
+            let f = flate2::read::GzDecoder::new(f);
+            let dest = self.workspaces_path().join(&name);
+            if dest.exists() {
+                fs::remove_dir_all(&dest).unwrap();
+            }
+            let mut archive = tar::Archive::new(f);
+            archive.unpack(self.workspaces_path()).unwrap();
+        }
+    }
+
+    /// Vec of `(ws_name, ws_root)`.
+    pub fn workspaces(&self) -> Vec<(String, PathBuf)> {
+        // CARGO_BENCH_WORKSPACES can be used to override, otherwise it just uses
+        // the workspaces in the workspaces directory.
+        let mut ps: Vec<_> = match std::env::var_os("CARGO_BENCH_WORKSPACES") {
+            Some(s) => std::env::split_paths(&s).collect(),
+            None => fs::read_dir(self.workspaces_path())
+                .unwrap()
+                .map(|e| e.unwrap().path())
+                // These currently fail in most cases on Windows due to long
+                // filenames in the git checkouts.
+                .filter(|p| {
+                    !(cfg!(windows)
+                        && matches!(p.file_name().unwrap().to_str().unwrap(), "servo" | "tikv"))
+                })
+                .collect(),
+        };
+        // Sort so it is consistent.
+        ps.sort();
+        ps.into_iter()
+            .map(|p| (p.file_name().unwrap().to_str().unwrap().to_owned(), p))
+            .collect()
+    }
+
+    /// Creates a new Config.
+    pub fn make_config(&self, ws_root: &Path) -> Config {
+        let shell = cargo::core::Shell::new();
+        let mut config = Config::new(shell, ws_root.to_path_buf(), self.cargo_home());
+        // Configure is needed to set the target_dir which is needed to write
+        // the .rustc_info.json file which is very expensive.
+        config
+            .configure(
+                0,
+                false,
+                None,
+                false,
+                false,
+                false,
+                &Some(self.target_dir()),
+                &[],
+                &[],
+            )
+            .unwrap();
+        config
+    }
+}
diff --git a/benches/capture/Cargo.toml b/benches/capture/Cargo.toml
new file mode 100644
index 0000000..25a9017
--- /dev/null
+++ b/benches/capture/Cargo.toml
@@ -0,0 +1,12 @@
+[package]
+name = "capture"
+version = "0.1.0"
+edition = "2021"
+license = "MIT OR Apache-2.0"
+description = "Tool for capturing a real-world workspace for benchmarking."
+
+[dependencies]
+cargo_metadata = "0.14.0"
+flate2 = { version = "1.0.3", default-features = false, features = ["zlib"] }
+tar = { version = "0.4.38", default-features = false }
+toml = "0.7.0"
diff --git a/benches/capture/src/main.rs b/benches/capture/src/main.rs
new file mode 100644
index 0000000..f6f02c4
--- /dev/null
+++ b/benches/capture/src/main.rs
@@ -0,0 +1,164 @@
+//! This tool helps to capture the `Cargo.toml` files of a workspace.
+//!
+//! Run it by passing a list of workspaces to capture.
+//! Use the `-f` flag to allow it to overwrite existing captures.
+//! The workspace will be saved in a `.tgz` file in the `../workspaces` directory.
+
+use flate2::{Compression, GzBuilder};
+use std::fs;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+fn main() {
+    let force = std::env::args().any(|arg| arg == "-f");
+    let dest = Path::new(env!("CARGO_MANIFEST_DIR"))
+        .parent()
+        .unwrap()
+        .join("workspaces");
+    if !dest.exists() {
+        panic!("expected {} to exist", dest.display());
+    }
+    for arg in std::env::args().skip(1).filter(|arg| !arg.starts_with("-")) {
+        let source_root = fs::canonicalize(arg).unwrap();
+        capture(&source_root, &dest, force);
+    }
+}
+
+fn capture(source_root: &Path, dest: &Path, force: bool) {
+    let name = Path::new(source_root.file_name().unwrap());
+    let mut dest_gz = PathBuf::from(dest);
+    dest_gz.push(name);
+    dest_gz.set_extension("tgz");
+    if dest_gz.exists() {
+        if !force {
+            panic!(
+                "dest {:?} already exists, use -f to force overwriting",
+                dest_gz
+            );
+        }
+        fs::remove_file(&dest_gz).unwrap();
+    }
+    let vcs_info = capture_vcs_info(source_root, force);
+    let dst = fs::File::create(&dest_gz).unwrap();
+    let encoder = GzBuilder::new()
+        .filename(format!("{}.tar", name.to_str().unwrap()))
+        .write(dst, Compression::best());
+    let mut ar = tar::Builder::new(encoder);
+    ar.mode(tar::HeaderMode::Deterministic);
+    if let Some(info) = &vcs_info {
+        add_ar_file(&mut ar, &name.join(".cargo_vcs_info.json"), info);
+    }
+
+    // Gather all local packages.
+    let metadata = cargo_metadata::MetadataCommand::new()
+        .manifest_path(source_root.join("Cargo.toml"))
+        .features(cargo_metadata::CargoOpt::AllFeatures)
+        .exec()
+        .expect("cargo_metadata failed");
+    let mut found_root = false;
+    for package in &metadata.packages {
+        if package.source.is_some() {
+            continue;
+        }
+        let manifest_path = package.manifest_path.as_std_path();
+        copy_manifest(&manifest_path, &mut ar, name, &source_root);
+        found_root |= manifest_path == source_root.join("Cargo.toml");
+    }
+    if !found_root {
+        // A virtual workspace.
+        let contents = fs::read_to_string(source_root.join("Cargo.toml")).unwrap();
+        assert!(!contents.contains("[package]"));
+        add_ar_file(&mut ar, &name.join("Cargo.toml"), &contents);
+    }
+    let lock = fs::read_to_string(source_root.join("Cargo.lock")).unwrap();
+    add_ar_file(&mut ar, &name.join("Cargo.lock"), &lock);
+    let encoder = ar.into_inner().unwrap();
+    encoder.finish().unwrap();
+    eprintln!("created {}", dest_gz.display());
+}
+
+fn copy_manifest<W: std::io::Write>(
+    manifest_path: &Path,
+    ar: &mut tar::Builder<W>,
+    name: &Path,
+    source_root: &Path,
+) {
+    let relative_path = manifest_path
+        .parent()
+        .unwrap()
+        .strip_prefix(source_root)
+        .expect("workspace member should be under workspace root");
+    let relative_path = name.join(relative_path);
+    let contents = fs::read_to_string(&manifest_path).unwrap();
+    let mut manifest: toml::Value = toml::from_str(&contents).unwrap();
+    let remove = |obj: &mut toml::Value, name| {
+        let table = obj.as_table_mut().unwrap();
+        if table.contains_key(name) {
+            table.remove(name);
+        }
+    };
+    remove(&mut manifest, "lib");
+    remove(&mut manifest, "bin");
+    remove(&mut manifest, "example");
+    remove(&mut manifest, "test");
+    remove(&mut manifest, "bench");
+    remove(&mut manifest, "profile");
+    if let Some(package) = manifest.get_mut("package") {
+        remove(package, "default-run");
+    }
+    let contents = toml::to_string(&manifest).unwrap();
+    add_ar_file(ar, &relative_path.join("Cargo.toml"), &contents);
+    add_ar_file(ar, &relative_path.join("src").join("lib.rs"), "");
+}
+
+fn add_ar_file<W: std::io::Write>(ar: &mut tar::Builder<W>, path: &Path, contents: &str) {
+    let mut header = tar::Header::new_gnu();
+    header.set_entry_type(tar::EntryType::file());
+    header.set_mode(0o644);
+    header.set_size(contents.len() as u64);
+    header.set_mtime(123456789);
+    header.set_cksum();
+    ar.append_data(&mut header, path, contents.as_bytes())
+        .unwrap();
+}
+
+fn capture_vcs_info(ws_root: &Path, force: bool) -> Option<String> {
+    let maybe_git = |command: &str| {
+        Command::new("git")
+            .current_dir(ws_root)
+            .args(command.split_whitespace().collect::<Vec<_>>())
+            .output()
+            .expect("git should be installed")
+    };
+    assert!(ws_root.join("Cargo.toml").exists());
+    let relative = maybe_git("ls-files --full-name Cargo.toml");
+    if !relative.status.success() {
+        if !force {
+            panic!("git repository not detected, use -f to force");
+        }
+        return None;
+    }
+    let p = Path::new(std::str::from_utf8(&relative.stdout).unwrap().trim());
+    let relative = p.parent().unwrap();
+    if !force {
+        let has_changes = !maybe_git("diff-index --quiet HEAD .").status.success();
+        if has_changes {
+            panic!("git repo appears to have changes, use -f to force, or clean the repo");
+        }
+    }
+    let commit = maybe_git("rev-parse HEAD");
+    assert!(commit.status.success());
+    let commit = std::str::from_utf8(&commit.stdout).unwrap().trim();
+    let remote = maybe_git("remote get-url origin");
+    assert!(remote.status.success());
+    let remote = std::str::from_utf8(&remote.stdout).unwrap().trim();
+    let info = format!(
+        "{{\n  \"git\": {{\n    \"sha1\": \"{}\",\n     \"remote\": \"{}\"\n  }},\
+         \n  \"path_in_vcs\": \"{}\"\n}}\n",
+        commit,
+        remote,
+        relative.display()
+    );
+    eprintln!("recording vcs info:\n{}", info);
+    Some(info)
+}
diff --git a/benches/workspaces/cargo.tgz b/benches/workspaces/cargo.tgz
new file mode 100644
index 0000000..653aff9
--- /dev/null
+++ b/benches/workspaces/cargo.tgz
diff --git a/benches/workspaces/diem.tgz b/benches/workspaces/diem.tgz
new file mode 100644
index 0000000..e047c6c
--- /dev/null
+++ b/benches/workspaces/diem.tgz
diff --git a/benches/workspaces/empty.tgz b/benches/workspaces/empty.tgz
new file mode 100644
index 0000000..1a7d555
--- /dev/null
+++ b/benches/workspaces/empty.tgz
diff --git a/benches/workspaces/gecko-dev.tgz b/benches/workspaces/gecko-dev.tgz
new file mode 100644
index 0000000..e89c676
--- /dev/null
+++ b/benches/workspaces/gecko-dev.tgz
diff --git a/benches/workspaces/rust-ws-inherit.tgz b/benches/workspaces/rust-ws-inherit.tgz
new file mode 100644
index 0000000..6e7b669
--- /dev/null
+++ b/benches/workspaces/rust-ws-inherit.tgz
diff --git a/benches/workspaces/rust.tgz b/benches/workspaces/rust.tgz
new file mode 100644
index 0000000..74da475
--- /dev/null
+++ b/benches/workspaces/rust.tgz
diff --git a/benches/workspaces/servo.tgz b/benches/workspaces/servo.tgz
new file mode 100644
index 0000000..5111643
--- /dev/null
+++ b/benches/workspaces/servo.tgz
diff --git a/benches/workspaces/substrate.tgz b/benches/workspaces/substrate.tgz
new file mode 100644
index 0000000..81c3874
--- /dev/null
+++ b/benches/workspaces/substrate.tgz
diff --git a/benches/workspaces/tikv.tgz b/benches/workspaces/tikv.tgz
new file mode 100644
index 0000000..74add19
--- /dev/null
+++ b/benches/workspaces/tikv.tgz
diff --git a/benches/workspaces/toml-rs.tgz b/benches/workspaces/toml-rs.tgz
new file mode 100644
index 0000000..9acab19
--- /dev/null
+++ b/benches/workspaces/toml-rs.tgz