summaryrefslogtreecommitdiffstats
path: root/src/spdk/test/common/config
diff options
context:
space:
mode:
Diffstat (limited to 'src/spdk/test/common/config')
-rw-r--r--src/spdk/test/common/config/README.md104
-rw-r--r--src/spdk/test/common/config/pkgdep/apt-get100
-rw-r--r--src/spdk/test/common/config/pkgdep/dnf72
-rw-r--r--src/spdk/test/common/config/pkgdep/git325
-rw-r--r--src/spdk/test/common/config/pkgdep/pacman62
-rw-r--r--src/spdk/test/common/config/pkgdep/pkg27
-rw-r--r--src/spdk/test/common/config/pkgdep/swupd21
-rw-r--r--src/spdk/test/common/config/pkgdep/yum67
-rw-r--r--src/spdk/test/common/config/vm_setup.conf12
-rwxr-xr-xsrc/spdk/test/common/config/vm_setup.sh176
10 files changed, 966 insertions, 0 deletions
diff --git a/src/spdk/test/common/config/README.md b/src/spdk/test/common/config/README.md
new file mode 100644
index 000000000..26a587709
--- /dev/null
+++ b/src/spdk/test/common/config/README.md
@@ -0,0 +1,104 @@
+# Virtual Test Configuration
+
+This readme and the associated bash script, vm_setup.sh, are intended to assist developers in quickly
+preparing a virtual test environment on which to run the SPDK validation tests rooted at autorun.sh.
+This file contains basic information about SPDK environment requirements, an introduction to the
+autorun-spdk.conf files used to moderate which tests are run by autorun.sh, and step-by-step instructions
+for spinning up a VM capable of running the SPDK test suite.
+There is no need for external hardware to run these tests. The linux kernel comes with the drivers necessary
+to emulate an RDMA enabled NIC. NVMe controllers can also be virtualized in emulators such as QEMU.
+
+## VM Envronment Requirements (Host)
+
+- 8 GiB of RAM (for DPDK)
+- Enable intel_kvm on the host machine from the bios.
+- Enable nesting for VMs in kernel command line (for vhost tests).
+ - In `/etc/default/grub` append the following to the GRUB_CMDLINE_LINUX line: intel_iommu=on kvm-intel.nested=1.
+
+## VM Specs
+
+When creating the user during the fedora installation, it is best to use the name sys_sgsw. Efforts are being made
+to remove all references to this user, or files specific to this user from the codebase, but there are still some
+trailing references to it.
+
+## Autorun-spdk.conf
+
+Every machine that runs the autotest scripts should include a file titled autorun-spdk.conf in the home directory
+of the user that will run them. This file consists of several lines of the form 'variable_name=0/1'. autorun.sh sources
+this file each time it is run, and determines which tests to attempt based on which variables are defined in the
+configuration file. For a full list of the variable declarations available for autorun-spdk.conf, please see
+`test/common/autotest_common.sh` starting at line 13.
+
+## Steps for Configuring the VM
+
+1. Download a fresh Fedora 26 image.
+2. Perform the installation of Fedora 26 server.
+3. Create an admin user sys_sgsw (enabling passwordless sudo for this account will make life easier during the tests).
+4. Run the vm_setup.sh script which will install all proper dependencies.
+5. Modify the autorun-spdk.conf file in the home directory.
+6. Reboot the VM.
+7. Run autorun.sh for SPDK. Any output files will be placed in `~/spdk_repo/output/`.
+
+## Additional Steps for Preparing the Vhost Tests
+
+The Vhost tests also require the creation of a second virtual machine nested inside of the test VM.
+Please follow the directions below to complete that installation. Note that host refers to the Fedora VM
+created above and guest or VM refer to the Ubuntu VM created in this section.
+
+1. Follow instructions from spdk/scripts/vagrant/README.md
+ - install all needed packages mentioned in "Mac OSX Setup" or "Windows 10 Setup" sections
+ - follow steps from "Configure Vagrant" section
+
+2. Use Vagrant scripts located in spdk/scripts/vagrant to automatically generate
+ VM image to use in SPDK vhost tests.
+ Example command:
+ ~~~{.sh}
+ spdk/scripts/vagrant/create_vhost_vm.sh --move-to-def-dirs ubuntu16
+ ~~~
+ This command will:
+ - Download a Ubuntu 16.04 image file
+ - upgrade the system and install needed dependencies (fio, sg3-utils, bc)
+ - add entry to VM's ~/.ssh/autorized_keys
+ - add appropriate options to GRUB command line and update grub
+ - convert the image to .qcow2 format
+ - move .qcow2 file and ssh keys to default locations used by vhost test scripts
+
+Alternatively it is possible to create the VM image manually using following steps:
+
+1. Create an image file for the VM. It does not have to be large, about 3.5G should suffice.
+2. Create an ssh keypair for host-guest communications (performed on the host):
+ - Generate an ssh keypair with the name spdk_vhost_id_rsa and save it in `/root/.ssh`.
+ - Make sure that only root has read access to the private key.
+3. Install the OS in the VM image (performed on guest):
+ - Use the latest Ubuntu server (Currently 16.04 LTS).
+ - When partitioning the disk, make one partion that consumes the whole disk mounted at /. Do not encrypt the disk or enable LVM.
+ - Choose the OpenSSH server packages during install.
+4. Post installation configuration (performed on guest):
+ - Run the following commands to enable all necessary dependencies:
+ ~~~{.sh}
+ sudo apt update
+ sudo apt upgrade
+ sudo apt install fio sg3-utils bc
+ ~~~
+ - Enable the root user: "sudo passwd root -> root".
+ - Enable root login over ssh: vim `/etc/ssh/sshd_config` -> PermitRootLogin=yes.
+ - Disable DNS for ssh: `/etc/ssh/sshd_config` -> UseDNS=no.
+ - Add the spdk_vhost key to root's known hosts: `/root/.ssh/authorized_keys` -> add spdk_vhost_id_rsa.pub key to authorized keys.
+ Remember to save the private key in `~/.ssh/spdk_vhost_id_rsa` on the host.
+ - Change the grub boot options for the guest as follows:
+ - Add "console=ttyS0 earlyprintk=ttyS0" to the boot options in `/etc/default/grub` (for serial output redirect).
+ - Add "scsi_mod.use_blk_mq=1" to boot options in `/etc/default/grub`.
+ ~~~{.sh}
+ sudo update-grub
+ ~~~
+ - Reboot the VM.
+ - Remove any unnecessary packages (this is to make booting the VM faster):
+ ~~~{.sh}
+ apt purge snapd
+ apt purge Ubuntu-core-launcher
+ apt purge squashfs-tools
+ apt purge unattended-upgrades
+ ~~~
+5. Copy the fio binary from the guest location `/usr/bin/fio` to the host location `/home/sys_sgsw/fio_ubuntu`.
+6. Place the guest VM in the host at the following location: `/home/sys_sgsw/vhost_vm_image.qcow2`.
+7. On the host, edit the `~/autorun-spdk.conf` file to include the following line: SPDK_TEST_VHOST=1.
diff --git a/src/spdk/test/common/config/pkgdep/apt-get b/src/spdk/test/common/config/pkgdep/apt-get
new file mode 100644
index 000000000..a1630620d
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/apt-get
@@ -0,0 +1,100 @@
+package_manager=apt-get
+
+update() {
+ sudo "$package_manager" update
+}
+
+install() {
+ (( $# )) || return 0
+
+ sudo "$package_manager" install -y "$@"
+}
+
+upgrade() {
+ sudo "$package_manager" update
+ sudo "$package_manager" upgrade -y
+}
+
+
+pre_install() {
+ echo "Package perl-open is not available at Ubuntu repositories" >&2
+
+ update
+
+ if [[ $INSTALL_TSOCKS == true ]]; then
+ install tsocks
+ fi
+
+ # asan an ubsan have to be installed together to not mix up gcc versions
+ if install libasan5; then
+ install libubsan1
+ else
+ echo "Latest libasan5 is not available" >&2
+ echo " installing libasan2 and corresponding libubsan0" >&2
+ install libasan2
+ install libubsan0
+ fi
+ if ! install rdma-core; then
+ echo "Package rdma-core is avaliable at Ubuntu 18 [universe] repositorium" >&2
+ install rdmacm-utils
+ install ibverbs-utils
+ else
+ LIBRXE_INSTALL=false
+ fi
+ if ! install libpmempool1; then
+ echo "Package libpmempool1 is available at Ubuntu 18 [universe] repositorium" >&2
+ fi
+ if ! install clang-tools; then
+ echo "Package clang-tools is available at Ubuntu 18 [universe] repositorium" >&2
+ fi
+ if ! install --no-install-suggests --no-install-recommends open-isns-utils; then
+ echo "Package open-isns-utils is available at Ubuntu 18 [universe] repositorium" >&2
+ fi
+
+ # Package name for Ubuntu 18 is targetcli-fb but for Ubuntu 16 it's targetcli
+ if ! install targetcli-fb; then
+ install targetcli
+ fi
+
+ # On Ubuntu 20.04 (focal) btrfs-tools are available under different name - btrfs-progs
+ if ! install btrfs-tools; then
+ install btrfs-progs
+ fi
+}
+
+packages=(
+ valgrind
+ jq
+ nvme-cli
+ ceph
+ gdb
+ fio
+ librbd-dev
+ linux-headers-generic
+ libgflags-dev
+ autoconf
+ automake
+ libtool
+ libmount-dev
+ open-iscsi
+ libglib2.0-dev
+ libpixman-1-dev
+ astyle
+ elfutils
+ libelf-dev
+ flex
+ bison
+ libswitch-perl
+ gdisk
+ socat
+ sshfs
+ sshpass
+ python3-pandas
+ bc
+ smartmontools
+ wget
+)
+
+if [[ $OSID != ubuntu ]]; then
+ echo "Located apt-get package manager, but it was tested for Ubuntu only"
+fi
diff --git a/src/spdk/test/common/config/pkgdep/dnf b/src/spdk/test/common/config/pkgdep/dnf
new file mode 100644
index 000000000..b009f106e
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/dnf
@@ -0,0 +1,72 @@
+package_manager=dnf
+
+upgrade() {
+ sudo "$package_manager" upgrade -y
+}
+
+install() {
+ (($#)) || return 0
+
+ sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+ valgrind
+ jq
+ nvme-cli
+ ceph
+ gdb
+ fio
+ librbd-devel
+ kernel-devel
+ gflags-devel
+ libasan
+ libubsan
+ autoconf
+ automake
+ libtool
+ libmount-devel
+ iscsi-initiator-utils
+ isns-utils-devel
+ pmempool
+ perl-open
+ glib2-devel
+ pixman-devel
+ astyle-devel
+ elfutils
+ libabigail
+ elfutils-libelf-devel
+ flex
+ bison
+ targetcli
+ perl-Switch
+ librdmacm-utils
+ libibverbs-utils
+ gdisk
+ socat
+ sshfs
+ sshpass
+ python3-pandas
+ btrfs-progs
+ rpm-build
+ iptables
+ clang-analyzer
+ bc
+ kernel-modules-extra
+ systemd-devel
+ smartmontools
+ wget
+)
+
+pre_install() {
+ if [[ $INTSALL_TSOCKS == true ]]; then
+ # currently, tsocks package is retired in fedora 31, so don't exit in case
+ # installation failed
+ # FIXME: Review when fedora starts to successfully build this package again.
+ install tsocks || echo "Installation of the tsocks package failed, proxy may not be available"
+ fi
+}
+
+if [[ $OSID != fedora ]]; then
+ echo "Located dnf package manager, but it was tested for Fedora only"
+fi
diff --git a/src/spdk/test/common/config/pkgdep/git b/src/spdk/test/common/config/pkgdep/git
new file mode 100644
index 000000000..f46183ac8
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/git
@@ -0,0 +1,325 @@
+function install_spdk() {
+ mkdir -p "$GIT_REPOS/spdk_repo/output" || echo "Can not create spdk_repo/output directory."
+
+ if [[ -d $GIT_REPOS/spdk_repo/spdk ]]; then
+ echo "spdk source already present, not cloning"
+ else
+ git -C "$GIT_REPOS/spdk_repo" clone "${GIT_REPO_SPDK}"
+ fi
+ git -C "$GIT_REPOS/spdk_repo/spdk" config submodule.dpdk.url "${GIT_REPO_DPDK}"
+ git -C "$GIT_REPOS/spdk_repo/spdk" config submodule.intel-ipsec-mb.url "${GIT_REPO_INTEL_IPSEC_MB}"
+ git -C "$GIT_REPOS/spdk_repo/spdk" submodule update --init --recursive
+}
+
+function install_refspdk() {
+ local last_release
+ local output_dir
+ local config_params
+ local rootdir
+
+ # Create a reference SPDK build for ABI tests
+ git -C "$GIT_REPOS/spdk_repo/spdk" fetch --tags
+ last_release=$(git -C "$GIT_REPOS/spdk_repo/spdk" tag | sort --version-sort | grep -v rc | tail -n1)
+ output_dir="$GIT_REPOS/spdk_$(tr . _ < <(tr -d '[:alpha:]' <<< $last_release))"
+
+ if [[ ! -d $output_dir ]]; then
+ cp -r "$GIT_REPOS/spdk_repo/spdk" "$output_dir"
+ fi
+
+ git -C "$output_dir" checkout "$last_release"
+ git -C "$output_dir" submodule update --init
+
+ cat > $HOME/autorun-spdk.conf <<- EOF
+ SPDK_BUILD_SHARED_OBJECT=1
+ SPDK_TEST_AUTOBUILD=1
+ SPDK_TEST_UNITTEST=1
+ SPDK_TEST_BLOCKDEV=1
+ SPDK_TEST_PMDK=1
+ SPDK_TEST_ISAL=1
+ SPDK_TEST_REDUCE=1
+ SPDK_TEST_CRYPTO=1
+ SPDK_TEST_FTL=1
+ SPDK_TEST_OCF=1
+ SPDK_TEST_RAID5=1
+ SPDK_TEST_RBD=1
+ SPDK_RUN_ASAN=1
+ SPDK_RUN_UBSAN=1
+ EOF
+
+ mkdir -p $HOME/output
+
+ (
+ rootdir="$output_dir"
+ source $HOME/autorun-spdk.conf
+ source $output_dir/test/common/autotest_common.sh
+
+ # Prepare separate, fixed, cmdline for the FreeBSD, Issue #1397.
+ if [[ $OSID == freebsd ]]; then
+ config_params="--enable-debug --enable-werror"
+ config_params+=" --with-idxd --with-fio=/usr/src/fio"
+ config_params+=" --disable-unit-tests --without-isal"
+ MAKE=gmake
+ else
+ config_params="$(get_config_params)"
+ fi
+ $output_dir/configure $(echo $config_params | sed 's/--enable-coverage//g')
+ if [[ $OSID != freebsd ]]; then
+ $MAKE -C $output_dir $MAKEFLAGS include/spdk/config.h
+ CONFIG_OCF_PATH="$output_dir/ocf" $MAKE -C $output_dir/lib/env_ocf $MAKEFLAGS exportlib O=$output_dir/build/ocf.a
+ $output_dir/configure $config_params --with-ocf=$output_dir/build/ocf.a --with-shared
+ fi
+ $MAKE -C $output_dir $MAKEFLAGS
+ )
+}
+
+function install_qat() {
+
+ kernel_maj=$(uname -r | cut -d'.' -f1)
+ kernel_min=$(uname -r | cut -d'.' -f2)
+
+ if [[ -e /sys/module/qat_c62x ]]; then
+ sudo modprobe -r qat_c62x || :
+ fi
+ if [[ -d $GIT_REPOS/QAT ]]; then
+ sudo rm -rf "$GIT_REPOS/QAT"
+ fi
+
+ mkdir "$GIT_REPOS/QAT"
+
+ tar -C "$GIT_REPOS/QAT" -xzof - < <(wget -O- "$DRIVER_LOCATION_QAT")
+
+ #The driver version 1.7.l.4.3.0-00033 contains a reference to a deprecated function. Remove it so the build won't fail.
+ if [ $kernel_maj -le 4 ]; then
+ if [ $kernel_min -le 17 ]; then
+ sudo sed -i 's/rdtscll(timestamp);/timestamp = rdtsc_ordered();/g' \
+ "$GIT_REPOS/QAT/quickassist/utilities/osal/src/linux/kernel_space/OsalServices.c" || true
+ fi
+ fi
+
+ (cd "$GIT_REPOS/QAT" && sudo ./configure --enable-icp-sriov=host && sudo make install)
+
+ if sudo service qat_service start; then
+ echo "failed to start the qat service. Something may be wrong with your device or package."
+ fi
+}
+
+function install_rocksdb() {
+ # Rocksdb is installed for use with the blobfs tests.
+ if [ ! -d /usr/src/rocksdb ]; then
+ git clone "${GIT_REPO_ROCKSDB}" "$GIT_REPOS/rocksdb"
+ git -C "$GIT_REPOS/rocksdb" checkout spdk-v5.6.1
+ sudo mv "$GIT_REPOS/rocksdb" /usr/src/
+ else
+ sudo git -C /usr/src/rocksdb checkout spdk-v5.6.1
+ echo "rocksdb already in /usr/src. Not checking out again"
+ fi
+}
+
+function install_fio() {
+ # This version of fio is installed in /usr/src/fio to enable
+ # building the spdk fio plugin.
+ local fio_version="fio-3.19"
+
+ if [ ! -d /usr/src/fio ]; then
+ if [ ! -d fio ]; then
+ git clone "${GIT_REPO_FIO}" "$GIT_REPOS/fio"
+ sudo mv "$GIT_REPOS/fio" /usr/src/
+ else
+ sudo mv "$GIT_REPOS/fio" /usr/src/
+ fi
+ (
+ git -C /usr/src/fio checkout master \
+ && git -C /usr/src/fio pull \
+ && git -C /usr/src/fio checkout $fio_version \
+ && if [ $OSID == 'freebsd' ]; then
+ gmake -C /usr/src/fio -j${jobs} \
+ && sudo gmake -C /usr/src/fio install
+ else
+ make -C /usr/src/fio -j${jobs} \
+ && sudo make -C /usr/src/fio install
+ fi
+ )
+ else
+ echo "fio already in /usr/src/fio. Not installing"
+ fi
+}
+
+function install_flamegraph() {
+ # Flamegraph is used when printing out timing graphs for the tests.
+ if [ ! -d /usr/local/FlameGraph ]; then
+ git clone "${GIT_REPO_FLAMEGRAPH}" "$GIT_REPOS/FlameGraph"
+ mkdir -p /usr/local
+ sudo mv "$GIT_REPOS/FlameGraph" /usr/local/FlameGraph
+ else
+ echo "flamegraph already installed. Skipping"
+ fi
+}
+
+function install_qemu() {
+ # Two versions of QEMU are used in the tests.
+ # Stock QEMU is used for vhost. A special fork
+ # is used to test OCSSDs. Install both.
+
+ # Forked QEMU
+ SPDK_QEMU_BRANCH=spdk-5.0.0
+ mkdir -p "$GIT_REPOS/qemu"
+ if [[ ! -d $GIT_REPOS/qemu/$SPDK_QEMU_BRANCH ]]; then
+ git clone "${GIT_REPO_QEMU}" -b "$SPDK_QEMU_BRANCH" "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH"
+ else
+ echo "qemu already checked out. Skipping"
+ fi
+
+ declare -a opt_params=("--prefix=/usr/local/qemu/$SPDK_QEMU_BRANCH")
+ if ((gcc_version >= 9)); then
+ # GCC 9 fails to compile Qemu due to some old warnings which were not detected by older versions.
+ opt_params+=("--extra-cflags=-Wno-error=stringop-truncation -Wno-error=deprecated-declarations -Wno-error=incompatible-pointer-types -Wno-error=format-truncation")
+ opt_params+=("--disable-glusterfs")
+ fi
+
+ # Most tsocks proxies rely on a configuration file in /etc/tsocks.conf.
+ # If using tsocks, please make sure to complete this config before trying to build qemu.
+ if [[ $INSTALL_TSOCKS == true && $NO_TSOCKS != true ]]; then
+ if hash tsocks 2> /dev/null; then
+ opt_params+=("--with-git='tsocks git'")
+ fi
+ fi
+
+ sed -i s@git://git.qemu.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.gitmodules"
+ sed -i s@git://git.qemu.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.git/config"
+ sed -i s@git://git.qemu-project.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.gitmodules"
+ sed -i s@git://git.qemu-project.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.git/config"
+ # The qemu configure script places several output files in the CWD.
+ (cd "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" && ./configure "${opt_params[@]}" --target-list="x86_64-softmmu" --enable-kvm --enable-linux-aio --enable-numa)
+
+ make -C "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" -j${jobs}
+ sudo make -C "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" install
+}
+
+function install_nvmecli() {
+ SPDK_NVME_CLI_BRANCH=spdk-1.6
+ if [[ ! -d $GIT_REPOS/nvme-cli ]]; then
+ git clone "${GIT_REPO_SPDK_NVME_CLI}" -b "$SPDK_NVME_CLI_BRANCH" "$GIT_REPOS/nvme-cli"
+ else
+ echo "nvme-cli already checked out. Skipping"
+ fi
+ if [ ! -d "/usr/local/src/nvme-cli" ]; then
+ # Changes required for SPDK are already merged on top of
+ # nvme-cli, however not released yet.
+ # Support for SPDK should be released in nvme-cli >1.11.1
+ if [[ ! -d $GIT_REPOS/nvme-cli-cuse ]]; then
+ git clone "https://github.com/linux-nvme/nvme-cli.git" "$GIT_REPOS/nvme-cli-cuse"
+ fi
+ git -C "$GIT_REPOS/nvme-cli-cuse" checkout "e770466615096a6d41f038a28819b00bc3078e1d"
+ make -C "$GIT_REPOS/nvme-cli-cuse"
+ sudo mv "$GIT_REPOS/nvme-cli-cuse" /usr/local/src/nvme-cli
+ fi
+}
+
+function install_libiscsi() {
+ # We currently don't make any changes to the libiscsi repository for our tests, but it is possible that we will need
+ # to later. Cloning from git is just future proofing the machines.
+ if [[ ! -d $GIT_REPOS/libiscsi ]]; then
+ git clone "${GIT_REPO_LIBISCSI}" "$GIT_REPOS/libiscsi"
+ else
+ echo "libiscsi already checked out. Skipping"
+ fi
+ (cd "$GIT_REPOS/libiscsi" && ./autogen.sh && ./configure --prefix=/usr/local/libiscsi)
+ make -C "$GIT_REPOS/libiscsi" -j${jobs}
+ sudo make -C "$GIT_REPOS/libiscsi" install
+}
+
+function install_git() {
+ install zlib-devel curl-devel
+ tar -C "$GIT_REPOS" -xzof <(wget -qO- "$GIT_REPO_GIT")
+ (cd "$GIT_REPOS/git-$GIT_VERSION" \
+ && make configure \
+ && ./configure --prefix=/usr/local/git \
+ && sudo make -j${jobs} install)
+ sudo sh -c "echo 'export PATH=/usr/local/git/bin:$PATH' >> /etc/bashrc"
+ export "PATH=/usr/local/git/bin:$PATH"
+}
+
+function install_extra_pkgs() {
+ if [[ $INSTALL_QAT == true ]]; then
+ install libudev-devel || install libudev-dev
+ fi
+
+ if [[ $INSTALL_QEMU == true ]]; then
+ install qemu-system-x86 qemu-img \
+ || install qemu-system-x86 qemu-utils \
+ || install qemu
+ fi
+}
+
+GIT_VERSION=2.25.1
+: ${GIT_REPO_SPDK=https://github.com/spdk/spdk.git}
+export GIT_REPO_SPDK
+: ${GIT_REPO_DPDK=https://github.com/spdk/dpdk.git}
+export GIT_REPO_DPDK
+: ${GIT_REPO_ROCKSDB=https://review.spdk.io/spdk/rocksdb}
+export GIT_REPO_ROCKSDB
+: ${GIT_REPO_FIO=http://git.kernel.dk/fio.git}
+export GIT_REPO_FIO
+: ${GIT_REPO_FLAMEGRAPH=https://github.com/brendangregg/FlameGraph.git}
+export GIT_REPO_FLAMEGRAPH
+: ${GIT_REPO_QEMU=https://github.com/spdk/qemu}
+export GIT_REPO_QEMU
+: ${GIT_REPO_LIBISCSI=https://github.com/sahlberg/libiscsi}
+export GIT_REPO_LIBISCSI
+: ${GIT_REPO_SPDK_NVME_CLI=https://github.com/spdk/nvme-cli}
+export GIT_REPO_SPDK_NVME_CLI
+: ${GIT_REPO_INTEL_IPSEC_MB=https://github.com/spdk/intel-ipsec-mb.git}
+export GIT_REPO_INTEL_IPSEC_MB
+: ${DRIVER_LOCATION_QAT=https://01.org/sites/default/files/downloads//qat1.7.l.4.9.0-00008.tar.gz}
+export DRIVER_LOCATION_QAT
+: ${GIT_REPO_GIT=https://github.com/git/git/archive/v${GIT_VERSION}.tar.gz}
+export GIT_REPO_GIT
+GIT_REPOS=${GIT_REPOS:-$HOME}
+
+gcc_version=$(gcc -dumpversion) gcc_version=${gcc_version%%.*}
+if [[ $ID == centos ]] && (( VERSION_ID == 7 )); then
+ # install proper version of the git first
+ install_git
+fi
+
+IFS="," read -ra conf_env <<< "$CONF"
+for conf in "${conf_env[@]}"; do
+ export "INSTALL_${conf^^}=true"
+done
+sources=(install_refspdk)
+
+if [[ $OS == FreeBSD ]]; then
+ jobs=$(($(sysctl -n hw.ncpu) * 2))
+else
+ jobs=$(($(nproc) * 2))
+ sources+=(
+ install_libiscsi
+ install_nvmecli
+ install_qat
+ install_rocksdb
+ install_flamegraph
+ install_qemu
+ )
+fi
+sources+=(install_fio)
+
+sudo mkdir -p /usr/{,local}/src
+sudo mkdir -p "$GIT_REPOS"
+
+install_extra_pkgs
+
+if [[ $INSTALL_REFSPDK == true ]]; then
+ # Serialize builds as refspdk depends on spdk
+ install_spdk
+ install_refspdk
+else
+ sources+=(install_spdk)
+fi
+
+for source in "${sources[@]}"; do
+ source_conf=${source^^}
+ if [[ ${!source_conf} == true ]]; then
+ "$source" &
+ fi
+done
+wait
diff --git a/src/spdk/test/common/config/pkgdep/pacman b/src/spdk/test/common/config/pkgdep/pacman
new file mode 100644
index 000000000..43d3db2f5
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/pacman
@@ -0,0 +1,62 @@
+package_manager=pacman
+
+upgrade() {
+ sudo "$package_manager" -Syu --noconfirm --needed
+}
+
+install() {
+ (($#)) || return 0
+
+ sudo "$package_manager" -Sy --noconfirm --needed "$@"
+}
+
+pre_install() {
+ if [[ $INTSALL_TSOCKS == true ]]; then
+ install tsocks
+ fi
+}
+
+packages=(
+ valgrind
+ jq
+ nvme-cli
+ ceph
+ gdb
+ fio
+ linux-headers
+ gflags
+ autoconf
+ automake
+ libtool
+ libutil-linux
+ libiscsi
+ open-isns
+ glib2
+ pixman
+ flex
+ bison
+ elfutils
+ libelf
+ astyle
+ gptfdisk
+ socat
+ sshfs
+ sshpass
+ python-pandas
+ btrfs-progs
+ iptables
+ clang
+ bc
+ perl-switch
+ open-iscsi
+ smartmontools
+ parted
+ wget
+)
+
+# TODO:
+# These are either missing or require some other installation method
+# than pacman:
+# librbd-devel
+# perl-open
+# targetcli
diff --git a/src/spdk/test/common/config/pkgdep/pkg b/src/spdk/test/common/config/pkgdep/pkg
new file mode 100644
index 000000000..3f3f41725
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/pkg
@@ -0,0 +1,27 @@
+package_manager=pkg
+
+upgrade() {
+ sudo "$package_manager" upgrade -y
+}
+
+install() {
+ (($#)) || return 0
+
+ sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+ pciutils
+ jq
+ gdb
+ fio
+ p5-extutils-pkgconfig
+ libtool
+ flex
+ bison
+ gdisk
+ socat
+ sshpass
+ py37-pandas
+ wget
+)
diff --git a/src/spdk/test/common/config/pkgdep/swupd b/src/spdk/test/common/config/pkgdep/swupd
new file mode 100644
index 000000000..c1d2a8a6b
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/swupd
@@ -0,0 +1,21 @@
+package_manager=swupd
+
+upgrade() {
+ sudo "$package_manager" update -y
+}
+
+install() {
+ (($#)) || return 0
+
+ sudo "$package_manager" bundle-add -y "$@"
+}
+
+packages=(
+ jq
+)
+
+pre_install() {
+ if [[ $INTSALL_TSOCKS == true ]]; then
+ install tsocks || echo "Installation of the tsocks package failed, proxy may not be available"
+ fi
+}
diff --git a/src/spdk/test/common/config/pkgdep/yum b/src/spdk/test/common/config/pkgdep/yum
new file mode 100644
index 000000000..32e89bc15
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/yum
@@ -0,0 +1,67 @@
+package_manager=yum
+
+upgrade() {
+ sudo "$package_manager" upgrade -y
+}
+
+install() {
+ (($#)) || return 0
+
+ sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+ pciutils
+ valgrind
+ jq
+ nvme-cli
+ gdb
+ fio
+ librbd-devel
+ kernel-devel
+ gflags-devel
+ libasan
+ libubsan
+ autoconf
+ automake
+ libtool
+ libmount-devel
+ iscsi-initiator-utils
+ isns-utils-devel pmempool
+ perl-open
+ glib2-devel
+ pixman-devel
+ astyle-devel
+ elfutils
+ elfutils-libelf-devel
+ flex
+ bison
+ targetcli
+ perl-Switch
+ librdmacm-utils
+ libibverbs-utils
+ gdisk
+ socat
+ sshfs
+ sshpass
+ python3-pandas
+ rpm-build
+ iptables
+ clang-analyzer
+ bc
+ kernel-modules-extra
+ systemd-devel
+ python3
+ wget
+)
+
+pre_install() {
+ if [[ $ID == centos ]] && (( VERSION_ID == 8 )); then
+ "$package_manager" update -y --refresh
+ fi
+
+ install nbd || {
+ wget -O nbd.rpm https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/n/nbd-3.14-2.el7.x86_64.rpm
+ install nbd.rpm
+ }
+}
diff --git a/src/spdk/test/common/config/vm_setup.conf b/src/spdk/test/common/config/vm_setup.conf
new file mode 100644
index 000000000..a8e58d82a
--- /dev/null
+++ b/src/spdk/test/common/config/vm_setup.conf
@@ -0,0 +1,12 @@
+# This configuration file is provided for reference purposes.
+GIT_REPO_SPDK=https://review.spdk.io/gerrit/spdk/spdk
+GIT_REPO_DPDK=https://github.com/spdk/dpdk.git
+GIT_REPO_OPEN_ISCSI=https://github.com/open-iscsi/open-iscsi
+GIT_REPO_ROCKSDB=https://review.gerrithub.io/spdk/rocksdb
+GIT_REPO_FIO=http://git.kernel.dk/fio.git
+GIT_REPO_FLAMEGRAPH=https://github.com/brendangregg/FlameGraph.git
+GIT_REPO_QEMU=https://github.com/spdk/qemu
+GIT_REPO_VPP=https://gerrit.fd.io/r/vpp
+GIT_REPO_LIBISCSI=https://github.com/sahlberg/libiscsi
+GIT_REPO_SPDK_NVME_CLI=https://github.com/spdk/nvme-cli
+DRIVER_LOCATION_QAT=https://01.org/sites/default/files/downloads/intelr-quickassist-technology/qat1.7.l.4.3.0-00033.tar.gz
diff --git a/src/spdk/test/common/config/vm_setup.sh b/src/spdk/test/common/config/vm_setup.sh
new file mode 100755
index 000000000..b2b3a8fc1
--- /dev/null
+++ b/src/spdk/test/common/config/vm_setup.sh
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+
+# Virtual Machine environment requirements:
+# 8 GiB of RAM (for DPDK)
+# enable intel_kvm on your host machine
+
+# The purpose of this script is to provide a simple procedure for spinning up a new
+# virtual test environment capable of running our whole test suite. This script, when
+# applied to a fresh install of fedora 26 or ubuntu 16,18 server will install all of the
+# necessary dependencies to run almost the complete test suite. The main exception being VHost.
+# Vhost requires the configuration of a second virtual machine. instructions for how to configure
+# that vm are included in the file TEST_ENV_SETUP_README inside this repository
+
+# it is important to enable nesting for vms in kernel command line of your machine for the vhost tests.
+# in /etc/default/grub
+# append the following to the GRUB_CMDLINE_LINUX line
+# intel_iommu=on kvm-intel.nested=1
+
+# We have made a lot of progress with removing hardcoded paths from the tests,
+
+sudo() {
+ "$(type -P sudo)" -E "$@"
+}
+
+set -e
+
+VM_SETUP_PATH=$(readlink -f ${BASH_SOURCE%/*})
+
+UPGRADE=false
+INSTALL=false
+CONF="rocksdb,fio,flamegraph,tsocks,qemu,libiscsi,nvmecli,qat,spdk,refspdk"
+
+if [[ -e /etc/os-release ]]; then
+ source /etc/os-release
+fi
+
+if [ $(uname -s) == "FreeBSD" ]; then
+ OSID="freebsd"
+ OSVERSION=$(freebsd-version | cut -d. -f1)
+else
+ OSID=$(source /etc/os-release && echo $ID)
+ OSVERSION=$(source /etc/os-release && echo $VERSION_ID)
+fi
+
+function usage() {
+ echo "This script is intended to automate the environment setup for a linux virtual machine."
+ echo "Please run this script as your regular user. The script will make calls to sudo as needed."
+ echo ""
+ echo "./vm_setup.sh"
+ echo " -h --help"
+ echo " -u --upgrade Run $package_manager upgrade"
+ echo " -i --install-deps Install $package_manager based dependencies"
+ echo " -t --test-conf List of test configurations to enable (${CONF})"
+ echo " -c --conf-path Path to configuration file"
+ echo " -d --dir-git Path to where git sources should be saved"
+ echo " -s --disable-tsocks Disable use of tsocks"
+ exit 0
+}
+
+vmsetupdir=$(readlink -f "$(dirname "$0")")
+rootdir=$(readlink -f "$vmsetupdir/../../../")
+
+managers=("$vmsetupdir/pkgdep/"*)
+# Get package manager #
+if hash dnf &> /dev/null; then
+ source "$vmsetupdir/pkgdep/dnf"
+elif hash yum &> /dev/null; then
+ source "$vmsetupdir/pkgdep/yum"
+elif hash apt-get &> /dev/null; then
+ source "$vmsetupdir/pkgdep/apt-get"
+elif hash pacman &> /dev/null; then
+ source "$vmsetupdir/pkgdep/pacman"
+elif hash pkg &> /dev/null; then
+ source "$vmsetupdir/pkgdep/pkg"
+elif hash swupd &> /dev/null; then
+ source "$vmsetupdir/pkgdep/swupd"
+else
+ package_manager="undefined"
+fi
+
+# Parse input arguments #
+while getopts 'd:siuht:c:-:' optchar; do
+ case "$optchar" in
+ -)
+ case "$OPTARG" in
+ help) usage ;;
+ upgrade) UPGRADE=true ;;
+ install-deps) INSTALL=true ;;
+ test-conf=*) CONF="${OPTARG#*=}" ;;
+ conf-path=*) CONF_PATH="${OPTARG#*=}" ;;
+ dir-git=*) GIT_REPOS="${OPTARG#*=}" ;;
+ disable-tsocks) NO_TSOCKS=true ;;
+ *)
+ echo "Invalid argument '$OPTARG'"
+ usage
+ ;;
+ esac
+ ;;
+ h) usage ;;
+ u) UPGRADE=true ;;
+ i) INSTALL=true ;;
+ t) CONF="$OPTARG" ;;
+ c) CONF_PATH="$OPTARG" ;;
+ d) GIT_REPOS="$OPTARG" ;;
+ s) NO_TSOCKS=true ;;
+ *)
+ echo "Invalid argument '$OPTARG'"
+ usage
+ ;;
+ esac
+done
+
+if [[ "$package_manager" == "undefined" ]]; then
+ echo "Supported package manager not found. Script supports:"
+ printf ' * %s\n' "${managers[@]##*/}"
+ exit 1
+fi
+
+if [ -n "$CONF_PATH" ]; then
+ if [ ! -f "$CONF_PATH" ]; then
+ echo Configuration file does not exist: "$CONF_PATH"
+ exit 1
+ else
+ source "$CONF_PATH"
+ fi
+fi
+
+if $UPGRADE; then
+ upgrade
+fi
+
+if $INSTALL; then
+ sudo "$rootdir/scripts/pkgdep.sh" --all
+ pre_install
+ install "${packages[@]}"
+fi
+
+source "$vmsetupdir/pkgdep/git"
+
+# create autorun-spdk.conf in home folder. This is sourced by the autotest_common.sh file.
+# By setting any one of the values below to 0, you can skip that specific test. If you are
+# using your autotest platform to do sanity checks before uploading to the build pool, it is
+# probably best to only run the tests that you believe your changes have modified along with
+# Scanbuild and check format. This is because running the whole suite of tests in series can
+# take ~40 minutes to complete.
+if [ ! -e ~/autorun-spdk.conf ]; then
+ cat > ~/autorun-spdk.conf << EOF
+# assign a value of 1 to all of the pertinent tests
+SPDK_RUN_VALGRIND=1
+SPDK_TEST_CRYPTO=1
+SPDK_RUN_FUNCTIONAL_TEST=1
+SPDK_TEST_AUTOBUILD=1
+SPDK_TEST_UNITTEST=1
+SPDK_TEST_ISCSI=1
+SPDK_TEST_ISCSI_INITIATOR=1
+SPDK_TEST_NVME=1
+SPDK_TEST_NVME_CLI=1
+SPDK_TEST_NVMF=1
+SPDK_TEST_RBD=1
+SPDK_TEST_BLOCKDEV=1
+SPDK_TEST_BLOBFS=1
+SPDK_TEST_PMDK=1
+SPDK_TEST_LVOL=1
+SPDK_TEST_JSON=1
+SPDK_RUN_ASAN=1
+SPDK_RUN_UBSAN=1
+# doesn't work on vm
+SPDK_TEST_IOAT=0
+# requires some extra configuration. see TEST_ENV_SETUP_README
+SPDK_TEST_VHOST=0
+SPDK_TEST_VHOST_INIT=0
+# Not configured here
+SPDK_RUN_INSTALLED_DPDK=0
+
+EOF
+fi