Adding upstream version 16.2.11+ds.upstream/16.2.11+dsupstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

10 files changed, 966 insertions, 0 deletions

diff --git a/src/spdk/test/common/config/README.md b/src/spdk/test/common/config/README.md
new file mode 100644
index 000000000..26a587709
--- /dev/null
+++ b/src/spdk/test/common/config/README.md
@@ -0,0 +1,104 @@
+# Virtual Test Configuration
+
+This readme and the associated bash script, vm_setup.sh, are intended to assist developers in quickly
+preparing a virtual test environment on which to run the SPDK validation tests rooted at autorun.sh.
+This file contains basic information about SPDK environment requirements, an introduction to the
+autorun-spdk.conf files used to moderate which tests are run by autorun.sh, and step-by-step instructions
+for spinning up a VM capable of running the SPDK test suite.
+There is no need for external hardware to run these tests. The linux kernel comes with the drivers necessary
+to emulate an RDMA enabled NIC. NVMe controllers can also be virtualized in emulators such as QEMU.
+
+## VM Envronment Requirements (Host)
+
+- 8 GiB of RAM (for DPDK)
+- Enable intel_kvm on the host machine from the bios.
+- Enable nesting for VMs in kernel command line (for vhost tests).
+  - In `/etc/default/grub` append the following to the GRUB_CMDLINE_LINUX line: intel_iommu=on kvm-intel.nested=1.
+
+## VM Specs
+
+When creating the user during the fedora installation, it is best to use the name sys_sgsw. Efforts are being made
+to remove all references to this user, or files specific to this user from the codebase, but there are still some
+trailing references to it.
+
+## Autorun-spdk.conf
+
+Every machine that runs the autotest scripts should include a file titled autorun-spdk.conf in the home directory
+of the user that will run them. This file consists of several lines of the form 'variable_name=0/1'. autorun.sh sources
+this file each time it is run, and determines which tests to attempt based on which variables are defined in the
+configuration file. For a full list of the variable declarations available for autorun-spdk.conf, please see
+`test/common/autotest_common.sh` starting at line 13.
+
+## Steps for Configuring the VM
+
+1. Download a fresh Fedora 26 image.
+2. Perform the installation of Fedora 26 server.
+3. Create an admin user sys_sgsw (enabling passwordless sudo for this account will make life easier during the tests).
+4. Run the vm_setup.sh script which will install all proper dependencies.
+5. Modify the autorun-spdk.conf file in the home directory.
+6. Reboot the VM.
+7. Run autorun.sh for SPDK. Any output files will be placed in `~/spdk_repo/output/`.
+
+## Additional Steps for Preparing the Vhost Tests
+
+The Vhost tests also require the creation of a second virtual machine nested inside of the test VM.
+Please follow the directions below to complete that installation. Note that host refers to the Fedora VM
+created above and guest or VM refer to the Ubuntu VM created in this section.
+
+1. Follow instructions from spdk/scripts/vagrant/README.md
+	- install all needed packages mentioned in "Mac OSX Setup" or "Windows 10 Setup" sections
+	- follow steps from "Configure Vagrant" section
+
+2. Use Vagrant scripts located in spdk/scripts/vagrant to automatically generate
+	VM image to use in SPDK vhost tests.
+	Example command:
+	~~~{.sh}
+	spdk/scripts/vagrant/create_vhost_vm.sh --move-to-def-dirs ubuntu16
+	~~~
+	This command will:
+		- Download a Ubuntu 16.04 image file
+		- upgrade the system and install needed dependencies (fio, sg3-utils, bc)
+		- add entry to VM's ~/.ssh/autorized_keys
+		- add appropriate options to GRUB command line and update grub
+		- convert the image to .qcow2 format
+		- move .qcow2 file and ssh keys to default locations used by vhost test scripts
+
+Alternatively it is possible to create the VM image manually using following steps:
+
+1. Create an image file for the VM. It does not have to be large, about 3.5G should suffice.
+2. Create an ssh keypair for host-guest communications (performed on the host):
+    - Generate an ssh keypair with the name spdk_vhost_id_rsa and save it in `/root/.ssh`.
+    - Make sure that only root has read access to the private key.
+3. Install the OS in the VM image (performed on guest):
+    - Use the latest Ubuntu server (Currently 16.04 LTS).
+    - When partitioning the disk, make one partion that consumes the whole disk mounted at /. Do not encrypt the disk or enable LVM.
+    - Choose the OpenSSH server packages during install.
+4. Post installation configuration (performed on guest):
+    - Run the following commands to enable all necessary dependencies:
+      ~~~{.sh}
+      sudo apt update
+      sudo apt upgrade
+      sudo apt install fio sg3-utils bc
+      ~~~
+    - Enable the root user: "sudo passwd root -> root".
+    - Enable root login over ssh: vim `/etc/ssh/sshd_config` -> PermitRootLogin=yes.
+    - Disable DNS for ssh: `/etc/ssh/sshd_config` -> UseDNS=no.
+    - Add the spdk_vhost key to root's known hosts: `/root/.ssh/authorized_keys` -> add spdk_vhost_id_rsa.pub key to authorized keys.
+    Remember to save the private key in `~/.ssh/spdk_vhost_id_rsa` on the host.
+    - Change the grub boot options for the guest as follows:
+      - Add "console=ttyS0 earlyprintk=ttyS0" to the boot options in `/etc/default/grub` (for serial output redirect).
+      - Add "scsi_mod.use_blk_mq=1" to boot options in `/etc/default/grub`.
+      ~~~{.sh}
+      sudo update-grub
+      ~~~
+    - Reboot the VM.
+    - Remove any unnecessary packages (this is to make booting the VM faster):
+      ~~~{.sh}
+      apt purge snapd
+      apt purge Ubuntu-core-launcher
+      apt purge squashfs-tools
+      apt purge unattended-upgrades
+      ~~~
+5. Copy the fio binary from the guest location `/usr/bin/fio` to the host location `/home/sys_sgsw/fio_ubuntu`.
+6. Place the guest VM in the host at the following location: `/home/sys_sgsw/vhost_vm_image.qcow2`.
+7. On the host, edit the `~/autorun-spdk.conf` file to include the following line: SPDK_TEST_VHOST=1.
diff --git a/src/spdk/test/common/config/pkgdep/apt-get b/src/spdk/test/common/config/pkgdep/apt-get
new file mode 100644
index 000000000..a1630620d
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/apt-get
@@ -0,0 +1,100 @@
+package_manager=apt-get
+
+update() {
+	sudo "$package_manager" update
+}
+
+install() {
+	(( $# )) || return 0
+
+	sudo "$package_manager" install -y "$@"
+}
+
+upgrade() {
+	sudo "$package_manager" update
+	sudo "$package_manager" upgrade -y
+}
+
+
+pre_install() {
+	echo "Package perl-open is not available at Ubuntu repositories" >&2
+
+	update
+
+	if [[ $INSTALL_TSOCKS == true ]]; then
+		install tsocks
+	fi
+
+	# asan an ubsan have to be installed together to not mix up gcc versions
+	if install libasan5; then
+		install libubsan1
+	else
+		echo "Latest libasan5 is not available" >&2
+		echo "  installing libasan2 and corresponding libubsan0" >&2
+		install libasan2
+		install libubsan0
+	fi
+	if ! install rdma-core; then
+		echo "Package rdma-core is avaliable at Ubuntu 18 [universe] repositorium" >&2
+		install rdmacm-utils
+		install ibverbs-utils
+	else
+		LIBRXE_INSTALL=false
+	fi
+	if ! install libpmempool1; then
+		echo "Package libpmempool1 is available at Ubuntu 18 [universe] repositorium" >&2
+	fi
+	if ! install clang-tools; then
+		echo "Package clang-tools is available at Ubuntu 18 [universe] repositorium" >&2
+	fi
+	if ! install --no-install-suggests --no-install-recommends open-isns-utils; then
+		echo "Package open-isns-utils is available at Ubuntu 18 [universe] repositorium" >&2
+	fi
+
+	# Package name for Ubuntu 18 is targetcli-fb but for Ubuntu 16 it's targetcli
+	if ! install targetcli-fb; then
+		install targetcli
+	fi
+
+	# On Ubuntu 20.04 (focal) btrfs-tools are available under different name - btrfs-progs
+	if ! install btrfs-tools; then
+		install btrfs-progs
+	fi
+}
+
+packages=(
+	valgrind
+	jq
+	nvme-cli
+	ceph
+	gdb
+	fio
+	librbd-dev
+	linux-headers-generic
+	libgflags-dev
+	autoconf
+	automake
+	libtool
+	libmount-dev
+	open-iscsi
+	libglib2.0-dev
+	libpixman-1-dev
+	astyle
+	elfutils
+	libelf-dev
+	flex
+	bison
+	libswitch-perl
+	gdisk
+	socat
+	sshfs
+	sshpass
+	python3-pandas
+	bc
+	smartmontools
+	wget
+)
+
+if [[ $OSID != ubuntu ]]; then
+	echo "Located apt-get package manager, but it was tested for Ubuntu only"
+fi
diff --git a/src/spdk/test/common/config/pkgdep/dnf b/src/spdk/test/common/config/pkgdep/dnf
new file mode 100644
index 000000000..b009f106e
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/dnf
@@ -0,0 +1,72 @@
+package_manager=dnf
+
+upgrade() {
+	sudo "$package_manager" upgrade -y
+}
+
+install() {
+	(($#)) || return 0
+
+	sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+	valgrind
+	jq
+	nvme-cli
+	ceph
+	gdb
+	fio
+	librbd-devel
+	kernel-devel
+	gflags-devel
+	libasan
+	libubsan
+	autoconf
+	automake
+	libtool
+	libmount-devel
+	iscsi-initiator-utils
+	isns-utils-devel
+	pmempool
+	perl-open
+	glib2-devel
+	pixman-devel
+	astyle-devel
+	elfutils
+	libabigail
+	elfutils-libelf-devel
+	flex
+	bison
+	targetcli
+	perl-Switch
+	librdmacm-utils
+	libibverbs-utils
+	gdisk
+	socat
+	sshfs
+	sshpass
+	python3-pandas
+	btrfs-progs
+	rpm-build
+	iptables
+	clang-analyzer
+	bc
+	kernel-modules-extra
+	systemd-devel
+	smartmontools
+	wget
+)
+
+pre_install() {
+	if [[ $INTSALL_TSOCKS == true ]]; then
+		# currently, tsocks package is retired in fedora 31, so don't exit in case
+		# installation failed
+		# FIXME: Review when fedora starts to successfully build this package again.
+		install tsocks || echo "Installation of the tsocks package failed, proxy may not be available"
+	fi
+}
+
+if [[ $OSID != fedora ]]; then
+	echo "Located dnf package manager, but it was tested for Fedora only"
+fi
diff --git a/src/spdk/test/common/config/pkgdep/git b/src/spdk/test/common/config/pkgdep/git
new file mode 100644
index 000000000..f46183ac8
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/git
@@ -0,0 +1,325 @@
+function install_spdk() {
+	mkdir -p "$GIT_REPOS/spdk_repo/output" || echo "Can not create spdk_repo/output directory."
+
+	if [[ -d $GIT_REPOS/spdk_repo/spdk ]]; then
+		echo "spdk source already present, not cloning"
+	else
+		git -C "$GIT_REPOS/spdk_repo" clone "${GIT_REPO_SPDK}"
+	fi
+	git -C "$GIT_REPOS/spdk_repo/spdk" config submodule.dpdk.url "${GIT_REPO_DPDK}"
+	git -C "$GIT_REPOS/spdk_repo/spdk" config submodule.intel-ipsec-mb.url "${GIT_REPO_INTEL_IPSEC_MB}"
+	git -C "$GIT_REPOS/spdk_repo/spdk" submodule update --init --recursive
+}
+
+function install_refspdk() {
+	local last_release
+	local output_dir
+	local config_params
+	local rootdir
+
+	# Create a reference SPDK build for ABI tests
+	git -C "$GIT_REPOS/spdk_repo/spdk" fetch --tags
+	last_release=$(git -C "$GIT_REPOS/spdk_repo/spdk" tag | sort --version-sort | grep -v rc | tail -n1)
+	output_dir="$GIT_REPOS/spdk_$(tr . _ < <(tr -d '[:alpha:]' <<< $last_release))"
+
+	if [[ ! -d $output_dir ]]; then
+		cp -r "$GIT_REPOS/spdk_repo/spdk" "$output_dir"
+	fi
+
+	git -C "$output_dir" checkout "$last_release"
+	git -C "$output_dir" submodule update --init
+
+	cat > $HOME/autorun-spdk.conf <<- EOF
+		SPDK_BUILD_SHARED_OBJECT=1
+		SPDK_TEST_AUTOBUILD=1
+		SPDK_TEST_UNITTEST=1
+		SPDK_TEST_BLOCKDEV=1
+		SPDK_TEST_PMDK=1
+		SPDK_TEST_ISAL=1
+		SPDK_TEST_REDUCE=1
+		SPDK_TEST_CRYPTO=1
+		SPDK_TEST_FTL=1
+		SPDK_TEST_OCF=1
+		SPDK_TEST_RAID5=1
+		SPDK_TEST_RBD=1
+		SPDK_RUN_ASAN=1
+		SPDK_RUN_UBSAN=1
+	EOF
+
+	mkdir -p $HOME/output
+
+	(
+		rootdir="$output_dir"
+		source $HOME/autorun-spdk.conf
+		source $output_dir/test/common/autotest_common.sh
+
+		# Prepare separate, fixed, cmdline for the FreeBSD, Issue #1397.
+		if [[ $OSID == freebsd ]]; then
+			config_params="--enable-debug --enable-werror"
+			config_params+=" --with-idxd --with-fio=/usr/src/fio"
+			config_params+=" --disable-unit-tests --without-isal"
+			MAKE=gmake
+		else
+			config_params="$(get_config_params)"
+		fi
+		$output_dir/configure $(echo $config_params | sed 's/--enable-coverage//g')
+		if [[ $OSID != freebsd ]]; then
+			$MAKE -C $output_dir $MAKEFLAGS include/spdk/config.h
+			CONFIG_OCF_PATH="$output_dir/ocf" $MAKE -C $output_dir/lib/env_ocf $MAKEFLAGS exportlib O=$output_dir/build/ocf.a
+			$output_dir/configure $config_params --with-ocf=$output_dir/build/ocf.a --with-shared
+		fi
+		$MAKE -C $output_dir $MAKEFLAGS
+	)
+}
+
+function install_qat() {
+
+	kernel_maj=$(uname -r | cut -d'.' -f1)
+	kernel_min=$(uname -r | cut -d'.' -f2)
+
+	if [[ -e /sys/module/qat_c62x ]]; then
+		sudo modprobe -r qat_c62x || :
+	fi
+	if [[ -d $GIT_REPOS/QAT ]]; then
+		sudo rm -rf "$GIT_REPOS/QAT"
+	fi
+
+	mkdir "$GIT_REPOS/QAT"
+
+	tar -C "$GIT_REPOS/QAT" -xzof - < <(wget -O- "$DRIVER_LOCATION_QAT")
+
+	#The driver version 1.7.l.4.3.0-00033 contains a reference to a deprecated function. Remove it so the build won't fail.
+	if [ $kernel_maj -le 4 ]; then
+		if [ $kernel_min -le 17 ]; then
+			sudo sed -i 's/rdtscll(timestamp);/timestamp = rdtsc_ordered();/g' \
+				"$GIT_REPOS/QAT/quickassist/utilities/osal/src/linux/kernel_space/OsalServices.c" || true
+		fi
+	fi
+
+	(cd "$GIT_REPOS/QAT" && sudo ./configure --enable-icp-sriov=host && sudo make install)
+
+	if sudo service qat_service start; then
+		echo "failed to start the qat service. Something may be wrong with your device or package."
+	fi
+}
+
+function install_rocksdb() {
+	# Rocksdb is installed for use with the blobfs tests.
+	if [ ! -d /usr/src/rocksdb ]; then
+		git clone "${GIT_REPO_ROCKSDB}" "$GIT_REPOS/rocksdb"
+		git -C "$GIT_REPOS/rocksdb" checkout spdk-v5.6.1
+		sudo mv "$GIT_REPOS/rocksdb" /usr/src/
+	else
+		sudo git -C /usr/src/rocksdb checkout spdk-v5.6.1
+		echo "rocksdb already in /usr/src. Not checking out again"
+	fi
+}
+
+function install_fio() {
+	# This version of fio is installed in /usr/src/fio to enable
+	# building the spdk fio plugin.
+	local fio_version="fio-3.19"
+
+	if [ ! -d /usr/src/fio ]; then
+		if [ ! -d fio ]; then
+			git clone "${GIT_REPO_FIO}" "$GIT_REPOS/fio"
+			sudo mv "$GIT_REPOS/fio" /usr/src/
+		else
+			sudo mv "$GIT_REPOS/fio" /usr/src/
+		fi
+		(
+			git -C /usr/src/fio checkout master \
+				&& git -C /usr/src/fio pull \
+				&& git -C /usr/src/fio checkout $fio_version \
+				&& if [ $OSID == 'freebsd' ]; then
+					gmake -C /usr/src/fio -j${jobs} \
+						&& sudo gmake -C /usr/src/fio install
+				else
+					make -C /usr/src/fio -j${jobs} \
+						&& sudo make -C /usr/src/fio install
+				fi
+		)
+	else
+		echo "fio already in /usr/src/fio. Not installing"
+	fi
+}
+
+function install_flamegraph() {
+	# Flamegraph is used when printing out timing graphs for the tests.
+	if [ ! -d /usr/local/FlameGraph ]; then
+		git clone "${GIT_REPO_FLAMEGRAPH}" "$GIT_REPOS/FlameGraph"
+		mkdir -p /usr/local
+		sudo mv "$GIT_REPOS/FlameGraph" /usr/local/FlameGraph
+	else
+		echo "flamegraph already installed. Skipping"
+	fi
+}
+
+function install_qemu() {
+	# Two versions of QEMU are used in the tests.
+	# Stock QEMU is used for vhost. A special fork
+	# is used to test OCSSDs. Install both.
+
+	# Forked QEMU
+	SPDK_QEMU_BRANCH=spdk-5.0.0
+	mkdir -p "$GIT_REPOS/qemu"
+	if [[ ! -d $GIT_REPOS/qemu/$SPDK_QEMU_BRANCH ]]; then
+		git clone "${GIT_REPO_QEMU}" -b "$SPDK_QEMU_BRANCH" "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH"
+	else
+		echo "qemu already checked out. Skipping"
+	fi
+
+	declare -a opt_params=("--prefix=/usr/local/qemu/$SPDK_QEMU_BRANCH")
+	if ((gcc_version >= 9)); then
+		# GCC 9 fails to compile Qemu due to some old warnings which were not detected by older versions.
+		opt_params+=("--extra-cflags=-Wno-error=stringop-truncation -Wno-error=deprecated-declarations -Wno-error=incompatible-pointer-types -Wno-error=format-truncation")
+		opt_params+=("--disable-glusterfs")
+	fi
+
+	# Most tsocks proxies rely on a configuration file in /etc/tsocks.conf.
+	# If using tsocks, please make sure to complete this config before trying to build qemu.
+	if [[ $INSTALL_TSOCKS == true && $NO_TSOCKS != true ]]; then
+		if hash tsocks 2> /dev/null; then
+			opt_params+=("--with-git='tsocks git'")
+		fi
+	fi
+
+	sed -i s@git://git.qemu.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.gitmodules"
+	sed -i s@git://git.qemu.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.git/config"
+	sed -i s@git://git.qemu-project.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.gitmodules"
+	sed -i s@git://git.qemu-project.org/@https://github.com/qemu/@g "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH/.git/config"
+	# The qemu configure script places several output files in the CWD.
+	(cd "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" && ./configure "${opt_params[@]}" --target-list="x86_64-softmmu" --enable-kvm --enable-linux-aio --enable-numa)
+
+	make -C "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" -j${jobs}
+	sudo make -C "$GIT_REPOS/qemu/$SPDK_QEMU_BRANCH" install
+}
+
+function install_nvmecli() {
+	SPDK_NVME_CLI_BRANCH=spdk-1.6
+	if [[ ! -d $GIT_REPOS/nvme-cli ]]; then
+		git clone "${GIT_REPO_SPDK_NVME_CLI}" -b "$SPDK_NVME_CLI_BRANCH" "$GIT_REPOS/nvme-cli"
+	else
+		echo "nvme-cli already checked out. Skipping"
+	fi
+	if [ ! -d "/usr/local/src/nvme-cli" ]; then
+		# Changes required for SPDK are already merged on top of
+		# nvme-cli, however not released yet.
+		# Support for SPDK should be released in nvme-cli >1.11.1
+		if [[ ! -d $GIT_REPOS/nvme-cli-cuse ]]; then
+			git clone "https://github.com/linux-nvme/nvme-cli.git" "$GIT_REPOS/nvme-cli-cuse"
+		fi
+		git -C "$GIT_REPOS/nvme-cli-cuse" checkout "e770466615096a6d41f038a28819b00bc3078e1d"
+		make -C "$GIT_REPOS/nvme-cli-cuse"
+		sudo mv "$GIT_REPOS/nvme-cli-cuse" /usr/local/src/nvme-cli
+	fi
+}
+
+function install_libiscsi() {
+	# We currently don't make any changes to the libiscsi repository for our tests, but it is possible that we will need
+	# to later. Cloning from git is just future proofing the machines.
+	if [[ ! -d $GIT_REPOS/libiscsi ]]; then
+		git clone "${GIT_REPO_LIBISCSI}" "$GIT_REPOS/libiscsi"
+	else
+		echo "libiscsi already checked out. Skipping"
+	fi
+	(cd "$GIT_REPOS/libiscsi" && ./autogen.sh && ./configure --prefix=/usr/local/libiscsi)
+	make -C "$GIT_REPOS/libiscsi" -j${jobs}
+	sudo make -C "$GIT_REPOS/libiscsi" install
+}
+
+function install_git() {
+	install zlib-devel curl-devel
+	tar -C "$GIT_REPOS" -xzof <(wget -qO- "$GIT_REPO_GIT")
+	(cd "$GIT_REPOS/git-$GIT_VERSION" \
+		&& make configure \
+		&& ./configure --prefix=/usr/local/git \
+		&& sudo make -j${jobs} install)
+	sudo sh -c "echo 'export PATH=/usr/local/git/bin:$PATH' >> /etc/bashrc"
+	export "PATH=/usr/local/git/bin:$PATH"
+}
+
+function install_extra_pkgs() {
+	if [[ $INSTALL_QAT == true ]]; then
+		install libudev-devel || install libudev-dev
+	fi
+
+	if [[ $INSTALL_QEMU == true ]]; then
+		install qemu-system-x86 qemu-img \
+			|| install qemu-system-x86 qemu-utils \
+			|| install qemu
+	fi
+}
+
+GIT_VERSION=2.25.1
+: ${GIT_REPO_SPDK=https://github.com/spdk/spdk.git}
+export GIT_REPO_SPDK
+: ${GIT_REPO_DPDK=https://github.com/spdk/dpdk.git}
+export GIT_REPO_DPDK
+: ${GIT_REPO_ROCKSDB=https://review.spdk.io/spdk/rocksdb}
+export GIT_REPO_ROCKSDB
+: ${GIT_REPO_FIO=http://git.kernel.dk/fio.git}
+export GIT_REPO_FIO
+: ${GIT_REPO_FLAMEGRAPH=https://github.com/brendangregg/FlameGraph.git}
+export GIT_REPO_FLAMEGRAPH
+: ${GIT_REPO_QEMU=https://github.com/spdk/qemu}
+export GIT_REPO_QEMU
+: ${GIT_REPO_LIBISCSI=https://github.com/sahlberg/libiscsi}
+export GIT_REPO_LIBISCSI
+: ${GIT_REPO_SPDK_NVME_CLI=https://github.com/spdk/nvme-cli}
+export GIT_REPO_SPDK_NVME_CLI
+: ${GIT_REPO_INTEL_IPSEC_MB=https://github.com/spdk/intel-ipsec-mb.git}
+export GIT_REPO_INTEL_IPSEC_MB
+: ${DRIVER_LOCATION_QAT=https://01.org/sites/default/files/downloads//qat1.7.l.4.9.0-00008.tar.gz}
+export DRIVER_LOCATION_QAT
+: ${GIT_REPO_GIT=https://github.com/git/git/archive/v${GIT_VERSION}.tar.gz}
+export GIT_REPO_GIT
+GIT_REPOS=${GIT_REPOS:-$HOME}
+
+gcc_version=$(gcc -dumpversion) gcc_version=${gcc_version%%.*}
+if [[ $ID == centos ]] && (( VERSION_ID == 7 )); then
+	# install proper version of the git first
+	install_git
+fi
+
+IFS="," read -ra conf_env <<< "$CONF"
+for conf in "${conf_env[@]}"; do
+	export "INSTALL_${conf^^}=true"
+done
+sources=(install_refspdk)
+
+if [[ $OS == FreeBSD ]]; then
+	jobs=$(($(sysctl -n hw.ncpu) * 2))
+else
+	jobs=$(($(nproc) * 2))
+	sources+=(
+		install_libiscsi
+		install_nvmecli
+		install_qat
+		install_rocksdb
+		install_flamegraph
+		install_qemu
+	)
+fi
+sources+=(install_fio)
+
+sudo mkdir -p /usr/{,local}/src
+sudo mkdir -p "$GIT_REPOS"
+
+install_extra_pkgs
+
+if [[ $INSTALL_REFSPDK == true ]]; then
+	# Serialize builds as refspdk depends on spdk
+	install_spdk
+	install_refspdk
+else
+	sources+=(install_spdk)
+fi
+
+for source in "${sources[@]}"; do
+	source_conf=${source^^}
+	if [[ ${!source_conf} == true ]]; then
+		"$source" &
+	fi
+done
+wait
diff --git a/src/spdk/test/common/config/pkgdep/pacman b/src/spdk/test/common/config/pkgdep/pacman
new file mode 100644
index 000000000..43d3db2f5
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/pacman
@@ -0,0 +1,62 @@
+package_manager=pacman
+
+upgrade() {
+	sudo "$package_manager" -Syu --noconfirm --needed
+}
+
+install() {
+	(($#)) || return 0
+
+	sudo "$package_manager" -Sy --noconfirm --needed "$@"
+}
+
+pre_install() {
+	if [[ $INTSALL_TSOCKS == true ]]; then
+		install tsocks
+	fi
+}
+
+packages=(
+	valgrind
+	jq
+	nvme-cli
+	ceph
+	gdb
+	fio
+	linux-headers
+	gflags
+	autoconf
+	automake
+	libtool
+	libutil-linux
+	libiscsi
+	open-isns
+	glib2
+	pixman
+	flex
+	bison
+	elfutils
+	libelf
+	astyle
+	gptfdisk
+	socat
+	sshfs
+	sshpass
+	python-pandas
+	btrfs-progs
+	iptables
+	clang
+	bc
+	perl-switch
+	open-iscsi
+	smartmontools
+	parted
+	wget
+)
+
+# TODO:
+# These are either missing or require some other installation method
+# than pacman:
+# librbd-devel
+# perl-open
+# targetcli
diff --git a/src/spdk/test/common/config/pkgdep/pkg b/src/spdk/test/common/config/pkgdep/pkg
new file mode 100644
index 000000000..3f3f41725
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/pkg
@@ -0,0 +1,27 @@
+package_manager=pkg
+
+upgrade() {
+	sudo "$package_manager" upgrade -y
+}
+
+install() {
+	(($#)) || return 0
+
+	sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+	pciutils
+	jq
+	gdb
+	fio
+	p5-extutils-pkgconfig
+	libtool
+	flex
+	bison
+	gdisk
+	socat
+	sshpass
+	py37-pandas
+	wget
+)
diff --git a/src/spdk/test/common/config/pkgdep/swupd b/src/spdk/test/common/config/pkgdep/swupd
new file mode 100644
index 000000000..c1d2a8a6b
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/swupd
@@ -0,0 +1,21 @@
+package_manager=swupd
+
+upgrade() {
+	sudo "$package_manager" update -y
+}
+
+install() {
+	(($#)) || return 0
+
+	sudo "$package_manager" bundle-add -y "$@"
+}
+
+packages=(
+	jq
+)
+
+pre_install() {
+	if [[ $INTSALL_TSOCKS == true ]]; then
+		install tsocks || echo "Installation of the tsocks package failed, proxy may not be available"
+	fi
+}
diff --git a/src/spdk/test/common/config/pkgdep/yum b/src/spdk/test/common/config/pkgdep/yum
new file mode 100644
index 000000000..32e89bc15
--- /dev/null
+++ b/src/spdk/test/common/config/pkgdep/yum
@@ -0,0 +1,67 @@
+package_manager=yum
+
+upgrade() {
+	sudo "$package_manager" upgrade -y
+}
+
+install() {
+	(($#)) || return 0
+
+	sudo "$package_manager" install -y "$@"
+}
+
+packages=(
+	pciutils
+	valgrind
+	jq
+	nvme-cli
+	gdb
+	fio
+	librbd-devel
+	kernel-devel
+	gflags-devel
+	libasan
+	libubsan
+	autoconf
+	automake
+	libtool
+	libmount-devel
+	iscsi-initiator-utils
+	isns-utils-devel pmempool
+	perl-open
+	glib2-devel
+	pixman-devel
+	astyle-devel
+	elfutils
+	elfutils-libelf-devel
+	flex
+	bison
+	targetcli
+	perl-Switch
+	librdmacm-utils
+	libibverbs-utils
+	gdisk
+	socat
+	sshfs
+	sshpass
+	python3-pandas
+	rpm-build
+	iptables
+	clang-analyzer
+	bc
+	kernel-modules-extra
+	systemd-devel
+	python3
+	wget
+)
+
+pre_install() {
+	if [[ $ID == centos ]] && (( VERSION_ID == 8 )); then
+		"$package_manager" update -y --refresh
+	fi
+
+	install nbd || {
+		wget -O nbd.rpm https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/n/nbd-3.14-2.el7.x86_64.rpm
+		install nbd.rpm
+	}
+}
diff --git a/src/spdk/test/common/config/vm_setup.conf b/src/spdk/test/common/config/vm_setup.conf
new file mode 100644
index 000000000..a8e58d82a
--- /dev/null
+++ b/src/spdk/test/common/config/vm_setup.conf
@@ -0,0 +1,12 @@
+# This configuration file is provided for reference purposes.
+GIT_REPO_SPDK=https://review.spdk.io/gerrit/spdk/spdk
+GIT_REPO_DPDK=https://github.com/spdk/dpdk.git
+GIT_REPO_OPEN_ISCSI=https://github.com/open-iscsi/open-iscsi
+GIT_REPO_ROCKSDB=https://review.gerrithub.io/spdk/rocksdb
+GIT_REPO_FIO=http://git.kernel.dk/fio.git
+GIT_REPO_FLAMEGRAPH=https://github.com/brendangregg/FlameGraph.git
+GIT_REPO_QEMU=https://github.com/spdk/qemu
+GIT_REPO_VPP=https://gerrit.fd.io/r/vpp
+GIT_REPO_LIBISCSI=https://github.com/sahlberg/libiscsi
+GIT_REPO_SPDK_NVME_CLI=https://github.com/spdk/nvme-cli
+DRIVER_LOCATION_QAT=https://01.org/sites/default/files/downloads/intelr-quickassist-technology/qat1.7.l.4.3.0-00033.tar.gz
diff --git a/src/spdk/test/common/config/vm_setup.sh b/src/spdk/test/common/config/vm_setup.sh
new file mode 100755
index 000000000..b2b3a8fc1
--- /dev/null
+++ b/src/spdk/test/common/config/vm_setup.sh
@@ -0,0 +1,176 @@
+#!/usr/bin/env bash
+
+# Virtual Machine environment requirements:
+# 8 GiB of RAM (for DPDK)
+# enable intel_kvm on your host machine
+
+# The purpose of this script is to provide a simple procedure for spinning up a new
+# virtual test environment capable of running our whole test suite. This script, when
+# applied to a fresh install of fedora 26 or ubuntu 16,18 server will install all of the
+# necessary dependencies to run almost the complete test suite. The main exception being VHost.
+# Vhost requires the configuration of a second virtual machine. instructions for how to configure
+# that vm are included in the file TEST_ENV_SETUP_README inside this repository
+
+# it is important to enable nesting for vms in kernel command line of your machine for the vhost tests.
+#     in /etc/default/grub
+#     append the following to the GRUB_CMDLINE_LINUX line
+#     intel_iommu=on kvm-intel.nested=1
+
+# We have made a lot of progress with removing hardcoded paths from the tests,
+
+sudo() {
+	"$(type -P sudo)" -E "$@"
+}
+
+set -e
+
+VM_SETUP_PATH=$(readlink -f ${BASH_SOURCE%/*})
+
+UPGRADE=false
+INSTALL=false
+CONF="rocksdb,fio,flamegraph,tsocks,qemu,libiscsi,nvmecli,qat,spdk,refspdk"
+
+if [[ -e /etc/os-release ]]; then
+	source /etc/os-release
+fi
+
+if [ $(uname -s) == "FreeBSD" ]; then
+	OSID="freebsd"
+	OSVERSION=$(freebsd-version | cut -d. -f1)
+else
+	OSID=$(source /etc/os-release && echo $ID)
+	OSVERSION=$(source /etc/os-release && echo $VERSION_ID)
+fi
+
+function usage() {
+	echo "This script is intended to automate the environment setup for a linux virtual machine."
+	echo "Please run this script as your regular user. The script will make calls to sudo as needed."
+	echo ""
+	echo "./vm_setup.sh"
+	echo "  -h --help"
+	echo "  -u --upgrade Run $package_manager upgrade"
+	echo "  -i --install-deps Install $package_manager based dependencies"
+	echo "  -t --test-conf List of test configurations to enable (${CONF})"
+	echo "  -c --conf-path Path to configuration file"
+	echo "  -d --dir-git Path to where git sources should be saved"
+	echo "  -s --disable-tsocks Disable use of tsocks"
+	exit 0
+}
+
+vmsetupdir=$(readlink -f "$(dirname "$0")")
+rootdir=$(readlink -f "$vmsetupdir/../../../")
+
+managers=("$vmsetupdir/pkgdep/"*)
+# Get package manager #
+if hash dnf &> /dev/null; then
+	source "$vmsetupdir/pkgdep/dnf"
+elif hash yum &> /dev/null; then
+	source "$vmsetupdir/pkgdep/yum"
+elif hash apt-get &> /dev/null; then
+	source "$vmsetupdir/pkgdep/apt-get"
+elif hash pacman &> /dev/null; then
+	source "$vmsetupdir/pkgdep/pacman"
+elif hash pkg &> /dev/null; then
+	source "$vmsetupdir/pkgdep/pkg"
+elif hash swupd &> /dev/null; then
+	source "$vmsetupdir/pkgdep/swupd"
+else
+	package_manager="undefined"
+fi
+
+# Parse input arguments #
+while getopts 'd:siuht:c:-:' optchar; do
+	case "$optchar" in
+		-)
+			case "$OPTARG" in
+				help) usage ;;
+				upgrade) UPGRADE=true ;;
+				install-deps) INSTALL=true ;;
+				test-conf=*) CONF="${OPTARG#*=}" ;;
+				conf-path=*) CONF_PATH="${OPTARG#*=}" ;;
+				dir-git=*) GIT_REPOS="${OPTARG#*=}" ;;
+				disable-tsocks) NO_TSOCKS=true ;;
+				*)
+					echo "Invalid argument '$OPTARG'"
+					usage
+					;;
+			esac
+			;;
+		h) usage ;;
+		u) UPGRADE=true ;;
+		i) INSTALL=true ;;
+		t) CONF="$OPTARG" ;;
+		c) CONF_PATH="$OPTARG" ;;
+		d) GIT_REPOS="$OPTARG" ;;
+		s) NO_TSOCKS=true ;;
+		*)
+			echo "Invalid argument '$OPTARG'"
+			usage
+			;;
+	esac
+done
+
+if [[ "$package_manager" == "undefined" ]]; then
+	echo "Supported package manager not found. Script supports:"
+	printf ' * %s\n' "${managers[@]##*/}"
+	exit 1
+fi
+
+if [ -n "$CONF_PATH" ]; then
+	if [ ! -f "$CONF_PATH" ]; then
+		echo Configuration file does not exist: "$CONF_PATH"
+		exit 1
+	else
+		source "$CONF_PATH"
+	fi
+fi
+
+if $UPGRADE; then
+	upgrade
+fi
+
+if $INSTALL; then
+	sudo "$rootdir/scripts/pkgdep.sh" --all
+	pre_install
+	install "${packages[@]}"
+fi
+
+source "$vmsetupdir/pkgdep/git"
+
+# create autorun-spdk.conf in home folder. This is sourced by the autotest_common.sh file.
+# By setting any one of the values below to 0, you can skip that specific test. If you are
+# using your autotest platform to do sanity checks before uploading to the build pool, it is
+# probably best to only run the tests that you believe your changes have modified along with
+# Scanbuild and check format. This is because running the whole suite of tests in series can
+# take ~40 minutes to complete.
+if [ ! -e ~/autorun-spdk.conf ]; then
+	cat > ~/autorun-spdk.conf << EOF
+# assign a value of 1 to all of the pertinent tests
+SPDK_RUN_VALGRIND=1
+SPDK_TEST_CRYPTO=1
+SPDK_RUN_FUNCTIONAL_TEST=1
+SPDK_TEST_AUTOBUILD=1
+SPDK_TEST_UNITTEST=1
+SPDK_TEST_ISCSI=1
+SPDK_TEST_ISCSI_INITIATOR=1
+SPDK_TEST_NVME=1
+SPDK_TEST_NVME_CLI=1
+SPDK_TEST_NVMF=1
+SPDK_TEST_RBD=1
+SPDK_TEST_BLOCKDEV=1
+SPDK_TEST_BLOBFS=1
+SPDK_TEST_PMDK=1
+SPDK_TEST_LVOL=1
+SPDK_TEST_JSON=1
+SPDK_RUN_ASAN=1
+SPDK_RUN_UBSAN=1
+# doesn't work on vm
+SPDK_TEST_IOAT=0
+# requires some extra configuration. see TEST_ENV_SETUP_README
+SPDK_TEST_VHOST=0
+SPDK_TEST_VHOST_INIT=0
+# Not configured here
+SPDK_RUN_INSTALLED_DPDK=0
+
+EOF
+fi