summaryrefslogtreecommitdiffstats
path: root/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.txt
diff options
context:
space:
mode:
Diffstat (limited to '')
-rw-r--r--src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.txt207
1 files changed, 207 insertions, 0 deletions
diff --git a/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.txt b/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.txt
new file mode 100644
index 00000000..2797f0a9
--- /dev/null
+++ b/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.txt
@@ -0,0 +1,207 @@
+Audio Testing of VirtualBox
+===========================
+
+
+Overview / Goal
+---------------
+
+The goal is to create a flexible testing framework to test the
+VirtualBox audio stack.
+
+It should be runnable with an easy-to-use setup so that also regular users
+can perform tests on request, without having to install or set up additional
+dependencies.
+
+That framework must be runnable on all host/guest combinations together with all
+audio drivers ("backends") and device emulations being offered. This makes it a
+rather big testing matrix which therefore has to be processed in an automated
+fashion.
+
+Additionally it should be flexible enough to add more (custom) tests later on.
+
+
+Operation
+---------
+
+The framework consists of several components which try to make use as much of
+the existing audio stack code as possible. This allows the following
+operation modes:
+
+Standalone
+ Playing back / recording audio data (test tones / .WAV files) in a
+ standalone scenario, i.e. no VirtualBox / VMs required). This mode is using
+ VirtualBox' audio (mixing) stack and available backend drivers without the
+ need of VirtualBox being installed.
+
+Manual
+ Performing single / multiple tests manually on a local machine.
+ Requires a running and set up test VM.
+
+Automated
+ Performs single / multiple tests via the Validation Kit audio test
+ driver and can be triggered via the Validation Kit Test Manager.
+
+(Re-)validation of previously ran tests
+ This takes two test sets and runs the validation / analysis on them.
+
+Self testing mode
+ Performs standalone self tests to verify / debug the involved components.
+
+
+Components and Terminology
+--------------------------
+
+The following components are in charge for performing the audio tests
+(depends on the operation mode, see above):
+
+- VBoxAudioTest (also known as VKAT, "Validation Kit Audio Test"):
+ A binary which can perform the standalone audio tests mentioned above, as well
+ as acting as the guest and host service(s) when performing manual or automated
+ tests. It also includes the analysis / verification of audio test sets.
+ VKAT also is included in host installations and Guest Additions since
+ VirtualBox 7.0 to give customers and end users the opportunity to test and
+ verify the audio stack.
+
+ Additional features include:
+ * Automatic probing of audio backends ("--probe-backends")
+ * Manual playback of test tones ("play -t")
+ * Manual playback of .WAV files ("play <WAV-File>")
+ * Manual recording to .WAV files ("recording <WAV-File>")
+ * Manual device enumeration (sub command "enum")
+ * Manual (re-)verification of test sets (sub command "verify")
+ * Self-contained self tests (sub command "selftest")
+
+ See the syntax help ("--help") for more.
+
+- ATS ("Audio Testing Service"): Component which is being used by 1 and the
+ Validation Kit audio driver (backend) to communicate across guest and host
+ boundaries. Currently using a TCP/IP transport layer. Also works with VMs
+ which are configured with NAT networking ("reverse connection").
+
+- Validation Kit audio test driver (tdAudioTest.py): Used for integrating and
+ invoking VKAT for manual and automated tests via the Validation Kit framework
+ (Test Manager). Optional. The test driver can be found at [1]_.
+
+- Validation Kit audio driver (backend): A dedicated audio backend which
+ communicates with VKAT running on the same host to perform the actual audio
+ tests on a VirtualBox installation. This makes it possible to test the full
+ audio stack on a running VM without any additional / external tools.
+
+ On guest playback, data will be recorded, on guest recording, data will be
+ injected from the host into the audio stack.
+
+- Test sets contain
+ - a test manifest with all information required (vkat_manifest.ini)
+ - the generated / captured audio data (as raw PCM)
+
+ and are either packed as .tar.gz archives or consist of a dedicated directory
+ per test set.
+
+ There always must be at least two test sets - one from the host side and one
+ from the guest side - to perform a verification.
+
+ Each test set contains a test tag so that matching test sets can be
+ identified.
+
+The above components are also included in VirtualBox release builds and can be
+optionally enabled (disabled by default).
+
+.. [1] src/VBox/ValidationKit/tests/audio/tdAudioTest.py
+
+
+Setup instructions
+------------------
+
+- VM needs to be configured to have audio emulation and audio testing enabled
+ (via extra-data, set "VBoxInternal2/Audio/Debug/Enabled" to "true").
+- Audio input / output for the VM needs to be enabled (depending on the test).
+- Start VBoxAudioTest on the guest, for example:
+
+ VBoxAudioTest test --mode guest --tcp-connect-address 10.0.2.2
+
+ Note: VBoxAudioTest is included with the Guest Additions starting at
+ VirtualBox 7.0.
+ Note: Depending on the VM's networking configuration there might be further
+ steps necessary in order to be able to reach the host from the guest.
+ See the VirtualBox manual for more information.
+
+
+Performing a manual test
+------------------------
+
+- Follow "Setup instructions".
+- Start VBoxAudioTest on the host with selected test(s), for example:
+
+ VBoxAudioTest test --mode host
+
+ Note: VBoxAudioTest is included with the VirtualBox 7.0 host installers and
+ will be installed by default.
+
+- By default the test verification will be done automatically after running the
+ tests.
+
+
+Advanced: Performing manual verification
+----------------------------------------
+
+VBoxAudioTest can manually be used with the "verify" sub command in order to
+(re-)verify previously generated test sets. It then will return different exit
+codes based on the verification result.
+
+
+Advanced: Performing an automated test
+--------------------------------------
+
+- TxS (Test E[x]ecution Service) has to be up and running (part of the
+ Validation Kit) on the guest.
+- Invoke the tdAudioTest.py test driver, either manually or fully automated
+ via Test Manager.
+
+
+Internals: Workflow for a single test
+-------------------------------------
+
+When a single test is being executed on a running VM, the following (simplified)
+workflow applies:
+
+- VKAT on the host connects to VKAT running on the guest (via ATS, also can be a
+ remote machine in theory).
+- VKAT on the host connects to Validation Kit audio driver on the host
+ (via ATS, also can be a remote machine in theory).
+- For example, when doing playback tests, VKAT on the host ...
+ * ... tells the Validation Kit audio driver to start recording
+ guest playback.
+ * ... tells the VKAT on the guest to start playing back audio data.
+ * ... gathers all test data (generated from/by the guest and recorded from
+ the host) as separate test sets.
+ * ... starts verification / analysis of the test sets.
+
+
+Current status / limitations
+----------------------------
+
+- The following test types are currently implemented:
+ * Test tone (sine wave) playback from the guest
+ * Test tone (sine wave) recording by the guest (injected from the host)
+- Only the HDA device emulation has been verified so far.
+- Only the ALSA audio stack on Debian 10 has been verified so far.
+ Note: This is different from PulseAudio using the ALSA plugin!
+
+
+Troubleshooting
+---------------
+
+- Make sure that audio device emulation is enabled and can be used within the
+ guest. Also, audio input / output has to be enabled, depending on the tests.
+- Make sure that the guest's VBoxAudioTest's instance can reach the host via
+ the selected transport layer (TCP/IP by default).
+- Increase the hosts audio logging level
+ (via extra-data, set "VBoxInternal2/Audio/Debug/Level" to "5").
+- Increase VBoxAudioTest's verbosity level (add "-v", can be specified
+ multiple times).
+- Check if the VBox release log contains any warnings / errors with the
+ "ValKit:" prefix.
+
+
+:Status: $Id: VBoxAudioValidationKitReadMe.txt $
+:Copyright: Copyright (C) 2021-2023 Oracle Corporation.