diff options
Diffstat (limited to '')
| -rw-r--r-- | src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.html | 601 |
1 files changed, 601 insertions, 0 deletions
diff --git a/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.html b/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.html new file mode 100644 index 00000000..fe4c2f6d --- /dev/null +++ b/src/VBox/ValidationKit/docs/VBoxAudioValidationKitReadMe.html @@ -0,0 +1,601 @@ +<?xml version="1.0" encoding="utf-8" ?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" /> +<title>Audio Testing of VirtualBox</title> +<style type="text/css"> + +/* +:Author: David Goodger (goodger@python.org) +:Id: $Id: VBoxAudioValidationKitReadMe.html $ +:Copyright: This stylesheet has been placed in the public domain. + +Default cascading style sheet for the HTML output of Docutils. + +See https://docutils.sourceforge.io/docs/howto/html-stylesheets.html for how to +customize this style sheet. +*/ + +/* used to remove borders from tables and images */ +.borderless, table.borderless td, table.borderless th { + border: 0 } + +table.borderless td, table.borderless th { + /* Override padding for "table.docutils td" with "! important". + The right padding separates the table cells. */ + padding: 0 0.5em 0 0 ! important } + +.first { + /* Override more specific margin styles with "! important". */ + margin-top: 0 ! important } + +.last, .with-subtitle { + margin-bottom: 0 ! important } + +.hidden { + display: none } + +.subscript { + vertical-align: sub; + font-size: smaller } + +.superscript { + vertical-align: super; + font-size: smaller } + +a.toc-backref { + text-decoration: none ; + color: black } + +blockquote.epigraph { + margin: 2em 5em ; } + +dl.docutils dd { + margin-bottom: 0.5em } + +object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] { + overflow: hidden; +} + +/* Uncomment (and remove this text!) to get bold-faced definition list terms +dl.docutils dt { + font-weight: bold } +*/ + +div.abstract { + margin: 2em 5em } + +div.abstract p.topic-title { + font-weight: bold ; + text-align: center } + +div.admonition, div.attention, div.caution, div.danger, div.error, +div.hint, div.important, div.note, div.tip, div.warning { + margin: 2em ; + border: medium outset ; + padding: 1em } + +div.admonition p.admonition-title, div.hint p.admonition-title, +div.important p.admonition-title, div.note p.admonition-title, +div.tip p.admonition-title { + font-weight: bold ; + font-family: sans-serif } + +div.attention p.admonition-title, div.caution p.admonition-title, +div.danger p.admonition-title, div.error p.admonition-title, +div.warning p.admonition-title, .code .error { + color: red ; + font-weight: bold ; + font-family: sans-serif } + +/* Uncomment (and remove this text!) to get reduced vertical space in + compound paragraphs. +div.compound .compound-first, div.compound .compound-middle { + margin-bottom: 0.5em } + +div.compound .compound-last, div.compound .compound-middle { + margin-top: 0.5em } +*/ + +div.dedication { + margin: 2em 5em ; + text-align: center ; + font-style: italic } + +div.dedication p.topic-title { + font-weight: bold ; + font-style: normal } + +div.figure { + margin-left: 2em ; + margin-right: 2em } + +div.footer, div.header { + clear: both; + font-size: smaller } + +div.line-block { + display: block ; + margin-top: 1em ; + margin-bottom: 1em } + +div.line-block div.line-block { + margin-top: 0 ; + margin-bottom: 0 ; + margin-left: 1.5em } + +div.sidebar { + margin: 0 0 0.5em 1em ; + border: medium outset ; + padding: 1em ; + background-color: #ffffee ; + width: 40% ; + float: right ; + clear: right } + +div.sidebar p.rubric { + font-family: sans-serif ; + font-size: medium } + +div.system-messages { + margin: 5em } + +div.system-messages h1 { + color: red } + +div.system-message { + border: medium outset ; + padding: 1em } + +div.system-message p.system-message-title { + color: red ; + font-weight: bold } + +div.topic { + margin: 2em } + +h1.section-subtitle, h2.section-subtitle, h3.section-subtitle, +h4.section-subtitle, h5.section-subtitle, h6.section-subtitle { + margin-top: 0.4em } + +h1.title { + text-align: center } + +h2.subtitle { + text-align: center } + +hr.docutils { + width: 75% } + +img.align-left, .figure.align-left, object.align-left, table.align-left { + clear: left ; + float: left ; + margin-right: 1em } + +img.align-right, .figure.align-right, object.align-right, table.align-right { + clear: right ; + float: right ; + margin-left: 1em } + +img.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left } + +.align-center { + clear: both ; + text-align: center } + +.align-right { + text-align: right } + +/* reset inner alignment in figures */ +div.align-right { + text-align: inherit } + +/* div.align-center * { */ +/* text-align: left } */ + +.align-top { + vertical-align: top } + +.align-middle { + vertical-align: middle } + +.align-bottom { + vertical-align: bottom } + +ol.simple, ul.simple { + margin-bottom: 1em } + +ol.arabic { + list-style: decimal } + +ol.loweralpha { + list-style: lower-alpha } + +ol.upperalpha { + list-style: upper-alpha } + +ol.lowerroman { + list-style: lower-roman } + +ol.upperroman { + list-style: upper-roman } + +p.attribution { + text-align: right ; + margin-left: 50% } + +p.caption { + font-style: italic } + +p.credits { + font-style: italic ; + font-size: smaller } + +p.label { + white-space: nowrap } + +p.rubric { + font-weight: bold ; + font-size: larger ; + color: maroon ; + text-align: center } + +p.sidebar-title { + font-family: sans-serif ; + font-weight: bold ; + font-size: larger } + +p.sidebar-subtitle { + font-family: sans-serif ; + font-weight: bold } + +p.topic-title { + font-weight: bold } + +pre.address { + margin-bottom: 0 ; + margin-top: 0 ; + font: inherit } + +pre.literal-block, pre.doctest-block, pre.math, pre.code { + margin-left: 2em ; + margin-right: 2em } + +pre.code .ln { color: grey; } /* line numbers */ +pre.code, code { background-color: #eeeeee } +pre.code .comment, code .comment { color: #5C6576 } +pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold } +pre.code .literal.string, code .literal.string { color: #0C5404 } +pre.code .name.builtin, code .name.builtin { color: #352B84 } +pre.code .deleted, code .deleted { background-color: #DEB0A1} +pre.code .inserted, code .inserted { background-color: #A3D289} + +span.classifier { + font-family: sans-serif ; + font-style: oblique } + +span.classifier-delimiter { + font-family: sans-serif ; + font-weight: bold } + +span.interpreted { + font-family: sans-serif } + +span.option { + white-space: nowrap } + +span.pre { + white-space: pre } + +span.problematic { + color: red } + +span.section-subtitle { + /* font-size relative to parent (h1..h6 element) */ + font-size: 80% } + +table.citation { + border-left: solid 1px gray; + margin-left: 1px } + +table.docinfo { + margin: 2em 4em } + +table.docutils { + margin-top: 0.5em ; + margin-bottom: 0.5em } + +table.footnote { + border-left: solid 1px black; + margin-left: 1px } + +table.docutils td, table.docutils th, +table.docinfo td, table.docinfo th { + padding-left: 0.5em ; + padding-right: 0.5em ; + vertical-align: top } + +table.docutils th.field-name, table.docinfo th.docinfo-name { + font-weight: bold ; + text-align: left ; + white-space: nowrap ; + padding-left: 0 } + +/* "booktabs" style (no vertical lines) */ +table.docutils.booktabs { + border: 0px; + border-top: 2px solid; + border-bottom: 2px solid; + border-collapse: collapse; +} +table.docutils.booktabs * { + border: 0px; +} +table.docutils.booktabs th { + border-bottom: thin solid; + text-align: left; +} + +h1 tt.docutils, h2 tt.docutils, h3 tt.docutils, +h4 tt.docutils, h5 tt.docutils, h6 tt.docutils { + font-size: 100% } + +ul.auto-toc { + list-style-type: none } + +</style> +</head> +<body> +<div class="document" id="audio-testing-of-virtualbox"> +<h1 class="title">Audio Testing of VirtualBox</h1> + +<div class="section" id="overview-goal"> +<h1>Overview / Goal</h1> +<p>The goal is to create a flexible testing framework to test the +VirtualBox audio stack.</p> +<p>It should be runnable with an easy-to-use setup so that also regular users +can perform tests on request, without having to install or set up additional +dependencies.</p> +<p>That framework must be runnable on all host/guest combinations together with all +audio drivers ("backends") and device emulations being offered. This makes it a +rather big testing matrix which therefore has to be processed in an automated +fashion.</p> +<p>Additionally it should be flexible enough to add more (custom) tests later on.</p> +</div> +<div class="section" id="operation"> +<h1>Operation</h1> +<p>The framework consists of several components which try to make use as much of +the existing audio stack code as possible. This allows the following +operation modes:</p> +<dl class="docutils"> +<dt>Standalone</dt> +<dd>Playing back / recording audio data (test tones / .WAV files) in a +standalone scenario, i.e. no VirtualBox / VMs required). This mode is using +VirtualBox' audio (mixing) stack and available backend drivers without the +need of VirtualBox being installed.</dd> +<dt>Manual</dt> +<dd>Performing single / multiple tests manually on a local machine. +Requires a running and set up test VM.</dd> +<dt>Automated</dt> +<dd>Performs single / multiple tests via the Validation Kit audio test +driver and can be triggered via the Validation Kit Test Manager.</dd> +<dt>(Re-)validation of previously ran tests</dt> +<dd>This takes two test sets and runs the validation / analysis on them.</dd> +<dt>Self testing mode</dt> +<dd>Performs standalone self tests to verify / debug the involved components.</dd> +</dl> +</div> +<div class="section" id="components-and-terminology"> +<h1>Components and Terminology</h1> +<p>The following components are in charge for performing the audio tests +(depends on the operation mode, see above):</p> +<ul> +<li><p class="first">VBoxAudioTest (also known as VKAT, "Validation Kit Audio Test"): +A binary which can perform the standalone audio tests mentioned above, as well +as acting as the guest and host service(s) when performing manual or automated +tests. It also includes the analysis / verification of audio test sets. +VKAT also is included in host installations and Guest Additions since +VirtualBox 7.0 to give customers and end users the opportunity to test and +verify the audio stack.</p> +<dl class="docutils"> +<dt>Additional features include:</dt> +<dd><ul class="first last simple"> +<li>Automatic probing of audio backends ("--probe-backends")</li> +<li>Manual playback of test tones ("play -t")</li> +<li>Manual playback of .WAV files ("play <WAV-File>")</li> +<li>Manual recording to .WAV files ("recording <WAV-File>")</li> +<li>Manual device enumeration (sub command "enum")</li> +<li>Manual (re-)verification of test sets (sub command "verify")</li> +<li>Self-contained self tests (sub command "selftest")</li> +</ul> +</dd> +</dl> +<p>See the syntax help ("--help") for more.</p> +</li> +<li><p class="first">ATS ("Audio Testing Service"): Component which is being used by 1 and the +Validation Kit audio driver (backend) to communicate across guest and host +boundaries. Currently using a TCP/IP transport layer. Also works with VMs +which are configured with NAT networking ("reverse connection").</p> +</li> +<li><p class="first">Validation Kit audio test driver (tdAudioTest.py): Used for integrating and +invoking VKAT for manual and automated tests via the Validation Kit framework +(Test Manager). Optional. The test driver can be found at <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p> +</li> +<li><p class="first">Validation Kit audio driver (backend): A dedicated audio backend which +communicates with VKAT running on the same host to perform the actual audio +tests on a VirtualBox installation. This makes it possible to test the full +audio stack on a running VM without any additional / external tools.</p> +<p>On guest playback, data will be recorded, on guest recording, data will be +injected from the host into the audio stack.</p> +</li> +<li><dl class="first docutils"> +<dt>Test sets contain</dt> +<dd><ul class="first last simple"> +<li>a test manifest with all information required (vkat_manifest.ini)</li> +<li>the generated / captured audio data (as raw PCM)</li> +</ul> +</dd> +</dl> +<p>and are either packed as .tar.gz archives or consist of a dedicated directory +per test set.</p> +<p>There always must be at least two test sets - one from the host side and one +from the guest side - to perform a verification.</p> +<p>Each test set contains a test tag so that matching test sets can be +identified.</p> +</li> +</ul> +<p>The above components are also included in VirtualBox release builds and can be +optionally enabled (disabled by default).</p> +<table class="docutils footnote" frame="void" id="footnote-1" rules="none"> +<colgroup><col class="label" /><col /></colgroup> +<tbody valign="top"> +<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>src/VBox/ValidationKit/tests/audio/tdAudioTest.py</td></tr> +</tbody> +</table> +</div> +<div class="section" id="setup-instructions"> +<h1>Setup instructions</h1> +<ul> +<li><p class="first">VM needs to be configured to have audio emulation and audio testing enabled +(via extra-data, set "VBoxInternal2/Audio/Debug/Enabled" to "true").</p> +</li> +<li><p class="first">Audio input / output for the VM needs to be enabled (depending on the test).</p> +</li> +<li><p class="first">Start VBoxAudioTest on the guest, for example:</p> +<p>VBoxAudioTest test --mode guest --tcp-connect-address 10.0.2.2</p> +<dl class="docutils"> +<dt>Note: VBoxAudioTest is included with the Guest Additions starting at</dt> +<dd><p class="first last">VirtualBox 7.0.</p> +</dd> +<dt>Note: Depending on the VM's networking configuration there might be further</dt> +<dd><p class="first last">steps necessary in order to be able to reach the host from the guest. +See the VirtualBox manual for more information.</p> +</dd> +</dl> +</li> +</ul> +</div> +<div class="section" id="performing-a-manual-test"> +<h1>Performing a manual test</h1> +<ul> +<li><p class="first">Follow "Setup instructions".</p> +</li> +<li><p class="first">Start VBoxAudioTest on the host with selected test(s), for example:</p> +<p>VBoxAudioTest test --mode host</p> +<blockquote> +<dl class="docutils"> +<dt>Note: VBoxAudioTest is included with the VirtualBox 7.0 host installers and</dt> +<dd><p class="first last">will be installed by default.</p> +</dd> +</dl> +</blockquote> +</li> +<li><p class="first">By default the test verification will be done automatically after running the +tests.</p> +</li> +</ul> +</div> +<div class="section" id="advanced-performing-manual-verification"> +<h1>Advanced: Performing manual verification</h1> +<p>VBoxAudioTest can manually be used with the "verify" sub command in order to +(re-)verify previously generated test sets. It then will return different exit +codes based on the verification result.</p> +</div> +<div class="section" id="advanced-performing-an-automated-test"> +<h1>Advanced: Performing an automated test</h1> +<ul class="simple"> +<li>TxS (Test E[x]ecution Service) has to be up and running (part of the +Validation Kit) on the guest.</li> +<li>Invoke the tdAudioTest.py test driver, either manually or fully automated +via Test Manager.</li> +</ul> +</div> +<div class="section" id="internals-workflow-for-a-single-test"> +<h1>Internals: Workflow for a single test</h1> +<p>When a single test is being executed on a running VM, the following (simplified) +workflow applies:</p> +<ul class="simple"> +<li>VKAT on the host connects to VKAT running on the guest (via ATS, also can be a +remote machine in theory).</li> +<li>VKAT on the host connects to Validation Kit audio driver on the host +(via ATS, also can be a remote machine in theory).</li> +<li><dl class="first docutils"> +<dt>For example, when doing playback tests, VKAT on the host ...</dt> +<dd><ul class="first last"> +<li><dl class="first docutils"> +<dt>... tells the Validation Kit audio driver to start recording</dt> +<dd>guest playback.</dd> +</dl> +</li> +<li>... tells the VKAT on the guest to start playing back audio data.</li> +<li><dl class="first docutils"> +<dt>... gathers all test data (generated from/by the guest and recorded from</dt> +<dd>the host) as separate test sets.</dd> +</dl> +</li> +<li>... starts verification / analysis of the test sets.</li> +</ul> +</dd> +</dl> +</li> +</ul> +</div> +<div class="section" id="current-status-limitations"> +<h1>Current status / limitations</h1> +<ul class="simple"> +<li><dl class="first docutils"> +<dt>The following test types are currently implemented:</dt> +<dd><ul class="first last"> +<li>Test tone (sine wave) playback from the guest</li> +<li>Test tone (sine wave) recording by the guest (injected from the host)</li> +</ul> +</dd> +</dl> +</li> +<li>Only the HDA device emulation has been verified so far.</li> +<li>Only the ALSA audio stack on Debian 10 has been verified so far. +Note: This is different from PulseAudio using the ALSA plugin!</li> +</ul> +</div> +<div class="section" id="troubleshooting"> +<h1>Troubleshooting</h1> +<ul class="simple"> +<li>Make sure that audio device emulation is enabled and can be used within the +guest. Also, audio input / output has to be enabled, depending on the tests.</li> +<li>Make sure that the guest's VBoxAudioTest's instance can reach the host via +the selected transport layer (TCP/IP by default).</li> +<li>Increase the hosts audio logging level +(via extra-data, set "VBoxInternal2/Audio/Debug/Level" to "5").</li> +<li>Increase VBoxAudioTest's verbosity level (add "-v", can be specified +multiple times).</li> +<li>Check if the VBox release log contains any warnings / errors with the +"ValKit:" prefix.</li> +</ul> +<table class="docutils field-list" frame="void" rules="none"> +<col class="field-name" /> +<col class="field-body" /> +<tbody valign="top"> +<tr class="field"><th class="field-name">Status:</th><td class="field-body">$Id: VBoxAudioValidationKitReadMe.html $</td> +</tr> +<tr class="field"><th class="field-name">Copyright:</th><td class="field-body">Copyright (C) 2021-2023 Oracle Corporation.</td> +</tr> +</tbody> +</table> +</div> +</div> +</body> +</html> |