summaryrefslogtreecommitdiffstats
path: root/third_party/libwebrtc/test/pc/e2e/g3doc
diff options
context:
space:
mode:
authorDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-21 11:44:51 +0000
committerDaniel Baumann <daniel.baumann@progress-linux.org>2024-04-21 11:44:51 +0000
commit9e3c08db40b8916968b9f30096c7be3f00ce9647 (patch)
treea68f146d7fa01f0134297619fbe7e33db084e0aa /third_party/libwebrtc/test/pc/e2e/g3doc
parentInitial commit. (diff)
downloadthunderbird-9e3c08db40b8916968b9f30096c7be3f00ce9647.tar.xz
thunderbird-9e3c08db40b8916968b9f30096c7be3f00ce9647.zip
Adding upstream version 1:115.7.0.upstream/1%115.7.0upstream
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'third_party/libwebrtc/test/pc/e2e/g3doc')
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md209
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md197
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.pngbin0 -> 39236 bytes
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/index.md224
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.pngbin0 -> 78481 bytes
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.pngbin0 -> 35899 bytes
-rw-r--r--third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.pngbin0 -> 79641 bytes
7 files changed, 630 insertions, 0 deletions
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md b/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md
new file mode 100644
index 0000000000..1b68c6db2c
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md
@@ -0,0 +1,209 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-04-12'} *-->
+
+# PeerConnection level framework fixture architecture
+
+## Overview
+
+The main implementation of
+[`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture`][1] is
+[`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTest`][2]. Internally it owns
+the next main pieces:
+
+* [`MediaHelper`][3] - responsible for adding audio and video tracks to the
+ peers.
+* [`VideoQualityAnalyzerInjectionHelper`][4] and
+ [`SingleProcessEncodedImageDataInjector`][5] - used to inject video quality
+ analysis and properly match captured and rendered video frames. You can read
+ more about it in
+ [DefaultVideoQualityAnalyzer](default_video_quality_analyzer.md) section.
+* [`AudioQualityAnalyzerInterface`][6] - used to measure audio quality metrics
+* [`TestActivitiesExecutor`][7] - used to support [`ExecuteAt(...)`][8] and
+ [`ExecuteEvery(...)`][9] API of `PeerConnectionE2EQualityTestFixture` to run
+ any arbitrary action during test execution timely synchronized with a test
+ call.
+* A vector of [`QualityMetricsReporter`][10] added by the
+ `PeerConnectionE2EQualityTestFixture` user.
+* Two peers: Alice and Bob represented by instances of [`TestPeer`][11]
+ object.
+
+Also it keeps a reference to [`webrtc::TimeController`][12], which is used to
+create all required threads, task queues, task queue factories and time related
+objects.
+
+## TestPeer
+
+Call participants are represented by instances of `TestPeer` object.
+[`TestPeerFactory`][13] is used to create them. `TestPeer` owns all instances
+related to the `webrtc::PeerConnection`, including required listeners and
+callbacks. Also it provides an API to do offer/answer exchange and ICE candidate
+exchange. For this purposes internally it uses an instance of
+[`webrtc::PeerConnectionWrapper`][14].
+
+The `TestPeer` also owns the `PeerConnection` worker thread. The signaling
+thread for all `PeerConnection`'s is owned by
+`PeerConnectionE2EQualityTestFixture` and shared between all participants in the
+call. The network thread is owned by the network layer (it maybe either emulated
+network provided by [Network Emulation Framework][24] or network thread and
+`rtc::NetworkManager` provided by user) and provided when peer is added to the
+fixture via [`AddPeer(...)`][15] API.
+
+## GetStats API based metrics reporters
+
+`PeerConnectionE2EQualityTestFixture` gives the user ability to provide
+different `QualityMetricsReporter`s which will listen for `PeerConnection`
+[`GetStats`][16] API. Then such reporters will be able to report various metrics
+that user wants to measure.
+
+`PeerConnectionE2EQualityTestFixture` itself also uses this mechanism to
+measure:
+
+* Audio quality metrics
+* Audio/Video sync metrics (with help of [`CrossMediaMetricsReporter`][17])
+
+Also framework provides a [`StatsBasedNetworkQualityMetricsReporter`][18] to
+measure network related WebRTC metrics and print debug raw emulated network
+statistic. This reporter should be added by user via
+[`AddQualityMetricsReporter(...)`][19] API if requried.
+
+Internally stats gathering is done by [`StatsPoller`][20]. Stats are requested
+once per second for each `PeerConnection` and then resulted object is provided
+into each stats listener.
+
+## Offer/Answer exchange
+
+`PeerConnectionE2EQualityTest` provides ability to test Simulcast and SVC for
+video. These features aren't supported by P2P call and in general requires a
+Selective Forwarding Unit (SFU). So special logic is applied to mimic SFU
+behavior in P2P call. This logic is located inside [`SignalingInterceptor`][21],
+[`QualityAnalyzingVideoEncoder`][22] and [`QualityAnalyzingVideoDecoder`][23]
+and consist of SDP modification during offer/answer exchange and special
+handling of video frames from unrelated Simulcast/SVC streams during decoding.
+
+### Simulcast
+
+In case of Simulcast we have a video track, which internally contains multiple
+video streams, for example low resolution, medium resolution and high
+resolution. WebRTC client doesn't support receiving an offer with multiple
+streams in it, because usually SFU will keep only single stream for the client.
+To bypass it framework will modify offer by converting a single track with three
+video streams into three independent video tracks. Then sender will think that
+it send simulcast, but receiver will think that it receives 3 independent
+tracks.
+
+To achieve such behavior some extra tweaks are required:
+
+* MID RTP header extension from original offer have to be removed
+* RID RTP header extension from original offer is replaced with MID RTP header
+ extension, so the ID that sender uses for RID on receiver will be parsed as
+ MID.
+* Answer have to be modified in the opposite way.
+
+Described modifications are illustrated on the picture below.
+
+![VP8 Simulcast offer modification](vp8_simulcast_offer_modification.png "VP8 Simulcast offer modification")
+
+The exchange will look like this:
+
+1. Alice creates an offer
+2. Alice sets offer as local description
+3. Do described offer modification
+4. Alice sends modified offer to Bob
+5. Bob sets modified offer as remote description
+6. Bob creates answer
+7. Bob sets answer as local description
+8. Do reverse modifications on answer
+9. Bob sends modified answer to Alice
+10. Alice sets modified answer as remote description
+
+Such mechanism put a constraint that RTX streams are not supported, because they
+don't have RID RTP header extension in their packets.
+
+### SVC
+
+In case of SVC the framework will update the sender's offer before even setting
+it as local description on the sender side. Then no changes to answer will be
+required.
+
+`ssrc` is a 32 bit random value that is generated in RTP to denote a specific
+source used to send media in an RTP connection. In original offer video track
+section will look like this:
+
+```
+m=video 9 UDP/TLS/RTP/SAVPF 98 100 99 101
+...
+a=ssrc-group:FID <primary ssrc> <retransmission ssrc>
+a=ssrc:<primary ssrc> cname:...
+....
+a=ssrc:<retransmission ssrc> cname:...
+....
+```
+
+To enable SVC for such video track framework will add extra `ssrc`s for each SVC
+stream that is required like this:
+
+```
+a=ssrc-group:FID <Low resolution primary ssrc> <Low resolution retransmission ssrc>
+a=ssrc:<Low resolution primary ssrc> cname:...
+....
+a=ssrc:<Low resolution retransmission ssrc> cname:....
+...
+a=ssrc-group:FID <Medium resolution primary ssrc> <Medium resolution retransmission ssrc>
+a=ssrc:<Medium resolution primary ssrc> cname:...
+....
+a=ssrc:<Medium resolution retransmission ssrc> cname:....
+...
+a=ssrc-group:FID <High resolution primary ssrc> <High resolution retransmission ssrc>
+a=ssrc:<High resolution primary ssrc> cname:...
+....
+a=ssrc:<High resolution retransmission ssrc> cname:....
+...
+```
+
+The next line will also be added to the video track section of the offer:
+
+```
+a=ssrc-group:SIM <Low resolution primary ssrc> <Medium resolution primary ssrc> <High resolution primary ssrc>
+```
+
+It will tell PeerConnection that this track should be configured as SVC. It
+utilize WebRTC Plan B offer structure to achieve SVC behavior, also it modifies
+offer before setting it as local description which violates WebRTC standard.
+Also it adds limitations that on lossy networks only top resolution streams can
+be analyzed, because WebRTC won't try to restore low resolution streams in case
+of loss, because it still receives higher stream.
+
+### Handling in encoder/decoder
+
+In the encoder, the framework for each encoded video frame will propagate
+information requried for the fake SFU to know if it belongs to an interesting
+simulcast stream/spatial layer of if it should be "discarded".
+
+On the decoder side frames that should be "discarded" by fake SFU will be auto
+decoded into single pixel images and only the interesting simulcast
+stream/spatial layer will go into real decoder and then will be analyzed.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=55;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[2]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/peer_connection_quality_test.h;l=44;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[3]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/media/media_helper.h;l=27;drc=d46db9f1523ae45909b4a6fdc90a140443068bc6
+[4]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/video_quality_analyzer_injection_helper.h;l=38;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[5]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/single_process_encoded_image_data_injector.h;l=40;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[6]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/audio_quality_analyzer_interface.h;l=23;drc=20f45823e37fd7272aa841831c029c21f29742c2
+[7]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[8]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=439;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[9]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=445;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[10]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=413;drc=9438fb3fff97c803d1ead34c0e4f223db168526f
+[11]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[12]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[13]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_peer_factory.h;l=46;drc=0ef4a2488a466a24ab97b31fdddde55440d451f9
+[14]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/pc/peer_connection_wrapper.h;l=47;drc=5ab79e62f691875a237ea28ca3975ea1f0ed62ec
+[15]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=459;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[16]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/peer_connection_interface.h;l=886;drc=9438fb3fff97c803d1ead34c0e4f223db168526f
+[17]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/cross_media_metrics_reporter.h;l=29;drc=9d777620236ec76754cfce19f6e82dd18e52d22c
+[18]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/cross_media_metrics_reporter.h;l=29;drc=9d777620236ec76754cfce19f6e82dd18e52d22c
+[19]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=450;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[20]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/stats_poller.h;l=52;drc=9b526180c9e9722d3fc7f8689da6ec094fc7fc0a
+[21]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/sdp/sdp_changer.h;l=79;drc=ee558dcca89fd8b105114ededf9e74d948da85e8
+[22]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/quality_analyzing_video_encoder.h;l=54;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[23]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/quality_analyzing_video_decoder.h;l=50;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[24]: /test/network/g3doc/index.md
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md b/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md
new file mode 100644
index 0000000000..67596777f2
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md
@@ -0,0 +1,197 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-02-21'} *-->
+
+# DefaultVideoQualityAnalyzer
+
+## Audience
+
+This document is for users of
+[`webrtc::webrtc_pc_e2e::DefaultVideoQualityAnalyzer`][1].
+
+## Overview
+
+`DefaultVideoQualityAnalyzer` implements
+[`webrtc::VideoQualityAnalyzerInterface`][2] and is a main
+implementation of video quality analyzer for WebRTC. To operate correctly it
+requires to receive video frame on each step:
+
+1. On frame captured - analyzer will generate a unique ID for the frame, that
+ caller should attach to the it.
+2. Immediately before frame enter the encoder.
+3. Immediately after the frame was encoded.
+4. After the frame was received and immediately before it entered the decoder.
+5. Immediately after the frame was decoded.
+6. When the frame was rendered.
+
+![VideoQualityAnalyzerInterface pipeline](video_quality_analyzer_pipeline.png "VideoQualityAnalyzerInterface pipeline")
+
+The analyzer updates its internal metrics per frame when it was rendered and
+reports all of them after it was stopped through
+[WebRTC perf results reporting system][10].
+
+To properly inject `DefaultVideoQualityAnalyzer` into pipeline the following helpers can be used:
+
+### VideoQualityAnalyzerInjectionHelper
+
+[`webrtc::webrtc_pc_e2e::VideoQualityAnalyzerInjectionHelper`][3] provides
+factory methods for components, that will be used to inject
+`VideoQualityAnalyzerInterface` into the `PeerConnection` pipeline:
+
+* Wrappers for [`webrtc::VideoEncoderFactory`][4] and
+ [`webrtc::VideoDecodeFactory`][5] which will properly pass
+ [`webrtc::VideoFrame`][6] and [`webrtc::EncodedImage`][7] into analyzer
+ before and after real video encode and decoder.
+* [`webrtc::test::TestVideoCapturer::FramePreprocessor`][8] which is used to
+ pass generated frames into analyzer on capturing and then set the returned
+ frame ID. It also configures dumping of captured frames if requried.
+* [`rtc::VideoSinkInterface<VideoFrame>`][9] which is used to pass frames to
+ the analyzer before they will be rendered to compute per frame metrics. It
+ also configures dumping of rendered video if requried.
+
+Besides factories `VideoQualityAnalyzerInjectionHelper` has method to
+orchestrate `VideoQualityAnalyzerInterface` workflow:
+
+* `Start` - to start video analyzer, so it will be able to receive and analyze
+ video frames.
+* `RegisterParticipantInCall` - to add new participants after analyzer was
+ started.
+* `Stop` - to stop analyzer, compute all metrics for frames that were recevied
+ before and report them.
+
+Also `VideoQualityAnalyzerInjectionHelper` implements
+[`webrtc::webrtc_pc_e2e::StatsObserverInterface`][11] to propagate WebRTC stats
+to `VideoQualityAnalyzerInterface`.
+
+### EncodedImageDataInjector and EncodedImageDataExtractor
+
+[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][14] and
+[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][15] are used to inject and
+extract data into `webrtc::EncodedImage` to propagate frame ID and other
+required information through the network.
+
+By default [`webrtc::webrtc_pc_e2e::SingleProcessEncodedImageDataInjector`][16]
+is used. It assumes `webrtc::EncodedImage` payload as black box which is
+remaining unchanged from encoder to decoder and stores the information required
+for its work in the last 3 bytes of the payload, replacing the original data
+during injection and restoring it back during extraction. Also
+`SingleProcessEncodedImageDataInjector` requires that sender and receiver were
+inside single process.
+
+![SingleProcessEncodedImageDataInjector](single_process_encoded_image_data_injector.png "SingleProcessEncodedImageDataInjector")
+
+## Exported metrics
+
+Exported metrics are reported to WebRTC perf results reporting system.
+
+### General
+
+* *`cpu_usage`* - CPU usage excluding video analyzer
+
+### Video
+
+* *`psnr`* - peak signal-to-noise ratio:
+ [wikipedia](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio)
+* *`ssim`* - structural similarity:
+ [wikipedia](https://en.wikipedia.org/wiki/Structural_similarity).
+* *`min_psnr`* - minimum value of psnr across all frames of video stream.
+* *`encode_time`* - time to encode a single frame.
+* *`decode_time`* - time to decode a single frame.
+* *`transport_time`* - time from frame encoded to frame received for decoding.
+* *`receive_to_render_time`* - time from frame received for decoding to frame
+ rendered.
+* *`total_delay_incl_transport`* - time from frame was captured on device to
+ time when frame was displayed on device.
+* *`encode_frame_rate`* - frame rate after encoder.
+* *`harmonic_framerate`* - video duration divided on squared sum of interframe
+ delays. Reflects render frame rate penalized by freezes.
+* *`time_between_rendered_frames`* - time between frames out to renderer.
+* *`dropped_frames`* - amount of frames that were sent, but weren't rendered
+ and are known not to be “on the way” from sender to receiver.
+
+Freeze is a pause when no new frames from decoder arrived for 150ms + avg time
+between frames or 3 * avg time between frames.
+
+* *`time_between_freezes`* - mean time from previous freeze end to new freeze
+ start.
+* *`freeze_time_ms`* - total freeze time in ms.
+* *`max_skipped`* - frames skipped between two nearest rendered.
+* *`pixels_per_frame`* - amount of pixels on frame (width * height).
+* *`target_encode_bitrate`* - target encode bitrate provided by BWE to
+ encoder.
+* *`actual_encode_bitrate -`* - actual encode bitrate produced by encoder.
+* *`available_send_bandwidth -`* - available send bandwidth estimated by BWE.
+* *`transmission_bitrate`* - bitrate of media in the emulated network, not
+ counting retransmissions FEC, and RTCP messages
+* *`retransmission_bitrate`* - bitrate of retransmission streams only.
+
+### Framework stability
+
+* *`frames_in_flight`* - amount of frames that were captured but wasn't seen
+ on receiver.
+
+## Debug metrics
+
+Debug metrics are not reported to WebRTC perf results reporting system, but are
+available through `DefaultVideoQualityAnalyzer` API.
+
+### [FrameCounters][12]
+
+Frame counters consist of next counters:
+
+* *`captured`* - count of frames, that were passed into WebRTC pipeline by
+ video stream source
+* *`pre_encoded`* - count of frames that reached video encoder.
+* *`encoded`* - count of encoded images that were produced by encoder for all
+ requested spatial layers and simulcast streams.
+* *`received`* - count of encoded images received in decoder for all requested
+ spatial layers and simulcast streams.
+* *`decoded`* - count of frames that were produced by decoder.
+* *`rendered`* - count of frames that went out from WebRTC pipeline to video
+ sink.
+* *`dropped`* - count of frames that were dropped in any point between
+ capturing and rendering.
+
+`DefaultVideoQualityAnalyzer` exports these frame counters:
+
+* *`GlobalCounters`* - frame counters for frames met on each stage of analysis
+ for all media streams.
+* *`PerStreamCounters`* - frame counters for frames met on each stage of
+ analysis separated per individual video track (single media section in the
+ SDP offer).
+
+### [AnalyzerStats][13]
+
+Contains metrics about internal state of video analyzer during its work
+
+* *`comparisons_queue_size`* - size of analyzer internal queue used to perform
+ captured and rendered frames comparisons measured when new element is added
+ to the queue.
+* *`comparisons_done`* - number of performed comparisons of 2 video frames
+ from captured and rendered streams.
+* *`cpu_overloaded_comparisons_done`* - number of cpu overloaded comparisons.
+ Comparison is cpu overloaded if it is queued when there are too many not
+ processed comparisons in the queue. Overloaded comparison doesn't include
+ metrics like SSIM and PSNR that require heavy computations.
+* *`memory_overloaded_comparisons_done`* - number of memory overloaded
+ comparisons. Comparison is memory overloaded if it is queued when its
+ captured frame was already removed due to high memory usage for that video
+ stream.
+* *`frames_in_flight_left_count`* - count of frames in flight in analyzer
+ measured when new comparison is added and after analyzer was stopped.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=188;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[2]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/video_quality_analyzer_interface.h;l=56;drc=d7808f1c464a07c8f1e2f97ec7ee92fda998d590
+[3]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/video_quality_analyzer_injection_helper.h;l=39;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[4]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[5]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_decoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[6]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_frame.h;l=30;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[7]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/encoded_image.h;l=71;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[8]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_video_capturer.h;l=28;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[9]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_sink_interface.h;l=19;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[10]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[11]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/stats_observer_interface.h;l=21;drc=9b526180c9e9722d3fc7f8689da6ec094fc7fc0a
+[12]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=57;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[13]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=113;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[14]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=23;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
+[15]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=46;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
+[16]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/single_process_encoded_image_data_injector.h;l=40;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png b/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png
new file mode 100644
index 0000000000..3f36725727
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png
Binary files differ
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/index.md b/third_party/libwebrtc/test/pc/e2e/g3doc/index.md
new file mode 100644
index 0000000000..678262bb2b
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/index.md
@@ -0,0 +1,224 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-04-12'} *-->
+
+# PeerConnection Level Framework
+
+## API
+
+* [Fixture][1]
+* [Fixture factory function][2]
+
+## Documentation
+
+The PeerConnection level framework is designed for end-to-end media quality
+testing through the PeerConnection level public API. The framework uses the
+*Unified plan* API to generate offers/answers during the signaling phase. The
+framework also wraps the video encoder/decoder and inject it into
+*`webrtc::PeerConnection`* to measure video quality, performing 1:1 frames
+matching between captured and rendered frames without any extra requirements to
+input video. For audio quality evaluation the standard `GetStats()` API from
+PeerConnection is used.
+
+The framework API is located in the namespace *`webrtc::webrtc_pc_e2e`*.
+
+### Supported features
+
+* Single or bidirectional media in the call
+* RTC Event log dump per peer
+* AEC dump per peer
+* Compatible with *`webrtc::TimeController`* for both real and simulated time
+* Media
+ * AV sync
+* Video
+ * Any amount of video tracks both from caller and callee sides
+ * Input video from
+ * Video generator
+ * Specified file
+ * Any instance of *`webrtc::test::FrameGeneratorInterface`*
+ * Dumping of captured/rendered video into file
+ * Screen sharing
+ * Vp8 simulcast from caller side
+ * Vp9 SVC from caller side
+ * Choosing of video codec (name and parameters), having multiple codecs
+ negotiated to support codec-switching testing.
+ * FEC (ULP or Flex)
+ * Forced codec overshooting (for encoder overshoot emulation on some
+ mobile devices, when hardware encoder can overshoot target bitrate)
+* Audio
+ * Up to 1 audio track both from caller and callee sides
+ * Generated audio
+ * Audio from specified file
+ * Dumping of captured/rendered audio into file
+ * Parameterizing of `cricket::AudioOptions`
+ * Echo emulation
+* Injection of various WebRTC components into underlying
+ *`webrtc::PeerConnection`* or *`webrtc::PeerConnectionFactory`*. You can see
+ the full list [here][11]
+* Scheduling of events, that can happen during the test, for example:
+ * Changes in network configuration
+ * User statistics measurements
+ * Custom defined actions
+* User defined statistics reporting via
+ *`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter`*
+ interface
+
+## Exported metrics
+
+### General
+
+* *`<peer_name>_connected`* - peer successfully established connection to
+ remote side
+* *`cpu_usage`* - CPU usage excluding video analyzer
+* *`audio_ahead_ms`* - Used to estimate how much audio and video is out of
+ sync when the two tracks were from the same source. Stats are polled
+ periodically during a call. The metric represents how much earlier was audio
+ played out on average over the call. If, during a stats poll, video is
+ ahead, then audio_ahead_ms will be equal to 0 for this poll.
+* *`video_ahead_ms`* - Used to estimate how much audio and video is out of
+ sync when the two tracks were from the same source. Stats are polled
+ periodically during a call. The metric represents how much earlier was video
+ played out on average over the call. If, during a stats poll, audio is
+ ahead, then video_ahead_ms will be equal to 0 for this poll.
+
+### Video
+
+See documentation for
+[*`DefaultVideoQualityAnalyzer`*](default_video_quality_analyzer.md#exported-metrics)
+
+### Audio
+
+* *`accelerate_rate`* - when playout is sped up, this counter is increased by
+ the difference between the number of samples received and the number of
+ samples played out. If speedup is achieved by removing samples, this will be
+ the count of samples removed. Rate is calculated as difference between
+ nearby samples divided on sample interval.
+* *`expand_rate`* - the total number of samples that are concealed samples
+ over time. A concealed sample is a sample that was replaced with synthesized
+ samples generated locally before being played out. Examples of samples that
+ have to be concealed are samples from lost packets or samples from packets
+ that arrive too late to be played out
+* *`speech_expand_rate`* - the total number of samples that are concealed
+ samples minus the total number of concealed samples inserted that are
+ "silent" over time. Playing out silent samples results in silence or comfort
+ noise.
+* *`preemptive_rate`* - when playout is slowed down, this counter is increased
+ by the difference between the number of samples received and the number of
+ samples played out. If playout is slowed down by inserting samples, this
+ will be the number of inserted samples. Rate is calculated as difference
+ between nearby samples divided on sample interval.
+* *`average_jitter_buffer_delay_ms`* - average size of NetEQ jitter buffer.
+* *`preferred_buffer_size_ms`* - preferred size of NetEQ jitter buffer.
+* *`visqol_mos`* - proxy for audio quality itself.
+* *`asdm_samples`* - measure of how much acceleration/deceleration was in the
+ signal.
+* *`word_error_rate`* - measure of how intelligible the audio was (percent of
+ words that could not be recognized in output audio).
+
+### Network
+
+* *`bytes_sent`* - represents the total number of payload bytes sent on this
+ PeerConnection, i.e., not including headers or padding
+* *`packets_sent`* - represents the total number of packets sent over this
+ PeerConnection’s transports.
+* *`average_send_rate`* - average send rate calculated on bytes_sent divided
+ by test duration.
+* *`payload_bytes_sent`* - total number of bytes sent for all SSRC plus total
+ number of RTP header and padding bytes sent for all SSRC. This does not
+ include the size of transport layer headers such as IP or UDP.
+* *`sent_packets_loss`* - packets_sent minus corresponding packets_received.
+* *`bytes_received`* - represents the total number of bytes received on this
+ PeerConnection, i.e., not including headers or padding.
+* *`packets_received`* - represents the total number of packets received on
+ this PeerConnection’s transports.
+* *`average_receive_rate`* - average receive rate calculated on bytes_received
+ divided by test duration.
+* *`payload_bytes_received`* - total number of bytes received for all SSRC
+ plus total number of RTP header and padding bytes received for all SSRC.
+ This does not include the size of transport layer headers such as IP or UDP.
+
+### Framework stability
+
+* *`frames_in_flight`* - amount of frames that were captured but wasn't seen
+ on receiver in the way that also all frames after also weren't seen on
+ receiver.
+* *`bytes_discarded_no_receiver`* - total number of bytes that were received
+ on network interfaces related to the peer, but destination port was closed.
+* *`packets_discarded_no_receiver`* - total number of packets that were
+ received on network interfaces related to the peer, but destination port was
+ closed.
+
+## Examples
+
+Examples can be found in
+
+* [peer_connection_e2e_smoke_test.cc][3]
+* [pc_full_stack_tests.cc][4]
+
+## Stats plotting
+
+### Description
+
+Stats plotting provides ability to plot statistic collected during the test.
+Right now it is used in PeerConnection level framework and give ability to see
+how video quality metrics changed during test execution.
+
+### Usage
+
+To make any metrics plottable you need:
+
+1. Collect metric data with [SamplesStatsCounter][5] which internally will
+ store all intermediate points and timestamps when these points were added.
+2. Then you need to report collected data with
+ [`webrtc::test::PrintResult(...)`][6]. By using these method you will also
+ specify name of the plottable metric.
+
+After these steps it will be possible to export your metric for plotting. There
+are several options how you can do this:
+
+1. Use [`webrtc::TestMain::Create()`][7] as `main` function implementation, for
+ example use [`test/test_main.cc`][8] as `main` function for your test.
+
+ In such case your binary will have flag `--plot`, where you can provide a
+ list of metrics, that you want to plot or specify `all` to plot all
+ available metrics.
+
+ If `--plot` is specified, the binary will output metrics data into `stdout`.
+ Then you need to pipe this `stdout` into python plotter script
+ [`rtc_tools/metrics_plotter.py`][9], which will plot data.
+
+ Examples:
+
+ ```shell
+ $ ./out/Default/test_support_unittests \
+ --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
+ --nologs \
+ --plot=all \
+ | python rtc_tools/metrics_plotter.py
+ ```
+
+ ```shell
+ $ ./out/Default/test_support_unittests \
+ --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
+ --nologs \
+ --plot=psnr,ssim \
+ | python rtc_tools/metrics_plotter.py
+ ```
+
+ Example chart: ![PSNR changes during the test](in_test_psnr_plot.png)
+
+2. Use API from [`test/testsupport/perf_test.h`][10] directly by invoking
+ `webrtc::test::PrintPlottableResults(const std::vector<std::string>&
+ desired_graphs)` to print plottable metrics to stdout. Then as in previous
+ option you need to pipe result into plotter script.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[2]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/create_peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[3]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/peer_connection_e2e_smoke_test.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[4]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/pc_full_stack_tests.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[5]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/numerics/samples_stats_counter.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[6]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/testsupport/perf_test.h;l=86;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[7]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/test_main_lib.h;l=23;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
+[8]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/test_main.cc;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
+[9]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/rtc_tools/metrics_plotter.py;drc=8cc6695652307929edfc877cd64b75cd9ec2d615
+[10]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/testsupport/perf_test.h;l=105;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[11]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=272;drc=484acf27231d931dbc99aedce85bc27e06486b96
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png b/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png
new file mode 100644
index 0000000000..73480bafbe
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png
Binary files differ
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png b/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png
new file mode 100644
index 0000000000..6cddb91110
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png
Binary files differ
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png b/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png
new file mode 100644
index 0000000000..c7eaa04c0e
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png
Binary files differ