Adding upstream version 1:115.7.0.upstream/1%115.7.0upstream

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

7 files changed, 630 insertions, 0 deletions

diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md b/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md
new file mode 100644
index 0000000000..1b68c6db2c
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/architecture.md
@@ -0,0 +1,209 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-04-12'} *-->
+
+# PeerConnection level framework fixture architecture
+
+## Overview
+
+The main implementation of
+[`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture`][1] is
+[`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTest`][2]. Internally it owns
+the next main pieces:
+
+*   [`MediaHelper`][3] - responsible for adding audio and video tracks to the
+    peers.
+*   [`VideoQualityAnalyzerInjectionHelper`][4] and
+    [`SingleProcessEncodedImageDataInjector`][5] - used to inject video quality
+    analysis and properly match captured and rendered video frames. You can read
+    more about it in
+    [DefaultVideoQualityAnalyzer](default_video_quality_analyzer.md) section.
+*   [`AudioQualityAnalyzerInterface`][6] - used to measure audio quality metrics
+*   [`TestActivitiesExecutor`][7] - used to support [`ExecuteAt(...)`][8] and
+    [`ExecuteEvery(...)`][9] API of `PeerConnectionE2EQualityTestFixture` to run
+    any arbitrary action during test execution timely synchronized with a test
+    call.
+*   A vector of [`QualityMetricsReporter`][10] added by the
+    `PeerConnectionE2EQualityTestFixture` user.
+*   Two peers: Alice and Bob represented by instances of [`TestPeer`][11]
+    object.
+
+Also it keeps a reference to [`webrtc::TimeController`][12], which is used to
+create all required threads, task queues, task queue factories and time related
+objects.
+
+## TestPeer
+
+Call participants are represented by instances of `TestPeer` object.
+[`TestPeerFactory`][13] is used to create them. `TestPeer` owns all instances
+related to the `webrtc::PeerConnection`, including required listeners and
+callbacks. Also it provides an API to do offer/answer exchange and ICE candidate
+exchange. For this purposes internally it uses an instance of
+[`webrtc::PeerConnectionWrapper`][14].
+
+The `TestPeer` also owns the `PeerConnection` worker thread. The signaling
+thread for all `PeerConnection`'s is owned by
+`PeerConnectionE2EQualityTestFixture` and shared between all participants in the
+call. The network thread is owned by the network layer (it maybe either emulated
+network provided by [Network Emulation Framework][24] or network thread and
+`rtc::NetworkManager` provided by user) and provided when peer is added to the
+fixture via [`AddPeer(...)`][15] API.
+
+## GetStats API based metrics reporters
+
+`PeerConnectionE2EQualityTestFixture` gives the user ability to provide
+different `QualityMetricsReporter`s which will listen for `PeerConnection`
+[`GetStats`][16] API. Then such reporters will be able to report various metrics
+that user wants to measure.
+
+`PeerConnectionE2EQualityTestFixture` itself also uses this mechanism to
+measure:
+
+*   Audio quality metrics
+*   Audio/Video sync metrics (with help of [`CrossMediaMetricsReporter`][17])
+
+Also framework provides a [`StatsBasedNetworkQualityMetricsReporter`][18] to
+measure network related WebRTC metrics and print debug raw emulated network
+statistic. This reporter should be added by user via
+[`AddQualityMetricsReporter(...)`][19] API if requried.
+
+Internally stats gathering is done by [`StatsPoller`][20]. Stats are requested
+once per second for each `PeerConnection` and then resulted object is provided
+into each stats listener.
+
+## Offer/Answer exchange
+
+`PeerConnectionE2EQualityTest` provides ability to test Simulcast and SVC for
+video. These features aren't supported by P2P call and in general requires a
+Selective Forwarding Unit (SFU). So special logic is applied to mimic SFU
+behavior in P2P call. This logic is located inside [`SignalingInterceptor`][21],
+[`QualityAnalyzingVideoEncoder`][22] and [`QualityAnalyzingVideoDecoder`][23]
+and consist of SDP modification during offer/answer exchange and special
+handling of video frames from unrelated Simulcast/SVC streams during decoding.
+
+### Simulcast
+
+In case of Simulcast we have a video track, which internally contains multiple
+video streams, for example low resolution, medium resolution and high
+resolution. WebRTC client doesn't support receiving an offer with multiple
+streams in it, because usually SFU will keep only single stream for the client.
+To bypass it framework will modify offer by converting a single track with three
+video streams into three independent video tracks. Then sender will think that
+it send simulcast, but receiver will think that it receives 3 independent
+tracks.
+
+To achieve such behavior some extra tweaks are required:
+
+*   MID RTP header extension from original offer have to be removed
+*   RID RTP header extension from original offer is replaced with MID RTP header
+    extension, so the ID that sender uses for RID on receiver will be parsed as
+    MID.
+*   Answer have to be modified in the opposite way.
+
+Described modifications are illustrated on the picture below.
+
+![VP8 Simulcast offer modification](vp8_simulcast_offer_modification.png "VP8 Simulcast offer modification")
+
+The exchange will look like this:
+
+1.  Alice creates an offer
+2.  Alice sets offer as local description
+3.  Do described offer modification
+4.  Alice sends modified offer to Bob
+5.  Bob sets modified offer as remote description
+6.  Bob creates answer
+7.  Bob sets answer as local description
+8.  Do reverse modifications on answer
+9.  Bob sends modified answer to Alice
+10. Alice sets modified answer as remote description
+
+Such mechanism put a constraint that RTX streams are not supported, because they
+don't have RID RTP header extension in their packets.
+
+### SVC
+
+In case of SVC the framework will update the sender's offer before even setting
+it as local description on the sender side. Then no changes to answer will be
+required.
+
+`ssrc` is a 32 bit random value that is generated in RTP to denote a specific
+source used to send media in an RTP connection. In original offer video track
+section will look like this:
+
+```
+m=video 9 UDP/TLS/RTP/SAVPF 98 100 99 101
+...
+a=ssrc-group:FID <primary ssrc> <retransmission ssrc>
+a=ssrc:<primary ssrc> cname:...
+....
+a=ssrc:<retransmission ssrc> cname:...
+....
+```
+
+To enable SVC for such video track framework will add extra `ssrc`s for each SVC
+stream that is required like this:
+
+```
+a=ssrc-group:FID <Low resolution primary ssrc> <Low resolution retransmission ssrc>
+a=ssrc:<Low resolution primary ssrc> cname:...
+....
+a=ssrc:<Low resolution retransmission ssrc> cname:....
+...
+a=ssrc-group:FID <Medium resolution primary ssrc> <Medium resolution retransmission ssrc>
+a=ssrc:<Medium resolution primary ssrc> cname:...
+....
+a=ssrc:<Medium resolution retransmission ssrc> cname:....
+...
+a=ssrc-group:FID <High resolution primary ssrc> <High resolution retransmission ssrc>
+a=ssrc:<High resolution primary ssrc> cname:...
+....
+a=ssrc:<High resolution retransmission ssrc> cname:....
+...
+```
+
+The next line will also be added to the video track section of the offer:
+
+```
+a=ssrc-group:SIM <Low resolution primary ssrc> <Medium resolution primary ssrc> <High resolution primary ssrc>
+```
+
+It will tell PeerConnection that this track should be configured as SVC. It
+utilize WebRTC Plan B offer structure to achieve SVC behavior, also it modifies
+offer before setting it as local description which violates WebRTC standard.
+Also it adds limitations that on lossy networks only top resolution streams can
+be analyzed, because WebRTC won't try to restore low resolution streams in case
+of loss, because it still receives higher stream.
+
+### Handling in encoder/decoder
+
+In the encoder, the framework for each encoded video frame will propagate
+information requried for the fake SFU to know if it belongs to an interesting
+simulcast stream/spatial layer of if it should be "discarded".
+
+On the decoder side frames that should be "discarded" by fake SFU will be auto
+decoded into single pixel images and only the interesting simulcast
+stream/spatial layer will go into real decoder and then will be analyzed.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=55;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[2]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/peer_connection_quality_test.h;l=44;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[3]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/media/media_helper.h;l=27;drc=d46db9f1523ae45909b4a6fdc90a140443068bc6
+[4]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/video_quality_analyzer_injection_helper.h;l=38;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[5]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/single_process_encoded_image_data_injector.h;l=40;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[6]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/audio_quality_analyzer_interface.h;l=23;drc=20f45823e37fd7272aa841831c029c21f29742c2
+[7]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[8]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=439;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[9]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=445;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[10]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=413;drc=9438fb3fff97c803d1ead34c0e4f223db168526f
+[11]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[12]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_activities_executor.h;l=28;drc=6cc893ad778a0965e2b7a8e614f3c98aa81bee5b
+[13]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/test_peer_factory.h;l=46;drc=0ef4a2488a466a24ab97b31fdddde55440d451f9
+[14]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/pc/peer_connection_wrapper.h;l=47;drc=5ab79e62f691875a237ea28ca3975ea1f0ed62ec
+[15]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=459;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[16]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/peer_connection_interface.h;l=886;drc=9438fb3fff97c803d1ead34c0e4f223db168526f
+[17]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/cross_media_metrics_reporter.h;l=29;drc=9d777620236ec76754cfce19f6e82dd18e52d22c
+[18]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/cross_media_metrics_reporter.h;l=29;drc=9d777620236ec76754cfce19f6e82dd18e52d22c
+[19]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=450;drc=484acf27231d931dbc99aedce85bc27e06486b96
+[20]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/stats_poller.h;l=52;drc=9b526180c9e9722d3fc7f8689da6ec094fc7fc0a
+[21]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/sdp/sdp_changer.h;l=79;drc=ee558dcca89fd8b105114ededf9e74d948da85e8
+[22]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/quality_analyzing_video_encoder.h;l=54;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[23]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/analyzer/video/quality_analyzing_video_decoder.h;l=50;drc=79020414fd5c71f9ec1f25445ea5f1c8001e1a49
+[24]: /test/network/g3doc/index.md
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md b/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md
new file mode 100644
index 0000000000..67596777f2
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/default_video_quality_analyzer.md
@@ -0,0 +1,197 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-02-21'} *-->
+
+# DefaultVideoQualityAnalyzer
+
+## Audience
+
+This document is for users of
+[`webrtc::webrtc_pc_e2e::DefaultVideoQualityAnalyzer`][1].
+
+## Overview
+
+`DefaultVideoQualityAnalyzer` implements
+[`webrtc::VideoQualityAnalyzerInterface`][2] and is a main
+implementation of video quality analyzer for WebRTC. To operate correctly it
+requires to receive video frame on each step:
+
+1.  On frame captured - analyzer will generate a unique ID for the frame, that
+    caller should attach to the it.
+2.  Immediately before frame enter the encoder.
+3.  Immediately after the frame was encoded.
+4.  After the frame was received and immediately before it entered the decoder.
+5.  Immediately after the frame was decoded.
+6.  When the frame was rendered.
+
+![VideoQualityAnalyzerInterface pipeline](video_quality_analyzer_pipeline.png "VideoQualityAnalyzerInterface pipeline")
+
+The analyzer updates its internal metrics per frame when it was rendered and
+reports all of them after it was stopped through
+[WebRTC perf results reporting system][10].
+
+To properly inject `DefaultVideoQualityAnalyzer` into pipeline the following helpers can be used:
+
+### VideoQualityAnalyzerInjectionHelper
+
+[`webrtc::webrtc_pc_e2e::VideoQualityAnalyzerInjectionHelper`][3] provides
+factory methods for components, that will be used to inject
+`VideoQualityAnalyzerInterface` into the `PeerConnection` pipeline:
+
+*   Wrappers for [`webrtc::VideoEncoderFactory`][4] and
+    [`webrtc::VideoDecodeFactory`][5] which will properly pass
+    [`webrtc::VideoFrame`][6] and [`webrtc::EncodedImage`][7] into analyzer
+    before and after real video encode and decoder.
+*   [`webrtc::test::TestVideoCapturer::FramePreprocessor`][8] which is used to
+    pass generated frames into analyzer on capturing and then set the returned
+    frame ID. It also configures dumping of captured frames if requried.
+*   [`rtc::VideoSinkInterface<VideoFrame>`][9] which is used to pass frames to
+    the analyzer before they will be rendered to compute per frame metrics. It
+    also configures dumping of rendered video if requried.
+
+Besides factories `VideoQualityAnalyzerInjectionHelper` has method to
+orchestrate `VideoQualityAnalyzerInterface` workflow:
+
+*   `Start` - to start video analyzer, so it will be able to receive and analyze
+    video frames.
+*   `RegisterParticipantInCall` - to add new participants after analyzer was
+    started.
+*   `Stop` - to stop analyzer, compute all metrics for frames that were recevied
+    before and report them.
+
+Also `VideoQualityAnalyzerInjectionHelper` implements
+[`webrtc::webrtc_pc_e2e::StatsObserverInterface`][11] to propagate WebRTC stats
+to `VideoQualityAnalyzerInterface`.
+
+### EncodedImageDataInjector and EncodedImageDataExtractor
+
+[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][14] and
+[`webrtc::webrtc_pc_e2e::EncodedImageDataInjector`][15] are used to inject and
+extract data into `webrtc::EncodedImage` to propagate frame ID and other
+required information through the network.
+
+By default [`webrtc::webrtc_pc_e2e::SingleProcessEncodedImageDataInjector`][16]
+is used. It assumes `webrtc::EncodedImage` payload as black box which is
+remaining unchanged from encoder to decoder and stores the information required
+for its work in the last 3 bytes of the payload, replacing the original data
+during injection and restoring it back during extraction. Also
+`SingleProcessEncodedImageDataInjector` requires that sender and receiver were
+inside single process.
+
+![SingleProcessEncodedImageDataInjector](single_process_encoded_image_data_injector.png "SingleProcessEncodedImageDataInjector")
+
+## Exported metrics
+
+Exported metrics are reported to WebRTC perf results reporting system.
+
+### General
+
+*   *`cpu_usage`* - CPU usage excluding video analyzer
+
+### Video
+
+*   *`psnr`* - peak signal-to-noise ratio:
+    [wikipedia](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio)
+*   *`ssim`* - structural similarity:
+    [wikipedia](https://en.wikipedia.org/wiki/Structural_similarity).
+*   *`min_psnr`* - minimum value of psnr across all frames of video stream.
+*   *`encode_time`* - time to encode a single frame.
+*   *`decode_time`* - time to decode a single frame.
+*   *`transport_time`* - time from frame encoded to frame received for decoding.
+*   *`receive_to_render_time`* - time from frame received for decoding to frame
+    rendered.
+*   *`total_delay_incl_transport`* - time from frame was captured on device to
+    time when frame was displayed on device.
+*   *`encode_frame_rate`* - frame rate after encoder.
+*   *`harmonic_framerate`* - video duration divided on squared sum of interframe
+    delays. Reflects render frame rate penalized by freezes.
+*   *`time_between_rendered_frames`* - time between frames out to renderer.
+*   *`dropped_frames`* - amount of frames that were sent, but weren't rendered
+    and are known not to be “on the way” from sender to receiver.
+
+Freeze is a pause when no new frames from decoder arrived for 150ms + avg time
+between frames or 3 * avg time between frames.
+
+*   *`time_between_freezes`* - mean time from previous freeze end to new freeze
+    start.
+*   *`freeze_time_ms`* - total freeze time in ms.
+*   *`max_skipped`* - frames skipped between two nearest rendered.
+*   *`pixels_per_frame`* - amount of pixels on frame (width * height).
+*   *`target_encode_bitrate`* - target encode bitrate provided by BWE to
+    encoder.
+*   *`actual_encode_bitrate -`* - actual encode bitrate produced by encoder.
+*   *`available_send_bandwidth -`* - available send bandwidth estimated by BWE.
+*   *`transmission_bitrate`* - bitrate of media in the emulated network, not
+    counting retransmissions FEC, and RTCP messages
+*   *`retransmission_bitrate`* - bitrate of retransmission streams only.
+
+### Framework stability
+
+*   *`frames_in_flight`* - amount of frames that were captured but wasn't seen
+    on receiver.
+
+## Debug metrics
+
+Debug metrics are not reported to WebRTC perf results reporting system, but are
+available through `DefaultVideoQualityAnalyzer` API.
+
+### [FrameCounters][12]
+
+Frame counters consist of next counters:
+
+*   *`captured`* - count of frames, that were passed into WebRTC pipeline by
+    video stream source
+*   *`pre_encoded`* - count of frames that reached video encoder.
+*   *`encoded`* - count of encoded images that were produced by encoder for all
+    requested spatial layers and simulcast streams.
+*   *`received`* - count of encoded images received in decoder for all requested
+    spatial layers and simulcast streams.
+*   *`decoded`* - count of frames that were produced by decoder.
+*   *`rendered`* - count of frames that went out from WebRTC pipeline to video
+    sink.
+*   *`dropped`* - count of frames that were dropped in any point between
+    capturing and rendering.
+
+`DefaultVideoQualityAnalyzer` exports these frame counters:
+
+*   *`GlobalCounters`* - frame counters for frames met on each stage of analysis
+    for all media streams.
+*   *`PerStreamCounters`* - frame counters for frames met on each stage of
+    analysis separated per individual video track (single media section in the
+    SDP offer).
+
+### [AnalyzerStats][13]
+
+Contains metrics about internal state of video analyzer during its work
+
+*   *`comparisons_queue_size`* - size of analyzer internal queue used to perform
+    captured and rendered frames comparisons measured when new element is added
+    to the queue.
+*   *`comparisons_done`* - number of performed comparisons of 2 video frames
+    from captured and rendered streams.
+*   *`cpu_overloaded_comparisons_done`* - number of cpu overloaded comparisons.
+    Comparison is cpu overloaded if it is queued when there are too many not
+    processed comparisons in the queue. Overloaded comparison doesn't include
+    metrics like SSIM and PSNR that require heavy computations.
+*   *`memory_overloaded_comparisons_done`* - number of memory overloaded
+    comparisons. Comparison is memory overloaded if it is queued when its
+    captured frame was already removed due to high memory usage for that video
+    stream.
+*   *`frames_in_flight_left_count`* - count of frames in flight in analyzer
+    measured when new comparison is added and after analyzer was stopped.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=188;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[2]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/video_quality_analyzer_interface.h;l=56;drc=d7808f1c464a07c8f1e2f97ec7ee92fda998d590
+[3]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/video_quality_analyzer_injection_helper.h;l=39;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[4]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[5]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_decoder_factory.h;l=27;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[6]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_frame.h;l=30;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[7]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/encoded_image.h;l=71;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[8]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_video_capturer.h;l=28;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[9]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_sink_interface.h;l=19;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[10]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[11]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/stats_observer_interface.h;l=21;drc=9b526180c9e9722d3fc7f8689da6ec094fc7fc0a
+[12]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=57;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[13]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/default_video_quality_analyzer.h;l=113;drc=08f46909a8735cf181b99ef2f7e1791c5a7531d2
+[14]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=23;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
+[15]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/encoded_image_data_injector.h;l=46;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
+[16]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/analyzer/video/single_process_encoded_image_data_injector.h;l=40;drc=c57089a97a3df454f4356d882cc8df173e8b3ead
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png b/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png
new file mode 100644
index 0000000000..3f36725727
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/in_test_psnr_plot.png
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/index.md b/third_party/libwebrtc/test/pc/e2e/g3doc/index.md
new file mode 100644
index 0000000000..678262bb2b
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/index.md
@@ -0,0 +1,224 @@
+<!-- go/cmark -->
+<!--* freshness: {owner: 'titovartem' reviewed: '2021-04-12'} *-->
+
+# PeerConnection Level Framework
+
+## API
+
+*   [Fixture][1]
+*   [Fixture factory function][2]
+
+## Documentation
+
+The PeerConnection level framework is designed for end-to-end media quality
+testing through the PeerConnection level public API. The framework uses the
+*Unified plan* API to generate offers/answers during the signaling phase. The
+framework also wraps the video encoder/decoder and inject it into
+*`webrtc::PeerConnection`* to measure video quality, performing 1:1 frames
+matching between captured and rendered frames without any extra requirements to
+input video. For audio quality evaluation the standard `GetStats()` API from
+PeerConnection is used.
+
+The framework API is located in the namespace *`webrtc::webrtc_pc_e2e`*.
+
+### Supported features
+
+*   Single or bidirectional media in the call
+*   RTC Event log dump per peer
+*   AEC dump per peer
+*   Compatible with *`webrtc::TimeController`* for both real and simulated time
+*   Media
+    *   AV sync
+*   Video
+    *   Any amount of video tracks both from caller and callee sides
+    *   Input video from
+        *   Video generator
+        *   Specified file
+        *   Any instance of *`webrtc::test::FrameGeneratorInterface`*
+    *   Dumping of captured/rendered video into file
+    *   Screen sharing
+    *   Vp8 simulcast from caller side
+    *   Vp9 SVC from caller side
+    *   Choosing of video codec (name and parameters), having multiple codecs
+        negotiated to support codec-switching testing.
+    *   FEC (ULP or Flex)
+    *   Forced codec overshooting (for encoder overshoot emulation on some
+        mobile devices, when hardware encoder can overshoot target bitrate)
+*   Audio
+    *   Up to 1 audio track both from caller and callee sides
+    *   Generated audio
+    *   Audio from specified file
+    *   Dumping of captured/rendered audio into file
+    *   Parameterizing of `cricket::AudioOptions`
+    *   Echo emulation
+*   Injection of various WebRTC components into underlying
+    *`webrtc::PeerConnection`* or *`webrtc::PeerConnectionFactory`*. You can see
+    the full list [here][11]
+*   Scheduling of events, that can happen during the test, for example:
+    *   Changes in network configuration
+    *   User statistics measurements
+    *   Custom defined actions
+*   User defined statistics reporting via
+    *`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter`*
+    interface
+
+## Exported metrics
+
+### General
+
+*   *`<peer_name>_connected`* - peer successfully established connection to
+    remote side
+*   *`cpu_usage`* - CPU usage excluding video analyzer
+*   *`audio_ahead_ms`* - Used to estimate how much audio and video is out of
+    sync when the two tracks were from the same source. Stats are polled
+    periodically during a call. The metric represents how much earlier was audio
+    played out on average over the call. If, during a stats poll, video is
+    ahead, then audio_ahead_ms will be equal to 0 for this poll.
+*   *`video_ahead_ms`* - Used to estimate how much audio and video is out of
+    sync when the two tracks were from the same source. Stats are polled
+    periodically during a call. The metric represents how much earlier was video
+    played out on average over the call. If, during a stats poll, audio is
+    ahead, then video_ahead_ms will be equal to 0 for this poll.
+
+### Video
+
+See documentation for
+[*`DefaultVideoQualityAnalyzer`*](default_video_quality_analyzer.md#exported-metrics)
+
+### Audio
+
+*   *`accelerate_rate`* - when playout is sped up, this counter is increased by
+    the difference between the number of samples received and the number of
+    samples played out. If speedup is achieved by removing samples, this will be
+    the count of samples removed. Rate is calculated as difference between
+    nearby samples divided on sample interval.
+*   *`expand_rate`* - the total number of samples that are concealed samples
+    over time. A concealed sample is a sample that was replaced with synthesized
+    samples generated locally before being played out. Examples of samples that
+    have to be concealed are samples from lost packets or samples from packets
+    that arrive too late to be played out
+*   *`speech_expand_rate`* - the total number of samples that are concealed
+    samples minus the total number of concealed samples inserted that are
+    "silent" over time. Playing out silent samples results in silence or comfort
+    noise.
+*   *`preemptive_rate`* - when playout is slowed down, this counter is increased
+    by the difference between the number of samples received and the number of
+    samples played out. If playout is slowed down by inserting samples, this
+    will be the number of inserted samples. Rate is calculated as difference
+    between nearby samples divided on sample interval.
+*   *`average_jitter_buffer_delay_ms`* - average size of NetEQ jitter buffer.
+*   *`preferred_buffer_size_ms`* - preferred size of NetEQ jitter buffer.
+*   *`visqol_mos`* - proxy for audio quality itself.
+*   *`asdm_samples`* - measure of how much acceleration/deceleration was in the
+    signal.
+*   *`word_error_rate`* - measure of how intelligible the audio was (percent of
+    words that could not be recognized in output audio).
+
+### Network
+
+*   *`bytes_sent`* - represents the total number of payload bytes sent on this
+    PeerConnection, i.e., not including headers or padding
+*   *`packets_sent`* - represents the total number of packets sent over this
+    PeerConnection’s transports.
+*   *`average_send_rate`* - average send rate calculated on bytes_sent divided
+    by test duration.
+*   *`payload_bytes_sent`* - total number of bytes sent for all SSRC plus total
+    number of RTP header and padding bytes sent for all SSRC. This does not
+    include the size of transport layer headers such as IP or UDP.
+*   *`sent_packets_loss`* - packets_sent minus corresponding packets_received.
+*   *`bytes_received`* - represents the total number of bytes received on this
+    PeerConnection, i.e., not including headers or padding.
+*   *`packets_received`* - represents the total number of packets received on
+    this PeerConnection’s transports.
+*   *`average_receive_rate`* - average receive rate calculated on bytes_received
+    divided by test duration.
+*   *`payload_bytes_received`* - total number of bytes received for all SSRC
+    plus total number of RTP header and padding bytes received for all SSRC.
+    This does not include the size of transport layer headers such as IP or UDP.
+
+### Framework stability
+
+*   *`frames_in_flight`* - amount of frames that were captured but wasn't seen
+    on receiver in the way that also all frames after also weren't seen on
+    receiver.
+*   *`bytes_discarded_no_receiver`* - total number of bytes that were received
+    on network interfaces related to the peer, but destination port was closed.
+*   *`packets_discarded_no_receiver`* - total number of packets that were
+    received on network interfaces related to the peer, but destination port was
+    closed.
+
+## Examples
+
+Examples can be found in
+
+*   [peer_connection_e2e_smoke_test.cc][3]
+*   [pc_full_stack_tests.cc][4]
+
+## Stats plotting
+
+### Description
+
+Stats plotting provides ability to plot statistic collected during the test.
+Right now it is used in PeerConnection level framework and give ability to see
+how video quality metrics changed during test execution.
+
+### Usage
+
+To make any metrics plottable you need:
+
+1.  Collect metric data with [SamplesStatsCounter][5] which internally will
+    store all intermediate points and timestamps when these points were added.
+2.  Then you need to report collected data with
+    [`webrtc::test::PrintResult(...)`][6]. By using these method you will also
+    specify name of the plottable metric.
+
+After these steps it will be possible to export your metric for plotting. There
+are several options how you can do this:
+
+1.  Use [`webrtc::TestMain::Create()`][7] as `main` function implementation, for
+    example use [`test/test_main.cc`][8] as `main` function for your test.
+
+    In such case your binary will have flag `--plot`, where you can provide a
+    list of metrics, that you want to plot or specify `all` to plot all
+    available metrics.
+
+    If `--plot` is specified, the binary will output metrics data into `stdout`.
+    Then you need to pipe this `stdout` into python plotter script
+    [`rtc_tools/metrics_plotter.py`][9], which will plot data.
+
+    Examples:
+
+    ```shell
+    $ ./out/Default/test_support_unittests \
+      --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
+      --nologs \
+      --plot=all \
+    | python rtc_tools/metrics_plotter.py
+    ```
+
+    ```shell
+    $ ./out/Default/test_support_unittests \
+      --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
+      --nologs \
+      --plot=psnr,ssim \
+    | python rtc_tools/metrics_plotter.py
+    ```
+
+    Example chart: ![PSNR changes during the test](in_test_psnr_plot.png)
+
+2.  Use API from [`test/testsupport/perf_test.h`][10] directly by invoking
+    `webrtc::test::PrintPlottableResults(const std::vector<std::string>&
+    desired_graphs)` to print plottable metrics to stdout. Then as in previous
+    option you need to pipe result into plotter script.
+
+[1]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[2]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/create_peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[3]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/pc/e2e/peer_connection_e2e_smoke_test.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[4]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/pc_full_stack_tests.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[5]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/numerics/samples_stats_counter.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
+[6]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/testsupport/perf_test.h;l=86;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[7]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/test_main_lib.h;l=23;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
+[8]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/test_main.cc;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
+[9]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/rtc_tools/metrics_plotter.py;drc=8cc6695652307929edfc877cd64b75cd9ec2d615
+[10]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/test/testsupport/perf_test.h;l=105;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
+[11]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=272;drc=484acf27231d931dbc99aedce85bc27e06486b96
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png b/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png
new file mode 100644
index 0000000000..73480bafbe
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/single_process_encoded_image_data_injector.png
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png b/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png
new file mode 100644
index 0000000000..6cddb91110
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/video_quality_analyzer_pipeline.png
diff --git a/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png b/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png
new file mode 100644
index 0000000000..c7eaa04c0e
--- /dev/null
+++ b/third_party/libwebrtc/test/pc/e2e/g3doc/vp8_simulcast_offer_modification.png