diff options
Diffstat (limited to 'third_party/libwebrtc/video/g3doc/adaptation.md')
| -rw-r--r-- | third_party/libwebrtc/video/g3doc/adaptation.md | 114 |
1 files changed, 114 insertions, 0 deletions
diff --git a/third_party/libwebrtc/video/g3doc/adaptation.md b/third_party/libwebrtc/video/g3doc/adaptation.md new file mode 100644 index 0000000000..cb06e886b2 --- /dev/null +++ b/third_party/libwebrtc/video/g3doc/adaptation.md @@ -0,0 +1,114 @@ +<!-- go/cmark --> +<!--* freshness: {owner: 'eshr' reviewed: '2021-04-13'} *--> + +# Video Adaptation + +Video adaptation is a mechanism which reduces the bandwidth or CPU consumption +by reducing encoded video quality. + +## Overview + +Adaptation occurs when a _Resource_ signals that it is currently underused or +overused. When overused, the video quality is decreased and when underused, the +video quality is increased. There are currently two dimensions in which the +quality can be adapted: frame-rate and resolution. The dimension that is adapted +is based on the degradation preference for the video track. + +## Resources + +_Resources_ monitor metrics from the system or the video stream. For example, a +resource could monitor system temperature or the bandwidth usage of the video +stream. A resource implements the [Resource][resource.h] interface. When a +resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When +the resource is no longer overused, it can signal this using +`SetUsageState(kUnderuse)`. + +There are two resources that are used by default on all video tracks: Quality +scaler resource and encode overuse resource. + +### QP Scaler Resource + +The quality scaler resource monitors the quantization parameter (QP) of the +encoded video frames for video send stream and ensures that the quality of the +stream is acceptable for the current resolution. After each frame is encoded the +[QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse +or underuse is signalled when the average QP is outside of the +[QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the +_high_ threshold, the QP scaler signals _overuse_, and when below the _low_ +threshold the QP scaler signals _underuse_. + +The thresholds are set by the video encoder in the `scaling_settings` property +of the [EncoderInfo][EncoderInfo]. + +*Note:* that the QP scaler is only enabled when the degradation preference is +`MAINTAIN_FRAMERATE` or `BALANCED`. + +### Encode Usage Resource + +The [encoder usage resource][encode_usage_resource.h] monitors how long it takes +to encode a video frame. This works as a good proxy measurement for CPU usage as +contention increases when CPU usage is high, increasing the encode times of the +video frames. + +The time is tracked from when frame encoding starts to when it is completed. If +the average encoder usage exceeds the thresholds set, *overuse* is triggered. + +### Injecting other Resources + +A custom resource can be injected into the call using the +[Call::AddAdaptationResource][Call::AddAdaptationResource] method. + +## Adaptation + +When a a *resource* signals the it is over or underused, this signal reaches the +`ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the +[VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the +degradation preference of the video stream. `ResourceAdaptationProcessor` will +determine if the `Adaptation` should be applied based on the current adaptation +status and the `Adaptation` proposal. + +### Degradation Preference + +There are 3 degradation preferences, described in the +[RtpParameters][RtpParameters] header. These are + +* `MAINTIAIN_FRAMERATE`: Adapt video resolution +* `MAINTIAIN_RESOLUTION`: Adapt video frame-rate. +* `BALANCED`: Adapt video frame-rate or resolution. + +The degradation preference is set for a video track using the +`degradation_preference` property in the [RtpParameters][RtpParameters]. + +## VideoSinkWants and video stream adaptation + +Once an adaptation is applied it notifies the video stream. The video stream +converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants +indicate to the video stream that some restrictions should be applied to the +stream before it is sent to encoding. It has a few properties, but for +adaptation the properties that might be set are: + +* `target_pixel_count`: The desired number of pixels for each video frame. The + actual pixel count should be close to this but does not have to be exact so + that aspect ratio can be maintained. +* `max_pixel_count`: The maximum number of pixels in each video frame. This + value can not be exceeded if set. +* `max_framerate_fps`: The maximum frame-rate for the video source. The source + is expected to drop frames that cause this threshold to be exceeded. + +The `VideoSinkWants` can be applied by any video source, or one may use the +[AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class +for sources that need video adaptation. + +[RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22 +[resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/adaptation/resource.h +[Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource +[quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/quality_scaler.h +[VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds +[EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo +[encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/encode_usage_resource.h +[VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/video_stream_adapter.h +[adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/adaptation_constraint.h +[bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/bitrate_constraint.h +[AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink +[VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22 +[adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/media/base/adapted_video_track_source.h |