# Video coding in WebRTC ## Introduction to layered video coding [Video coding][video-coding-wiki] is the process of encoding a stream of uncompressed video frames into a compressed bitstream, whose bitrate is lower than that of the original stream. ### Block-based hybrid video coding All video codecs in WebRTC are based on the block-based hybrid video coding paradigm, which entails prediction of the original video frame using either [information from previously encoded frames][motion-compensation-wiki] or information from previously encoded portions of the current frame, subtraction of the prediction from the original video, and [transform][transform-coding-wiki] and [quantization][quantization-wiki] of the resulting difference. The output of the quantization process, quantized transform coefficients, is losslessly [entropy coded][entropy-coding-wiki] along with other encoder parameters (e.g., those related to the prediction process) and then a reconstruction is constructed by inverse quantizing and inverse transforming the quantized transform coefficients and adding the result to the prediction. Finally, in-loop filtering is applied and the resulting reconstruction is stored as a reference frame to be used to develop predictions for future frames. ### Frame types When an encoded frame depends on previously encoded frames (i.e., it has one or more inter-frame dependencies), the prior frames must be available at the receiver before the current frame can be decoded. In order for a receiver to start decoding an encoded bitstream, a frame which has no prior dependencies is required. Such a frame is called a "key frame". For real-time-communications encoding, key frames typically compress less efficiently than "delta frames" (i.e., frames whose predictions are derived from previously encoded frames). ### Single-layer coding In 1:1 calls, the encoded bitstream has a single recipient. Using end-to-end bandwidth estimation, the target bitrate can thus be well tailored for the intended recipient. The number of key frames can be kept to a minimum and the compressability of the stream can be maximized. One way of achiving this is by using "single-layer coding", where each delta frame only depends on the frame that was most recently encoded. ### Scalable video coding In multiway conferences, on the other hand, the encoded bitstream has multiple recipients each of whom may have different downlink bandwidths. In order to tailor the encoded bitstreams to a heterogeneous network of receivers, [scalable video coding][svc-wiki] can be used. The idea is to introduce structure into the dependency graph of the encoded bitstream, such that _layers_ of the full stream can be decoded using only available lower layers. This structure allows for a [selective forwarding unit][sfu-webrtc-glossary] to discard upper layers of the of the bitstream in order to achieve the intended downlink bandwidth. There are multiple types of scalability: * _Temporal scalability_ are layers whose framerate (and bitrate) is lower than that of the upper layer(s) * _Spatial scalability_ are layers whose resolution (and bitrate) is lower than that of the upper layer(s) * _Quality scalability_ are layers whose bitrate is lower than that of the upper layer(s) WebRTC supports temporal scalability for `VP8`, `VP9` and `AV1`, and spatial scalability for `VP9` and `AV1`. ### Simulcast Simulcast is another approach for multiway conferencing, where multiple _independent_ bitstreams are produced by the encoder. In cases where multiple encodings of the same source are required (e.g., uplink transmission in a multiway call), spatial scalability with inter-layer prediction generally offers superior coding efficiency compared with simulcast. When a single encoding is required (e.g., downlink transmission in any call), simulcast generally provides better coding efficiency for the upper spatial layers. The `K-SVC` concept, where spatial inter-layer dependencies are only used to encode key frames, for which inter-layer prediction is typically significantly more effective than it is for delta frames, can be seen as a compromise between full spatial scalability and simulcast. ## Overview of implementation in `modules/video_coding` Given the general introduction to video coding above, we now describe some specifics of the [`modules/video_coding`][modules-video-coding] folder in WebRTC. ### Built-in software codecs in [`modules/video_coding/codecs`][modules-video-coding-codecs] This folder contains WebRTC-specific classes that wrap software codec implementations for different video coding standards: * [libaom][libaom-src] for [AV1][av1-spec] * [libvpx][libvpx-src] for [VP8][vp8-spec] and [VP9][vp9-spec] * [OpenH264][openh264-src] for [H.264 constrained baseline profile][h264-spec] Users of the library can also inject their own codecs, using the [VideoEncoderFactory][video-encoder-factory-interface] and [VideoDecoderFactory][video-decoder-factory-interface] interfaces. This is how platform-supported codecs, such as hardware backed codecs, are implemented. ### Video codec test framework in [`modules/video_coding/codecs/test`][modules-video-coding-codecs-test] This folder contains a test framework that can be used to evaluate video quality performance of different video codec implementations. ### SVC helper classes in [`modules/video_coding/svc`][modules-video-coding-svc] * [`ScalabilityStructure*`][scalabilitystructure] - different [standardized scalability structures][scalability-structure-spec] * [`ScalableVideoController`][scalablevideocontroller] - provides instructions to the video encoder how to create a scalable stream * [`SvcRateAllocator`][svcrateallocator] - bitrate allocation to different spatial and temporal layers ### Utility classes in [`modules/video_coding/utility`][modules-video-coding-utility] * [`FrameDropper`][framedropper] - drops incoming frames when encoder systematically overshoots its target bitrate * [`FramerateController`][frameratecontroller] - drops incoming frames to achieve a target framerate * [`QpParser`][qpparser] - parses the quantization parameter from a bitstream * [`QualityScaler`][qualityscaler] - signals when an encoder generates encoded frames whose quantization parameter is outside the window of acceptable values * [`SimulcastRateAllocator`][simulcastrateallocator] - bitrate allocation to simulcast layers ### General helper classes in [`modules/video_coding`][modules-video-coding] * [`FecControllerDefault`][feccontrollerdefault] - provides a default implementation for rate allocation to [forward error correction][fec-wiki] * [`VideoCodecInitializer`][videocodecinitializer] - converts between different encoder configuration structs ### Receiver buffer classes in [`modules/video_coding`][modules-video-coding] * [`PacketBuffer`][packetbuffer] - (re-)combines RTP packets into frames * [`RtpFrameReferenceFinder`][rtpframereferencefinder] - determines dependencies between frames based on information in the RTP header, payload header and RTP extensions * [`FrameBuffer`][framebuffer] - order frames based on their dependencies to be fed to the decoder [video-coding-wiki]: https://en.wikipedia.org/wiki/Video_coding_format [motion-compensation-wiki]: https://en.wikipedia.org/wiki/Motion_compensation [transform-coding-wiki]: https://en.wikipedia.org/wiki/Transform_coding [motion-vector-wiki]: https://en.wikipedia.org/wiki/Motion_vector [mpeg-wiki]: https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group [svc-wiki]: https://en.wikipedia.org/wiki/Scalable_Video_Coding [sfu-webrtc-glossary]: https://webrtcglossary.com/sfu/ [libvpx-src]: https://chromium.googlesource.com/webm/libvpx/ [libaom-src]: https://aomedia.googlesource.com/aom/ [openh264-src]: https://github.com/cisco/openh264 [vp8-spec]: https://tools.ietf.org/html/rfc6386 [vp9-spec]: https://storage.googleapis.com/downloads.webmproject.org/docs/vp9/vp9-bitstream-specification-v0.6-20160331-draft.pdf [av1-spec]: https://aomediacodec.github.io/av1-spec/ [h264-spec]: https://www.itu.int/rec/T-REC-H.264-201906-I/en [video-encoder-factory-interface]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder_factory.h;l=27;drc=afadfb24a5e608da6ae102b20b0add53a083dcf3 [video-decoder-factory-interface]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_decoder_factory.h;l=27;drc=49c293f03d8f593aa3aca282577fcb14daa63207 [scalability-structure-spec]: https://w3c.github.io/webrtc-svc/#scalabilitymodes* [fec-wiki]: https://en.wikipedia.org/wiki/Error_correction_code#Forward_error_correction [entropy-coding-wiki]: https://en.wikipedia.org/wiki/Entropy_encoding [modules-video-coding]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/ [modules-video-coding-codecs]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/codecs/ [modules-video-coding-codecs-test]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/codecs/test/ [modules-video-coding-svc]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/svc/ [modules-video-coding-utility]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/ [scalabilitystructure]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/svc/create_scalability_structure.h?q=CreateScalabilityStructure [scalablevideocontroller]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/svc/scalable_video_controller.h?q=ScalableVideoController [svcrateallocator]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/svc/svc_rate_allocator.h?q=SvcRateAllocator [framedropper]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/frame_dropper.h?q=FrameDropper [frameratecontroller]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/framerate_controller.h?q=FramerateController [qpparser]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/qp_parser.h?q=QpParser [qualityscaler]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/quality_scaler.h?q=QualityScaler [simulcastrateallocator]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/simulcast_rate_allocator.h?q=SimulcastRateAllocator [feccontrollerdefault]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/fec_controller_default.h?q=FecControllerDefault [videocodecinitializer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/include/video_codec_initializer.h?q=VideoCodecInitializer [packetbuffer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/packet_buffer.h?q=PacketBuffer [rtpframereferencefinder]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/rtp_frame_reference_finder.h?q=RtpFrameReferenceFinder [framebuffer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/frame_buffer.h [quantization-wiki]: https://en.wikipedia.org/wiki/Quantization_(signal_processing)