diff options
Diffstat (limited to 'Documentation/userspace-api/media/v4l/dev-subdev.rst')
| -rw-r--r-- | Documentation/userspace-api/media/v4l/dev-subdev.rst | 669 |
1 files changed, 669 insertions, 0 deletions
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst new file mode 100644 index 0000000000..a4f1df7093 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -0,0 +1,669 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later + +.. _subdev: + +******************** +Sub-device Interface +******************** + +The complex nature of V4L2 devices, where hardware is often made of +several integrated circuits that need to interact with each other in a +controlled way, leads to complex V4L2 drivers. The drivers usually +reflect the hardware model in software, and model the different hardware +components as software blocks called sub-devices. + +V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver +implements the media device API, they will automatically inherit from +media entities. Applications will be able to enumerate the sub-devices +and discover the hardware topology using the media entities, pads and +links enumeration API. + +In addition to make sub-devices discoverable, drivers can also choose to +make them directly configurable by applications. When both the +sub-device driver and the V4L2 device driver support this, sub-devices +will feature a character device node on which ioctls can be called to + +- query, read and write sub-devices controls + +- subscribe and unsubscribe to events and retrieve them + +- negotiate image formats on individual pads + +- inspect and modify internal data routing between pads of the same entity + +Sub-device character device nodes, conventionally named +``/dev/v4l-subdev*``, use major number 81. + +Drivers may opt to limit the sub-device character devices to only expose +operations that do not modify the device state. In such a case the sub-devices +are referred to as ``read-only`` in the rest of this documentation, and the +related restrictions are documented in individual ioctls. + + +Controls +======== + +Most V4L2 controls are implemented by sub-device hardware. Drivers +usually merge all controls and expose them through video device nodes. +Applications can control all sub-devices through a single interface. + +Complex devices sometimes implement the same control in different pieces +of hardware. This situation is common in embedded platforms, where both +sensors and image processing hardware implement identical functions, +such as contrast adjustment, white balance or faulty pixels correction. +As the V4L2 controls API doesn't support several identical controls in a +single device, all but one of the identical controls are hidden. + +Applications can access those hidden controls through the sub-device +node with the V4L2 control API described in :ref:`control`. The ioctls +behave identically as when issued on V4L2 device nodes, with the +exception that they deal only with controls implemented in the +sub-device. + +Depending on the driver, those controls might also be exposed through +one (or several) V4L2 device nodes. + + +Events +====== + +V4L2 sub-devices can notify applications of events as described in +:ref:`event`. The API behaves identically as when used on V4L2 device +nodes, with the exception that it only deals with events generated by +the sub-device. Depending on the driver, those events might also be +reported on one (or several) V4L2 device nodes. + + +.. _pad-level-formats: + +Pad-level Formats +================= + +.. warning:: + + Pad-level formats are only applicable to very complex devices that + need to expose low-level format configuration to user space. Generic + V4L2 applications do *not* need to use the API described in this + section. + +.. note:: + + For the purpose of this section, the term *format* means the + combination of media bus data format, frame width and frame height. + +Image formats are typically negotiated on video capture and output +devices using the format and +:ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver is +responsible for configuring every block in the video pipeline according +to the requested format at the pipeline input and/or output. + +For complex devices, such as often found in embedded systems, identical +image sizes at the output of a pipeline can be achieved using different +hardware configurations. One such example is shown on +:ref:`pipeline-scaling`, where image scaling can be performed on both +the video sensor and the host image processing hardware. + + +.. _pipeline-scaling: + +.. kernel-figure:: pipeline.dot + :alt: pipeline.dot + :align: center + + Image Format Negotiation on Pipelines + + High quality and high speed pipeline configuration + + + +The sensor scaler is usually of less quality than the host scaler, but +scaling on the sensor is required to achieve higher frame rates. +Depending on the use case (quality vs. speed), the pipeline must be +configured differently. Applications need to configure the formats at +every point in the pipeline explicitly. + +Drivers that implement the :ref:`media API <media-controller-intro>` +can expose pad-level image format configuration to applications. When +they do, applications can use the +:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and +:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. to +negotiate formats on a per-pad basis. + +Applications are responsible for configuring coherent parameters on the +whole pipeline and making sure that connected pads have compatible +formats. The pipeline is checked for formats mismatch at +:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` error +code is then returned if the configuration is invalid. + +Pad-level image format configuration support can be tested by calling +the :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad +0. If the driver returns an ``EINVAL`` error code pad-level format +configuration is not supported by the sub-device. + + +Format Negotiation +------------------ + +Acceptable formats on pads can (and usually do) depend on a number of +external parameters, such as formats on other pads, active links, or +even controls. Finding a combination of formats on all pads in a video +pipeline, acceptable to both application and driver, can't rely on +formats enumeration only. A format negotiation mechanism is required. + +Central to the format negotiation mechanism are the get/set format +operations. When called with the ``which`` argument set to +:ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the +:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and +:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate on +a set of formats parameters that are not connected to the hardware +configuration. Modifying those 'try' formats leaves the device state +untouched (this applies to both the software state stored in the driver +and the hardware state stored in the device itself). + +While not kept as part of the device state, try formats are stored in +the sub-device file handles. A +:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will return +the last try format set *on the same sub-device file handle*. Several +applications querying the same sub-device at the same time will thus not +interact with each other. + +To find out whether a particular format is supported by the device, +applications use the +:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Drivers +verify and, if needed, change the requested ``format`` based on device +requirements and return the possibly modified value. Applications can +then choose to try a different format or accept the returned value and +continue. + +Formats returned by the driver during a negotiation iteration are +guaranteed to be supported by the device. In particular, drivers +guarantee that a returned format will not be further changed if passed +to an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is +(as long as external parameters, such as formats on other pads or links' +configuration are not changed). + +Drivers automatically propagate formats inside sub-devices. When a try +or active format is set on a pad, corresponding formats on other pads of +the same sub-device can be modified by the driver. Drivers are free to +modify formats as required by the device. However, they should comply +with the following rules when possible: + +- Formats should be propagated from sink pads to source pads. Modifying + a format on a source pad should not modify the format on any sink + pad. + +- Sub-devices that scale frames using variable scaling factors should + reset the scale factors to default values when sink pads formats are + modified. If the 1:1 scaling ratio is supported, this means that + source pads formats should be reset to the sink pads formats. + +Formats are not propagated across links, as that would involve +propagating them from one sub-device file handle to another. +Applications must then take care to configure both ends of every link +explicitly with compatible formats. Identical formats on the two ends of +a link are guaranteed to be compatible. Drivers are free to accept +different formats matching device requirements as being compatible. + +:ref:`sample-pipeline-config` shows a sample configuration sequence +for the pipeline described in :ref:`pipeline-scaling` (table columns +list entity names and pad numbers). + + +.. raw:: latex + + \begingroup + \scriptsize + \setlength{\tabcolsep}{2pt} + +.. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}| + +.. _sample-pipeline-config: + +.. flat-table:: Sample Pipeline Configuration + :header-rows: 1 + :stub-columns: 0 + :widths: 5 5 5 5 5 5 5 + + * - + - Sensor/0 + + format + - Frontend/0 + + format + - Frontend/1 + + format + - Scaler/0 + + format + - Scaler/0 + + compose selection rectangle + - Scaler/1 + + format + * - Initial state + - 2048x1536 + + SGRBG8_1X8 + - (default) + - (default) + - (default) + - (default) + - (default) + * - Configure frontend sink format + - 2048x1536 + + SGRBG8_1X8 + - *2048x1536* + + *SGRBG8_1X8* + - *2046x1534* + + *SGRBG8_1X8* + - (default) + - (default) + - (default) + * - Configure scaler sink format + - 2048x1536 + + SGRBG8_1X8 + - 2048x1536 + + SGRBG8_1X8 + - 2046x1534 + + SGRBG8_1X8 + - *2046x1534* + + *SGRBG8_1X8* + - *0,0/2046x1534* + - *2046x1534* + + *SGRBG8_1X8* + * - Configure scaler sink compose selection + - 2048x1536 + + SGRBG8_1X8 + - 2048x1536 + + SGRBG8_1X8 + - 2046x1534 + + SGRBG8_1X8 + - 2046x1534 + + SGRBG8_1X8 + - *0,0/1280x960* + - *1280x960* + + *SGRBG8_1X8* + +.. raw:: latex + + \endgroup + +1. Initial state. The sensor source pad format is set to its native 3MP + size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the + host frontend and scaler sink and source pads have the default + values, as well as the compose rectangle on the scaler's sink pad. + +2. The application configures the frontend sink pad format's size to + 2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The + driver propagates the format to the frontend source pad. + +3. The application configures the scaler sink pad format's size to + 2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to + match the frontend source size and media bus code. The media bus code + on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver + propagates the size to the compose selection rectangle on the + scaler's sink pad, and the format to the scaler source pad. + +4. The application configures the size of the compose selection + rectangle of the scaler's sink pad 1280x960. The driver propagates + the size to the scaler's source pad format. + +When satisfied with the try results, applications can set the active +formats by setting the ``which`` argument to +``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try +formats by drivers. To avoid modifying the hardware state during format +negotiation, applications should negotiate try formats first and then +modify the active settings using the try formats returned during the +last negotiation iteration. This guarantees that the active format will +be applied as-is by the driver without being modified. + + +.. _v4l2-subdev-selections: + +Selections: cropping, scaling and composition +--------------------------------------------- + +Many sub-devices support cropping frames on their input or output pads +(or possible even on both). Cropping is used to select the area of +interest in an image, typically on an image sensor or a video decoder. +It can also be used as part of digital zoom implementations to select +the area of the image that will be scaled up. + +Crop settings are defined by a crop rectangle and represented in a +struct :c:type:`v4l2_rect` by the coordinates of the top +left corner and the rectangle size. Both the coordinates and sizes are +expressed in pixels. + +As for pad formats, drivers store try and active rectangles for the +selection targets :ref:`v4l2-selections-common`. + +On sink pads, cropping is applied relative to the current pad format. +The pad format represents the image size as received by the sub-device +from the previous block in the pipeline, and the crop rectangle +represents the sub-image that will be transmitted further inside the +sub-device for processing. + +The scaling operation changes the size of the image by scaling it to new +dimensions. The scaling ratio isn't specified explicitly, but is implied +from the original and scaled image sizes. Both sizes are represented by +struct :c:type:`v4l2_rect`. + +Scaling support is optional. When supported by a subdev, the crop +rectangle on the subdev's sink pad is scaled to the size configured +using the +:ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTL +using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the +subdev supports scaling but not composing, the top and left values are +not used and must always be set to zero. + +On source pads, cropping is similar to sink pads, with the exception +that the source size from which the cropping is performed, is the +COMPOSE rectangle on the sink pad. In both sink and source pads, the +crop rectangle must be entirely contained inside the source image size +for the crop operation. + +The drivers should always use the closest possible rectangle the user +requests on all selection targets, unless specifically told otherwise. +``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round +the image size either up or down. :ref:`v4l2-selection-flags` + + +Types of selection targets +-------------------------- + + +Actual targets +^^^^^^^^^^^^^^ + +Actual targets (without a postfix) reflect the actual hardware +configuration at any point of time. There is a BOUNDS target +corresponding to every actual target. + + +BOUNDS targets +^^^^^^^^^^^^^^ + +BOUNDS targets is the smallest rectangle that contains all valid actual +rectangles. It may not be possible to set the actual rectangle as large +as the BOUNDS rectangle, however. This may be because e.g. a sensor's +pixel array is not rectangular but cross-shaped or round. The maximum +size may also be smaller than the BOUNDS rectangle. + + +.. _format-propagation: + +Order of configuration and format propagation +--------------------------------------------- + +Inside subdevs, the order of image processing steps will always be from +the sink pad towards the source pad. This is also reflected in the order +in which the configuration must be performed by the user: the changes +made will be propagated to any subsequent stages. If this behaviour is +not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This +flag causes no propagation of the changes are allowed in any +circumstances. This may also cause the accessed rectangle to be adjusted +by the driver, depending on the properties of the underlying hardware. + +The coordinates to a step always refer to the actual size of the +previous step. The exception to this rule is the sink compose +rectangle, which refers to the sink compose bounds rectangle --- if it +is supported by the hardware. + +1. Sink pad format. The user configures the sink pad format. This format + defines the parameters of the image the entity receives through the + pad for further processing. + +2. Sink pad actual crop selection. The sink pad crop defines the crop + performed to the sink pad format. + +3. Sink pad actual compose selection. The size of the sink pad compose + rectangle defines the scaling ratio compared to the size of the sink + pad crop rectangle. The location of the compose rectangle specifies + the location of the actual sink compose rectangle in the sink compose + bounds rectangle. + +4. Source pad actual crop selection. Crop on the source pad defines crop + performed to the image in the sink compose bounds rectangle. + +5. Source pad format. The source pad format defines the output pixel + format of the subdev, as well as the other parameters with the + exception of the image width and height. Width and height are defined + by the size of the source pad actual crop selection. + +Accessing any of the above rectangles not supported by the subdev will +return ``EINVAL``. Any rectangle referring to a previous unsupported +rectangle coordinates will instead refer to the previous supported +rectangle. For example, if sink crop is not supported, the compose +selection will refer to the sink pad format dimensions instead. + + +.. _subdev-image-processing-crop: + +.. kernel-figure:: subdev-image-processing-crop.svg + :alt: subdev-image-processing-crop.svg + :align: center + + **Figure 4.5. Image processing in subdevs: simple crop example** + +In the above example, the subdev supports cropping on its sink pad. To +configure it, the user sets the media bus format on the subdev's sink +pad. Now the actual crop rectangle can be set on the sink pad --- the +location and size of this rectangle reflect the location and size of a +rectangle to be cropped from the sink format. The size of the sink crop +rectangle will also be the size of the format of the subdev's source +pad. + + +.. _subdev-image-processing-scaling-multi-source: + +.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg + :alt: subdev-image-processing-scaling-multi-source.svg + :align: center + + **Figure 4.6. Image processing in subdevs: scaling with multiple sources** + +In this example, the subdev is capable of first cropping, then scaling +and finally cropping for two source pads individually from the resulting +scaled image. The location of the scaled image in the cropped image is +ignored in sink compose target. Both of the locations of the source crop +rectangles refer to the sink scaling rectangle, independently cropping +an area at location specified by the source crop rectangle from it. + + +.. _subdev-image-processing-full: + +.. kernel-figure:: subdev-image-processing-full.svg + :alt: subdev-image-processing-full.svg + :align: center + + **Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources** + +The subdev driver supports two sink pads and two source pads. The images +from both of the sink pads are individually cropped, then scaled and +further composed on the composition bounds rectangle. From that, two +independent streams are cropped and sent out of the subdev from the +source pads. + + +.. toctree:: + :maxdepth: 1 + + subdev-formats + +Streams, multiplexed media pads and internal routing +---------------------------------------------------- + +Simple V4L2 sub-devices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. + +Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple +data streams are transmitted on the same bus, which is represented by a media +link connecting a transmitter source pad with a sink pad on the receiver. For +example, a camera sensor can produce two distinct streams, a pixel stream and a +metadata stream, which are transmitted on the multiplexed data bus, represented +by a media link which connects the single sensor's source pad with the receiver +sink pad. The stream-aware receiver will de-multiplex the streams received on +the its sink pad and allows to route them individually to one of its source +pads. + +Subdevice drivers that support multiplexed streams are compatible with +non-multiplexed subdev drivers, but, of course, require a routing configuration +where the link between those two types of drivers contains only a single +stream. + +Understanding streams +^^^^^^^^^^^^^^^^^^^^^ + +A stream is a stream of content (e.g. pixel data or metadata) flowing through +the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a +receiver and demultiplexer in a SoC). Each media link carries all the enabled +streams from one end of the link to the other, and sub-devices have routing +tables which describe how the incoming streams from sink pads are routed to the +source pads. + +A stream ID is a media pad-local identifier for a stream. Streams IDs of +the same stream must be equal on both ends of a link. In other words, +a particular stream ID must exist on both sides of a media +link, but another stream ID can be used for the same stream at the other side +of the sub-device. + +A stream at a specific point in the media pipeline is identified by the +sub-device and a (pad, stream) pair. For sub-devices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, streams, formats and selections +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of streams to the V4L2 sub-device interface moves the sub-device +formats and selections from pads to (pad, stream) pairs. Besides the +usual pad, also the stream ID needs to be provided for setting formats and +selections. The order of configuring formats and selections along a stream is +the same as without streams (see :ref:`format-propagation`). + +Instead of the sub-device wide merging of streams from all sink pads +towards all source pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a source pad, however, only a single route is allowed. + +Any configurations of a stream within a pad, such as format or selections, +are independent of similar configurations on other streams. This is +subject to change in the future. + +Configuring streams +^^^^^^^^^^^^^^^^^^^ + +The configuration of the streams is done individually for each sub-device and +the validity of the streams between sub-devices is validated when the pipeline +is started. + +There are three steps in configuring the streams: + +1) Set up links. Connect the pads between sub-devices using the :ref:`Media +Controller API <media_controller>` + +2) Streams. Streams are declared and their routing is configured by +setting the routing table for the sub-device using +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that +setting the routing table will reset formats and selections in the +sub-device to default values. + +3) Configure formats and selections. Formats and selections of each stream +are configured separately as documented for plain sub-devices in +:ref:`format-propagation`. The stream ID is set to the same stream ID +associated with either sink or source pads of routes configured using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. + +Multiplexed streams setup example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A simple example of a multiplexed stream setup might be as follows: + +- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source + pad (pad 0) which carries a pixel data stream. + +- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the + sensors (pads 0, 1), and one source pad (pad 2), which outputs two streams. + +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), + connected to the bridge, and two source pads (pads 1-2), going to the DMA + engine. The receiver demultiplexes the incoming streams to the source pads. + +- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is + connected to a single source pad in the receiver. + +The sensors, the bridge and the receiver are modeled as V4L2 sub-devices, +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. + +To configure this pipeline, the userspace must take the following steps: + +1) Set up media links between entities: connect the sensors to the bridge, +bridge to the receiver, and the receiver to the DMA engines. This step does +not differ from normal non-multiplexed media controller setup. + +2) Configure routing + +.. flat-table:: Bridge routing table + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 0/0 + - 2/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from Sensor A + * - 1/0 + - 2/1 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from Sensor B + +.. flat-table:: Receiver routing table + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 0/0 + - 1/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from Sensor A + * - 0/1 + - 2/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from Sensor B + +3) Configure formats and selections + +After configuring routing, the next step is configuring the formats and +selections for the streams. This is similar to performing this step without +streams, with just one exception: the ``stream`` field needs to be assigned +to the value of the stream ID. + +A common way to accomplish this is to start from the sensors and propagate the +configurations along the stream towards the receiver, +using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each +stream endpoint in each sub-device. |