diff options
Diffstat (limited to 'Documentation/userspace-api')
16 files changed, 800 insertions, 65 deletions
diff --git a/Documentation/userspace-api/dma-buf-alloc-exchange.rst b/Documentation/userspace-api/dma-buf-alloc-exchange.rst new file mode 100644 index 000000000..fdff19fce --- /dev/null +++ b/Documentation/userspace-api/dma-buf-alloc-exchange.rst @@ -0,0 +1,389 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. Copyright 2021-2023 Collabora Ltd. + +======================== +Exchanging pixel buffers +======================== + +As originally designed, the Linux graphics subsystem had extremely limited +support for sharing pixel-buffer allocations between processes, devices, and +subsystems. Modern systems require extensive integration between all three +classes; this document details how applications and kernel subsystems should +approach this sharing for two-dimensional image data. + +It is written with reference to the DRM subsystem for GPU and display devices, +V4L2 for media devices, and also to Vulkan, EGL and Wayland, for userspace +support, however any other subsystems should also follow this design and advice. + + +Glossary of terms +================= + +.. glossary:: + + image: + Conceptually a two-dimensional array of pixels. The pixels may be stored + in one or more memory buffers. Has width and height in pixels, pixel + format and modifier (implicit or explicit). + + row: + A span along a single y-axis value, e.g. from co-ordinates (0,100) to + (200,100). + + scanline: + Synonym for row. + + column: + A span along a single x-axis value, e.g. from co-ordinates (100,0) to + (100,100). + + memory buffer: + A piece of memory for storing (parts of) pixel data. Has stride and size + in bytes and at least one handle in some API. May contain one or more + planes. + + plane: + A two-dimensional array of some or all of an image's color and alpha + channel values. + + pixel: + A picture element. Has a single color value which is defined by one or + more color channels values, e.g. R, G and B, or Y, Cb and Cr. May also + have an alpha value as an additional channel. + + pixel data: + Bytes or bits that represent some or all of the color/alpha channel values + of a pixel or an image. The data for one pixel may be spread over several + planes or memory buffers depending on format and modifier. + + color value: + A tuple of numbers, representing a color. Each element in the tuple is a + color channel value. + + color channel: + One of the dimensions in a color model. For example, RGB model has + channels R, G, and B. Alpha channel is sometimes counted as a color + channel as well. + + pixel format: + A description of how pixel data represents the pixel's color and alpha + values. + + modifier: + A description of how pixel data is laid out in memory buffers. + + alpha: + A value that denotes the color coverage in a pixel. Sometimes used for + translucency instead. + + stride: + A value that denotes the relationship between pixel-location co-ordinates + and byte-offset values. Typically used as the byte offset between two + pixels at the start of vertically-consecutive tiling blocks. For linear + layouts, the byte offset between two vertically-adjacent pixels. For + non-linear formats the stride must be computed in a consistent way, which + usually is done as-if the layout was linear. + + pitch: + Synonym for stride. + + +Formats and modifiers +===================== + +Each buffer must have an underlying format. This format describes the color +values provided for each pixel. Although each subsystem has its own format +descriptions (e.g. V4L2 and fbdev), the ``DRM_FORMAT_*`` tokens should be reused +wherever possible, as they are the standard descriptions used for interchange. +These tokens are described in the ``drm_fourcc.h`` file, which is a part of +DRM's uAPI. + +Each ``DRM_FORMAT_*`` token describes the translation between a pixel +co-ordinate in an image, and the color values for that pixel contained within +its memory buffers. The number and type of color channels are described: +whether they are RGB or YUV, integer or floating-point, the size of each channel +and their locations within the pixel memory, and the relationship between color +planes. + +For example, ``DRM_FORMAT_ARGB8888`` describes a format in which each pixel has +a single 32-bit value in memory. Alpha, red, green, and blue, color channels are +available at 8-bit precision per channel, ordered respectively from most to +least significant bits in little-endian storage. ``DRM_FORMAT_*`` is not +affected by either CPU or device endianness; the byte pattern in memory is +always as described in the format definition, which is usually little-endian. + +As a more complex example, ``DRM_FORMAT_NV12`` describes a format in which luma +and chroma YUV samples are stored in separate planes, where the chroma plane is +stored at half the resolution in both dimensions (i.e. one U/V chroma +sample is stored for each 2x2 pixel grouping). + +Format modifiers describe a translation mechanism between these per-pixel memory +samples, and the actual memory storage for the buffer. The most straightforward +modifier is ``DRM_FORMAT_MOD_LINEAR``, describing a scheme in which each plane +is laid out row-sequentially, from the top-left to the bottom-right corner. +This is considered the baseline interchange format, and most convenient for CPU +access. + +Modern hardware employs much more sophisticated access mechanisms, typically +making use of tiled access and possibly also compression. For example, the +``DRM_FORMAT_MOD_VIVANTE_TILED`` modifier describes memory storage where pixels +are stored in 4x4 blocks arranged in row-major ordering, i.e. the first tile in +a plane stores pixels (0,0) to (3,3) inclusive, and the second tile in a plane +stores pixels (4,0) to (7,3) inclusive. + +Some modifiers may modify the number of planes required for an image; for +example, the ``I915_FORMAT_MOD_Y_TILED_CCS`` modifier adds a second plane to RGB +formats in which it stores data about the status of every tile, notably +including whether the tile is fully populated with pixel data, or can be +expanded from a single solid color. + +These extended layouts are highly vendor-specific, and even specific to +particular generations or configurations of devices per-vendor. For this reason, +support of modifiers must be explicitly enumerated and negotiated by all users +in order to ensure a compatible and optimal pipeline, as discussed below. + + +Dimensions and size +=================== + +Each pixel buffer must be accompanied by logical pixel dimensions. This refers +to the number of unique samples which can be extracted from, or stored to, the +underlying memory storage. For example, even though a 1920x1080 +``DRM_FORMAT_NV12`` buffer has a luma plane containing 1920x1080 samples for the Y +component, and 960x540 samples for the U and V components, the overall buffer is +still described as having dimensions of 1920x1080. + +The in-memory storage of a buffer is not guaranteed to begin immediately at the +base address of the underlying memory, nor is it guaranteed that the memory +storage is tightly clipped to either dimension. + +Each plane must therefore be described with an ``offset`` in bytes, which will be +added to the base address of the memory storage before performing any per-pixel +calculations. This may be used to combine multiple planes into a single memory +buffer; for example, ``DRM_FORMAT_NV12`` may be stored in a single memory buffer +where the luma plane's storage begins immediately at the start of the buffer +with an offset of 0, and the chroma plane's storage follows within the same buffer +beginning from the byte offset for that plane. + +Each plane must also have a ``stride`` in bytes, expressing the offset in memory +between two contiguous row. For example, a ``DRM_FORMAT_MOD_LINEAR`` buffer +with dimensions of 1000x1000 may have been allocated as if it were 1024x1000, in +order to allow for aligned access patterns. In this case, the buffer will still +be described with a width of 1000, however the stride will be ``1024 * bpp``, +indicating that there are 24 pixels at the positive extreme of the x axis whose +values are not significant. + +Buffers may also be padded further in the y dimension, simply by allocating a +larger area than would ordinarily be required. For example, many media decoders +are not able to natively output buffers of height 1080, but instead require an +effective height of 1088 pixels. In this case, the buffer continues to be +described as having a height of 1080, with the memory allocation for each buffer +being increased to account for the extra padding. + + +Enumeration +=========== + +Every user of pixel buffers must be able to enumerate a set of supported formats +and modifiers, described together. Within KMS, this is achieved with the +``IN_FORMATS`` property on each DRM plane, listing the supported DRM formats, and +the modifiers supported for each format. In userspace, this is supported through +the `EGL_EXT_image_dma_buf_import_modifiers`_ extension entrypoints for EGL, the +`VK_EXT_image_drm_format_modifier`_ extension for Vulkan, and the +`zwp_linux_dmabuf_v1`_ extension for Wayland. + +Each of these interfaces allows users to query a set of supported +format+modifier combinations. + + +Negotiation +=========== + +It is the responsibility of userspace to negotiate an acceptable format+modifier +combination for its usage. This is performed through a simple intersection of +lists. For example, if a user wants to use Vulkan to render an image to be +displayed on a KMS plane, it must: + + - query KMS for the ``IN_FORMATS`` property for the given plane + - query Vulkan for the supported formats for its physical device, making sure + to pass the ``VkImageUsageFlagBits`` and ``VkImageCreateFlagBits`` + corresponding to the intended rendering use + - intersect these formats to determine the most appropriate one + - for this format, intersect the lists of supported modifiers for both KMS and + Vulkan, to obtain a final list of acceptable modifiers for that format + +This intersection must be performed for all usages. For example, if the user +also wishes to encode the image to a video stream, it must query the media API +it intends to use for encoding for the set of modifiers it supports, and +additionally intersect against this list. + +If the intersection of all lists is an empty list, it is not possible to share +buffers in this way, and an alternate strategy must be considered (e.g. using +CPU access routines to copy data between the different uses, with the +corresponding performance cost). + +The resulting modifier list is unsorted; the order is not significant. + + +Allocation +========== + +Once userspace has determined an appropriate format, and corresponding list of +acceptable modifiers, it must allocate the buffer. As there is no universal +buffer-allocation interface available at either kernel or userspace level, the +client makes an arbitrary choice of allocation interface such as Vulkan, GBM, or +a media API. + +Each allocation request must take, at a minimum: the pixel format, a list of +acceptable modifiers, and the buffer's width and height. Each API may extend +this set of properties in different ways, such as allowing allocation in more +than two dimensions, intended usage patterns, etc. + +The component which allocates the buffer will make an arbitrary choice of what +it considers the 'best' modifier within the acceptable list for the requested +allocation, any padding required, and further properties of the underlying +memory buffers such as whether they are stored in system or device-specific +memory, whether or not they are physically contiguous, and their cache mode. +These properties of the memory buffer are not visible to userspace, however the +``dma-heaps`` API is an effort to address this. + +After allocation, the client must query the allocator to determine the actual +modifier selected for the buffer, as well as the per-plane offset and stride. +Allocators are not permitted to vary the format in use, to select a modifier not +provided within the acceptable list, nor to vary the pixel dimensions other than +the padding expressed through offset, stride, and size. + +Communicating additional constraints, such as alignment of stride or offset, +placement within a particular memory area, etc, is out of scope of dma-buf, +and is not solved by format and modifier tokens. + + +Import +====== + +To use a buffer within a different context, device, or subsystem, the user +passes these parameters (format, modifier, width, height, and per-plane offset +and stride) to an importing API. + +Each memory buffer is referred to by a buffer handle, which may be unique or +duplicated within an image. For example, a ``DRM_FORMAT_NV12`` buffer may have +the luma and chroma buffers combined into a single memory buffer by use of the +per-plane offset parameters, or they may be completely separate allocations in +memory. For this reason, each import and allocation API must provide a separate +handle for each plane. + +Each kernel subsystem has its own types and interfaces for buffer management. +DRM uses GEM buffer objects (BOs), V4L2 has its own references, etc. These types +are not portable between contexts, processes, devices, or subsystems. + +To address this, ``dma-buf`` handles are used as the universal interchange for +buffers. Subsystem-specific operations are used to export native buffer handles +to a ``dma-buf`` file descriptor, and to import those file descriptors into a +native buffer handle. dma-buf file descriptors can be transferred between +contexts, processes, devices, and subsystems. + +For example, a Wayland media player may use V4L2 to decode a video frame into a +``DRM_FORMAT_NV12`` buffer. This will result in two memory planes (luma and +chroma) being dequeued by the user from V4L2. These planes are then exported to +one dma-buf file descriptor per plane, these descriptors are then sent along +with the metadata (format, modifier, width, height, per-plane offset and stride) +to the Wayland server. The Wayland server will then import these file +descriptors as an EGLImage for use through EGL/OpenGL (ES), a VkImage for use +through Vulkan, or a KMS framebuffer object; each of these import operations +will take the same metadata and convert the dma-buf file descriptors into their +native buffer handles. + +Having a non-empty intersection of supported modifiers does not guarantee that +import will succeed into all consumers; they may have constraints beyond those +implied by modifiers which must be satisfied. + + +Implicit modifiers +================== + +The concept of modifiers post-dates all of the subsystems mentioned above. As +such, it has been retrofitted into all of these APIs, and in order to ensure +backwards compatibility, support is needed for drivers and userspace which do +not (yet) support modifiers. + +As an example, GBM is used to allocate buffers to be shared between EGL for +rendering and KMS for display. It has two entrypoints for allocating buffers: +``gbm_bo_create`` which only takes the format, width, height, and a usage token, +and ``gbm_bo_create_with_modifiers`` which extends this with a list of modifiers. + +In the latter case, the allocation is as discussed above, being provided with a +list of acceptable modifiers that the implementation can choose from (or fail if +it is not possible to allocate within those constraints). In the former case +where modifiers are not provided, the GBM implementation must make its own +choice as to what is likely to be the 'best' layout. Such a choice is entirely +implementation-specific: some will internally use tiled layouts which are not +CPU-accessible if the implementation decides that is a good idea through +whatever heuristic. It is the implementation's responsibility to ensure that +this choice is appropriate. + +To support this case where the layout is not known because there is no awareness +of modifiers, a special ``DRM_FORMAT_MOD_INVALID`` token has been defined. This +pseudo-modifier declares that the layout is not known, and that the driver +should use its own logic to determine what the underlying layout may be. + +.. note:: + + ``DRM_FORMAT_MOD_INVALID`` is a non-zero value. The modifier value zero is + ``DRM_FORMAT_MOD_LINEAR``, which is an explicit guarantee that the image + has the linear layout. Care and attention should be taken to ensure that + zero as a default value is not mixed up with either no modifier or the linear + modifier. Also note that in some APIs the invalid modifier value is specified + with an out-of-band flag, like in ``DRM_IOCTL_MODE_ADDFB2``. + +There are four cases where this token may be used: + - during enumeration, an interface may return ``DRM_FORMAT_MOD_INVALID``, either + as the sole member of a modifier list to declare that explicit modifiers are + not supported, or as part of a larger list to declare that implicit modifiers + may be used + - during allocation, a user may supply ``DRM_FORMAT_MOD_INVALID``, either as the + sole member of a modifier list (equivalent to not supplying a modifier list + at all) to declare that explicit modifiers are not supported and must not be + used, or as part of a larger list to declare that an allocation using implicit + modifiers is acceptable + - in a post-allocation query, an implementation may return + ``DRM_FORMAT_MOD_INVALID`` as the modifier of the allocated buffer to declare + that the underlying layout is implementation-defined and that an explicit + modifier description is not available; per the above rules, this may only be + returned when the user has included ``DRM_FORMAT_MOD_INVALID`` as part of the + list of acceptable modifiers, or not provided a list + - when importing a buffer, the user may supply ``DRM_FORMAT_MOD_INVALID`` as the + buffer modifier (or not supply a modifier) to indicate that the modifier is + unknown for whatever reason; this is only acceptable when the buffer has + not been allocated with an explicit modifier + +It follows from this that for any single buffer, the complete chain of operations +formed by the producer and all the consumers must be either fully implicit or fully +explicit. For example, if a user wishes to allocate a buffer for use between +GPU, display, and media, but the media API does not support modifiers, then the +user **must not** allocate the buffer with explicit modifiers and attempt to +import the buffer into the media API with no modifier, but either perform the +allocation using implicit modifiers, or allocate the buffer for media use +separately and copy between the two buffers. + +As one exception to the above, allocations may be 'upgraded' from implicit +to explicit modifiers. For example, if the buffer is allocated with +``gbm_bo_create`` (taking no modifiers), the user may then query the modifier with +``gbm_bo_get_modifier`` and then use this modifier as an explicit modifier token +if a valid modifier is returned. + +When allocating buffers for exchange between different users and modifiers are +not available, implementations are strongly encouraged to use +``DRM_FORMAT_MOD_LINEAR`` for their allocation, as this is the universal baseline +for exchange. However, it is not guaranteed that this will result in the correct +interpretation of buffer content, as implicit modifier operation may still be +subject to driver-specific heuristics. + +Any new users - userspace programs and protocols, kernel subsystems, etc - +wishing to exchange buffers must offer interoperability through dma-buf file +descriptors for memory planes, DRM format tokens to describe the format, DRM +format modifiers to describe the layout in memory, at least width and height for +dimensions, and at least offset and stride for each memory plane. + +.. _zwp_linux_dmabuf_v1: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml +.. _VK_EXT_image_drm_format_modifier: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_EXT_image_drm_format_modifier.html +.. _EGL_EXT_image_dma_buf_import_modifiers: https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst index 72a65db0c..031df47a7 100644 --- a/Documentation/userspace-api/index.rst +++ b/Documentation/userspace-api/index.rst @@ -22,6 +22,7 @@ place where this information is gathered. unshare spec_ctrl accelerators/ocxl + dma-buf-alloc-exchange ebpf/index ELF ioctl/index diff --git a/Documentation/userspace-api/landlock.rst b/Documentation/userspace-api/landlock.rst index d8cd8cd9c..2e3822677 100644 --- a/Documentation/userspace-api/landlock.rst +++ b/Documentation/userspace-api/landlock.rst @@ -8,13 +8,13 @@ Landlock: unprivileged access control ===================================== :Author: Mickaël Salaün -:Date: October 2022 +:Date: October 2023 The goal of Landlock is to enable to restrict ambient rights (e.g. global -filesystem access) for a set of processes. Because Landlock is a stackable -LSM, it makes possible to create safe security sandboxes as new security layers -in addition to the existing system-wide access-controls. This kind of sandbox -is expected to help mitigate the security impact of bugs or +filesystem or network access) for a set of processes. Because Landlock +is a stackable LSM, it makes possible to create safe security sandboxes as new +security layers in addition to the existing system-wide access-controls. This +kind of sandbox is expected to help mitigate the security impact of bugs or unexpected/malicious behaviors in user space applications. Landlock empowers any process, including unprivileged ones, to securely restrict themselves. @@ -28,20 +28,34 @@ appropriately <kernel_support>`. Landlock rules ============== -A Landlock rule describes an action on an object. An object is currently a -file hierarchy, and the related filesystem actions are defined with `access -rights`_. A set of rules is aggregated in a ruleset, which can then restrict +A Landlock rule describes an action on an object which the process intends to +perform. A set of rules is aggregated in a ruleset, which can then restrict the thread enforcing it, and its future children. +The two existing types of rules are: + +Filesystem rules + For these rules, the object is a file hierarchy, + and the related filesystem actions are defined with + `filesystem access rights`. + +Network rules (since ABI v4) + For these rules, the object is a TCP port, + and the related actions are defined with `network access rights`. + Defining and enforcing a security policy ---------------------------------------- -We first need to define the ruleset that will contain our rules. For this -example, the ruleset will contain rules that only allow read actions, but write -actions will be denied. The ruleset then needs to handle both of these kind of -actions. This is required for backward and forward compatibility (i.e. the -kernel and user space may not know each other's supported restrictions), hence -the need to be explicit about the denied-by-default access rights. +We first need to define the ruleset that will contain our rules. + +For this example, the ruleset will contain rules that only allow filesystem +read actions and establish a specific TCP connection. Filesystem write +actions and other TCP actions will be denied. + +The ruleset then needs to handle both these kinds of actions. This is +required for backward and forward compatibility (i.e. the kernel and user +space may not know each other's supported restrictions), hence the need +to be explicit about the denied-by-default access rights. .. code-block:: c @@ -62,6 +76,9 @@ the need to be explicit about the denied-by-default access rights. LANDLOCK_ACCESS_FS_MAKE_SYM | LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_TRUNCATE, + .handled_access_net = + LANDLOCK_ACCESS_NET_BIND_TCP | + LANDLOCK_ACCESS_NET_CONNECT_TCP, }; Because we may not know on which kernel version an application will be @@ -70,9 +87,7 @@ should try to protect users as much as possible whatever the kernel they are using. To avoid binary enforcement (i.e. either all security features or none), we can leverage a dedicated Landlock command to get the current version of the Landlock ABI and adapt the handled accesses. Let's check if we should -remove the ``LANDLOCK_ACCESS_FS_REFER`` or ``LANDLOCK_ACCESS_FS_TRUNCATE`` -access rights, which are only supported starting with the second and third -version of the ABI. +remove access rights which are only supported in higher versions of the ABI. .. code-block:: c @@ -92,6 +107,12 @@ version of the ABI. case 2: /* Removes LANDLOCK_ACCESS_FS_TRUNCATE for ABI < 3 */ ruleset_attr.handled_access_fs &= ~LANDLOCK_ACCESS_FS_TRUNCATE; + __attribute__((fallthrough)); + case 3: + /* Removes network support for ABI < 4 */ + ruleset_attr.handled_access_net &= + ~(LANDLOCK_ACCESS_NET_BIND_TCP | + LANDLOCK_ACCESS_NET_CONNECT_TCP); } This enables to create an inclusive ruleset that will contain our rules. @@ -143,10 +164,23 @@ for the ruleset creation, by filtering access rights according to the Landlock ABI version. In this example, this is not required because all of the requested ``allowed_access`` rights are already available in ABI 1. -We now have a ruleset with one rule allowing read access to ``/usr`` while -denying all other handled accesses for the filesystem. The next step is to -restrict the current thread from gaining more privileges (e.g. thanks to a SUID -binary). +For network access-control, we can add a set of rules that allow to use a port +number for a specific action: HTTPS connections. + +.. code-block:: c + + struct landlock_net_port_attr net_port = { + .allowed_access = LANDLOCK_ACCESS_NET_CONNECT_TCP, + .port = 443, + }; + + err = landlock_add_rule(ruleset_fd, LANDLOCK_RULE_NET_PORT, + &net_port, 0); + +The next step is to restrict the current thread from gaining more privileges +(e.g. through a SUID binary). We now have a ruleset with the first rule +allowing read access to ``/usr`` while denying all other handled accesses for +the filesystem, and a second rule allowing HTTPS connections. .. code-block:: c @@ -355,7 +389,7 @@ Access rights ------------- .. kernel-doc:: include/uapi/linux/landlock.h - :identifiers: fs_access + :identifiers: fs_access net_access Creating a new ruleset ---------------------- @@ -374,6 +408,7 @@ Extending a ruleset .. kernel-doc:: include/uapi/linux/landlock.h :identifiers: landlock_rule_type landlock_path_beneath_attr + landlock_net_port_attr Enforcing a ruleset ------------------- @@ -387,9 +422,9 @@ Current limitations Filesystem topology modification -------------------------------- -As for file renaming and linking, a sandboxed thread cannot modify its -filesystem topology, whether via :manpage:`mount(2)` or -:manpage:`pivot_root(2)`. However, :manpage:`chroot(2)` calls are not denied. +Threads sandboxed with filesystem restrictions cannot modify filesystem +topology, whether via :manpage:`mount(2)` or :manpage:`pivot_root(2)`. +However, :manpage:`chroot(2)` calls are not denied. Special filesystems ------------------- @@ -451,6 +486,14 @@ always allowed when using a kernel that only supports the first or second ABI. Starting with the Landlock ABI version 3, it is now possible to securely control truncation thanks to the new ``LANDLOCK_ACCESS_FS_TRUNCATE`` access right. +Network support (ABI < 4) +------------------------- + +Starting with the Landlock ABI version 4, it is now possible to restrict TCP +bind and connect actions to only a set of allowed ports thanks to the new +``LANDLOCK_ACCESS_NET_BIND_TCP`` and ``LANDLOCK_ACCESS_NET_CONNECT_TCP`` +access rights. + .. _kernel_support: Kernel support @@ -469,6 +512,12 @@ still enable it by adding ``lsm=landlock,[...]`` to Documentation/admin-guide/kernel-parameters.rst thanks to the bootloader configuration. +To be able to explicitly allow TCP operations (e.g., adding a network rule with +``LANDLOCK_ACCESS_NET_BIND_TCP``), the kernel must support TCP +(``CONFIG_INET=y``). Otherwise, sys_landlock_add_rule() returns an +``EAFNOSUPPORT`` error, which can safely be ignored because this kind of TCP +operation is already not possible. + Questions and answers ===================== diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst new file mode 100644 index 000000000..919a50e8b --- /dev/null +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst @@ -0,0 +1,104 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _media_using_camera_sensor_drivers: + +Using camera sensor drivers +=========================== + +This section describes common practices for how the V4L2 sub-device interface is +used to control the camera sensor drivers. + +You may also find :ref:`media_writing_camera_sensor_drivers` useful. + +Frame size +---------- + +There are two distinct ways to configure the frame size produced by camera +sensors. + +Freely configurable camera sensor drivers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Freely configurable camera sensor drivers expose the device's internal +processing pipeline as one or more sub-devices with different cropping and +scaling configurations. The output size of the device is the result of a series +of cropping and scaling operations from the device's pixel array's size. + +An example of such a driver is the CCS driver. + +Register list based drivers +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Register list based drivers generally, instead of able to configure the device +they control based on user requests, are limited to a number of preset +configurations that combine a number of different parameters that on hardware +level are independent. How a driver picks such configuration is based on the +format set on a source pad at the end of the device's internal pipeline. + +Most sensor drivers are implemented this way. + +Frame interval configuration +---------------------------- + +There are two different methods for obtaining possibilities for different frame +intervals as well as configuring the frame interval. Which one to implement +depends on the type of the device. + +Raw camera sensors +~~~~~~~~~~~~~~~~~~ + +Instead of a high level parameter such as frame interval, the frame interval is +a result of the configuration of a number of camera sensor implementation +specific parameters. Luckily, these parameters tend to be the same for more or +less all modern raw camera sensors. + +The frame interval is calculated using the following equation:: + + frame interval = (analogue crop width + horizontal blanking) * + (analogue crop height + vertical blanking) / pixel rate + +The formula is bus independent and is applicable for raw timing parameters on +large variety of devices beyond camera sensors. Devices that have no analogue +crop, use the full source image size, i.e. pixel array size. + +Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and +``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control +is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in +the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same +sub-device. The unit of that control is pixels per second. + +Register list based drivers need to implement read-only sub-device nodes for the +purpose. Devices that are not register list based need these to configure the +device's internal processing pipeline. + +The first entity in the linear pipeline is the pixel array. The pixel array may +be followed by other entities that are there to allow configuring binning, +skipping, scaling or digital crop, see :ref:`VIDIOC_SUBDEV_G_SELECTION +<VIDIOC_SUBDEV_G_SELECTION>`. + +USB cameras etc. devices +~~~~~~~~~~~~~~~~~~~~~~~~ + +USB video class hardware, as well as many cameras offering a similar higher +level interface natively, generally use the concept of frame interval (or frame +rate) on device level in firmware or hardware. This means lower level controls +implemented by raw cameras may not be used on uAPI (or even kAPI) to control the +frame interval on these devices. + +Rotation, orientation and flipping +---------------------------------- + +Some systems have the camera sensor mounted upside down compared to its natural +mounting rotation. In such cases, drivers shall expose the information to +userspace with the :ref:`V4L2_CID_CAMERA_SENSOR_ROTATION +<v4l2-camera-sensor-rotation>` control. + +Sensor drivers shall also report the sensor's mounting orientation with the +:ref:`V4L2_CID_CAMERA_SENSOR_ORIENTATION <v4l2-camera-sensor-orientation>`. + +Sensor drivers that have any vertical or horizontal flips embedded in the +register programming sequences shall initialize the :ref:`V4L2_CID_HFLIP +<v4l2-cid-hflip>` and :ref:`V4L2_CID_VFLIP <v4l2-cid-vflip>` controls with the +values programmed by the register sequences. The default values of these +controls shall be 0 (disabled). Especially these controls shall not be inverted, +independently of the sensor's mounting rotation. diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst index 6708d649a..1726f8ec8 100644 --- a/Documentation/userspace-api/media/drivers/index.rst +++ b/Documentation/userspace-api/media/drivers/index.rst @@ -32,11 +32,13 @@ For more details see the file COPYING in the source distribution of Linux. :numbered: aspeed-video + camera-sensor ccs cx2341x-uapi dw100 imx-uapi max2175 + npcm-video omap3isp-uapi st-vgxy61 uvcvideo diff --git a/Documentation/userspace-api/media/drivers/npcm-video.rst b/Documentation/userspace-api/media/drivers/npcm-video.rst new file mode 100644 index 000000000..b47771dd8 --- /dev/null +++ b/Documentation/userspace-api/media/drivers/npcm-video.rst @@ -0,0 +1,66 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. include:: <isonum.txt> + +NPCM video driver +================= + +This driver is used to control the Video Capture/Differentiation (VCD) engine +and Encoding Compression Engine (ECE) present on Nuvoton NPCM SoCs. The VCD can +capture a frame from digital video input and compare two frames in memory, and +the ECE can compress the frame data into HEXTILE format. + +Driver-specific Controls +------------------------ + +V4L2_CID_NPCM_CAPTURE_MODE +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The VCD engine supports two modes: + +- COMPLETE mode: + + Capture the next complete frame into memory. + +- DIFF mode: + + Compare the incoming frame with the frame stored in memory, and updates the + differentiated frame in memory. + +Application can use ``V4L2_CID_NPCM_CAPTURE_MODE`` control to set the VCD mode +with different control values (enum v4l2_npcm_capture_mode): + +- ``V4L2_NPCM_CAPTURE_MODE_COMPLETE``: will set VCD to COMPLETE mode. +- ``V4L2_NPCM_CAPTURE_MODE_DIFF``: will set VCD to DIFF mode. + +V4L2_CID_NPCM_RECT_COUNT +~~~~~~~~~~~~~~~~~~~~~~~~ + +If using V4L2_PIX_FMT_HEXTILE format, VCD will capture frame data and then ECE +will compress the data into HEXTILE rectangles and store them in V4L2 video +buffer with the layout defined in Remote Framebuffer Protocol: +:: + + (RFC 6143, https://www.rfc-editor.org/rfc/rfc6143.html#section-7.6.1) + + +--------------+--------------+-------------------+ + | No. of bytes | Type [Value] | Description | + +--------------+--------------+-------------------+ + | 2 | U16 | x-position | + | 2 | U16 | y-position | + | 2 | U16 | width | + | 2 | U16 | height | + | 4 | S32 | encoding-type (5) | + +--------------+--------------+-------------------+ + | HEXTILE rectangle data | + +-------------------------------------------------+ + +Application can get the video buffer through VIDIOC_DQBUF, and followed by +calling ``V4L2_CID_NPCM_RECT_COUNT`` control to get the number of HEXTILE +rectangles in this buffer. + +References +---------- +include/uapi/linux/npcm-video.h + +**Copyright** |copy| 2022 Nuvoton Technologies diff --git a/Documentation/userspace-api/media/gen-errors.rst b/Documentation/userspace-api/media/gen-errors.rst index e595d0bea..4e8defd36 100644 --- a/Documentation/userspace-api/media/gen-errors.rst +++ b/Documentation/userspace-api/media/gen-errors.rst @@ -59,9 +59,7 @@ Generic Error Codes - - ``ENOTTY`` - - The ioctl is not supported by the driver, actually meaning that - the required functionality is not available, or the file - descriptor is not for a media device. + - The ioctl is not supported by the file descriptor. - - ``ENOSPC`` diff --git a/Documentation/userspace-api/media/v4l/buffer.rst b/Documentation/userspace-api/media/v4l/buffer.rst index 04dec3e57..52bbee81c 100644 --- a/Documentation/userspace-api/media/v4l/buffer.rst +++ b/Documentation/userspace-api/media/v4l/buffer.rst @@ -549,9 +549,9 @@ Buffer Flags - 0x00000400 - The buffer has been prepared for I/O and can be queued by the application. Drivers set or clear this flag when the - :ref:`VIDIOC_QUERYBUF`, + :ref:`VIDIOC_QUERYBUF <VIDIOC_QUERYBUF>`, :ref:`VIDIOC_PREPARE_BUF <VIDIOC_QBUF>`, - :ref:`VIDIOC_QBUF` or + :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` or :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl is called. * .. _`V4L2-BUF-FLAG-NO-CACHE-INVALIDATE`: diff --git a/Documentation/userspace-api/media/v4l/control.rst b/Documentation/userspace-api/media/v4l/control.rst index 4463fce69..57893814a 100644 --- a/Documentation/userspace-api/media/v4l/control.rst +++ b/Documentation/userspace-api/media/v4l/control.rst @@ -143,9 +143,13 @@ Control IDs recognise the difference between digital and analogue gain use controls ``V4L2_CID_DIGITAL_GAIN`` and ``V4L2_CID_ANALOGUE_GAIN``. +.. _v4l2-cid-hflip: + ``V4L2_CID_HFLIP`` ``(boolean)`` Mirror the picture horizontally. +.. _v4l2-cid-vflip: + ``V4L2_CID_VFLIP`` ``(boolean)`` Mirror the picture vertically. diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst index a4f1df709..43988516a 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -579,20 +579,19 @@ is started. There are three steps in configuring the streams: -1) Set up links. Connect the pads between sub-devices using the :ref:`Media -Controller API <media_controller>` +1. Set up links. Connect the pads between sub-devices using the + :ref:`Media Controller API <media_controller>` -2) Streams. Streams are declared and their routing is configured by -setting the routing table for the sub-device using -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that -setting the routing table will reset formats and selections in the -sub-device to default values. +2. Streams. Streams are declared and their routing is configured by setting the + routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING + <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will + reset formats and selections in the sub-device to default values. -3) Configure formats and selections. Formats and selections of each stream -are configured separately as documented for plain sub-devices in -:ref:`format-propagation`. The stream ID is set to the same stream ID -associated with either sink or source pads of routes configured using the -:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. +3. Configure formats and selections. Formats and selections of each stream are + configured separately as documented for plain sub-devices in + :ref:`format-propagation`. The stream ID is set to the same stream ID + associated with either sink or source pads of routes configured using the + :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -618,11 +617,11 @@ modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. To configure this pipeline, the userspace must take the following steps: -1) Set up media links between entities: connect the sensors to the bridge, -bridge to the receiver, and the receiver to the DMA engines. This step does -not differ from normal non-multiplexed media controller setup. +1. Set up media links between entities: connect the sensors to the bridge, + bridge to the receiver, and the receiver to the DMA engines. This step does + not differ from normal non-multiplexed media controller setup. -2) Configure routing +2. Configure routing .. flat-table:: Bridge routing table :header-rows: 1 @@ -656,14 +655,14 @@ not differ from normal non-multiplexed media controller setup. - V4L2_SUBDEV_ROUTE_FL_ACTIVE - Pixel data stream from Sensor B -3) Configure formats and selections +3. Configure formats and selections -After configuring routing, the next step is configuring the formats and -selections for the streams. This is similar to performing this step without -streams, with just one exception: the ``stream`` field needs to be assigned -to the value of the stream ID. + After configuring routing, the next step is configuring the formats and + selections for the streams. This is similar to performing this step without + streams, with just one exception: the ``stream`` field needs to be assigned + to the value of the stream ID. -A common way to accomplish this is to start from the sensors and propagate the -configurations along the stream towards the receiver, -using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each -stream endpoint in each sub-device. + A common way to accomplish this is to start from the sensors and propagate + the configurations along the stream towards the receiver, using + :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each + stream endpoint in each sub-device. diff --git a/Documentation/userspace-api/media/v4l/dv-timings.rst b/Documentation/userspace-api/media/v4l/dv-timings.rst index e17f056b1..4b19bcb4b 100644 --- a/Documentation/userspace-api/media/v4l/dv-timings.rst +++ b/Documentation/userspace-api/media/v4l/dv-timings.rst @@ -33,6 +33,27 @@ current DV timings they use the the DV timings as seen by the video receiver applications use the :ref:`VIDIOC_QUERY_DV_TIMINGS` ioctl. +When the hardware detects a video source change (e.g. the video +signal appears or disappears, or the video resolution changes), then +it will issue a `V4L2_EVENT_SOURCE_CHANGE` event. Use the +:ref:`ioctl VIDIOC_SUBSCRIBE_EVENT <VIDIOC_SUBSCRIBE_EVENT>` and the +:ref:`VIDIOC_DQEVENT` to check if this event was reported. + +If the video signal changed, then the application has to stop +streaming, free all buffers, and call the :ref:`VIDIOC_QUERY_DV_TIMINGS` +to obtain the new video timings, and if they are valid, it can set +those by calling the :ref:`ioctl VIDIOC_S_DV_TIMINGS <VIDIOC_G_DV_TIMINGS>`. +This will also update the format, so use the :ref:`ioctl VIDIOC_G_FMT <VIDIOC_G_FMT>` +to obtain the new format. Now the application can allocate new buffers +and start streaming again. + +The :ref:`VIDIOC_QUERY_DV_TIMINGS` will just report what the +hardware detects, it will never change the configuration. If the +currently set timings and the actually detected timings differ, then +typically this will mean that you will not be able to capture any +video. The correct approach is to rely on the `V4L2_EVENT_SOURCE_CHANGE` +event so you know when something changed. + Applications can make use of the :ref:`input-capabilities` and :ref:`output-capabilities` flags to determine whether the digital video ioctls can be used with the given input or output. diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst index 296ad2025..886ba7b08 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst @@ -288,6 +288,13 @@ please make a proposal on the linux-media mailing list. - 'MT2110R' - This format is two-planar 10-Bit raster mode and having similitude with ``V4L2_PIX_FMT_MM21`` in term of alignment and tiling. Used for AVC. + * .. _V4L2-PIX-FMT-HEXTILE: + + - ``V4L2_PIX_FMT_HEXTILE`` + - 'HXTL' + - Compressed format used by Nuvoton NPCM video driver. This format is + defined in Remote Framebuffer Protocol (RFC 6143, chapter 7.7.4 Hextile + Encoding). .. raw:: latex \normalsize diff --git a/Documentation/userspace-api/media/v4l/pixfmt-srggb12p.rst b/Documentation/userspace-api/media/v4l/pixfmt-srggb12p.rst index b6e79e2f8..7c3810ff7 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-srggb12p.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-srggb12p.rst @@ -60,7 +60,7 @@ Each cell is one byte. G\ :sub:`10low`\ (bits 3--0) - G\ :sub:`12high` - R\ :sub:`13high` - - R\ :sub:`13low`\ (bits 3--2) + - R\ :sub:`13low`\ (bits 7--4) G\ :sub:`12low`\ (bits 3--0) - - start + 12: @@ -82,6 +82,6 @@ Each cell is one byte. G\ :sub:`30low`\ (bits 3--0) - G\ :sub:`32high` - R\ :sub:`33high` - - R\ :sub:`33low`\ (bits 3--2) + - R\ :sub:`33low`\ (bits 7--4) G\ :sub:`32low`\ (bits 3--0) diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst index a3a35eeed..eb3cd20b0 100644 --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst @@ -949,6 +949,78 @@ The following tables list existing packed RGB formats. - b\ :sub:`2` - b\ :sub:`1` - b\ :sub:`0` + * .. _MEDIA-BUS-FMT-RGB666-2X9-BE: + + - MEDIA_BUS_FMT_RGB666_2X9_BE + - 0x1025 + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - r\ :sub:`5` + - r\ :sub:`4` + - r\ :sub:`3` + - r\ :sub:`2` + - r\ :sub:`1` + - r\ :sub:`0` + - g\ :sub:`5` + - g\ :sub:`4` + - g\ :sub:`3` + * - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - g\ :sub:`2` + - g\ :sub:`1` + - g\ :sub:`0` + - b\ :sub:`5` + - b\ :sub:`4` + - b\ :sub:`3` + - b\ :sub:`2` + - b\ :sub:`1` + - b\ :sub:`0` * .. _MEDIA-BUS-FMT-BGR666-1X18: - MEDIA_BUS_FMT_BGR666_1X18 diff --git a/Documentation/userspace-api/netlink/genetlink-legacy.rst b/Documentation/userspace-api/netlink/genetlink-legacy.rst index 40b82ad5d..70a77387f 100644 --- a/Documentation/userspace-api/netlink/genetlink-legacy.rst +++ b/Documentation/userspace-api/netlink/genetlink-legacy.rst @@ -11,6 +11,20 @@ the ``genetlink-legacy`` protocol level. Specification ============= +Globals +------- + +Attributes listed directly at the root level of the spec file. + +version +~~~~~~~ + +Generic Netlink family version, default is 1. + +``version`` has historically been used to introduce family changes +which may break backwards compatibility. Since compatibility breaking changes +are generally not allowed ``version`` is very rarely used. + Attribute type nests -------------------- @@ -168,7 +182,7 @@ members - ``name`` - The attribute name of the struct member - ``type`` - One of the scalar types ``u8``, ``u16``, ``u32``, ``u64``, ``s8``, - ``s16``, ``s32``, ``s64``, ``string`` or ``binary``. + ``s16``, ``s32``, ``s64``, ``string``, ``binary`` or ``bitfield32``. - ``byte-order`` - ``big-endian`` or ``little-endian`` - ``doc``, ``enum``, ``enum-as-flags``, ``display-hint`` - Same as for :ref:`attribute definitions <attribute_properties>` diff --git a/Documentation/userspace-api/netlink/specs.rst b/Documentation/userspace-api/netlink/specs.rst index cc4e24309..c1b951649 100644 --- a/Documentation/userspace-api/netlink/specs.rst +++ b/Documentation/userspace-api/netlink/specs.rst @@ -86,11 +86,6 @@ name Name of the family. Name identifies the family in a unique way, since the Family IDs are allocated dynamically. -version -~~~~~~~ - -Generic Netlink family version, default is 1. - protocol ~~~~~~~~ @@ -408,10 +403,21 @@ This section describes the attribute types supported by the ``genetlink`` compatibility level. Refer to documentation of different levels for additional attribute types. -Scalar integer types +Common integer types -------------------- -Fixed-width integer types: +``sint`` and ``uint`` represent signed and unsigned 64 bit integers. +If the value can fit on 32 bits only 32 bits are carried in netlink +messages, otherwise full 64 bits are carried. Note that the payload +is only aligned to 4B, so the full 64 bit value may be unaligned! + +Common integer types should be preferred over fix-width types in majority +of cases. + +Fix-width integer types +----------------------- + +Fixed-width integer types include: ``u8``, ``u16``, ``u32``, ``u64``, ``s8``, ``s16``, ``s32``, ``s64``. Note that types smaller than 32 bit should be avoided as using them @@ -421,6 +427,9 @@ See :ref:`pad_type` for padding of 64 bit attributes. The payload of the attribute is the integer in host order unless ``byte-order`` specifies otherwise. +64 bit values are usually aligned by the kernel but it is recommended +that the user space is able to deal with unaligned values. + .. _pad_type: pad |