1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
|
/** \page page_dma_buf DMA-BUF Sharing
PipeWire supports sharing Direct Memory Access buffers (DMA-BUFs) between
clients via the `SPA_DATA_DmaBuf` data type. However properly negotiating
DMA-BUF support on both the producer and the consumer side require following
a specific procedure. This page describes said procedure by using events and
methods from the filter or stream API.
Note: This article focuses mostly on DMA-BUF sharing from arbitrary devices,
like discrete GPUs. For using DMA-BUFs created by v4l2 please refer to the
corresponding paragraph.
# Capability Negotiations
The capability negotiation for DMA-BUFs is complicated by the fact that a
usable and preferred optimal modifier for a given format can only be
determined by the allocator. This allocator has to be invoked with the intersection
of all supported modifiers for every client. As a result, the fixation of the
modifier is delegated from PipeWire to the node responsible for
allocating the buffers.
## pw_stream_connect
The stream parameters should contain two `SPA_PARAM_EnumFormat` objects for
each format: one for DMA-BUFs, one for shared memory buffers as a fallback.
Query the list of all supported modifiers from your graphics API of choice.
Add a `SPA_FORMAT_VIDEO_modifier` property to the first stream parameter with
the flags `SPA_POD_PROP_FLAG_MANDATORY | SPA_POD_PROP_FLAG_DONT_FIXATE`. The
value of the property should be set to a `SPA_CHOICE_Enum` containing one
`long` choice per supported modifier, plus `DRM_FORMAT_MOD_INVALID` if the
graphics API supports modifier-less buffers.
Note: When a producer is only supporting modifier-less buffers it can omit
the `SPA_POD_PROP_FLAG_DONT_FIXATE` (see param_changed hook, For producers).
The second stream parameter should not contain any `SPA_FORMAT_VIDEO_modifier`
property.
To prioritise DMA-BUFs place those `SPA_PARAM_EnumFormat` containing modifiers
first, when emitting them to PipeWire.
## param_changed Hook
When the `param_changed` hook is called for a `SPA_PARAM_Format` the client
has to parse the `spa_pod` directly. Use
`spa_pod_find_prop(param, NULL, SPA_FORMAT_VIDEO_modifier)` to check
whether modifiers were negotiated. If they were negotiated, set the
`SPA_PARAM_BUFFERS_dataType` property to `1 << SPA_DATA_DmaBuf`. If they were
not negotiated, fall back to shared memory by setting the
`SPA_PARAM_BUFFERS_dataType` property to `1 << SPA_DATA_MemFd`,
`1 << SPA_DATA_MemPtr`, or both.
While consumers only have to parse the resulting `SPA_PARAM_Format` for any
format related information, it's up to the producer to fixate onto a single
format modifier pair. The producer is also responsible to check if all clients
announce sufficient capabilities or fallback to shared memory buffers when
possible.
### For Consumers
Use `spa_format_video_raw_parse` to get the format and modifier.
### For Producers
Producers have to handle two cases when it comes to modifiers wrt. to the
previous announced capabilities: Using only the modifier-less API, only the
modifier-aware one, or supporting both.
- modifier-less:
In this case only the modifier `DRM_FORMAT_MOD_INVALID` was announced with
the format.
It is sufficient to check if the `SPA_PARAM_Format` contains the modifier
property as described above. If that is the case, use DMA-BUFs for screen-sharing,
else fall back to SHM, if possible.
- modifier-aware:
In this case a list with all supported modifiers will be returned in the format.
(using `DRM_FORMAT_MOD_INVALID` as the token for the modifier-less API).
On the `param_changed` event check if the modifier key is present and has the flag
`SPA_POD_PROP_FLAG_DONT_FIXATE` attached to it. In this case, extract all modifiers
from the list and do a test allocation with your allocator to choose the preferred
modifier. Fixate on that `EnumFormat` by announcing a `SPA_PARAM_EnumFormat` with
only one modifier in the `SPA_CHOICE_Enum` and without the
`SPA_POD_PROP_FLAG_DONT_FIXATE` flag, followed by the previous announced
`EnumFormat`. This will retrigger the `param_changed` event with an
`SPA_PARAM_Format` as described below.
If the `SPA_PARAM_Format` contains a modifier key, without the flag
`SPA_POD_PROP_FLAG_DONT_FIXATE`, it should only contain one value in the
`SPA_CHOICE_Enum`. In this case announce the `SPA_PARAM_Buffers` accordingly
to the selected format and modifier. It is important to query the plane count
of the used format modifier pair and set `SPA_PARAM_BUFFERS_blocks` accordingly.
Note: When test allocating a buffer, collect all possible modifiers, while omitting
`DRM_FORMAT_MOD_INVALID` from the `SPA_FORMAT_VIDEO_modifier` property and
pass them all to the graphics API. If the allocation fails and the list of
possible modifiers contains `DRM_FORMAT_MOD_INVALID`, fall back to allocating
without an explicit modifier if the graphics API allows it.
## add_buffer Hook
This is relevant for producers.
Allocate a DMA-BUF only using the negotiated format and modifier.
## on_event Hook
This is relevant for consumers.
Check the type of the dequeued buffer. If its `SPA_DATA_MemFd` or
`SPA_DATA_MemPtr` use the fallback SHM import mechanism.
If it's `SPA_DATA_DmaBuf`
get the DMA-BUF FDs (the plane count is encoded in the `n_datas` variable of the
`spa_buffer` struct) and import them with the graphics API.
Note: Some graphics APIs have separated functions for the modifier-less case
(`DRM_FORMAT_MOD_INVALID`) or are omitting the modifier, since it might be used
for error handling.
## Example Programs
- \ref video-src-fixate.c "": \snippet{doc} video-src-fixate.c title
- \ref video-play-fixate.c "": \snippet{doc} video-play-fixate.c title
# DMA-BUF Mapping Warning
It's important to make sure all consumers of the PipeWire stream are prepared
to deal with DMA-BUFs. Most DMA-BUFs cannot be treated like shared memory in general
because of the following issues:
- DMA-BUFs can use hardware-specific tiling and compression as described by
modifiers. Thus, a `mmap(3)` on the DMA-BUF FD will not give a linear view of
the buffer contents.
- DMA-BUFs need to be properly synchronized with the asynchronous reads and
writes of the hardware. A `mmap(3)` call is not enough to guarantee proper
synchronization. (Maybe add link to linux syscall doc??)
- Blindly accessing the DMA-BUFs via `mmap(3)` can be extremely slow if the
buffer has been allocated on discrete hardware. Consumers are better off
using a proper graphics API (such as EGL, Vulkan or VA-API) to process the
DMA-BUFs.
# Size of DMA-BUFs
When importing a DMA-BUF with a proper graphics API the size of a single buffer plane
is no relevant property since it will be derived by the driver from the other properties.
Therefore consumers should ignore the field `maxsize` of a `spa_data` and the field
`size` of a `spa_chunk` struct. Producers are allowed to set both to 0.
In cases where mapping a single plane is required the size should be obtained locally
via the filedescriptor.
# v4l2
Another use case for streaming via DMA-BUFs are exporting a camera feed from v4l2
as DMA-BUFs. Those are located in the main memory where it is possible to mmap them.
This should be done as follows: Neither producer nor consumer should announce a
modifier, but both should include `1 << SPA_DATA_DmaBuf` in the
`SPA_PARAM_BUFFERS_dataType` property. It's the the responsibility of the producer
while the `add_buffer` event to choose DMA-BUF as the used buffer type even though
no modifier is present, if it can guarantee, that the used buffer is mmapable.
Note: For now v4l2 uses planar buffers without modifiers. This is the reason for
this special case.
*/
|
