1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
|
.TH PCILMR 8 "05 April 2024" "pciutils-3.12.0" "The PCI Utilities"
.SH NAME
pcilmr \- margin PCIe Links
.SH SYNOPSIS
.B pcilmr
.RB [ "--margin" ]
.RI [ "<margining options>" ] " <downstream component> ..."
.br
.B pcilmr --full
.RI [ "<margining options>" ]
.br
.B pcilmr --scan
.SH CONFIGURATION
List of the requirements for links and system settings
to run the margining test.
.B BIOS settings
(depends on the system, relevant for server baseboards
with Xeon CPUs):
.IP \[bu] 3
Turn off PCIe Leaky Bucket Feature, Re-Equalization and Link Degradation;
.IP \[bu]
Set Error Thresholds to 0;
.IP \[bu]
Intel VMD for NVMe SSDs - in case of strange behavior of the
.BR pcilmr,
try to run it with the VMD turned off.
.PP
.B Device (link) requirements:
.IP
.I "Configured by the user before running the utility, the utility does not change them:"
.RS
.IP \[bu] 3
The current Link data rate must be 16.0 GT/s or higher (right now
utility supports 16 GT/s and 32 GT/s Links);
.IP \[bu]
Link Downstream Component must be at D0 Power Management State.
.RE
.IP
.I "Configured by the utility during operation, utility set them to their original "
.I "state after receiving the results:"
.RS
.IP \[bu] 3
The ASPM must be disabled in both the Downstream Port and Upstream Port;
.IP \[bu]
The Hardware Autonomous Speed Disable bit of the Link Control 2 register must be Set in both the
Downstream Port and Upstream Port;
.IP \[bu]
The Hardware Autonomous Width Disable bit of the Link Control register must be Set in both the
Downstream Port and Upstream Port.
.SH DESCRIPTION
.B pcilmr
utility allows you to take advantage of the PCIe Lane Margining at the Receiver
capability which is mandatory for all Ports supporting a data rate of 16.0 GT/s or
higher, including Pseudo Ports (Retimers). Lane Margining at Receiver enables system
software to obtain the margin information of a given Receiver while the Link is in the
L0 state. The margin information includes both voltage and time, in either direction from
the current Receiver position. Margining support for timing is required, while support
for voltage is optional at 16.0 GT/s and required at 32.0 GT/s and higher data rates. Also,
independent time margining and independent voltage margining is optional.
Utility allows to get an approximation of the eye margin diagram in the form of a rhombus
(by four points). Lane Margining at the Receiver capability enables users to margin PCIe
links without a hardware debugger and without the need to stop the target system. Utility
can be useful to debug link issues due to receiver margins.
However, the utility results may be not particularly accurate and, as it was found out during
testing, specific devices provide rather dubious capability support and the reliability of
the information they provide is questionable. The PCIe specification provides reference values
for the eye diagram, which are also used by the
.B pcilmr
to evaluate the results, but it seems that it makes sense to contact the
manufacturer of a particular device for references.
The PCIe Base Specification Revision 5.0 sets allowed range for Timing Margin from 20%\~UI to 50%\~UI and
for Voltage Margin from 50\~mV to 500\~mV. Utility uses 30%\~UI as the recommended
value for Timing - taken from NVIDIA presentation ("PCIe 4.0 Mass Electrical Margins Data
Collection").
.B pcilmr
requires root privileges (to access Extended Configuration Space), but during our testing
there were no problems with the devices and they successfully returned to their normal initial
state after the end of testing.
.SH OPTIONS
.SS Device Specifier
.B "<device/component>" \t
.RI [ "<domain>" :] <bus> : <dev> . <func>
(see
.BR lspci (8))
.SS Utility Modes
.TP
.BI --margin " <downstream component> ..."
Margin selected Links.
.TP
.B --full
Margin all ready for testing (in a meaning similar to the
.B --scan
option) Links in the system (one by one).
.TP
.B --scan
Scan for Links with negotiated speed 16 GT/s or higher. Mark "Ready" those of them
in which at least one of the Link sides have Margining Ready bit set meaning that
these Links are ready for testing and you can run utility on them.
.SS Margining Test options
.TP
.B -c
Print Device Lane Margining Capabilities only. Do not run margining.
.TP
\fB\-l\fI <lane>\fP[\fI,<lane>...\fP]
Specify lanes for margining.
.br
Remember that Device may use Lane Reversal for Lane numbering. However, utility
uses logical lane numbers in arguments and for logging. Utility will automatically
determine Lane Reversal and tune its calls.
.br
Default: all link lanes.
.TP
.BI -e " <errors>"
Specify Error Count Limit for margining.
.br
Default: 4.
.TP
\fB-r\fI <recvn>\fP[\fI,<recvn>...\fP]
Specify Receivers to select margining targets.
.br
Default: all available Receivers (including Retimers).
.TP
.BI -p " <parallel_lanes>"
Specify number of lanes to margin simultaneously.
.br
According to spec it's possible for Receiver to margin up to MaxLanes + 1
lanes simultaneously, but during testing, performing margining on several
lanes simultaneously led to results that were different from sequential
margining, so this feature requires additional verification and
.I -p
option right now is for experiments mostly.
.br
Default: 1.
.PP
.B "Use only one of -T/-t options at the same time (same for -V/-v)."
.br
.B "Without these options utility will use MaxSteps from Device"
.B "capabilities as test limit."
.TP
.B -T
Time Margining will continue until the Error Count is no more
than an Error Count Limit. Use this option to find Link limit.
.TP
.BI -t " <steps>"
Specify maximum number of steps for Time Margining.
.TP
.B -V
Same as
.I -T
option, but for Voltage.
.TP
.BI -v " <steps>"
Specify maximum number of steps for Voltage Margining.
.SS Margining Log options
.TP
.BI -o " <directory>"
Save margining results in csv form into the specified directory. Utility
will generate file with the name in form of
.RI "\[dq]lmr_" "<downstream component>" "_Rx" # _ <timestamp> ".csv\[dq]"
for each successfully tested receiver.
.SH EXAMPLES
Utility syntax example:
.RS
.BI "pcilmr -l" " 0,1 " "-r" " 1,6 " "-TV" " ab:0.0 52:0.0"
.RE
.UR https://gist.github.com/bombanya/f2b15263712757ffba1a11eea011c419
Examples of collected results on different systems.
.UE
.SH SEE ALSO
.nh
.BR lspci (8),
.B PCI Express Base Specification (Lane Margining at Receiver)
.hy
|
