summaryrefslogtreecommitdiffstats
path: root/man7/cpuset.7
blob: 2db2bfcfa0103922f9011dbfb4f3d38579012f2a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
.\" Copyright (c) 2008 Silicon Graphics, Inc.
.\"
.\" Author: Paul Jackson (http://oss.sgi.com/projects/cpusets)
.\"
.\" SPDX-License-Identifier: GPL-2.0-only
.\"
.TH cpuset 7 2023-10-31 "Linux man-pages 6.7"
.SH NAME
cpuset \- confine processes to processor and memory node subsets
.SH DESCRIPTION
The cpuset filesystem is a pseudo-filesystem interface
to the kernel cpuset mechanism,
which is used to control the processor placement
and memory placement of processes.
It is commonly mounted at
.IR /dev/cpuset .
.P
On systems with kernels compiled with built in support for cpusets,
all processes are attached to a cpuset, and cpusets are always present.
If a system supports cpusets, then it will have the entry
.B nodev cpuset
in the file
.IR /proc/filesystems .
By mounting the cpuset filesystem (see the
.B EXAMPLES
section below),
the administrator can configure the cpusets on a system
to control the processor and memory placement of processes
on that system.
By default, if the cpuset configuration
on a system is not modified or if the cpuset filesystem
is not even mounted, then the cpuset mechanism,
though present, has no effect on the system's behavior.
.P
A cpuset defines a list of CPUs and memory nodes.
.P
The CPUs of a system include all the logical processing
units on which a process can execute, including, if present,
multiple processor cores within a package and Hyper-Threads
within a processor core.
Memory nodes include all distinct
banks of main memory; small and SMP systems typically have
just one memory node that contains all the system's main memory,
while NUMA (non-uniform memory access) systems have multiple memory nodes.
.P
Cpusets are represented as directories in a hierarchical
pseudo-filesystem, where the top directory in the hierarchy
.RI ( /dev/cpuset )
represents the entire system (all online CPUs and memory nodes)
and any cpuset that is the child (descendant) of
another parent cpuset contains a subset of that parent's
CPUs and memory nodes.
The directories and files representing cpusets have normal
filesystem permissions.
.P
Every process in the system belongs to exactly one cpuset.
A process is confined to run only on the CPUs in
the cpuset it belongs to, and to allocate memory only
on the memory nodes in that cpuset.
When a process
.BR fork (2)s,
the child process is placed in the same cpuset as its parent.
With sufficient privilege, a process may be moved from one
cpuset to another and the allowed CPUs and memory nodes
of an existing cpuset may be changed.
.P
When the system begins booting, a single cpuset is
defined that includes all CPUs and memory nodes on the
system, and all processes are in that cpuset.
During the boot process, or later during normal system operation,
other cpusets may be created, as subdirectories of this top cpuset,
under the control of the system administrator,
and processes may be placed in these other cpusets.
.P
Cpusets are integrated with the
.BR sched_setaffinity (2)
scheduling affinity mechanism and the
.BR mbind (2)
and
.BR set_mempolicy (2)
memory-placement mechanisms in the kernel.
Neither of these mechanisms let a process make use
of a CPU or memory node that is not allowed by that process's cpuset.
If changes to a process's cpuset placement conflict with these
other mechanisms, then cpuset placement is enforced
even if it means overriding these other mechanisms.
The kernel accomplishes this overriding by silently
restricting the CPUs and memory nodes requested by
these other mechanisms to those allowed by the
invoking process's cpuset.
This can result in these
other calls returning an error, if for example, such
a call ends up requesting an empty set of CPUs or
memory nodes, after that request is restricted to
the invoking process's cpuset.
.P
Typically, a cpuset is used to manage
the CPU and memory-node confinement for a set of
cooperating processes such as a batch scheduler job, and these
other mechanisms are used to manage the placement of
individual processes or memory regions within that set or job.
.SH FILES
Each directory below
.I /dev/cpuset
represents a cpuset and contains a fixed set of pseudo-files
describing the state of that cpuset.
.P
New cpusets are created using the
.BR mkdir (2)
system call or the
.BR mkdir (1)
command.
The properties of a cpuset, such as its flags, allowed
CPUs and memory nodes, and attached processes, are queried and modified
by reading or writing to the appropriate file in that cpuset's directory,
as listed below.
.P
The pseudo-files in each cpuset directory are automatically created when
the cpuset is created, as a result of the
.BR mkdir (2)
invocation.
It is not possible to directly add or remove these pseudo-files.
.P
A cpuset directory that contains no child cpuset directories,
and has no attached processes, can be removed using
.BR rmdir (2)
or
.BR rmdir (1).
It is not necessary, or possible,
to remove the pseudo-files inside the directory before removing it.
.P
The pseudo-files in each cpuset directory are
small text files that may be read and
written using traditional shell utilities such as
.BR cat (1),
and
.BR echo (1),
or from a program by using file I/O library functions or system calls,
such as
.BR open (2),
.BR read (2),
.BR write (2),
and
.BR close (2).
.P
The pseudo-files in a cpuset directory represent internal kernel
state and do not have any persistent image on disk.
Each of these per-cpuset files is listed and described below.
.\" ====================== tasks ======================
.TP
.I tasks
List of the process IDs (PIDs) of the processes in that cpuset.
The list is formatted as a series of ASCII
decimal numbers, each followed by a newline.
A process may be added to a cpuset (automatically removing
it from the cpuset that previously contained it) by writing its
PID to that cpuset's
.I tasks
file (with or without a trailing newline).
.IP
.B Warning:
only one PID may be written to the
.I tasks
file at a time.
If a string is written that contains more
than one PID, only the first one will be used.
.\" =================== notify_on_release ===================
.TP
.I notify_on_release
Flag (0 or 1).
If set (1), that cpuset will receive special handling
after it is released, that is, after all processes cease using
it (i.e., terminate or are moved to a different cpuset)
and all child cpuset directories have been removed.
See the \fBNotify On Release\fR section, below.
.\" ====================== cpus ======================
.TP
.I cpuset.cpus
List of the physical numbers of the CPUs on which processes
in that cpuset are allowed to execute.
See \fBList Format\fR below for a description of the
format of
.IR cpus .
.IP
The CPUs allowed to a cpuset may be changed by
writing a new list to its
.I cpus
file.
.\" ==================== cpu_exclusive ====================
.TP
.I cpuset.cpu_exclusive
Flag (0 or 1).
If set (1), the cpuset has exclusive use of
its CPUs (no sibling or cousin cpuset may overlap CPUs).
By default, this is off (0).
Newly created cpusets also initially default this to off (0).
.IP
Two cpusets are
.I sibling
cpusets if they share the same parent cpuset in the
.I /dev/cpuset
hierarchy.
Two cpusets are
.I cousin
cpusets if neither is the ancestor of the other.
Regardless of the
.I cpu_exclusive
setting, if one cpuset is the ancestor of another,
and if both of these cpusets have nonempty
.IR cpus ,
then their
.I cpus
must overlap, because the
.I cpus
of any cpuset are always a subset of the
.I cpus
of its parent cpuset.
.\" ====================== mems ======================
.TP
.I cpuset.mems
List of memory nodes on which processes in this cpuset are
allowed to allocate memory.
See \fBList Format\fR below for a description of the
format of
.IR mems .
.\" ==================== mem_exclusive ====================
.TP
.I cpuset.mem_exclusive
Flag (0 or 1).
If set (1), the cpuset has exclusive use of
its memory nodes (no sibling or cousin may overlap).
Also if set (1), the cpuset is a \fBHardwall\fR cpuset (see below).
By default, this is off (0).
Newly created cpusets also initially default this to off (0).
.IP
Regardless of the
.I mem_exclusive
setting, if one cpuset is the ancestor of another,
then their memory nodes must overlap, because the memory
nodes of any cpuset are always a subset of the memory nodes
of that cpuset's parent cpuset.
.\" ==================== mem_hardwall ====================
.TP
.IR cpuset.mem_hardwall " (since Linux 2.6.26)"
Flag (0 or 1).
If set (1), the cpuset is a \fBHardwall\fR cpuset (see below).
Unlike \fBmem_exclusive\fR,
there is no constraint on whether cpusets
marked \fBmem_hardwall\fR may have overlapping
memory nodes with sibling or cousin cpusets.
By default, this is off (0).
Newly created cpusets also initially default this to off (0).
.\" ==================== memory_migrate ====================
.TP
.IR cpuset.memory_migrate " (since Linux 2.6.16)"
Flag (0 or 1).
If set (1), then memory migration is enabled.
By default, this is off (0).
See the \fBMemory Migration\fR section, below.
.\" ==================== memory_pressure ====================
.TP
.IR cpuset.memory_pressure " (since Linux 2.6.16)"
A measure of how much memory pressure the processes in this
cpuset are causing.
See the \fBMemory Pressure\fR section, below.
Unless
.I memory_pressure_enabled
is enabled, always has value zero (0).
This file is read-only.
See the
.B WARNINGS
section, below.
.\" ================= memory_pressure_enabled =================
.TP
.IR cpuset.memory_pressure_enabled " (since Linux 2.6.16)"
Flag (0 or 1).
This file is present only in the root cpuset, normally
.IR /dev/cpuset .
If set (1), the
.I memory_pressure
calculations are enabled for all cpusets in the system.
By default, this is off (0).
See the
\fBMemory Pressure\fR section, below.
.\" ================== memory_spread_page ==================
.TP
.IR cpuset.memory_spread_page " (since Linux 2.6.17)"
Flag (0 or 1).
If set (1), pages in the kernel page cache
(filesystem buffers) are uniformly spread across the cpuset.
By default, this is off (0) in the top cpuset,
and inherited from the parent cpuset in
newly created cpusets.
See the \fBMemory Spread\fR section, below.
.\" ================== memory_spread_slab ==================
.TP
.IR cpuset.memory_spread_slab " (since Linux 2.6.17)"
Flag (0 or 1).
If set (1), the kernel slab caches
for file I/O (directory and inode structures) are
uniformly spread across the cpuset.
By default, is off (0) in the top cpuset,
and inherited from the parent cpuset in
newly created cpusets.
See the \fBMemory Spread\fR section, below.
.\" ================== sched_load_balance ==================
.TP
.IR cpuset.sched_load_balance " (since Linux 2.6.24)"
Flag (0 or 1).
If set (1, the default) the kernel will
automatically load balance processes in that cpuset over
the allowed CPUs in that cpuset.
If cleared (0) the
kernel will avoid load balancing processes in this cpuset,
.I unless
some other cpuset with overlapping CPUs has its
.I sched_load_balance
flag set.
See \fBScheduler Load Balancing\fR, below, for further details.
.\" ================== sched_relax_domain_level ==================
.TP
.IR cpuset.sched_relax_domain_level " (since Linux 2.6.26)"
Integer, between \-1 and a small positive value.
The
.I sched_relax_domain_level
controls the width of the range of CPUs over which the kernel scheduler
performs immediate rebalancing of runnable tasks across CPUs.
If
.I sched_load_balance
is disabled, then the setting of
.I sched_relax_domain_level
does not matter, as no such load balancing is done.
If
.I sched_load_balance
is enabled, then the higher the value of the
.IR sched_relax_domain_level ,
the wider
the range of CPUs over which immediate load balancing is attempted.
See \fBScheduler Relax Domain Level\fR, below, for further details.
.\" ================== proc cpuset ==================
.P
In addition to the above pseudo-files in each directory below
.IR /dev/cpuset ,
each process has a pseudo-file,
.IR /proc/ pid /cpuset ,
that displays the path of the process's cpuset directory
relative to the root of the cpuset filesystem.
.\" ================== proc status ==================
.P
Also the
.IR /proc/ pid /status
file for each process has four added lines,
displaying the process's
.I Cpus_allowed
(on which CPUs it may be scheduled) and
.I Mems_allowed
(on which memory nodes it may obtain memory),
in the two formats \fBMask Format\fR and \fBList Format\fR (see below)
as shown in the following example:
.P
.in +4n
.EX
Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list:     0\-127
Mems_allowed:   ffffffff,ffffffff
Mems_allowed_list:     0\-63
.EE
.in
.P
The "allowed" fields were added in Linux 2.6.24;
the "allowed_list" fields were added in Linux 2.6.26.
.\" ================== EXTENDED CAPABILITIES ==================
.SH EXTENDED CAPABILITIES
In addition to controlling which
.I cpus
and
.I mems
a process is allowed to use, cpusets provide the following
extended capabilities.
.\" ================== Exclusive Cpusets ==================
.SS Exclusive cpusets
If a cpuset is marked
.I cpu_exclusive
or
.IR mem_exclusive ,
no other cpuset, other than a direct ancestor or descendant,
may share any of the same CPUs or memory nodes.
.P
A cpuset that is
.I mem_exclusive
restricts kernel allocations for
buffer cache pages and other internal kernel data pages
commonly shared by the kernel across
multiple users.
All cpusets, whether
.I mem_exclusive
or not, restrict allocations of memory for user space.
This enables configuring a
system so that several independent jobs can share common kernel data,
while isolating each job's user allocation in
its own cpuset.
To do this, construct a large
.I mem_exclusive
cpuset to hold all the jobs, and construct child,
.RI non- mem_exclusive
cpusets for each individual job.
Only a small amount of kernel memory,
such as requests from interrupt handlers, is allowed to be
placed on memory nodes
outside even a
.I mem_exclusive
cpuset.
.\" ================== Hardwall ==================
.SS Hardwall
A cpuset that has
.I mem_exclusive
or
.I mem_hardwall
set is a
.I hardwall
cpuset.
A
.I hardwall
cpuset restricts kernel allocations for page, buffer,
and other data commonly shared by the kernel across multiple users.
All cpusets, whether
.I hardwall
or not, restrict allocations of memory for user space.
.P
This enables configuring a system so that several independent
jobs can share common kernel data, such as filesystem pages,
while isolating each job's user allocation in its own cpuset.
To do this, construct a large
.I hardwall
cpuset to hold
all the jobs, and construct child cpusets for each individual
job which are not
.I hardwall
cpusets.
.P
Only a small amount of kernel memory, such as requests from
interrupt handlers, is allowed to be taken outside even a
.I hardwall
cpuset.
.\" ================== Notify On Release ==================
.SS Notify on release
If the
.I notify_on_release
flag is enabled (1) in a cpuset,
then whenever the last process in the cpuset leaves
(exits or attaches to some other cpuset)
and the last child cpuset of that cpuset is removed,
the kernel will run the command
.IR /sbin/cpuset_release_agent ,
supplying the pathname (relative to the mount point of the
cpuset filesystem) of the abandoned cpuset.
This enables automatic removal of abandoned cpusets.
.P
The default value of
.I notify_on_release
in the root cpuset at system boot is disabled (0).
The default value of other cpusets at creation
is the current value of their parent's
.I notify_on_release
setting.
.P
The command
.I /sbin/cpuset_release_agent
is invoked, with the name
.RI ( /dev/cpuset
relative path)
of the to-be-released cpuset in
.IR argv[1] .
.P
The usual contents of the command
.I /sbin/cpuset_release_agent
is simply the shell script:
.P
.in +4n
.EX
#!/bin/sh
rmdir /dev/cpuset/$1
.EE
.in
.P
As with other flag values below, this flag can
be changed by writing an ASCII
number 0 or 1 (with optional trailing newline)
into the file, to clear or set the flag, respectively.
.\" ================== Memory Pressure ==================
.SS Memory pressure
The
.I memory_pressure
of a cpuset provides a simple per-cpuset running average of
the rate that the processes in a cpuset are attempting to free up in-use
memory on the nodes of the cpuset to satisfy additional memory requests.
.P
This enables batch managers that are monitoring jobs running in dedicated
cpusets to efficiently detect what level of memory pressure that job
is causing.
.P
This is useful both on tightly managed systems running a wide mix of
submitted jobs, which may choose to terminate or reprioritize jobs that
are trying to use more memory than allowed on the nodes assigned them,
and with tightly coupled, long-running, massively parallel scientific
computing jobs that will dramatically fail to meet required performance
goals if they start to use more memory than allowed to them.
.P
This mechanism provides a very economical way for the batch manager
to monitor a cpuset for signs of memory pressure.
It's up to the batch manager or other user code to decide
what action to take if it detects signs of memory pressure.
.P
Unless memory pressure calculation is enabled by setting the pseudo-file
.IR /dev/cpuset/cpuset.memory_pressure_enabled ,
it is not computed for any cpuset, and reads from any
.I memory_pressure
always return zero, as represented by the ASCII string "0\en".
See the \fBWARNINGS\fR section, below.
.P
A per-cpuset, running average is employed for the following reasons:
.IP \[bu] 3
Because this meter is per-cpuset rather than per-process or per virtual
memory region, the system load imposed by a batch scheduler monitoring
this metric is sharply reduced on large systems, because a scan of
the tasklist can be avoided on each set of queries.
.IP \[bu]
Because this meter is a running average rather than an accumulating
counter, a batch scheduler can detect memory pressure with a
single read, instead of having to read and accumulate results
for a period of time.
.IP \[bu]
Because this meter is per-cpuset rather than per-process,
the batch scheduler can obtain the key information\[em]memory
pressure in a cpuset\[em]with a single read, rather than having to
query and accumulate results over all the (dynamically changing)
set of processes in the cpuset.
.P
The
.I memory_pressure
of a cpuset is calculated using a per-cpuset simple digital filter
that is kept within the kernel.
For each cpuset, this filter tracks
the recent rate at which processes attached to that cpuset enter the
kernel direct reclaim code.
.P
The kernel direct reclaim code is entered whenever a process has to
satisfy a memory page request by first finding some other page to
repurpose, due to lack of any readily available already free pages.
Dirty filesystem pages are repurposed by first writing them
to disk.
Unmodified filesystem buffer pages are repurposed
by simply dropping them, though if that page is needed again, it
will have to be reread from disk.
.P
The
.I cpuset.memory_pressure
file provides an integer number representing the recent (half-life of
10 seconds) rate of entries to the direct reclaim code caused by any
process in the cpuset, in units of reclaims attempted per second,
times 1000.
.\" ================== Memory Spread ==================
.SS Memory spread
There are two Boolean flag files per cpuset that control where the
kernel allocates pages for the filesystem buffers and related
in-kernel data structures.
They are called
.I cpuset.memory_spread_page
and
.IR cpuset.memory_spread_slab .
.P
If the per-cpuset Boolean flag file
.I cpuset.memory_spread_page
is set, then
the kernel will spread the filesystem buffers (page cache) evenly
over all the nodes that the faulting process is allowed to use, instead
of preferring to put those pages on the node where the process is running.
.P
If the per-cpuset Boolean flag file
.I cpuset.memory_spread_slab
is set,
then the kernel will spread some filesystem-related slab caches,
such as those for inodes and directory entries, evenly over all the nodes
that the faulting process is allowed to use, instead of preferring to
put those pages on the node where the process is running.
.P
The setting of these flags does not affect the data segment
(see
.BR brk (2))
or stack segment pages of a process.
.P
By default, both kinds of memory spreading are off and the kernel
prefers to allocate memory pages on the node local to where the
requesting process is running.
If that node is not allowed by the
process's NUMA memory policy or cpuset configuration or if there are
insufficient free memory pages on that node, then the kernel looks
for the nearest node that is allowed and has sufficient free memory.
.P
When new cpusets are created, they inherit the memory spread settings
of their parent.
.P
Setting memory spreading causes allocations for the affected page or
slab caches to ignore the process's NUMA memory policy and be spread
instead.
However, the effect of these changes in memory placement
caused by cpuset-specified memory spreading is hidden from the
.BR mbind (2)
or
.BR set_mempolicy (2)
calls.
These two NUMA memory policy calls always appear to behave as if
no cpuset-specified memory spreading is in effect, even if it is.
If cpuset memory spreading is subsequently turned off, the NUMA
memory policy most recently specified by these calls is automatically
reapplied.
.P
Both
.I cpuset.memory_spread_page
and
.I cpuset.memory_spread_slab
are Boolean flag files.
By default, they contain "0", meaning that the feature is off
for that cpuset.
If a "1" is written to that file, that turns the named feature on.
.P
Cpuset-specified memory spreading behaves similarly to what is known
(in other contexts) as round-robin or interleave memory placement.
.P
Cpuset-specified memory spreading can provide substantial performance
improvements for jobs that:
.IP \[bu] 3
need to place thread-local data on
memory nodes close to the CPUs which are running the threads that most
frequently access that data; but also
.IP \[bu]
need to access large filesystem data sets that must to be spread
across the several nodes in the job's cpuset in order to fit.
.P
Without this policy,
the memory allocation across the nodes in the job's cpuset
can become very uneven,
especially for jobs that might have just a single
thread initializing or reading in the data set.
.\" ================== Memory Migration ==================
.SS Memory migration
Normally, under the default setting (disabled) of
.IR cpuset.memory_migrate ,
once a page is allocated (given a physical page
of main memory), then that page stays on whatever node it
was allocated, so long as it remains allocated, even if the
cpuset's memory-placement policy
.I mems
subsequently changes.
.P
When memory migration is enabled in a cpuset, if the
.I mems
setting of the cpuset is changed, then any memory page in use by any
process in the cpuset that is on a memory node that is no longer
allowed will be migrated to a memory node that is allowed.
.P
Furthermore, if a process is moved into a cpuset with
.I memory_migrate
enabled, any memory pages it uses that were on memory nodes allowed
in its previous cpuset, but which are not allowed in its new cpuset,
will be migrated to a memory node allowed in the new cpuset.
.P
The relative placement of a migrated page within
the cpuset is preserved during these migration operations if possible.
For example,
if the page was on the second valid node of the prior cpuset,
then the page will be placed on the second valid node of the new cpuset,
if possible.
.\" ================== Scheduler Load Balancing ==================
.SS Scheduler load balancing
The kernel scheduler automatically load balances processes.
If one CPU is underutilized,
the kernel will look for processes on other more
overloaded CPUs and move those processes to the underutilized CPU,
within the constraints of such placement mechanisms as cpusets and
.BR sched_setaffinity (2).
.P
The algorithmic cost of load balancing and its impact on key shared
kernel data structures such as the process list increases more than
linearly with the number of CPUs being balanced.
For example, it
costs more to load balance across one large set of CPUs than it does
to balance across two smaller sets of CPUs, each of half the size
of the larger set.
(The precise relationship between the number of CPUs being balanced
and the cost of load balancing depends
on implementation details of the kernel process scheduler, which is
subject to change over time, as improved kernel scheduler algorithms
are implemented.)
.P
The per-cpuset flag
.I sched_load_balance
provides a mechanism to suppress this automatic scheduler load
balancing in cases where it is not needed and suppressing it would have
worthwhile performance benefits.
.P
By default, load balancing is done across all CPUs, except those
marked isolated using the kernel boot time "isolcpus=" argument.
(See \fBScheduler Relax Domain Level\fR, below, to change this default.)
.P
This default load balancing across all CPUs is not well suited to
the following two situations:
.IP \[bu] 3
On large systems, load balancing across many CPUs is expensive.
If the system is managed using cpusets to place independent jobs
on separate sets of CPUs, full load balancing is unnecessary.
.IP \[bu]
Systems supporting real-time on some CPUs need to minimize
system overhead on those CPUs, including avoiding process load
balancing if that is not needed.
.P
When the per-cpuset flag
.I sched_load_balance
is enabled (the default setting),
it requests load balancing across
all the CPUs in that cpuset's allowed CPUs,
ensuring that load balancing can move a process (not otherwise pinned,
as by
.BR sched_setaffinity (2))
from any CPU in that cpuset to any other.
.P
When the per-cpuset flag
.I sched_load_balance
is disabled, then the
scheduler will avoid load balancing across the CPUs in that cpuset,
\fIexcept\fR in so far as is necessary because some overlapping cpuset
has
.I sched_load_balance
enabled.
.P
So, for example, if the top cpuset has the flag
.I sched_load_balance
enabled, then the scheduler will load balance across all
CPUs, and the setting of the
.I sched_load_balance
flag in other cpusets has no effect,
as we're already fully load balancing.
.P
Therefore in the above two situations, the flag
.I sched_load_balance
should be disabled in the top cpuset, and only some of the smaller,
child cpusets would have this flag enabled.
.P
When doing this, you don't usually want to leave any unpinned processes in
the top cpuset that might use nontrivial amounts of CPU, as such processes
may be artificially constrained to some subset of CPUs, depending on
the particulars of this flag setting in descendant cpusets.
Even if such a process could use spare CPU cycles in some other CPUs,
the kernel scheduler might not consider the possibility of
load balancing that process to the underused CPU.
.P
Of course, processes pinned to a particular CPU can be left in a cpuset
that disables
.I sched_load_balance
as those processes aren't going anywhere else anyway.
.\" ================== Scheduler Relax Domain Level ==================
.SS Scheduler relax domain level
The kernel scheduler performs immediate load balancing whenever
a CPU becomes free or another task becomes runnable.
This load
balancing works to ensure that as many CPUs as possible are usefully
employed running tasks.
The kernel also performs periodic load
balancing off the software clock described in
.BR time (7).
The setting of
.I sched_relax_domain_level
applies only to immediate load balancing.
Regardless of the
.I sched_relax_domain_level
setting, periodic load balancing is attempted over all CPUs
(unless disabled by turning off
.IR sched_load_balance .)
In any case, of course, tasks will be scheduled to run only on
CPUs allowed by their cpuset, as modified by
.BR sched_setaffinity (2)
system calls.
.P
On small systems, such as those with just a few CPUs, immediate load
balancing is useful to improve system interactivity and to minimize
wasteful idle CPU cycles.
But on large systems, attempting immediate
load balancing across a large number of CPUs can be more costly than
it is worth, depending on the particular performance characteristics
of the job mix and the hardware.
.P
The exact meaning of the small integer values of
.I sched_relax_domain_level
will depend on internal
implementation details of the kernel scheduler code and on the
non-uniform architecture of the hardware.
Both of these will evolve
over time and vary by system architecture and kernel version.
.P
As of this writing, when this capability was introduced in Linux
2.6.26, on certain popular architectures, the positive values of
.I sched_relax_domain_level
have the following meanings.
.P
.PD 0
.TP
.B 1
Perform immediate load balancing across Hyper-Thread
siblings on the same core.
.TP
.B 2
Perform immediate load balancing across other cores in the same package.
.TP
.B 3
Perform immediate load balancing across other CPUs
on the same node or blade.
.TP
.B 4
Perform immediate load balancing across over several
(implementation detail) nodes [On NUMA systems].
.TP
.B 5
Perform immediate load balancing across over all CPUs
in system [On NUMA systems].
.PD
.P
The
.I sched_relax_domain_level
value of zero (0) always means
don't perform immediate load balancing,
hence that load balancing is done only periodically,
not immediately when a CPU becomes available or another task becomes
runnable.
.P
The
.I sched_relax_domain_level
value of minus one (\-1)
always means use the system default value.
The system default value can vary by architecture and kernel version.
This system default value can be changed by kernel
boot-time "relax_domain_level=" argument.
.P
In the case of multiple overlapping cpusets which have conflicting
.I sched_relax_domain_level
values, then the highest such value
applies to all CPUs in any of the overlapping cpusets.
In such cases,
.B \-1
is the lowest value,
overridden by any other value,
and
.B 0
is the next lowest value.
.SH FORMATS
The following formats are used to represent sets of
CPUs and memory nodes.
.\" ================== Mask Format ==================
.SS Mask format
The \fBMask Format\fR is used to represent CPU and memory-node bit masks
in the
.IR /proc/ pid /status
file.
.P
This format displays each 32-bit
word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f");
words are filled with leading zeros, if required.
For masks longer than one word, a comma separator is used between words.
Words are displayed in big-endian
order, which has the most significant bit first.
The hex digits within a word are also in big-endian order.
.P
The number of 32-bit words displayed is the minimum number needed to
display all bits of the bit mask, based on the size of the bit mask.
.P
Examples of the \fBMask Format\fR:
.P
.in +4n
.EX
00000001                        # just bit 0 set
40000000,00000000,00000000      # just bit 94 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32\-39 set
00000000,000e3862               # 1,5,6,11\-13,17\-19 set
.EE
.in
.P
A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
.P
.in +4n
.EX
00000001,00000001,00010117
.EE
.in
.P
The first "1" is for bit 64, the
second for bit 32, the third for bit 16, the fourth for bit 8, the
fifth for bit 4, and the "7" is for bits 2, 1, and 0.
.\" ================== List Format ==================
.SS List format
The \fBList Format\fR for
.I cpus
and
.I mems
is a comma-separated list of CPU or memory-node
numbers and ranges of numbers, in ASCII decimal.
.P
Examples of the \fBList Format\fR:
.P
.in +4n
.EX
0\-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0\-2,7,12\-14     # bits 0, 1, 2, 7, 12, 13, and 14 set
.EE
.in
.\" ================== RULES ==================
.SH RULES
The following rules apply to each cpuset:
.IP \[bu] 3
Its CPUs and memory nodes must be a (possibly equal)
subset of its parent's.
.IP \[bu]
It can be marked
.I cpu_exclusive
only if its parent is.
.IP \[bu]
It can be marked
.I mem_exclusive
only if its parent is.
.IP \[bu]
If it is
.IR cpu_exclusive ,
its CPUs may not overlap any sibling.
.IP \[bu]
If it is
.IR mem_exclusive ,
its memory nodes may not overlap any sibling.
.\" ================== PERMISSIONS ==================
.SH PERMISSIONS
The permissions of a cpuset are determined by the permissions
of the directories and pseudo-files in the cpuset filesystem,
normally mounted at
.IR /dev/cpuset .
.P
For instance, a process can put itself in some other cpuset (than
its current one) if it can write the
.I tasks
file for that cpuset.
This requires execute permission on the encompassing directories
and write permission on the
.I tasks
file.
.P
An additional constraint is applied to requests to place some
other process in a cpuset.
One process may not attach another to
a cpuset unless it would have permission to send that process
a signal (see
.BR kill (2)).
.P
A process may create a child cpuset if it can access and write the
parent cpuset directory.
It can modify the CPUs or memory nodes
in a cpuset if it can access that cpuset's directory (execute
permissions on the each of the parent directories) and write the
corresponding
.I cpus
or
.I mems
file.
.P
There is one minor difference between the manner in which these
permissions are evaluated and the manner in which normal filesystem
operation permissions are evaluated.
The kernel interprets
relative pathnames starting at a process's current working directory.
Even if one is operating on a cpuset file, relative pathnames
are interpreted relative to the process's current working directory,
not relative to the process's current cpuset.
The only ways that
cpuset paths relative to a process's current cpuset can be used are
if either the process's current working directory is its cpuset
(it first did a
.B cd
or
.BR chdir (2)
to its cpuset directory beneath
.IR /dev/cpuset ,
which is a bit unusual)
or if some user code converts the relative cpuset path to a
full filesystem path.
.P
In theory, this means that user code should specify cpusets
using absolute pathnames, which requires knowing the mount point of
the cpuset filesystem (usually, but not necessarily,
.IR /dev/cpuset ).
In practice, all user level code that this author is aware of
simply assumes that if the cpuset filesystem is mounted, then
it is mounted at
.IR /dev/cpuset .
Furthermore, it is common practice for carefully written
user code to verify the presence of the pseudo-file
.I /dev/cpuset/tasks
in order to verify that the cpuset pseudo-filesystem
is currently mounted.
.\" ================== WARNINGS ==================
.SH WARNINGS
.SS Enabling memory_pressure
By default, the per-cpuset file
.I cpuset.memory_pressure
always contains zero (0).
Unless this feature is enabled by writing "1" to the pseudo-file
.IR /dev/cpuset/cpuset.memory_pressure_enabled ,
the kernel does
not compute per-cpuset
.IR memory_pressure .
.SS Using the echo command
When using the
.B echo
command at the shell prompt to change the values of cpuset files,
beware that the built-in
.B echo
command in some shells does not display an error message if the
.BR write (2)
system call fails.
.\" Gack!  csh(1)'s echo does this
For example, if the command:
.P
.in +4n
.EX
echo 19 > cpuset.mems
.EE
.in
.P
failed because memory node 19 was not allowed (perhaps
the current system does not have a memory node 19), then the
.B echo
command might not display any error.
It is better to use the
.B /bin/echo
external command to change cpuset file settings, as this
command will display
.BR write (2)
errors, as in the example:
.P
.in +4n
.EX
/bin/echo 19 > cpuset.mems
/bin/echo: write error: Invalid argument
.EE
.in
.\" ================== EXCEPTIONS ==================
.SH EXCEPTIONS
.SS Memory placement
Not all allocations of system memory are constrained by cpusets,
for the following reasons.
.P
If hot-plug functionality is used to remove all the CPUs that are
currently assigned to a cpuset, then the kernel will automatically
update the
.I cpus_allowed
of all processes attached to CPUs in that cpuset
to allow all CPUs.
When memory hot-plug functionality for removing
memory nodes is available, a similar exception is expected to apply
there as well.
In general, the kernel prefers to violate cpuset placement,
rather than starving a process that has had all its allowed CPUs or
memory nodes taken offline.
User code should reconfigure cpusets to refer only to online CPUs
and memory nodes when using hot-plug to add or remove such resources.
.P
A few kernel-critical, internal memory-allocation requests, marked
GFP_ATOMIC, must be satisfied immediately.
The kernel may drop some
request or malfunction if one of these allocations fail.
If such a request cannot be satisfied within the current process's cpuset,
then we relax the cpuset, and look for memory anywhere we can find it.
It's better to violate the cpuset than stress the kernel.
.P
Allocations of memory requested by kernel drivers while processing
an interrupt lack any relevant process context, and are not confined
by cpusets.
.SS Renaming cpusets
You can use the
.BR rename (2)
system call to rename cpusets.
Only simple renaming is supported; that is, changing the name of a cpuset
directory is permitted, but moving a directory into
a different directory is not permitted.
.\" ================== ERRORS ==================
.SH ERRORS
The Linux kernel implementation of cpusets sets
.I errno
to specify the reason for a failed system call affecting cpusets.
.P
The possible
.I errno
settings and their meaning when set on
a failed cpuset call are as listed below.
.TP
.B E2BIG
Attempted a
.BR write (2)
on a special cpuset file
with a length larger than some kernel-determined upper
limit on the length of such writes.
.TP
.B EACCES
Attempted to
.BR write (2)
the process ID (PID) of a process to a cpuset
.I tasks
file when one lacks permission to move that process.
.TP
.B EACCES
Attempted to add, using
.BR write (2),
a CPU or memory node to a cpuset, when that CPU or memory node was
not already in its parent.
.TP
.B EACCES
Attempted to set, using
.BR write (2),
.I cpuset.cpu_exclusive
or
.I cpuset.mem_exclusive
on a cpuset whose parent lacks the same setting.
.TP
.B EACCES
Attempted to
.BR write (2)
a
.I cpuset.memory_pressure
file.
.TP
.B EACCES
Attempted to create a file in a cpuset directory.
.TP
.B EBUSY
Attempted to remove, using
.BR rmdir (2),
a cpuset with attached processes.
.TP
.B EBUSY
Attempted to remove, using
.BR rmdir (2),
a cpuset with child cpusets.
.TP
.B EBUSY
Attempted to remove
a CPU or memory node from a cpuset
that is also in a child of that cpuset.
.TP
.B EEXIST
Attempted to create, using
.BR mkdir (2),
a cpuset that already exists.
.TP
.B EEXIST
Attempted to
.BR rename (2)
a cpuset to a name that already exists.
.TP
.B EFAULT
Attempted to
.BR read (2)
or
.BR write (2)
a cpuset file using
a buffer that is outside the writing processes accessible address space.
.TP
.B EINVAL
Attempted to change a cpuset, using
.BR write (2),
in a way that would violate a
.I cpu_exclusive
or
.I mem_exclusive
attribute of that cpuset or any of its siblings.
.TP
.B EINVAL
Attempted to
.BR write (2)
an empty
.I cpuset.cpus
or
.I cpuset.mems
list to a cpuset which has attached processes or child cpusets.
.TP
.B EINVAL
Attempted to
.BR write (2)
a
.I cpuset.cpus
or
.I cpuset.mems
list which included a range with the second number smaller than
the first number.
.TP
.B EINVAL
Attempted to
.BR write (2)
a
.I cpuset.cpus
or
.I cpuset.mems
list which included an invalid character in the string.
.TP
.B EINVAL
Attempted to
.BR write (2)
a list to a
.I cpuset.cpus
file that did not include any online CPUs.
.TP
.B EINVAL
Attempted to
.BR write (2)
a list to a
.I cpuset.mems
file that did not include any online memory nodes.
.TP
.B EINVAL
Attempted to
.BR write (2)
a list to a
.I cpuset.mems
file that included a node that held no memory.
.TP
.B EIO
Attempted to
.BR write (2)
a string to a cpuset
.I tasks
file that
does not begin with an ASCII decimal integer.
.TP
.B EIO
Attempted to
.BR rename (2)
a cpuset into a different directory.
.TP
.B ENAMETOOLONG
Attempted to
.BR read (2)
a
.IR /proc/ pid /cpuset
file for a cpuset path that is longer than the kernel page size.
.TP
.B ENAMETOOLONG
Attempted to create, using
.BR mkdir (2),
a cpuset whose base directory name is longer than 255 characters.
.TP
.B ENAMETOOLONG
Attempted to create, using
.BR mkdir (2),
a cpuset whose full pathname,
including the mount point (typically "/dev/cpuset/") prefix,
is longer than 4095 characters.
.TP
.B ENODEV
The cpuset was removed by another process at the same time as a
.BR write (2)
was attempted on one of the pseudo-files in the cpuset directory.
.TP
.B ENOENT
Attempted to create, using
.BR mkdir (2),
a cpuset in a parent cpuset that doesn't exist.
.TP
.B ENOENT
Attempted to
.BR access (2)
or
.BR open (2)
a nonexistent file in a cpuset directory.
.TP
.B ENOMEM
Insufficient memory is available within the kernel; can occur
on a variety of system calls affecting cpusets, but only if the
system is extremely short of memory.
.TP
.B ENOSPC
Attempted to
.BR write (2)
the process ID (PID)
of a process to a cpuset
.I tasks
file when the cpuset had an empty
.I cpuset.cpus
or empty
.I cpuset.mems
setting.
.TP
.B ENOSPC
Attempted to
.BR write (2)
an empty
.I cpuset.cpus
or
.I cpuset.mems
setting to a cpuset that
has tasks attached.
.TP
.B ENOTDIR
Attempted to
.BR rename (2)
a nonexistent cpuset.
.TP
.B EPERM
Attempted to remove a file from a cpuset directory.
.TP
.B ERANGE
Specified a
.I cpuset.cpus
or
.I cpuset.mems
list to the kernel which included a number too large for the kernel
to set in its bit masks.
.TP
.B ESRCH
Attempted to
.BR write (2)
the process ID (PID) of a nonexistent process to a cpuset
.I tasks
file.
.\" ================== VERSIONS ==================
.SH VERSIONS
Cpusets appeared in Linux 2.6.12.
.\" ================== NOTES ==================
.SH NOTES
Despite its name, the
.I pid
parameter is actually a thread ID,
and each thread in a threaded group can be attached to a different
cpuset.
The value returned from a call to
.BR gettid (2)
can be passed in the argument
.IR pid .
.\" ================== BUGS ==================
.SH BUGS
.I cpuset.memory_pressure
cpuset files can be opened
for writing, creation, or truncation, but then the
.BR write (2)
fails with
.I errno
set to
.BR EACCES ,
and the creation and truncation options on
.BR open (2)
have no effect.
.\" ================== EXAMPLES ==================
.SH EXAMPLES
The following examples demonstrate querying and setting cpuset
options using shell commands.
.SS Creating and attaching to a cpuset.
To create a new cpuset and attach the current command shell to it,
the steps are:
.P
.PD 0
.IP (1) 5
mkdir /dev/cpuset (if not already done)
.IP (2)
mount \-t cpuset none /dev/cpuset (if not already done)
.IP (3)
Create the new cpuset using
.BR mkdir (1).
.IP (4)
Assign CPUs and memory nodes to the new cpuset.
.IP (5)
Attach the shell to the new cpuset.
.PD
.P
For example, the following sequence of commands will set up a cpuset
named "Charlie", containing just CPUs 2 and 3, and memory node 1,
and then attach the current shell to that cpuset.
.P
.in +4n
.EX
.RB "$" " mkdir /dev/cpuset"
.RB "$" " mount \-t cpuset cpuset /dev/cpuset"
.RB "$" " cd /dev/cpuset"
.RB "$" " mkdir Charlie"
.RB "$" " cd Charlie"
.RB "$" " /bin/echo 2\-3 > cpuset.cpus"
.RB "$" " /bin/echo 1 > cpuset.mems"
.RB "$" " /bin/echo $$ > tasks"
# The current shell is now running in cpuset Charlie
# The next line should display \[aq]/Charlie\[aq]
.RB "$" " cat /proc/self/cpuset"
.EE
.in
.\"
.SS Migrating a job to different memory nodes.
To migrate a job (the set of processes attached to a cpuset)
to different CPUs and memory nodes in the system, including moving
the memory pages currently allocated to that job,
perform the following steps.
.P
.PD 0
.IP (1) 5
Let's say we want to move the job in cpuset
.I alpha
(CPUs 4\[en]7 and memory nodes 2\[en]3) to a new cpuset
.I beta
(CPUs 16\[en]19 and memory nodes 8\[en]9).
.IP (2)
First create the new cpuset
.IR beta .
.IP (3)
Then allow CPUs 16\[en]19 and memory nodes 8\[en]9 in
.IR beta .
.IP (4)
Then enable
.I memory_migration
in
.IR beta .
.IP (5)
Then move each process from
.I alpha
to
.IR beta .
.PD
.P
The following sequence of commands accomplishes this.
.P
.in +4n
.EX
.RB "$" " cd /dev/cpuset"
.RB "$" " mkdir beta"
.RB "$" " cd beta"
.RB "$" " /bin/echo 16\-19 > cpuset.cpus"
.RB "$" " /bin/echo 8\-9 > cpuset.mems"
.RB "$" " /bin/echo 1 > cpuset.memory_migrate"
.RB "$" " while read i; do /bin/echo $i; done < ../alpha/tasks > tasks"
.EE
.in
.P
The above should move any processes in
.I alpha
to
.IR beta ,
and any memory held by these processes on memory nodes 2\[en]3 to memory
nodes 8\[en]9, respectively.
.P
Notice that the last step of the above sequence did not do:
.P
.in +4n
.EX
.RB "$" " cp ../alpha/tasks tasks"
.EE
.in
.P
The
.I while
loop, rather than the seemingly easier use of the
.BR cp (1)
command, was necessary because
only one process PID at a time may be written to the
.I tasks
file.
.P
The same effect (writing one PID at a time) as the
.I while
loop can be accomplished more efficiently, in fewer keystrokes and in
syntax that works on any shell, but alas more obscurely, by using the
.B \-u
(unbuffered) option of
.BR sed (1):
.P
.in +4n
.EX
.RB "$" " sed \-un p < ../alpha/tasks > tasks"
.EE
.in
.\" ================== SEE ALSO ==================
.SH SEE ALSO
.BR taskset (1),
.BR get_mempolicy (2),
.BR getcpu (2),
.BR mbind (2),
.BR sched_getaffinity (2),
.BR sched_setaffinity (2),
.BR sched_setscheduler (2),
.BR set_mempolicy (2),
.BR CPU_SET (3),
.BR proc (5),
.BR cgroups (7),
.BR numa (7),
.BR sched (7),
.BR migratepages (8),
.BR numactl (8)
.P
.I Documentation/admin\-guide/cgroup\-v1/cpusets.rst
in the Linux kernel source tree
.\" commit 45ce80fb6b6f9594d1396d44dd7e7c02d596fef8
(or
.I Documentation/cgroup\-v1/cpusets.txt
before Linux 4.18, and
.I Documentation/cpusets.txt
before Linux 2.6.29)