summaryrefslogtreecommitdiffstats
path: root/Documentation/admin-guide/mm/damon/usage.rst
blob: 87fdc258d1027b3de482d53527dc24e1a3aca3a8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
.. SPDX-License-Identifier: GPL-2.0

===============
Detailed Usages
===============

DAMON provides below interfaces for different users.

- *DAMON user space tool.*
  `This <https://github.com/awslabs/damo>`_ is for privileged people such as
  system administrators who want a just-working human-friendly interface.
  Using this, users can use the DAMON’s major features in a human-friendly way.
  It may not be highly tuned for special cases, though.  For more detail,
  please refer to its `usage document
  <https://github.com/awslabs/damo/blob/next/USAGE.md>`_.
- *sysfs interface.*
  :ref:`This <sysfs_interface>` is for privileged user space programmers who
  want more optimized use of DAMON.  Using this, users can use DAMON’s major
  features by reading from and writing to special sysfs files.  Therefore,
  you can write and use your personalized DAMON sysfs wrapper programs that
  reads/writes the sysfs files instead of you.  The `DAMON user space tool
  <https://github.com/awslabs/damo>`_ is one example of such programs.
- *Kernel Space Programming Interface.*
  :doc:`This </mm/damon/api>` is for kernel space programmers.  Using this,
  users can utilize every feature of DAMON most flexibly and efficiently by
  writing kernel space DAMON application programs for you.  You can even extend
  DAMON for various address spaces.  For detail, please refer to the interface
  :doc:`document </mm/damon/api>`.
- *debugfs interface. (DEPRECATED!)*
  :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface
  <sysfs_interface>`.  This is deprecated, so users should move to the
  :ref:`sysfs interface <sysfs_interface>`.  If you depend on this and cannot
  move, please report your usecase to damon@lists.linux.dev and
  linux-mm@kvack.org.

.. _sysfs_interface:

sysfs Interface
===============

DAMON sysfs interface is built when ``CONFIG_DAMON_SYSFS`` is defined.  It
creates multiple directories and files under its sysfs directory,
``<sysfs>/kernel/mm/damon/``.  You can control DAMON by writing to and reading
from the files under the directory.

For a short example, users can monitor the virtual address space of a given
workload as below. ::

    # cd /sys/kernel/mm/damon/admin/
    # echo 1 > kdamonds/nr_kdamonds && echo 1 > kdamonds/0/contexts/nr_contexts
    # echo vaddr > kdamonds/0/contexts/0/operations
    # echo 1 > kdamonds/0/contexts/0/targets/nr_targets
    # echo $(pidof <workload>) > kdamonds/0/contexts/0/targets/0/pid_target
    # echo on > kdamonds/0/state

Files Hierarchy
---------------

The files hierarchy of DAMON sysfs interface is shown below.  In the below
figure, parents-children relations are represented with indentations, each
directory is having ``/`` suffix, and files in each directory are separated by
comma (",").

.. parsed-literal::

    :ref:`/sys/kernel/mm/damon <sysfs_root>`/admin
    │ :ref:`kdamonds <sysfs_kdamonds>`/nr_kdamonds
    │ │ :ref:`0 <sysfs_kdamond>`/state,pid
    │ │ │ :ref:`contexts <sysfs_contexts>`/nr_contexts
    │ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations
    │ │ │ │ │ :ref:`monitoring_attrs <sysfs_monitoring_attrs>`/
    │ │ │ │ │ │ intervals/sample_us,aggr_us,update_us
    │ │ │ │ │ │ nr_regions/min,max
    │ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
    │ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target
    │ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions
    │ │ │ │ │ │ │ │ :ref:`0 <sysfs_region>`/start,end
    │ │ │ │ │ │ │ │ ...
    │ │ │ │ │ │ ...
    │ │ │ │ │ :ref:`schemes <sysfs_schemes>`/nr_schemes
    │ │ │ │ │ │ :ref:`0 <sysfs_scheme>`/action,apply_interval_us
    │ │ │ │ │ │ │ :ref:`access_pattern <sysfs_access_pattern>`/
    │ │ │ │ │ │ │ │ sz/min,max
    │ │ │ │ │ │ │ │ nr_accesses/min,max
    │ │ │ │ │ │ │ │ age/min,max
    │ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms
    │ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
    │ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals
    │ │ │ │ │ │ │ │ │ 0/target_value,current_value
    │ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low
    │ │ │ │ │ │ │ :ref:`filters <sysfs_filters>`/nr_filters
    │ │ │ │ │ │ │ │ 0/type,matching,memcg_id
    │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,qt_exceeds
    │ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes
    │ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age
    │ │ │ │ │ │ │ │ ...
    │ │ │ │ │ │ ...
    │ │ │ │ ...
    │ │ ...

.. _sysfs_root:

Root
----

The root of the DAMON sysfs interface is ``<sysfs>/kernel/mm/damon/``, and it
has one directory named ``admin``.  The directory contains the files for
privileged user space programs' control of DAMON.  User space tools or daemons
having the root permission could use this directory.

.. _sysfs_kdamonds:

kdamonds/
---------

Under the ``admin`` directory, one directory, ``kdamonds``, which has files for
controlling the kdamonds (refer to
:ref:`design <damon_design_execution_model_and_data_structures>` for more
details) exists.  In the beginning, this directory has only one file,
``nr_kdamonds``.  Writing a number (``N``) to the file creates the number of
child directories named ``0`` to ``N-1``.  Each directory represents each
kdamond.

.. _sysfs_kdamond:

kdamonds/<N>/
-------------

In each kdamond directory, two files (``state`` and ``pid``) and one directory
(``contexts``) exist.

Reading ``state`` returns ``on`` if the kdamond is currently running, or
``off`` if it is not running.

Users can write below commands for the kdamond to the ``state`` file.

- ``on``: Start running.
- ``off``: Stop running.
- ``commit``: Read the user inputs in the sysfs files except ``state`` file
  again.
- ``commit_schemes_quota_goals``: Read the DAMON-based operation schemes'
  :ref:`quota goals <sysfs_schemes_quota_goals>`.
- ``update_schemes_stats``: Update the contents of stats files for each
  DAMON-based operation scheme of the kdamond.  For details of the stats,
  please refer to :ref:`stats section <sysfs_schemes_stats>`.
- ``update_schemes_tried_regions``: Update the DAMON-based operation scheme
  action tried regions directory for each DAMON-based operation scheme of the
  kdamond.  For details of the DAMON-based operation scheme action tried
  regions directory, please refer to
  :ref:`tried_regions section <sysfs_schemes_tried_regions>`.
- ``update_schemes_tried_bytes``: Update only ``.../tried_regions/total_bytes``
  files.
- ``clear_schemes_tried_regions``: Clear the DAMON-based operating scheme
  action tried regions directory for each DAMON-based operation scheme of the
  kdamond.

If the state is ``on``, reading ``pid`` shows the pid of the kdamond thread.

``contexts`` directory contains files for controlling the monitoring contexts
that this kdamond will execute.

.. _sysfs_contexts:

kdamonds/<N>/contexts/
----------------------

In the beginning, this directory has only one file, ``nr_contexts``.  Writing a
number (``N``) to the file creates the number of child directories named as
``0`` to ``N-1``.  Each directory represents each monitoring context (refer to
:ref:`design <damon_design_execution_model_and_data_structures>` for more
details).  At the moment, only one context per kdamond is supported, so only
``0`` or ``1`` can be written to the file.

.. _sysfs_context:

contexts/<N>/
-------------

In each context directory, two files (``avail_operations`` and ``operations``)
and three directories (``monitoring_attrs``, ``targets``, and ``schemes``)
exist.

DAMON supports multiple types of monitoring operations, including those for
virtual address space and the physical address space.  You can get the list of
available monitoring operations set on the currently running kernel by reading
``avail_operations`` file.  Based on the kernel configuration, the file will
list some or all of below keywords.

 - vaddr: Monitor virtual address spaces of specific processes
 - fvaddr: Monitor fixed virtual address ranges
 - paddr: Monitor the physical address space of the system

Please refer to :ref:`regions sysfs directory <sysfs_regions>` for detailed
differences between the operations sets in terms of the monitoring target
regions.

You can set and get what type of monitoring operations DAMON will use for the
context by writing one of the keywords listed in ``avail_operations`` file and
reading from the ``operations`` file.

.. _sysfs_monitoring_attrs:

contexts/<N>/monitoring_attrs/
------------------------------

Files for specifying attributes of the monitoring including required quality
and efficiency of the monitoring are in ``monitoring_attrs`` directory.
Specifically, two directories, ``intervals`` and ``nr_regions`` exist in this
directory.

Under ``intervals`` directory, three files for DAMON's sampling interval
(``sample_us``), aggregation interval (``aggr_us``), and update interval
(``update_us``) exist.  You can set and get the values in micro-seconds by
writing to and reading from the files.

Under ``nr_regions`` directory, two files for the lower-bound and upper-bound
of DAMON's monitoring regions (``min`` and ``max``, respectively), which
controls the monitoring overhead, exist.  You can set and get the values by
writing to and rading from the files.

For more details about the intervals and monitoring regions range, please refer
to the Design document (:doc:`/mm/damon/design`).

.. _sysfs_targets:

contexts/<N>/targets/
---------------------

In the beginning, this directory has only one file, ``nr_targets``.  Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``.  Each directory represents each monitoring target.

.. _sysfs_target:

targets/<N>/
------------

In each target directory, one file (``pid_target``) and one directory
(``regions``) exist.

If you wrote ``vaddr`` to the ``contexts/<N>/operations``, each target should
be a process.  You can specify the process to DAMON by writing the pid of the
process to the ``pid_target`` file.

.. _sysfs_regions:

targets/<N>/regions
-------------------

When ``vaddr`` monitoring operations set is being used (``vaddr`` is written to
the ``contexts/<N>/operations`` file), DAMON automatically sets and updates the
monitoring target regions so that entire memory mappings of target processes
can be covered.  However, users could want to set the initial monitoring region
to specific address ranges.

In contrast, DAMON do not automatically sets and updates the monitoring target
regions when ``fvaddr`` or ``paddr`` monitoring operations sets are being used
(``fvaddr`` or ``paddr`` have written to the ``contexts/<N>/operations``).
Therefore, users should set the monitoring target regions by themselves in the
cases.

For such cases, users can explicitly set the initial monitoring target regions
as they want, by writing proper values to the files under this directory.

In the beginning, this directory has only one file, ``nr_regions``.  Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``.  Each directory represents each initial monitoring target region.

.. _sysfs_region:

regions/<N>/
------------

In each region directory, you will find two files (``start`` and ``end``).  You
can set and get the start and end addresses of the initial monitoring target
region by writing to and reading from the files, respectively.

Each region should not overlap with others.  ``end`` of directory ``N`` should
be equal or smaller than ``start`` of directory ``N+1``.

.. _sysfs_schemes:

contexts/<N>/schemes/
---------------------

The directory for DAMON-based Operation Schemes (:ref:`DAMOS
<damon_design_damos>`).  Users can get and set the schemes by reading from and
writing to files under this directory.

In the beginning, this directory has only one file, ``nr_schemes``.  Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``.  Each directory represents each DAMON-based operation scheme.

.. _sysfs_scheme:

schemes/<N>/
------------

In each scheme directory, five directories (``access_pattern``, ``quotas``,
``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files
(``action`` and ``apply_interval``) exist.

The ``action`` file is for setting and getting the scheme's :ref:`action
<damon_design_damos_action>`.  The keywords that can be written to and read
from the file and their meaning are as below.

Note that support of each action depends on the running DAMON operations set
:ref:`implementation <sysfs_context>`.

 - ``willneed``: Call ``madvise()`` for the region with ``MADV_WILLNEED``.
   Supported by ``vaddr`` and ``fvaddr`` operations set.
 - ``cold``: Call ``madvise()`` for the region with ``MADV_COLD``.
   Supported by ``vaddr`` and ``fvaddr`` operations set.
 - ``pageout``: Call ``madvise()`` for the region with ``MADV_PAGEOUT``.
   Supported by ``vaddr``, ``fvaddr`` and ``paddr`` operations set.
 - ``hugepage``: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``.
   Supported by ``vaddr`` and ``fvaddr`` operations set.
 - ``nohugepage``: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``.
   Supported by ``vaddr`` and ``fvaddr`` operations set.
 - ``lru_prio``: Prioritize the region on its LRU lists.
   Supported by ``paddr`` operations set.
 - ``lru_deprio``: Deprioritize the region on its LRU lists.
   Supported by ``paddr`` operations set.
 - ``stat``: Do nothing but count the statistics.
   Supported by all operations sets.

The ``apply_interval_us`` file is for setting and getting the scheme's
:ref:`apply_interval <damon_design_damos>` in microseconds.

.. _sysfs_access_pattern:

schemes/<N>/access_pattern/
---------------------------

The directory for the target access :ref:`pattern
<damon_design_damos_access_pattern>` of the given DAMON-based operation scheme.

Under the ``access_pattern`` directory, three directories (``sz``,
``nr_accesses``, and ``age``) each having two files (``min`` and ``max``)
exist.  You can set and get the access pattern for the given scheme by writing
to and reading from the ``min`` and ``max`` files under ``sz``,
``nr_accesses``, and ``age`` directories, respectively.  Note that the ``min``
and the ``max`` form a closed interval.

.. _sysfs_quotas:

schemes/<N>/quotas/
-------------------

The directory for the :ref:`quotas <damon_design_damos_quotas>` of the given
DAMON-based operation scheme.

Under ``quotas`` directory, three files (``ms``, ``bytes``,
``reset_interval_ms``) and two directores (``weights`` and ``goals``) exist.

You can set the ``time quota`` in milliseconds, ``size quota`` in bytes, and
``reset interval`` in milliseconds by writing the values to the three files,
respectively.  Then, DAMON tries to use only up to ``time quota`` milliseconds
for applying the ``action`` to memory regions of the ``access_pattern``, and to
apply the action to only up to ``bytes`` bytes of memory regions within the
``reset_interval_ms``.  Setting both ``ms`` and ``bytes`` zero disables the
quota limits.

Under ``weights`` directory, three files (``sz_permil``,
``nr_accesses_permil``, and ``age_permil``) exist.
You can set the :ref:`prioritization weights
<damon_design_damos_quotas_prioritization>` for size, access frequency, and age
in per-thousand unit by writing the values to the three files under the
``weights`` directory.

.. _sysfs_schemes_quota_goals:

schemes/<N>/quotas/goals/
-------------------------

The directory for the :ref:`automatic quota tuning goals
<damon_design_damos_quotas_auto_tuning>` of the given DAMON-based operation
scheme.

In the beginning, this directory has only one file, ``nr_goals``.  Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``.  Each directory represents each goal and current achievement.
Among the multiple feedback, the best one is used.

Each goal directory contains two files, namely ``target_value`` and
``current_value``.  Users can set and get any number to those files to set the
feedback.  User space main workload's latency or throughput, system metrics
like free memory ratio or memory pressure stall time (PSI) could be example
metrics for the values.  Note that users should write
``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond
directory <sysfs_kdamond>` to pass the feedback to DAMON.

.. _sysfs_watermarks:

schemes/<N>/watermarks/
-----------------------

The directory for the :ref:`watermarks <damon_design_damos_watermarks>` of the
given DAMON-based operation scheme.

Under the watermarks directory, five files (``metric``, ``interval_us``,
``high``, ``mid``, and ``low``) for setting the metric, the time interval
between check of the metric, and the three watermarks exist.  You can set and
get the five values by writing to the files, respectively.

Keywords and meanings of those that can be written to the ``metric`` file are
as below.

 - none: Ignore the watermarks
 - free_mem_rate: System's free memory rate (per thousand)

The ``interval`` should written in microseconds unit.

.. _sysfs_filters:

schemes/<N>/filters/
--------------------

The directory for the :ref:`filters <damon_design_damos_filters>` of the given
DAMON-based operation scheme.

In the beginning, this directory has only one file, ``nr_filters``.  Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``.  Each directory represents each filter.  The filters are evaluated
in the numeric order.

Each filter directory contains six files, namely ``type``, ``matcing``,
``memcg_path``, ``addr_start``, ``addr_end``, and ``target_idx``.  To ``type``
file, you can write one of four special keywords: ``anon`` for anonymous pages,
``memcg`` for specific memory cgroup, ``addr`` for specific address range (an
open-ended interval), or ``target`` for specific DAMON monitoring target
filtering.  In case of the memory cgroup filtering, you can specify the memory
cgroup of the interest by writing the path of the memory cgroup from the
cgroups mount point to ``memcg_path`` file.  In case of the address range
filtering, you can specify the start and end address of the range to
``addr_start`` and ``addr_end`` files, respectively.  For the DAMON monitoring
target filtering, you can specify the index of the target between the list of
the DAMON context's monitoring targets list to ``target_idx`` file.  You can
write ``Y`` or ``N`` to ``matching`` file to filter out pages that does or does
not match to the type, respectively.  Then, the scheme's action will not be
applied to the pages that specified to be filtered out.

For example, below restricts a DAMOS action to be applied to only non-anonymous
pages of all memory cgroups except ``/having_care_already``.::

    # echo 2 > nr_filters
    # # filter out anonymous pages
    echo anon > 0/type
    echo Y > 0/matching
    # # further filter out all cgroups except one at '/having_care_already'
    echo memcg > 1/type
    echo /having_care_already > 1/memcg_path
    echo Y > 1/matching

Note that ``anon`` and ``memcg`` filters are currently supported only when
``paddr`` :ref:`implementation <sysfs_context>` is being used.

Also, memory regions that are filtered out by ``addr`` or ``target`` filters
are not counted as the scheme has tried to those, while regions that filtered
out by other type filters are counted as the scheme has tried to.  The
difference is applied to :ref:`stats <damos_stats>` and
:ref:`tried regions <sysfs_schemes_tried_regions>`.

.. _sysfs_schemes_stats:

schemes/<N>/stats/
------------------

DAMON counts the total number and bytes of regions that each scheme is tried to
be applied, the two numbers for the regions that each scheme is successfully
applied, and the total number of the quota limit exceeds.  This statistics can
be used for online analysis or tuning of the schemes.

The statistics can be retrieved by reading the files under ``stats`` directory
(``nr_tried``, ``sz_tried``, ``nr_applied``, ``sz_applied``, and
``qt_exceeds``), respectively.  The files are not updated in real time, so you
should ask DAMON sysfs interface to update the content of the files for the
stats by writing a special keyword, ``update_schemes_stats`` to the relevant
``kdamonds/<N>/state`` file.

.. _sysfs_schemes_tried_regions:

schemes/<N>/tried_regions/
--------------------------

This directory initially has one file, ``total_bytes``.

When a special keyword, ``update_schemes_tried_regions``, is written to the
relevant ``kdamonds/<N>/state`` file, DAMON updates the ``total_bytes`` file so
that reading it returns the total size of the scheme tried regions, and creates
directories named integer starting from ``0`` under this directory.  Each
directory contains files exposing detailed information about each of the memory
region that the corresponding scheme's ``action`` has tried to be applied under
this directory, during next :ref:`apply interval <damon_design_damos>` of the
corresponding scheme.  The information includes address range, ``nr_accesses``,
and ``age`` of the region.

Writing ``update_schemes_tried_bytes`` to the relevant ``kdamonds/<N>/state``
file will only update the ``total_bytes`` file, and will not create the
subdirectories.

The directories will be removed when another special keyword,
``clear_schemes_tried_regions``, is written to the relevant
``kdamonds/<N>/state`` file.

The expected usage of this directory is investigations of schemes' behaviors,
and query-like efficient data access monitoring results retrievals.  For the
latter use case, in particular, users can set the ``action`` as ``stat`` and
set the ``access pattern`` as their interested pattern that they want to query.

.. _sysfs_schemes_tried_region:

tried_regions/<N>/
------------------

In each region directory, you will find four files (``start``, ``end``,
``nr_accesses``, and ``age``).  Reading the files will show the start and end
addresses, ``nr_accesses``, and ``age`` of the region that corresponding
DAMON-based operation scheme ``action`` has tried to be applied.

Example
~~~~~~~

Below commands applies a scheme saying "If a memory region of size in [4KiB,
8KiB] is showing accesses per aggregate interval in [0, 5] for aggregate
interval in [10, 20], page out the region.  For the paging out, use only up to
10ms per second, and also don't page out more than 1GiB per second.  Under the
limitation, page out memory regions having longer age first.  Also, check the
free memory rate of the system every 5 seconds, start the monitoring and paging
out when the free memory rate becomes lower than 50%, but stop it if the free
memory rate becomes larger than 60%, or lower than 30%". ::

    # cd <sysfs>/kernel/mm/damon/admin
    # # populate directories
    # echo 1 > kdamonds/nr_kdamonds; echo 1 > kdamonds/0/contexts/nr_contexts;
    # echo 1 > kdamonds/0/contexts/0/schemes/nr_schemes
    # cd kdamonds/0/contexts/0/schemes/0
    # # set the basic access pattern and the action
    # echo 4096 > access_pattern/sz/min
    # echo 8192 > access_pattern/sz/max
    # echo 0 > access_pattern/nr_accesses/min
    # echo 5 > access_pattern/nr_accesses/max
    # echo 10 > access_pattern/age/min
    # echo 20 > access_pattern/age/max
    # echo pageout > action
    # # set quotas
    # echo 10 > quotas/ms
    # echo $((1024*1024*1024)) > quotas/bytes
    # echo 1000 > quotas/reset_interval_ms
    # # set watermark
    # echo free_mem_rate > watermarks/metric
    # echo 5000000 > watermarks/interval_us
    # echo 600 > watermarks/high
    # echo 500 > watermarks/mid
    # echo 300 > watermarks/low

Please note that it's highly recommended to use user space tools like `damo
<https://github.com/awslabs/damo>`_ rather than manually reading and writing
the files as above.  Above is only for an example.

.. _tracepoint:

Tracepoints for Monitoring Results
==================================

Users can get the monitoring results via the :ref:`tried_regions
<sysfs_schemes_tried_regions>`.  The interface is useful for getting a
snapshot, but it could be inefficient for fully recording all the monitoring
results.  For the purpose, two trace points, namely ``damon:damon_aggregated``
and ``damon:damos_before_apply``, are provided.  ``damon:damon_aggregated``
provides the whole monitoring results, while ``damon:damos_before_apply``
provides the monitoring results for regions that each DAMON-based Operation
Scheme (:ref:`DAMOS <damon_design_damos>`) is gonna be applied.  Hence,
``damon:damos_before_apply`` is more useful for recording internal behavior of
DAMOS, or DAMOS target access
:ref:`pattern <damon_design_damos_access_pattern>` based query-like efficient
monitoring results recording.

While the monitoring is turned on, you could record the tracepoint events and
show results using tracepoint supporting tools like ``perf``.  For example::

    # echo on > monitor_on
    # perf record -e damon:damon_aggregated &
    # sleep 5
    # kill 9 $(pidof perf)
    # echo off > monitor_on
    # perf script
    kdamond.0 46568 [027] 79357.842179: damon:damon_aggregated: target_id=0 nr_regions=11 122509119488-135708762112: 0 864
    [...]

Each line of the perf script output represents each monitoring region.  The
first five fields are as usual other tracepoint outputs.  The sixth field
(``target_id=X``) shows the ide of the monitoring target of the region.  The
seventh field (``nr_regions=X``) shows the total number of monitoring regions
for the target.  The eighth field (``X-Y:``) shows the start (``X``) and end
(``Y``) addresses of the region in bytes.  The ninth field (``X``) shows the
``nr_accesses`` of the region (refer to
:ref:`design <damon_design_region_based_sampling>` for more details of the
counter).  Finally the tenth field (``X``) shows the ``age`` of the region
(refer to :ref:`design <damon_design_age_tracking>` for more details of the
counter).

If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would
be somewhat like below::

    kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136
    [...]

Each line of the output represents each monitoring region that each DAMON-based
Operation Scheme was about to be applied at the traced time.  The first five
fields are as usual.  It shows the index of the DAMON context (``ctx_idx=X``)
of the scheme in the list of the contexts of the context's kdamond, the index
of the scheme (``scheme_idx=X``) in the list of the schemes of the context, in
addition to the output of ``damon_aggregated`` tracepoint.


.. _debugfs_interface:

debugfs Interface (DEPRECATED!)
===============================

.. note::

  THIS IS DEPRECATED!

  DAMON debugfs interface is deprecated, so users should move to the
  :ref:`sysfs interface <sysfs_interface>`.  If you depend on this and cannot
  move, please report your usecase to damon@lists.linux.dev and
  linux-mm@kvack.org.

DAMON exports eight files, ``attrs``, ``target_ids``, ``init_regions``,
``schemes``, ``monitor_on``, ``kdamond_pid``, ``mk_contexts`` and
``rm_contexts`` under its debugfs directory, ``<debugfs>/damon/``.


Attributes
----------

Users can get and set the ``sampling interval``, ``aggregation interval``,
``update interval``, and min/max number of monitoring target regions by
reading from and writing to the ``attrs`` file.  To know about the monitoring
attributes in detail, please refer to the :doc:`/mm/damon/design`.  For
example, below commands set those values to 5 ms, 100 ms, 1,000 ms, 10 and
1000, and then check it again::

    # cd <debugfs>/damon
    # echo 5000 100000 1000000 10 1000 > attrs
    # cat attrs
    5000 100000 1000000 10 1000


Target IDs
----------

Some types of address spaces supports multiple monitoring target.  For example,
the virtual memory address spaces monitoring can have multiple processes as the
monitoring targets.  Users can set the targets by writing relevant id values of
the targets to, and get the ids of the current targets by reading from the
``target_ids`` file.  In case of the virtual address spaces monitoring, the
values should be pids of the monitoring target processes.  For example, below
commands set processes having pids 42 and 4242 as the monitoring targets and
check it again::

    # cd <debugfs>/damon
    # echo 42 4242 > target_ids
    # cat target_ids
    42 4242

Users can also monitor the physical memory address space of the system by
writing a special keyword, "``paddr\n``" to the file.  Because physical address
space monitoring doesn't support multiple targets, reading the file will show a
fake value, ``42``, as below::

    # cd <debugfs>/damon
    # echo paddr > target_ids
    # cat target_ids
    42

Note that setting the target ids doesn't start the monitoring.


Initial Monitoring Target Regions
---------------------------------

In case of the virtual address space monitoring, DAMON automatically sets and
updates the monitoring target regions so that entire memory mappings of target
processes can be covered.  However, users can want to limit the monitoring
region to specific address ranges, such as the heap, the stack, or specific
file-mapped area.  Or, some users can know the initial access pattern of their
workloads and therefore want to set optimal initial regions for the 'adaptive
regions adjustment'.

In contrast, DAMON do not automatically sets and updates the monitoring target
regions in case of physical memory monitoring.  Therefore, users should set the
monitoring target regions by themselves.

In such cases, users can explicitly set the initial monitoring target regions
as they want, by writing proper values to the ``init_regions`` file.  The input
should be a sequence of three integers separated by white spaces that represent
one region in below form.::

    <target idx> <start address> <end address>

The ``target idx`` should be the index of the target in ``target_ids`` file,
starting from ``0``, and the regions should be passed in address order.  For
example, below commands will set a couple of address ranges, ``1-100`` and
``100-200`` as the initial monitoring target region of pid 42, which is the
first one (index ``0``) in ``target_ids``, and another couple of address
ranges, ``20-40`` and ``50-100`` as that of pid 4242, which is the second one
(index ``1``) in ``target_ids``.::

    # cd <debugfs>/damon
    # cat target_ids
    42 4242
    # echo "0   1       100 \
            0   100     200 \
            1   20      40  \
            1   50      100" > init_regions

Note that this sets the initial monitoring target regions only.  In case of
virtual memory monitoring, DAMON will automatically updates the boundary of the
regions after one ``update interval``.  Therefore, users should set the
``update interval`` large enough in this case, if they don't want the
update.


Schemes
-------

Users can get and set the DAMON-based operation :ref:`schemes
<damon_design_damos>` by reading from and writing to ``schemes`` debugfs file.
Reading the file also shows the statistics of each scheme.  To the file, each
of the schemes should be represented in each line in below form::

    <target access pattern> <action> <quota> <watermarks>

You can disable schemes by simply writing an empty string to the file.

Target Access Pattern
~~~~~~~~~~~~~~~~~~~~~

The target access :ref:`pattern <damon_design_damos_access_pattern>` of the
scheme.  The ``<target access pattern>`` is constructed with three ranges in
below form::

    min-size max-size min-acc max-acc min-age max-age

Specifically, bytes for the size of regions (``min-size`` and ``max-size``),
number of monitored accesses per aggregate interval for access frequency
(``min-acc`` and ``max-acc``), number of aggregate intervals for the age of
regions (``min-age`` and ``max-age``) are specified.  Note that the ranges are
closed interval.

Action
~~~~~~

The ``<action>`` is a predefined integer for memory management :ref:`actions
<damon_design_damos_action>`.  The supported numbers and their meanings are as
below.

 - 0: Call ``madvise()`` for the region with ``MADV_WILLNEED``.  Ignored if
   ``target`` is ``paddr``.
 - 1: Call ``madvise()`` for the region with ``MADV_COLD``.  Ignored if
   ``target`` is ``paddr``.
 - 2: Call ``madvise()`` for the region with ``MADV_PAGEOUT``.
 - 3: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``.  Ignored if
   ``target`` is ``paddr``.
 - 4: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``.  Ignored if
   ``target`` is ``paddr``.
 - 5: Do nothing but count the statistics

Quota
~~~~~

Users can set the :ref:`quotas <damon_design_damos_quotas>` of the given scheme
via the ``<quota>`` in below form::

    <ms> <sz> <reset interval> <priority weights>

This makes DAMON to try to use only up to ``<ms>`` milliseconds for applying
the action to memory regions of the ``target access pattern`` within the
``<reset interval>`` milliseconds, and to apply the action to only up to
``<sz>`` bytes of memory regions within the ``<reset interval>``.  Setting both
``<ms>`` and ``<sz>`` zero disables the quota limits.

For the :ref:`prioritization <damon_design_damos_quotas_prioritization>`, users
can set the weights for the three properties in ``<priority weights>`` in below
form::

    <size weight> <access frequency weight> <age weight>

Watermarks
~~~~~~~~~~

Users can specify :ref:`watermarks <damon_design_damos_watermarks>` of the
given scheme via ``<watermarks>`` in below form::

    <metric> <check interval> <high mark> <middle mark> <low mark>

``<metric>`` is a predefined integer for the metric to be checked.  The
supported numbers and their meanings are as below.

 - 0: Ignore the watermarks
 - 1: System's free memory rate (per thousand)

The value of the metric is checked every ``<check interval>`` microseconds.

If the value is higher than ``<high mark>`` or lower than ``<low mark>``, the
scheme is deactivated.  If the value is lower than ``<mid mark>``, the scheme
is activated.

.. _damos_stats:

Statistics
~~~~~~~~~~

It also counts the total number and bytes of regions that each scheme is tried
to be applied, the two numbers for the regions that each scheme is successfully
applied, and the total number of the quota limit exceeds.  This statistics can
be used for online analysis or tuning of the schemes.

The statistics can be shown by reading the ``schemes`` file.  Reading the file
will show each scheme you entered in each line, and the five numbers for the
statistics will be added at the end of each line.

Example
~~~~~~~

Below commands applies a scheme saying "If a memory region of size in [4KiB,
8KiB] is showing accesses per aggregate interval in [0, 5] for aggregate
interval in [10, 20], page out the region.  For the paging out, use only up to
10ms per second, and also don't page out more than 1GiB per second.  Under the
limitation, page out memory regions having longer age first.  Also, check the
free memory rate of the system every 5 seconds, start the monitoring and paging
out when the free memory rate becomes lower than 50%, but stop it if the free
memory rate becomes larger than 60%, or lower than 30%".::

    # cd <debugfs>/damon
    # scheme="4096 8192  0 5    10 20    2"  # target access pattern and action
    # scheme+=" 10 $((1024*1024*1024)) 1000" # quotas
    # scheme+=" 0 0 100"                     # prioritization weights
    # scheme+=" 1 5000000 600 500 300"       # watermarks
    # echo "$scheme" > schemes


Turning On/Off
--------------

Setting the files as described above doesn't incur effect unless you explicitly
start the monitoring.  You can start, stop, and check the current status of the
monitoring by writing to and reading from the ``monitor_on`` file.  Writing
``on`` to the file starts the monitoring of the targets with the attributes.
Writing ``off`` to the file stops those.  DAMON also stops if every target
process is terminated.  Below example commands turn on, off, and check the
status of DAMON::

    # cd <debugfs>/damon
    # echo on > monitor_on
    # echo off > monitor_on
    # cat monitor_on
    off

Please note that you cannot write to the above-mentioned debugfs files while
the monitoring is turned on.  If you write to the files while DAMON is running,
an error code such as ``-EBUSY`` will be returned.


Monitoring Thread PID
---------------------

DAMON does requested monitoring with a kernel thread called ``kdamond``.  You
can get the pid of the thread by reading the ``kdamond_pid`` file.  When the
monitoring is turned off, reading the file returns ``none``. ::

    # cd <debugfs>/damon
    # cat monitor_on
    off
    # cat kdamond_pid
    none
    # echo on > monitor_on
    # cat kdamond_pid
    18594


Using Multiple Monitoring Threads
---------------------------------

One ``kdamond`` thread is created for each monitoring context.  You can create
and remove monitoring contexts for multiple ``kdamond`` required use case using
the ``mk_contexts`` and ``rm_contexts`` files.

Writing the name of the new context to the ``mk_contexts`` file creates a
directory of the name on the DAMON debugfs directory.  The directory will have
DAMON debugfs files for the context. ::

    # cd <debugfs>/damon
    # ls foo
    # ls: cannot access 'foo': No such file or directory
    # echo foo > mk_contexts
    # ls foo
    # attrs  init_regions  kdamond_pid  schemes  target_ids

If the context is not needed anymore, you can remove it and the corresponding
directory by putting the name of the context to the ``rm_contexts`` file. ::

    # echo foo > rm_contexts
    # ls foo
    # ls: cannot access 'foo': No such file or directory

Note that ``mk_contexts``, ``rm_contexts``, and ``monitor_on`` files are in the
root directory only.