blob: adc69ffd72bb43aba4a2b13752330ccdf35fda5b (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
# Database engine
The Database Engine works like a traditional
database. There is some amount of RAM dedicated to data caching and indexing and the rest of
the data reside compressed on disk. The number of history entries is not fixed in this case,
but depends on the configured disk space and the effective compression ratio of the data stored.
## Files
With the DB engine memory mode the metric data are stored in database files. These files are
organized in pairs, the datafiles and their corresponding journalfiles, e.g.:
```
datafile-1-0000000001.ndf
journalfile-1-0000000001.njf
datafile-1-0000000002.ndf
journalfile-1-0000000002.njf
datafile-1-0000000003.ndf
journalfile-1-0000000003.njf
...
```
They are located under their host's cache directory in the directory `./dbengine`
(e.g. for localhost the default location is `/var/cache/netdata/dbengine/*`). The higher
numbered filenames contain more recent metric data. The user can safely delete some pairs
of files when netdata is stopped to manually free up some space.
*Users should* **back up** *their `./dbengine` folders if they consider this data to be important.*
## Configuration
There is one DB engine instance per netdata host/node. That is, there is one `./dbengine` folder
per node, and all charts of `dbengine` memory mode in such a host share the same storage space
and DB engine instance memory state. You can select the memory mode for localhost by editing
netdata.conf and setting:
```
[global]
memory mode = dbengine
```
For setting the memory mode for the rest of the nodes you should look at
[streaming](../../streaming/).
The `history` configuration option is meaningless for `memory mode = dbengine` and is ignored
for any metrics being stored in the DB engine.
All DB engine instances, for localhost and all other streaming recipient nodes inherit their
configuration from `netdata.conf`:
```
[global]
page cache size = 32
dbengine disk space = 256
```
The above values are the default and minimum values for Page Cache size and DB engine disk space
quota. Both numbers are in **MiB**. All DB engine instances will allocate the configured resources
separately.
The `page cache size` option determines the amount of RAM in **MiB** that is dedicated to caching
netdata metric values themselves.
The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated
to storing netdata metric values and all related metadata describing them.
## Operation
The DB engine stores chart metric values in 4096-byte pages in memory. Each chart dimension gets
its own page to store consecutive values generated from the data collectors. Those pages comprise
the **Page Cache**.
When those pages fill up they are slowly compressed and flushed to disk.
It can take `4096 / 4 = 1024 seconds = 17 minutes`, for a chart dimension that is being collected
every 1 second, to fill a page. Pages can be cut short when we stop netdata or the DB engine
instance so as to not lose the data. When we query the DB engine for data we trigger disk read
I/O requests that fill the Page Cache with the requested pages and potentially evict cold
(not recently used) pages.
When the disk quota is exceeded the oldest values are removed from the DB engine at real time, by
automatically deleting the oldest datafile and journalfile pair. Any corresponding pages residing
in the Page Cache will also be invalidated and removed. The DB engine logic will try to maintain
between 10 and 20 file pairs at any point in time.
The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not
generate excessive I/O traffic so as to create the minimum possible interference with other
applications.
## Memory requirements
Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that
is much larger than the available memory.
There are explicit memory requirements **per** DB engine **instance**, meaning **per** netdata
**node** (e.g. localhost and streaming recipient nodes):
- `page cache size` must be at least `#dimensions-being-collected x 4096 x 2` bytes.
- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata.
- roughly speaking this is 3% of the uncompressed disk space taken by the DB files.
- for very highly compressible data (compression ratio > 90%) this RAM overhead
is comparable to the disk space footprint.
An important observation is that RAM usage depends on both the `page cache size` and the
`dbengine disk space` options.
## File descriptor requirements
The Database Engine may keep a **significant** amount of files open per instance (e.g. per streaming
slave or master server). When configuring your system you should make sure there are at least 50
file descriptors available per `dbengine` instance.
Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25%
of the file descriptors that are available to the Netdata service are accessible by dbengine instances.
You should take that into account when configuring your service
or system-wide file descriptor limits. You can roughly estimate that the netdata service needs 2048 file
descriptors for every 10 streaming slave hosts when streaming is configured to use `memory mode = dbengine`.
If for example one wants to allocate 65536 file descriptors to the netdata service on a systemd system
one needs to override the netdata service by running `sudo systemctl edit netdata` and creating a
file with contents:
```
[Service]
LimitNOFILE=65536
```
For other types of services one can add the line:
```
ulimit -n 65536
```
at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing `/etc/sysctl.conf`. For linux that would be:
```
fs.file-max = 65536
```
In FreeBSD and OS X you change the lines like this:
```
kern.maxfilesperproc=65536
kern.maxfiles=65536
```
You can apply the settings by running `sysctl -p` or by rebooting.
[]()
|
