diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2019-09-03 10:23:48 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2019-09-03 10:23:48 +0000 |
| commit | cd7ed12292aef11d9062b64f61215174e8cc1860 (patch) | |
| tree | 9998ab03d153956743d9319cf3a0279b9593ce36 /database/engine/README.md | |
| parent | Releasing debian version 1.16.1-6. (diff) | |
| download | netdata-cd7ed12292aef11d9062b64f61215174e8cc1860.tar.xz netdata-cd7ed12292aef11d9062b64f61215174e8cc1860.zip | |
Merging upstream version 1.17.0.
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'database/engine/README.md')
| -rw-r--r-- | database/engine/README.md | 42 |
1 files changed, 25 insertions, 17 deletions
diff --git a/database/engine/README.md b/database/engine/README.md index adc69ffd..7791a549 100644 --- a/database/engine/README.md +++ b/database/engine/README.md @@ -4,6 +4,8 @@ The Database Engine works like a traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data reside compressed on disk. The number of history entries is not fixed in this case, but depends on the configured disk space and the effective compression ratio of the data stored. +This is the **only mode** that supports changing the data collection update frequency +(`update_every`) **without losing** the previously stored metrics. ## Files @@ -18,18 +20,18 @@ journalfile-1-0000000002.njf datafile-1-0000000003.ndf journalfile-1-0000000003.njf ... -``` +``` They are located under their host's cache directory in the directory `./dbengine` (e.g. for localhost the default location is `/var/cache/netdata/dbengine/*`). The higher numbered filenames contain more recent metric data. The user can safely delete some pairs -of files when netdata is stopped to manually free up some space. +of files when Netdata is stopped to manually free up some space. -*Users should* **back up** *their `./dbengine` folders if they consider this data to be important.* +_Users should_ **back up** _their `./dbengine` folders if they consider this data to be important._ ## Configuration -There is one DB engine instance per netdata host/node. That is, there is one `./dbengine` folder +There is one DB engine instance per Netdata host/node. That is, there is one `./dbengine` folder per node, and all charts of `dbengine` memory mode in such a host share the same storage space and DB engine instance memory state. You can select the memory mode for localhost by editing netdata.conf and setting: @@ -59,10 +61,10 @@ quota. Both numbers are in **MiB**. All DB engine instances will allocate the co separately. The `page cache size` option determines the amount of RAM in **MiB** that is dedicated to caching -netdata metric values themselves. +Netdata metric values themselves. The `dbengine disk space` option determines the amount of disk space in **MiB** that is dedicated -to storing netdata metric values and all related metadata describing them. +to storing Netdata metric values and all related metadata describing them. ## Operation @@ -72,7 +74,7 @@ the **Page Cache**. When those pages fill up they are slowly compressed and flushed to disk. It can take `4096 / 4 = 1024 seconds = 17 minutes`, for a chart dimension that is being collected -every 1 second, to fill a page. Pages can be cut short when we stop netdata or the DB engine +every 1 second, to fill a page. Pages can be cut short when we stop Netdata or the DB engine instance so as to not lose the data. When we query the DB engine for data we trigger disk read I/O requests that fill the Page Cache with the requested pages and potentially evict cold (not recently used) pages. @@ -91,17 +93,17 @@ applications. Using memory mode `dbengine` we can overcome most memory restrictions and store a dataset that is much larger than the available memory. -There are explicit memory requirements **per** DB engine **instance**, meaning **per** netdata +There are explicit memory requirements **per** DB engine **instance**, meaning **per** Netdata **node** (e.g. localhost and streaming recipient nodes): -- `page cache size` must be at least `#dimensions-being-collected x 4096 x 2` bytes. +- `page cache size` must be at least `#dimensions-being-collected x 4096 x 2` bytes. -- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata. +- an additional `#pages-on-disk x 4096 x 0.03` bytes of RAM are allocated for metadata. - - roughly speaking this is 3% of the uncompressed disk space taken by the DB files. + - roughly speaking this is 3% of the uncompressed disk space taken by the DB files. - - for very highly compressible data (compression ratio > 90%) this RAM overhead - is comparable to the disk space footprint. + - for very highly compressible data (compression ratio > 90%) this RAM overhead + is comparable to the disk space footprint. An important observation is that RAM usage depends on both the `page cache size` and the `dbengine disk space` options. @@ -115,11 +117,11 @@ file descriptors available per `dbengine` instance. Netdata allocates 25% of the available file descriptors to its Database Engine instances. This means that only 25% of the file descriptors that are available to the Netdata service are accessible by dbengine instances. You should take that into account when configuring your service -or system-wide file descriptor limits. You can roughly estimate that the netdata service needs 2048 file +or system-wide file descriptor limits. You can roughly estimate that the Netdata service needs 2048 file descriptors for every 10 streaming slave hosts when streaming is configured to use `memory mode = dbengine`. -If for example one wants to allocate 65536 file descriptors to the netdata service on a systemd system -one needs to override the netdata service by running `sudo systemctl edit netdata` and creating a +If for example one wants to allocate 65536 file descriptors to the Netdata service on a systemd system +one needs to override the Netdata service by running `sudo systemctl edit netdata` and creating a file with contents: ``` @@ -128,18 +130,24 @@ LimitNOFILE=65536 ``` For other types of services one can add the line: + ``` ulimit -n 65536 ``` + at the beginning of the service file. Alternatively you can change the system-wide limits of the kernel by changing `/etc/sysctl.conf`. For linux that would be: + ``` fs.file-max = 65536 ``` + In FreeBSD and OS X you change the lines like this: + ``` kern.maxfilesperproc=65536 kern.maxfiles=65536 ``` + You can apply the settings by running `sysctl -p` or by rebooting. -[]() +[](<>) |