diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2021-02-07 11:45:55 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2021-02-07 11:45:55 +0000 |
| commit | a8220ab2d293bb7f4b014b79d16b2fb05090fa93 (patch) | |
| tree | 77f0a30f016c0925cf7ee9292e644bba183c2774 /database/README.md | |
| parent | Adding upstream version 1.19.0. (diff) | |
| download | netdata-a8220ab2d293bb7f4b014b79d16b2fb05090fa93.tar.xz netdata-a8220ab2d293bb7f4b014b79d16b2fb05090fa93.zip | |
Adding upstream version 1.29.0.upstream/1.29.0
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'database/README.md')
| -rw-r--r-- | database/README.md | 74 |
1 files changed, 30 insertions, 44 deletions
diff --git a/database/README.md b/database/README.md index 143615a0..9fef7058 100644 --- a/database/README.md +++ b/database/README.md @@ -1,60 +1,46 @@ -# Database - -Although `netdata` does all its calculations using `long double`, it stores all values using a [custom-made 32-bit -number](../libnetdata/storage_number/). - -So, for each dimension of a chart, Netdata will need: `4 bytes for the value * the entries of its history`. It will not -store any other data for each value in the time series database. Since all its values are stored in a time series with -fixed step, the time each value corresponds can be calculated at run time, using the position of a value in the round -robin database. - -The default history is 3.600 entries, thus it will need 14.4KB for each chart dimension. If you need 1.000 dimensions, -they will occupy just 14.4MB. +<!-- +title: "Database" +description: "The Netdata Agent leverages multiple, user-configurable time-series databases that use RAM and/or disk to store metrics on any type of node." +custom_edit_url: https://github.com/netdata/netdata/edit/master/database/README.md +--> -Of course, 3.600 entries is a very short history, especially if data collection frequency is set to 1 second. You will -have just one hour of data. - -For a day of data and 1.000 dimensions, you will need: `86.400 seconds * 4 bytes * 1.000 dimensions = 345MB of RAM`. - -One option you have to lower this number is to use **[Memory Deduplication - Kernel Same Page Merging - KSM](#ksm)**. -Another possibility is to use the **[Database Engine](engine/)**. +# Database -## Memory modes +Netdata is fully capable of long-term metrics storage, at per-second granularity, via its default database engine +(`dbengine`). But to remain as flexible as possible, Netdata supports a number of types of metrics storage: -Currently Netdata supports 6 memory modes: +1. `dbengine`, (the default) data are in database files. The [Database Engine](/database/engine/README.md) works like a + traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data + reside compressed on disk. The number of history entries is not fixed in this case, but depends on the configured + disk space and the effective compression ratio of the data stored. This is the **only mode** that supports changing + the data collection update frequency (`update_every`) **without losing** the previously stored metrics. For more + details see [here](/database/engine/README.md). -1. `ram`, data are purely in memory. Data are never saved on disk. This mode uses `mmap()` and supports [KSM](#ksm). +2. `ram`, data are purely in memory. Data are never saved on disk. This mode uses `mmap()` and supports [KSM](#ksm). -2. `save`, data are only in RAM while Netdata runs and are saved to / loaded from disk on Netdata +3. `save`, data are only in RAM while Netdata runs and are saved to / loaded from disk on Netdata restart. It also uses `mmap()` and supports [KSM](#ksm). -3. `map`, data are in memory mapped files. This works like the swap. Keep in mind though, this will have a constant +4. `map`, data are in memory mapped files. This works like the swap. Keep in mind though, this will have a constant write on your disk. When Netdata writes data on its memory, the Linux kernel marks the related memory pages as dirty and automatically starts updating them on disk. Unfortunately we cannot control how frequently this works. The Linux kernel uses exactly the same algorithm it uses for its swap memory. Check below for additional information on running a dedicated central Netdata server. This mode uses `mmap()` but does not support [KSM](#ksm). -4. `none`, without a database (collected metrics can only be streamed to another Netdata). +5. `none`, without a database (collected metrics can only be streamed to another Netdata). -5. `alloc`, like `ram` but it uses `calloc()` and does not support [KSM](#ksm). This mode is the fallback for all +6. `alloc`, like `ram` but it uses `calloc()` and does not support [KSM](#ksm). This mode is the fallback for all others except `none`. -6. `dbengine`, (the default) data are in database files. The [Database Engine](engine/) works like a traditional - database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data reside - compressed on disk. The number of history entries is not fixed in this case, but depends on the configured disk - space and the effective compression ratio of the data stored. This is the **only mode** that supports changing the - data collection update frequency (`update_every`) **without losing** the previously stored metrics. For more details - see [here](engine/). - You can select the memory mode by editing `netdata.conf` and setting: ```conf [global] - # ram, save (the default, save on exit, load on start), map (swap like) - memory mode = save + # dbengine (default), ram, save (the default if dbengine not available), map (swap like), none, alloc + memory mode = dbengine - # the directory where data are saved - cache directory = /var/cache/netdata + # the directory where data are saved + cache directory = /var/cache/netdata ``` ## Running Netdata in embedded devices @@ -73,7 +59,7 @@ seconds. This will **cut in half** both CPU and RAM resources consumed by Netdat weak devices you might have to use `update every = 5` and `history = 720` (still 1 hour of data, but 1/5 of the CPU and RAM resources). -You can also disable [data collection plugins](../collectors) you don't need. Disabling such plugins will also free both +You can also disable [data collection plugins](/collectors/README.md) you don't need. Disabling such plugins will also free both CPU and RAM resources. ## Running a dedicated central Netdata server @@ -161,15 +147,15 @@ There is another memory mode to help overcome the memory size problem. What is * ### dbengine -In this mode, the database of Netdata is stored in database files. The [Database Engine](engine/) works like a -traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of the data reside -compressed on disk. The number of history entries is not fixed in this case, but depends on the configured disk space -and the effective compression ratio of the data stored. +In this mode, the database of Netdata is stored in database files. The [Database Engine](/database/engine/README.md) +works like a traditional database. There is some amount of RAM dedicated to data caching and indexing and the rest of +the data reside compressed on disk. The number of history entries is not fixed in this case, but depends on the +configured disk space and the effective compression ratio of the data stored. We suggest to use **this** mode on nodes that also run other applications. The Database Engine uses direct I/O to avoid polluting the OS filesystem caches and does not generate excessive I/O traffic so as to create the minimum possible interference with other applications. Using memory mode `dbengine` we can overcome most memory restrictions. For more -details see [here](engine/). +details see [here](/database/engine/README.md). ## KSM @@ -226,4 +212,4 @@ Netdata will create charts for kernel memory de-duplication performance, like th  -[](<>)
\ No newline at end of file +[](<>) |