Adding upstream version 1.29.0.upstream/1.29.0

Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>

1 files changed, 365 insertions, 0 deletions

diff --git a/BUILD.md b/BUILD.md
new file mode 100644
index 000000000..049c86d3f
--- /dev/null
+++ b/BUILD.md
@@ -0,0 +1,365 @@
+<!--
+title: "The build system"
+custom_edit_url: https://github.com/netdata/netdata/edit/master/BUILD.md
+-->
+
+# The build system
+
+We are currently migrating from `autotools` to `CMake` as a build-system. This document 
+currently describes how we intend to perform this migration, and will be updated after
+the migration to explain how the new `CMake` configuration works.
+
+## Stages during the build
+
+1. The `netdata-installer.sh`, take in arguments and environment settings to control the
+   build.
+2. The configure step: `autoreconf -ivf ; ./configure` passing arguments into the configure
+   script. This becomes `generation-time` in CMake. This includes package / system detection
+   and configuration resulting in the `config.h` in the source root.
+3. The build step: recurse through the generated Makefiles and build the executable.
+4. The first install step: calls `make install` to handle all the install steps put into
+   the Makefiles by the configure step (puts binaries / libraries / config into target
+   tree structure).
+5. The second install step: the rest of the installer after the make install handles
+   system-level configuration (privilege setting, user / groups,  fetch/build/install `go.d`
+   plugins, telemetry, installing service for startup, uninstaller, auto-updates.
+
+The ideal migration result is to replace all of this with the following steps:
+```
+mkdir build ; cd build ; cmake .. -D... ; cmake --build . --target install
+```
+
+The `-D...` indicates where the command-line arguments for configuration are passed into
+`CMake`.
+
+## CMake generation time
+
+At generation time we need to solve the following issues:
+
+### Feature flags
+
+Every command-line switch on the installer and the configure script needs to becomes an
+argument to the CMake generation, we can do this with variables in the CMake cache:
+
+CMakeLists.txt:
+```
+option(ENABLE_DBENGINE "Enable the dbengine storage" ON)
+...
+if(${ENABLE_DBENGINE})
+...
+endif()
+```
+
+Command-line interface
+```
+cmake -DENABLE_DBENGINE
+```
+
+### Dependency detection
+
+We have a mixture of soft- and hard-dependencies on libraries. For most of these we expect
+`pkg-config` information, for some we manually probe for libraries and include files. We
+should treat all of the external dependencies consistently:
+
+1. Default to autodetect using `pkg-config` (e.g. the standard `jemalloc` drops a `.pc`
+   into the system but we do not check for it.
+2. If no `.pc` is found perform a manual search for libraries under known names, and
+   check for accessible symbols inside them.
+3. Check that include paths work.
+4. Allow a command-line override (e.g. `-DWITH_JEMALLOC=/...`).
+5. If none of the above work then fail the install if the dependency is hard, otherwise
+   indicate it is not present in the `config.h`.
+
+Before doing any dependency detection we need to determine which search paths are 
+really in use for the current compiler, after the `project` declaration we can use:
+```
+execute_process(COMMAND ${CMAKE_C_COMPILER} "--print-search-dirs"
+                COMMAND grep "^libraries:"
+                COMMAND sed "s/^libraries: =//"
+                COMMAND tr ":" " "
+                COMMAND tr -d "\n"
+                OUTPUT_VARIABLE CC_SEARCH_DIRS
+                RESULTS_VARIABLE CC_SEARCH_RES)
+string(REGEX MATCH   "^[0-9]+" CC_SEARCH_RES ${CC_SEARCH_RES})
+#string(STRIP "${CC_SEARCH_RES}" CC_SEARCH_RES)
+if(0 LESS ${CC_SEARCH_RES})
+    message(STATUS "Warning - cannot determine standard compiler library paths")
+    # Note: we will probably need a different method for Windows...
+endif()
+
+```
+
+The output format for this switch works on both `Clang` and `gcc`, it also includes
+the include search path, which can be extracted in a similar way. Standard advice here
+is to list the `ldconfig` cache or use the `-V` flag to check, but this does not work
+consistently across platforms - in particular `gcc` will reconfigure `ld` when it is
+called to gcc's internal view of search paths. During experiments each of these 
+alternative missed / added unused paths. Dumping the compiler's own estimate of the
+search paths seems to work consistently across clang/gcc/linux/freebsd configurations.
+
+The default behaviour in CMake is to search across predefined paths (e.g. `CMAKE_LIBRARY_PATH`)
+that are based on heuristics about the current platform. Most projects using CMake seem
+to overwrite this with their own estimates.
+
+We can use the extracted paths as a base, add our own heuristics based on OS and then
+`set(CMAKE_LIBRARY_PATH ${OUR_OWN_LIB_SEARCH})` to get the best results. Roughly we do
+the following for each external dependency:
+```
+set(WITH_JSONC "Detect" CACHE STRING "Manually set the path to a json-c installation")
+...
+if(${WITH_JSONC} STREQUAL "Detect")
+    pkg_check_modules(JSONC json-c)     # Don't set the REQUIRED flag
+    if(JSONC_FOUND)
+        message(STATUS "libjsonc found through .pc -> ${JSONC_CFLAGS_OTHER} ${JSONC_LIBRARIES}")
+        # ... setup using JSONC_CFLAGS_OTHER JSONC_LIBRARIES and JSONC_INCLUDE_DIRS
+    else()
+        find_library(LIB_JSONC
+                     NAMES json-c libjson-c
+                     PATHS ${CMAKE_LIBRARY_PATH})       # Includes our additions by this point
+        if(${LIB_JSONC} STREQUAL "LIB_JSONC-NOTFOUND")
+            message(STATUS "Library json-c not installed, disabling")
+        else()
+            check_library_exists(${LIB_JSONC} json_object_get_type "" HAVE_JSONC)
+            # ... setup using heuristics for CFLAGS and check include files are available
+        endif()
+    endif()
+else()
+    # ... use explicit path as base to check for library and includes ...
+endif()
+
+```
+
+For checking the include path we have two options, if we overwrite the `CMAKE_`... variables
+to change the internal search path we can use:
+```
+CHECK_INCLUDE_FILE(json/json.h HAVE_JSONC_H)
+```
+Or we can build a custom search path and then use:
+```
+find_file(HAVE_JSONC_H json/json.h PATHS ${OUR_INCLUDE_PATHS})
+```
+
+Note: we may have cases where there is no `.pc` but we have access to a `.cmake` (e.g. AWS SDK, mongodb,cmocka) - these need to be checked / pulled inside the repo while building a prototype.
+
+### Compiler compatibility checks
+
+In CMakeLists.txt:
+
+```
+CHECK_INCLUDE_FILE(sys/prctl.h HAVE_PRCTL_H)
+configure_file(cmake/config.in config.h)
+```
+
+In cmake/config.in:
+
+```
+#cmakedefine HAVE_PRCTL_H 1
+```
+
+If we want to check explicitly if something compiles (e.g. the accept4 check, or the 
+`strerror_r` typing issue) then we set the `CMAKE_`... paths and then use:
+```
+check_c_source_compiles(
+    "
+    #include <string.h>
+    int main() { char x = *strerror_r(0, &x, sizeof(x)); return 0; }
+    "
+    STRERROR_R_CHAR_P)
+
+```
+This produces a bool that we can use inside CMake or propagate into the `config.h`.
+
+We can handle the atomic checks with:
+```
+check_c_source_compiles(
+    "
+    int main (int argc, char **argv)
+    {
+      volatile unsigned long ul1 = 1, ul2 = 0, ul3 = 2;
+      __atomic_load_n(&ul1, __ATOMIC_SEQ_CST);
+      __atomic_compare_exchange(&ul1, &ul2, &ul3, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);
+      __atomic_fetch_add(&ul1, 1, __ATOMIC_SEQ_CST);
+      __atomic_fetch_sub(&ul3, 1, __ATOMIC_SEQ_CST);
+      __atomic_or_fetch(&ul1, ul2, __ATOMIC_SEQ_CST);
+      __atomic_and_fetch(&ul1, ul2, __ATOMIC_SEQ_CST);
+      volatile unsigned long long ull1 = 1, ull2 = 0, ull3 = 2;
+      __atomic_load_n(&ull1, __ATOMIC_SEQ_CST);
+      __atomic_compare_exchange(&ull1, &ull2, &ull3, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);
+      __atomic_fetch_add(&ull1, 1, __ATOMIC_SEQ_CST);
+      __atomic_fetch_sub(&ull3, 1, __ATOMIC_SEQ_CST);
+      __atomic_or_fetch(&ull1, ull2, __ATOMIC_SEQ_CST);
+      __atomic_and_fetch(&ull1, ull2, __ATOMIC_SEQ_CST);
+      return 0;
+    }
+    "
+    HAVE_C__ATOMIC)
+```
+
+For the specific problem of getting the correct type signature in log.c for the `strerror_r`
+calls we can replicate what we have now, or we can delete this code completely and use a 
+better solution that is documented [here](http://www.club.cc.cmu.edu/~cmccabe/blog_strerror.html).
+To replicate what we have now:
+```
+check_c_source_compiles(
+    "
+    #include <string.h>
+    int main() { char x = *strerror_r(0, &x, sizeof(x)); return 0; }
+    "
+    STRERROR_R_CHAR_P)
+
+check_c_source_compiles(
+    "
+    #include <string.h>
+    int main() { int x = strerror_r(0, &x, sizeof(x)); return 0; }
+    "
+    STRERROR_R_INT)
+
+if("${STRERROR_R_CHAR_P}" OR "${STRERROR_R_INT}")
+    set(HAVE_DECL_STRERROR_R 1)
+endif()
+message(STATUS "Result was ${HAVE_DECL_STRERROR_R}")
+
+```
+
+Note: I did not find an explicit way to select compiler when both `clang` and `gcc` are
+present. We might have an implicit way (like redirecting `cc`) but we should put one in.
+
+
+
+### Debugging problems in test compilations
+
+Test compilations attempt to feed a test-input into the targeted compiler and result
+in a yes/no decision, this is similar to `AC_LANG_SOURCE(.... if test $ac_...` in .`m4`.
+We have two techniques to use in CMake:
+```
+cmake_minimum_required(VERSION 3.1.0)
+include(CheckCCompilerFlag)
+project(empty C)
+
+check_c_source_compiles(
+    "
+    #include <string.h>
+    int main() { char x = *strerror_r(0, &x, sizeof(x)); return 0; }
+    "
+    STRERROR_R_CHAR_P)
+
+try_compile(HAVE_JEMALLOC ${CMAKE_CURRENT_BINARY_DIR}
+            ${CMAKE_CURRENT_SOURCE_DIR}/quickdemo.c
+            LINK_LIBRARIES jemalloc)
+```
+
+The `check_c_source_compiles` is light-weight:
+
+* Inline source for the test, easy to follow.
+* Build errors are reported in `CMakeFiles/CMakeErrors.log`
+
+But we cannot alter the include-paths / library-paths / compiler-flags specifically for
+the test without overwriting the current CMake settings. The alternative approach is
+slightly more heavy-weight:
+
+* Can't inline source for `try_compile` - it requires a `.c` file in the tree.
+* Build errors are not shown, the recovery process for them is somewhat difficult.
+
+```
+rm -rf * && cmake .. --debug-trycompile
+grep jemal CMakeFiles/CMakeTmp/CMakeFiles/*dir/*
+cd CMakeFiles/CMakeTmp/CMakeFiles/cmTC_d6f0e.dir  # for example
+cmake --build ../..
+```
+
+This implies that we can do this to diagnose problems / develop test-programs, but we
+have to make them *bullet-proof* as we cannot expose this to end-users. This means that
+the results of the compilation must be *crisp* - exactly yes/no if the feature we are
+testing is supported.
+
+### System configuration checks
+
+For any system configuration checks that fall outside of the above scope (includes, libraries,
+packages, test-compilation checks) we have a fall-back that we can use to glue any holes
+that we need, e.g. to pull out the packaging strings, inside the `CMakeLists.h`:
+```
+execute_process(COMMAND cat ${CMAKE_CURRENT_SOURCE_DIR}/packaging/version
+                COMMAND tr -d '\n'
+                OUTPUT_VARIABLE VERSION_FROM_FILE)
+message(STATUS "Packaging version ${VERSION_FROM_FILE}")
+```
+and this in the `config.h.in`:
+```
+#define VERSION_FROM_FILE "@VERSION_FROM_FILE@"
+```
+
+## CMake build time
+
+We have a working definition of the targets that is in use with CLion and works on modern
+CMake (3.15). It breaks on older CMake version (e.g. 3.7) with an error message (issue#7091).
+No PoC yet to fix this, but it looks like changing the target properties should do it (in the
+worst case we can drop the separate object completely and merge the sources directly into
+the final target).
+
+Steps needed for building a prototype:
+
+1. Pick a reasonable configuration.
+2. Use the PoC techniques above to do a full generation of `CMAKE_` variables in the cache
+   according to the feature options and dependencies.
+3. Push these into the project variables.
+4. Work on it until the build succeeds in at least one known configuration.
+5. Smoke-test that the output is valid (i.e. the executable loads and runs, and we can
+   access the dashboard).
+6. Do a full comparison of the `config.h` generated by autotools against the CMake version
+   and document / fix any deviations.
+
+## CMake install target
+
+I've only looked at this superficially as we do not have a prototype yet, but each of the
+first-stage install steps (in `make install`) and the second-stage (in `netdata-installer.sh`)
+look feasible.
+
+## General issues
+
+*   We need to choose a minimum CMake version that is an available package across all of our
+    supported environments. There is currently a build issue #7091 that documents a problem 
+    in the compilation phase (we cannot link in libnetdata as an object on old CMake versions
+    and need to find a different way to express this).
+
+*   The default variable-expansion / comparisons in CMake are awkward, we need this to make it
+    sane:
+    ```
+    cmake_policy(SET CMP0054 "NEW")
+    ```
+*   Default paths for libs / includes are not comprehensive on most environments, we still need
+    some heuristics for common locations, e.g. `/usr/local` on FreeBSD.
+
+# Recommendations
+
+We should follow these steps:
+
+1. Build a prototype.
+2. Build a test-environment to check the prototype against environments / configurations that
+   the team uses.
+3. Perform an "internal" release - merge the new CMake into master, but not announce it or 
+   offer to support it.
+4. Check it works for the team internally.
+5. Do a soft-release: offer it externally as a replacement option for autotools.
+6. Gather feedback and usage reports on a wider range of configurations.
+7. Do a hard-release: switch over the preferred build-system in the installation instructions.
+8. Gather feedback and usage reports on a wider range of configurations (again).
+9. Deprecate / remove the autotools build-system completely (so that we can support a single
+   build-system).
+
+Some smaller miscellaneous suggestions:
+
+1. Remove the `_Generic` / `strerror_r` config to make the system simpler (use the technique
+   on the blog post to make the standard version re-entrant so that it is thread-safe).
+2. Pull in jemalloc by source into the repo if it is our preferred malloc implementation.
+
+# Background
+
+* [Stack overflow starting point](https://stackoverflow.com/questions/7132862/how-do-i-convert-an-autotools-project-to-a-cmake-project#7680240)
+* [CMake wiki including previous autotools conversions](https://gitlab.kitware.com/cmake/community/wikis/Home)
+* [Commands section in old CMake docs](https://cmake.org/cmake/help/v2.8.8/cmake.html#section_Commands)
+* [try_compile in newer CMake docs](https://cmake.org/cmake/help/v3.7/command/try_compile.html)
+* [configure_file in newer CMake docs](https://cmake.org/cmake/help/v3.7/command/configure_file.html?highlight=configure_file)
+* [header checks in CMake](https://stackoverflow.com/questions/647892/how-to-check-header-files-and-library-functions-in-cmake-like-it-is-done-in-auto)
+* [how to write platform checks](https://gitlab.kitware.com/cmake/community/wikis/doc/tutorials/How-To-Write-Platform-Checks)
+
+[![analytics](https://www.google-analytics.com/collect?v=1&aip=1&t=pageview&_s=1&ds=github&dr=https%3A%2F%2Fgithub.com%2Fnetdata%2Fnetdata&dl=https%3A%2F%2Fmy-netdata.io%2Fgithub%2FBUILD&_u=MAC~&cid=5792dfd7-8dc4-476b-af31-da2fdb9f93d2&tid=UA-64295674-3)](<>)