diff options
author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-03-09 13:19:22 +0000 |
---|---|---|
committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-03-09 13:19:22 +0000 |
commit | c21c3b0befeb46a51b6bf3758ffa30813bea0ff0 (patch) | |
tree | 9754ff1ca740f6346cf8483ec915d4054bc5da2d /packaging | |
parent | Adding upstream version 1.43.2. (diff) | |
download | netdata-c21c3b0befeb46a51b6bf3758ffa30813bea0ff0.tar.xz netdata-c21c3b0befeb46a51b6bf3758ffa30813bea0ff0.zip |
Adding upstream version 1.44.3.upstream/1.44.3
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to '')
39 files changed, 758 insertions, 201 deletions
diff --git a/packaging/PLATFORM_SUPPORT.md b/packaging/PLATFORM_SUPPORT.md index 3c81d64fe..5448e5da4 100644 --- a/packaging/PLATFORM_SUPPORT.md +++ b/packaging/PLATFORM_SUPPORT.md @@ -98,7 +98,6 @@ with minimal user effort. | Alpine Linux | Edge | No | | | Alpine Linux | 3.17 | No | | | Alpine Linux | 3.16 | No | | -| Alpine Linux | 3.15 | No | | | Arch Linux | Latest | No | We officially recommend the community packages available for Arch Linux | | Manjaro Linux | Latest | No | We officially recommend the community packages available for Arch Linux | | openSUSE | Tumbleweed | x86\_64, AArch64 | Scheduled for promotion to Core tier at some point after the release of v1.41.0 of the Netdata Agent | diff --git a/packaging/VERSIONING_AND_PUBLIC_API.md b/packaging/VERSIONING_AND_PUBLIC_API.md new file mode 100644 index 000000000..79c537851 --- /dev/null +++ b/packaging/VERSIONING_AND_PUBLIC_API.md @@ -0,0 +1,145 @@ +# Netdata Agent Versioning Policy (DRAFT) + +This document outlines how versions are handled for the Netdata Agent. This policy applies to version 2.0.0 of +the Netdata Agent and newer versions. + +## Stable Releases + +Versions for stable releases of the Netdata Agent consist of three parts, a major version, a minor version, and +a patch version, presented like `<major>.<minor>.<patch>`. For example, a version of `1.42.3` has a major version +of 1, a minor version of 42, and a patch version of 3. + +The patch version is incremented when a new stable release is made that only contains bug fixes that do not alter +the public API in a backwards incompatible manner. Special exceptions may be made for critical security bugs, +but such exceptions will be prominently noted in the release notes for the versions for which they are made. + +The minor version is incremented when a new stable release is made that contains new features and functionality +that do not alter the strictly defined parts of public API in a backwards incompatible manner. A new minor version +may have changes to the loosely defined parts of the public API that are not backwards compatible, but unless +they are critical security fixes they will be announced ahead of time in the release notes for the previous minor +version. Once a new minor version is published, no new patch releases will be published for previous minor versions +unless they fix serious bugs. + +The major version is incremented when a new stable release is made that alters the strictly defined public API in +some backwards incompatible manner. Any backwards incompatible changes that will be included in a new major version +will be announced ahead of time in the release notes for the previous minor version. Once a given major version +is published, no new minor releases will be published for any prior major version, though new patch releases _may_ +be published for the latest minor release of any prior major version to fix serious bugs. + +In most cases, just prior to a new major version being published, a final stable minor release will be published +for the previous major version, including all non-breaking changes that will be in the new major version. This is +intended to ensure that users who choose to remain on the previous major version for an extended period of time +will be as up-to-date as possible. + +## Nightly Builds + +Versions for nightly builds of the Netdata Agent consist of four parts, a major version, a minor version, a revision +number, and an optional commit ID, presented like `<major>.<minor>.0-<revision>-<commit>`. For example, a version +of `1.43.0-11-gb15437502` has a major version of 1, a minor version of 43, a revision of 11, and a commit ID of +`gb15437502`. A commit ID consists of a lowercaase letter `g`, followed by the short commit hash for the corresponding +commit. If the commit ID is not included, it may be replaced by the word ‘nightly’. + +The major and minor version numbers for a nightly build correspond exactly to an associated stable release. A +given major version of a nightly build has the same compatibility guarantees as it would for a stable release. A +given minor version of a nightly build will generally include any backwards-incompatible changes to the loosely +defined public API that will be in the _next_ minor version of the associated stable release. + +The revision number indicates the number of commits on the main branch of the Netdata Agent git repository since +the associated stable release, and the commit ID, if included, should indicate the exact commit hash used for the +nightly build. + +Due to how our release process works, nightly version numbers do not track stable patch releases. For example, if the +latest stable release is `1.42.4`, the latest nightly version will still show something like `1.42.0-209-nightly`. The +first nightly build version published after an associated stable release will include all relevant fixes that were +in that stable release. In addition, in most cases, the last nightly build version published before an associated +stable patch release will include all relevant fixes that are in that patch release. + +Nightly builds are only published on days when changes have actually been committed to the main branch of the +Netdata Agent git repository. + +## Public API + +The remainder of the document outlines the public API of the Netdata agent. + +We define two categories of components within the public API: + +- Strictly defined components are guaranteed not to change in a backwards incompatible manner without an associated + major version bump, and will have impending changes announced in the release notes at least one minor release + before they are changed. +- Loosely defined components are guaranteed not to change in a backwards incompatible manner without an associated + minor version bump, and will have impending changes announced in the release notes at least one minor release + before they are changed. + +There are also a few things we handle specially, which will be noted later in the document. + +### Strictly Defined Public API Components + +The following aspects of the public API are strictly defined, and are guaranteed not to change in a backwards +incompatible manner without an associated major version increase, and such changes will be announced in the release +notes at least one minor release prior to being merged: + +- All mandatory build dependencies which are not vendored in the Netdata Agent code. This includes, but is not + limited to: + - The underlying build system (such as autotools or CMake). + - Primary library dependencies (such as libuv). + - Any external tooling that is required at build time. +- The REST API provided by the Netdata Agent’s internal web server, accessible via the `/api` endpoint. This + does not extend to the charts, labels, or other system-specific data returned by some API endpoints. +- The protocol used for streaming and replicating data between Netdata Agents. +- The protocol used for communicating with external data collection plugins. +- The APIs provided by the `python.d.plugin` and `charts.d.plugin` data collection frameworks. +- The set of optional features supported by the Agent which are provided by default in our pre-built packages. If + support for an optional feature is being completely removed from the agent, that is instead covered by what + component that feature is part of. + +### Loosely Defined Public API Components + +The following aspects of the public API are loosely defined. They are guaranteed to not change in a backwards +incompatible manner without an associated minor version increase, and such changes will be announced in the release +notes at least one minor release prior to being merged: + +- Configuration options in any configuration file normally located under `/etc/netdata` on a typical install, + as well as their default values. +- Environment variables that are interpreted by the Netdata Agent, or by the startup code in our official OCI + container images. +- The exact set of charts provided, including chart families, chart names, and provided metrics. +- The exact set of supported data collection sources and data export targets. +- The exact set of system service managers we officially support running the Netdata Agent under. +- The exact set of alert delivery mechanisms supported by the Netdata Agent. +- The high-level implementation of the Netdata Agent’s integrated web server. +- The v0 and v1 dashboard UIs provided through the Netdata Agent’s internal web server. + +All loosely defined API components may also change in a backwards incompatible manner if the major version is +increased. Large scale changes to these components may also warrant a major version increase even if there are no +backwards incompatible changes to strictly defined public API components. + +### Special Cases + +The following special exceptions to the public API exist: + +- When an internal on-disk file format (such as the dbengine data file format) is changed, the old format is + guaranteed to be supported for in-place updates for at least two minor versions after the change happens. The + new format is not guaranteed to be backwards compatible. +- The list of supported platforms is functionally a part of the public API, but our existing [platform support + policy](https://github.com/netdata/netdata/blob/master/packaging/PLATFORM_SUPPORT.md) dictates when and how + support for specific platforms is added or removed. +- The list of components provided as separate packages in our official native packages is considered part of our + strictly defined public API, but changes to our packaging that do not alter the functionality of existing installs + are considered to be backwards compatible. This means that we may choose to split a plugin out to it’s own + package at any time, but it will remain as a mandatory dependency until at least the next major release. +- Options and environment variables used by the `kickstart.sh` install script and the `netdata-updater.sh` script + are handled separately from regular Netdata Agent versioning. Backwards compatible changes may happen at any + time for these, while backwards incompatible changes will have a deprecation period during which the old behavior + will be preserved but will issue a warning about the impending change. + +### Things Not Covered By The Public API + +Any components which are not explicitly listed above as being part of the public API are not part of the public +API. This includes, but is not limited to: + +- Any mandatory build components which are vendored as part of the Netdata sources, such as SQLite3 or libJudy. This + extends to both the presence or abscence of such components, as well as the exact version being bundled. +- The exact installation mechanism that will be used on any given system when using our `kickstart.sh` installation + script. +- The exact underlying implementation of any data collection plugin. +- The exact underlying implementation of any data export mechanism. diff --git a/packaging/docker/Dockerfile b/packaging/docker/Dockerfile index ddc4a4f5c..8e7c9a7b1 100644 --- a/packaging/docker/Dockerfile +++ b/packaging/docker/Dockerfile @@ -3,7 +3,7 @@ # This image contains preinstalled dependencies # hadolint ignore=DL3007 -FROM netdata/builder:v1 as builder +FROM netdata/builder:v2 as builder # One of 'nightly' or 'stable' ARG RELEASE_CHANNEL=nightly @@ -29,7 +29,7 @@ RUN chmod +x netdata-installer.sh && \ cp -rp /deps/* /usr/local/ && \ /bin/echo -e "INSTALL_TYPE='oci'\nPREBUILT_ARCH='$(uname -m)'" > ./system/.install-type && \ CFLAGS="$(packaging/docker/gen-cflags.sh)" LDFLAGS="-Wl,--gc-sections" ./netdata-installer.sh --dont-wait --dont-start-it --use-system-protobuf \ - ${EXTRA_INSTALL_OPTS} --one-time-build --enable-lto "$([ "$RELEASE_CHANNEL" = stable ] && echo --stable-channel)" + ${EXTRA_INSTALL_OPTS} --disable-ebpf --one-time-build --enable-lto "$([ "$RELEASE_CHANNEL" = stable ] && echo --stable-channel)" # files to one directory RUN mkdir -p /app/usr/sbin/ \ @@ -49,6 +49,7 @@ RUN mkdir -p /app/usr/sbin/ \ mv /usr/sbin/netdata /app/usr/sbin/ && \ mv /usr/sbin/netdata-claim.sh /app/usr/sbin/ && \ mv /usr/sbin/netdatacli /app/usr/sbin/ && \ + mv /usr/sbin/systemd-cat-native /app/usr/sbin/ && \ mv packaging/docker/run.sh /app/usr/sbin/ && \ mv packaging/docker/health.sh /app/usr/sbin/ && \ mkdir -p /deps/etc && \ @@ -59,7 +60,7 @@ RUN mkdir -p /app/usr/sbin/ \ ##################################################################### # This image contains preinstalled dependencies # hadolint ignore=DL3007 -FROM netdata/base:v1 as base +FROM netdata/base:v2 as base LABEL org.opencontainers.image.authors="Netdatabot <bot@netdata.cloud>" LABEL org.opencontainers.image.url="https://netdata.cloud" @@ -83,23 +84,26 @@ ENV DOCKER_USR netdata ENV NETDATA_LISTENER_PORT 19999 EXPOSE $NETDATA_LISTENER_PORT -ENV NETDATA_EXTRA_APK_PACKAGES="" +ENV NETDATA_EXTRA_DEB_PACKAGES="" RUN mkdir -p /opt/src /var/log/netdata && \ ln -sf /dev/stdout /var/log/netdata/access.log && \ + ln -sf /dev/stdout /var/log/netdata/aclk.log && \ ln -sf /dev/stdout /var/log/netdata/debug.log && \ ln -sf /dev/stderr /var/log/netdata/error.log && \ + ln -sf /dev/stderr /var/log/netdata/daemon.log && \ ln -sf /dev/stdout /var/log/netdata/collector.log && \ - ln -sf /dev/stdout /var/log/netdata/health.log && \ - addgroup -g ${NETDATA_GID} -S "${DOCKER_GRP}" && \ - adduser -S -H -s /usr/sbin/nologin -u ${NETDATA_GID} -h /etc/netdata -G "${DOCKER_GRP}" "${DOCKER_USR}" + ln -sf /dev/stdout /var/log/netdata/fluentbit.log && \ + ln -sf /dev/stdout /var/log/netdata/health.log COPY --from=builder /app / -# Apply the permissions as described in +# Create netdata user and apply the permissions as described in # https://docs.netdata.cloud/docs/netdata-security/#netdata-directories, but own everything by root group due to https://github.com/netdata/netdata/pull/6543 # hadolint ignore=DL3013 -RUN chown -R root:root \ +RUN addgroup --gid ${NETDATA_GID} --system "${DOCKER_GRP}" && \ + adduser --system --no-create-home --shell /usr/sbin/nologin --uid ${NETDATA_UID} --home /etc/netdata --group "${DOCKER_USR}" && \ + chown -R root:root \ /etc/netdata \ /usr/share/netdata \ /usr/libexec/netdata && \ @@ -111,17 +115,17 @@ RUN chown -R root:root \ chown -R netdata:netdata /var/lib/netdata/cloud.d && \ chmod 0700 /var/lib/netdata/cloud.d && \ chmod 0755 /usr/libexec/netdata/plugins.d/*.plugin && \ - chmod 4755 \ - /usr/libexec/netdata/plugins.d/cgroup-network \ - /usr/libexec/netdata/plugins.d/local-listeners \ - /usr/libexec/netdata/plugins.d/apps.plugin \ - /usr/libexec/netdata/plugins.d/debugfs.plugin && \ - if [ -f /usr/libexec/netdata/plugins.d/freeipmi.plugin ]; then \ - chmod 4755 /usr/libexec/netdata/plugins.d/freeipmi.plugin; \ - fi && \ - if [ -f /usr/libexec/netdata/plugins.d/go.d.plugin ]; then \ - chmod 4755 /usr/libexec/netdata/plugins.d/go.d.plugin; \ - fi && \ + for name in cgroup-network \ + local-listeners \ + apps.plugin \ + debugfs.plugin \ + freeipmi.plugin \ + go.d.plugin \ + perf.plugin \ + slabinfo.plugin \ + systemd-journal.plugin; do \ + [ -f "/usr/libexec/netdata/plugins.d/$name" ] && chmod 4755 "/usr/libexec/netdata/plugins.d/$name"; \ + done && \ # Group write permissions due to: https://github.com/netdata/netdata/pull/6543 find /var/lib/netdata /var/cache/netdata -type d -exec chmod 0770 {} \; && \ find /var/lib/netdata /var/cache/netdata -type f -exec chmod 0660 {} \; && \ diff --git a/packaging/docker/README.md b/packaging/docker/README.md index b4ce58254..528ef5926 100644 --- a/packaging/docker/README.md +++ b/packaging/docker/README.md @@ -21,7 +21,7 @@ The Netdata container requires different privileges and mounts to provide functi Netdata installed on the host. Below you can find a list of Netdata components that need these privileges and mounts, along with their descriptions. -<details> +<details open> <summary>Privileges</summary> | Component | Privileges | Description | @@ -33,16 +33,18 @@ along with their descriptions. </details> -<details> +<details open> <summary>Mounts</summary> -| Component | Mounts | Description | -|:--------------:|:--------------------------:|-------------------------------------------------------------------------------------------------------------------------------------| -| netdata | /etc/os-release | Host info detection. | -| cgroups.plugin | /sys, /var/run/docker.sock | Docker containers monitoring and name resolution. | -| go.d.plugin | /var/run/docker.sock | Docker Engine and containers monitoring. See [docker](https://github.com/netdata/go.d.plugin/tree/master/modules/docker) collector. | -| apps.plugin | /etc/passwd, /etc/group | Monitoring of host system resource usage by each user and user group. | -| proc.plugin | /proc | Host system monitoring (CPU, memory, network interfaces, disks, etc.). | +| Component | Mounts | Description | +|:----------------------:|:--------------------------:|--------------------------------------------------------------------------------------------------------------------------------------------| +| netdata | /etc/os-release | Host info detection. | +| cgroups.plugin | /sys, /var/run/docker.sock | Docker containers monitoring and name resolution. | +| go.d.plugin | /var/run/docker.sock | Docker Engine and containers monitoring. See [docker](https://github.com/netdata/go.d.plugin/tree/master/modules/docker#readme) collector. | +| go.d.plugin | /var/log | Web servers logs tailing. See [weblog](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog#readme) collector. | +| apps.plugin | /etc/passwd, /etc/group | Monitoring of host system resource usage by each user and user group. | +| proc.plugin | /proc | Host system monitoring (CPU, memory, network interfaces, disks, etc.). | +| systemd-journal.plugin | /var/log | Viewing, exploring and analyzing systemd journal logs. | </details> @@ -69,9 +71,11 @@ docker run -d --name=netdata \ -v netdatacache:/var/cache/netdata \ -v /etc/passwd:/host/etc/passwd:ro \ -v /etc/group:/host/etc/group:ro \ + -v /etc/localtime:/etc/localtime:ro \ -v /proc:/host/proc:ro \ -v /sys:/host/sys:ro \ -v /etc/os-release:/host/etc/os-release:ro \ + -v /var/log:/host/var/log:ro \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ --restart unless-stopped \ --cap-add SYS_PTRACE \ @@ -108,9 +112,11 @@ services: - netdatacache:/var/cache/netdata - /etc/passwd:/host/etc/passwd:ro - /etc/group:/host/etc/group:ro + - /etc/localtime:/etc/localtime:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /etc/os-release:/host/etc/os-release:ro + - /var/log:/host/var/log:ro - /var/run/docker.sock:/var/run/docker.sock:ro volumes: @@ -127,6 +133,34 @@ volumes: > If you plan to Claim the node to Netdata Cloud, you can find the command with the right parameters by clicking the " > Add Nodes" button in your Space's "Nodes" view. +### With systemd units monitoring + +Monitoring systemd units requires mounting `/run/dbus`. This mount is not available on non-systemd systems, so we cannot +use it in the Recommended Way. + +Mounting `/run/dbus` provides: + +- [go.d/systemdunits](https://github.com/netdata/go.d.plugin/tree/master/modules/systemdunits#readme). +- Systemd-list-units function: information about all systemd units, including their active state, description, whether + they are enabled, and more. + +<Tabs> +<TabItem value="docker_run" label="docker run"> + +<h3> Using the <code>docker run</code> command </h3> + +Add `-v /run/dbus:/run/dbus:ro` to your `docker run`. + +</TabItem> +<TabItem value="docker compose" label="docker-compose"> + +<h3> Using the <code>docker-compose</code> command</h3> + +Add `- /run/dbus:/run/dbus:ro` to the netdata service `volumes`. + +</TabItem> +</Tabs> + ### With host-editable configuration Use a [bind mount](https://docs.docker.com/storage/bind-mounts/) for `/etc/netdata` rather than a volume. @@ -153,9 +187,11 @@ docker run -d --name=netdata \ -v netdatacache:/var/cache/netdata \ -v /etc/passwd:/host/etc/passwd:ro \ -v /etc/group:/host/etc/group:ro \ + -v /etc/localtime:/etc/localtime:ro \ -v /proc:/host/proc:ro \ -v /sys:/host/sys:ro \ -v /etc/os-release:/host/etc/os-release:ro \ + -v /var/log:/host/var/log:ro \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ --restart unless-stopped \ --cap-add SYS_PTRACE \ @@ -192,9 +228,11 @@ services: - netdatacache:/var/cache/netdata - /etc/passwd:/host/etc/passwd:ro - /etc/group:/host/etc/group:ro + - /etc/localtime:/etc/localtime:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /etc/os-release:/host/etc/os-release:ro + - /var/log:/host/var/log:ro - /var/run/docker.sock:/var/run/docker.sock:ro volumes: @@ -261,9 +299,11 @@ services: - netdatacache:/var/cache/netdata - /etc/passwd:/host/etc/passwd:ro - /etc/group:/host/etc/group:ro + - /etc/localtime:/etc/localtime:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /etc/os-release:/host/etc/os-release:ro + - /var/log:/host/var/log:ro - /var/run/docker.sock:/var/run/docker.sock:ro volumes: caddy_data: @@ -310,9 +350,11 @@ services: - netdatacache:/var/cache/netdata - /etc/passwd:/host/etc/passwd:ro - /etc/group:/host/etc/group:ro + - /etc/localtime:/etc/localtime:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /etc/os-release:/host/etc/os-release:ro + - /var/log:/host/var/log:ro environment: - DOCKER_HOST=localhost:2375 proxy: @@ -333,6 +375,8 @@ volumes: #### CetusGuard +> Note: This deployment method is supported by the community + ```yaml version: '3' services: @@ -353,9 +397,11 @@ services: - netdatacache:/var/cache/netdata - /etc/passwd:/host/etc/passwd:ro - /etc/group:/host/etc/group:ro + - /etc/localtime:/etc/localtime:ro - /proc:/host/proc:ro - /sys:/host/sys:ro - /etc/os-release:/host/etc/os-release:ro + - /var/log:/host/var/log:ro environment: - DOCKER_HOST=localhost:2375 cetusguard: @@ -444,22 +490,20 @@ above section on [configuring Agent containers](#configure-agent-containers) to how you created the container. Alternatively, you can directly use the hostname from the node running the container by mounting `/etc/hostname` from -the host in the container. With `docker run`, this can be done by adding `--volume /etc/hostname:/etc/hostname:ro` to +the host in the container. With `docker run`, this can be done by adding `--volume /etc/hostname:/host/etc/hostname:ro` to the options. If you are using Docker Compose, you can add an entry to the container's `volumes` section -reading `- /etc/hostname:/etc/hostname:ro`. +reading `- /etc/hostname:/host/etc/hostname:ro`. ## Adding extra packages at runtime By default, the official Netdata container images do not include a number of optional runtime dependencies. You -can add these dependencies, or any other APK packages, at runtime by listing them in the environment variable -`NETDATA_EXTRA_APK_PACKAGES`. +can add these dependencies, or any other APT packages, at runtime by listing them in the environment variable +`NETDATA_EXTRA_DEB_PACKAGES`. Commonly useful packages include: - `apcupsd`: For monitoring APC UPS devices. -- `libvirt-daemon`: For resolving cgroup names for libvirt domains. - `lm-sensors`: For monitoring hardware sensors. -- `msmtp`: For email alert support. - `netcat-openbsd`: For IRC alert support. ## Health Checks diff --git a/packaging/docker/run.sh b/packaging/docker/run.sh index 1b6ce7c66..415573320 100755 --- a/packaging/docker/run.sh +++ b/packaging/docker/run.sh @@ -41,11 +41,38 @@ export DOCKER_HOST if [ -n "${PGID}" ]; then echo "Creating docker group ${PGID}" - addgroup -g "${PGID}" "docker" || echo >&2 "Could not add group docker with ID ${PGID}, its already there probably" + addgroup --gid "${PGID}" "docker" || echo >&2 "Could not add group docker with ID ${PGID}, its already there probably" echo "Assign netdata user to docker group ${PGID}" - usermod -a -G "${PGID}" "${DOCKER_USR}" || echo >&2 "Could not add netdata user to group docker with ID ${PGID}" + usermod --append --groups "docker" "${DOCKER_USR}" || echo >&2 "Could not add netdata user to group docker with ID ${PGID}" fi +# Needed to read Proxmox VMs and (LXC) containers configuration files (name resolution + CPU and memory limits) +function add_netdata_to_proxmox_conf_files_group() { + group_guid="$(stat -c %g /host/etc/pve 2>/dev/null || true)" + [ -z "${group_guid}" ] && return + + if ! getent group "${group_guid}" >/dev/null; then + echo "Creating proxmox-etc-pve group with GID ${group_guid}" + if ! addgroup -g "${group_guid}" "proxmox-etc-pve"; then + echo >&2 "Failed to add group proxmox-etc-pve with GID ${group_guid}." + return + fi + fi + + if ! getent group "${group_guid}" | grep -q netdata; then + echo "Assign netdata user to group ${group_guid}" + if ! usermod -a -G "${group_guid}" "${DOCKER_USR}"; then + echo >&2 "Failed to add netdata user to group with GID ${group_guid}." + return + fi + fi +} + +if [ -d "/host/etc/pve" ]; then + add_netdata_to_proxmox_conf_files_group || true +fi + + if mountpoint -q /etc/netdata; then echo "Copying stock configuration to /etc/netdata" cp -an /etc/netdata.stock/* /etc/netdata @@ -71,13 +98,20 @@ if [ -n "${NETDATA_CLAIM_URL}" ] && [ -n "${NETDATA_CLAIM_TOKEN}" ] && [ ! -f /v fi if [ -n "${NETDATA_EXTRA_APK_PACKAGES}" ]; then - echo "Fetching APK repository metadata." - if ! apk update; then - echo "Failed to fetch APK repository metadata." + echo >&2 "WARNING: Netdata’s Docker images have switched from Alpine to Debian as a base platform. Supplementary package support is now handled through the NETDATA_EXTRA_DEB_PACKAGES variable instead of NETDATA_EXTRA_APK_PACKAGES." + echo >&2 "WARNING: The container will still run, but supplementary packages listed in NETDATA_EXTRA_APK_PACKAGES will not be installed." + echo >&2 "WARNING: To remove these messages, either undefine NETDATA_EXTRA_APK_PACKAGES, or define it to an empty string." +fi + +if [ -n "${NETDATA_EXTRA_DEB_PACKAGES}" ]; then + echo "Fetching APT repository metadata." + if ! apt-get update; then + echo "Failed to fetch APT repository metadata." else echo "Installing supplementary packages." + export DEBIAN_FRONTEND="noninteractive" # shellcheck disable=SC2086 - if ! apk add --no-cache ${NETDATA_EXTRA_APK_PACKAGES}; then + if ! apt-get install -y --no-install-recommends ${NETDATA_EXTRA_DEB_PACKAGES}; then echo "Failed to install supplementary packages." fi fi diff --git a/packaging/go.d.checksums b/packaging/go.d.checksums index b2171eaa9..addfc0a50 100644 --- a/packaging/go.d.checksums +++ b/packaging/go.d.checksums @@ -1,17 +1,20 @@ -836476aed93644135120bb3d765c6e93a4364f39a7c0863b4fad4a228f1f8328 *config.tar.gz -884537604f270d5158806432784c96679467112a80074a56378ed5a43b819727 *go.d.plugin-v0.56.3.darwin-amd64.tar.gz -eee7fa6216e5e3ff281b5203ed2381a8eedcc1c39417441701f82f8640ce3a51 *go.d.plugin-v0.56.3.darwin-arm64.tar.gz -5e9e1c6f6c51b65899a1b11c79c97b7a17f3e898dcb2a35028d2760ac15c14eb *go.d.plugin-v0.56.3.freebsd-386.tar.gz -369e1d74de745075570061312d31ce4ef3f2f7ce171be4fb63995d0c9c44689f *go.d.plugin-v0.56.3.freebsd-amd64.tar.gz -5ef79cdb6941dcb30e16641ee6217d67e674a29a178d07fb44eb8ce4329906bf *go.d.plugin-v0.56.3.freebsd-arm.tar.gz -a27f4470c9497f8f0e7f039245d98aec365186cd0081e99e44150d18088f4f62 *go.d.plugin-v0.56.3.freebsd-arm64.tar.gz -5ba987ab34e4ff0aa0f1a1b5700944afe0345080cd3ab83fb97fe36e3c708088 *go.d.plugin-v0.56.3.linux-386.tar.gz -ebb04e8b51a07b8d6b551e39b88343769ee4fec767d589ffa6cfa858aca6db1c *go.d.plugin-v0.56.3.linux-amd64.tar.gz -2edcaca8db297df6a2576182dcf2320154d171c171f43f1772606cebf23655c4 *go.d.plugin-v0.56.3.linux-arm.tar.gz -5bfd6ad65d3d7dc7f390e15ed580e58d6ca3f6026c48599566f331bf3d63e80f *go.d.plugin-v0.56.3.linux-arm64.tar.gz -9c392c729acc126fd44ae923c187dd07376a4095bc9515f5f5e7c4763f590c9f *go.d.plugin-v0.56.3.linux-mips.tar.gz -8270eb151d53ce83afea589ccd3a01122415c90e8df6d9024e5f2405ee209c25 *go.d.plugin-v0.56.3.linux-mips64.tar.gz -68bca1d848116288d44272af0edc1dac47c05898eb747f62af04e18492de7d1d *go.d.plugin-v0.56.3.linux-mips64le.tar.gz -11719aa0b01a1ed13cd9667fa83c09ddbd2c459165dedb9a7e98bbf196e47e60 *go.d.plugin-v0.56.3.linux-mipsle.tar.gz -8cb083d7a29fb5cdec6134946275e9d960acfeb1c3244da767e2f2b3fadb1f36 *go.d.plugin-v0.56.3.linux-ppc64.tar.gz -9bfe6539c09dafa3efbffd010ac2db0d1df15717375b89cd95ba5dfdd4c2252e *go.d.plugin-v0.56.3.linux-ppc64le.tar.gz +9eeb1a06764fcc7f75e25d56916d1566d4a7206de778856165f407409f091470 *config.tar.gz +9eeb1a06764fcc7f75e25d56916d1566d4a7206de778856165f407409f091470 *go.d.plugin-config-v0.58.0.tar.gz +cb9bbbc164e16fdef46ddb3a9aafa354fe83a2765a5aa25a7ceeaa4c60d90eb7 *go.d.plugin-v0.58.0.darwin-amd64.tar.gz +6d5123955f87ebf30e5faf17c8502616ca84f156ae5e6108cb4a83b79dd0fa6b *go.d.plugin-v0.58.0.darwin-arm64.tar.gz +04a3ceebb345556cfc3f5dd5230c31d06cf59f8f6a6d85c4e8cfb1497ac2c793 *go.d.plugin-v0.58.0.freebsd-386.tar.gz +9b530d2d7d387c81d0551888b0aa6b55290910f75c5f01a0d399ca29fa83757f *go.d.plugin-v0.58.0.freebsd-amd64.tar.gz +45f4d0884b3993d3758f63a453ace96207ebdb9f2d97f89d5e42795ca743c6b6 *go.d.plugin-v0.58.0.freebsd-arm.tar.gz +d4222a6812255946f5d367cd59e8d6284c36b44baaba2925f7268bc42368a41a *go.d.plugin-v0.58.0.freebsd-arm64.tar.gz +4d71efc97a8f32db36f1d3f925e97531f846d9c39d66fbe63f00097f9a6cd425 *go.d.plugin-v0.58.0.linux-386.tar.gz +287db876af5a5b093ee91ef937f4ee59ebc5fdf79e403a48042b9f3cf58c716f *go.d.plugin-v0.58.0.linux-amd64.tar.gz +c0a4f1a20e2d93e1df7adab651b9feb7ca481b0b04e4e12323cad7b8f39e8590 *go.d.plugin-v0.58.0.linux-arm.tar.gz +b94adb6df7fc3a04cda1078e82c2d97a514c12dcc12f5dba7cec2259a34c89bb *go.d.plugin-v0.58.0.linux-arm64.tar.gz +5570b0ebc7c1a45c00301b0212531ee178cc06cb47912330ebc3d3d20bed6b13 *go.d.plugin-v0.58.0.linux-mips.tar.gz +6a850631e1978fdb6ff27923c3779f85e985dd0adb3cfb3767a482777e1802c8 *go.d.plugin-v0.58.0.linux-mips64.tar.gz +1ac22842fa52b97efac45f39f36e9fe69bd9a47497d91653563e02c2855ea5ff *go.d.plugin-v0.58.0.linux-mips64le.tar.gz +2487214cf11430e4152fbccf17205764d91e731aa236b2edb994d8242d33db26 *go.d.plugin-v0.58.0.linux-mipsle.tar.gz +547e4196cd1ebe07054de74f64bcea5ff704376138a495d6b66a6d3f46b22c5f *go.d.plugin-v0.58.0.linux-ppc64.tar.gz +3917e4c798cca7d5f944eb983f8facc2227abff88fc12398a277ee38010540cd *go.d.plugin-v0.58.0.linux-ppc64le.tar.gz +089bff22c63c1b79a0081e3c52e26eacafbea3698f967e5d18cee0c7dd0f88f9 *go.d.plugin-vendor-v0.58.0.tar.gz +089bff22c63c1b79a0081e3c52e26eacafbea3698f967e5d18cee0c7dd0f88f9 *vendor.tar.gz diff --git a/packaging/go.d.version b/packaging/go.d.version index 891e4e1ae..0bf661714 100644 --- a/packaging/go.d.version +++ b/packaging/go.d.version @@ -1 +1 @@ -v0.56.3 +v0.58.0 diff --git a/packaging/installer/dependencies/alpine.sh b/packaging/installer/dependencies/alpine.sh index 321d57707..ee0504b34 100755 --- a/packaging/installer/dependencies/alpine.sh +++ b/packaging/installer/dependencies/alpine.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/sh # Package tree used for installing netdata on distribution: # << Alpine: [3.12] [3.13] [3.14] [3.15] [edge] >> @@ -31,6 +31,9 @@ package_tree=" util-linux-dev libmnl-dev json-c-dev + musl-fts-dev + bison + flex yaml-dev " @@ -67,7 +70,8 @@ check_flags() { done if [ "${DONT_WAIT}" -eq 0 ] && [ "${NON_INTERACTIVE}" -eq 0 ]; then - read -r -p "Press ENTER to run it > " || exit 1 + printf "Press ENTER to run it > " + read -r || exit 1 fi } @@ -76,8 +80,18 @@ check_flags ${@} packages_to_install= +handle_old_alpine() { + version="$(grep VERSION_ID /etc/os-release | cut -f 2 -d '=')" + major="$(echo "${version}" | cut -f 1 -d '.')" + minor="$(echo "${version}" | cut -f 2 -d '.')" + + if [ "${major}" -le 3 ] && [ "${minor}" -le 16 ]; then + package_tree="$(echo "${package_tree}" | sed 's/musl-fts-dev/fts-dev/')" + fi +} + for package in $package_tree; do - if apk -e info "$package" &> /dev/null; then + if apk -e info "$package" > /dev/null 2>&1 ; then echo "Package '${package}' is installed" else echo "Package '${package}' is NOT installed" @@ -85,7 +99,7 @@ for package in $package_tree; do fi done -if [[ -z $packages_to_install ]]; then +if [ -z "${packages_to_install}" ]; then echo "All required packages are already installed. Skipping .." else echo "packages_to_install:" "$packages_to_install" diff --git a/packaging/installer/dependencies/arch.sh b/packaging/installer/dependencies/arch.sh index c0890d925..30be834be 100755 --- a/packaging/installer/dependencies/arch.sh +++ b/packaging/installer/dependencies/arch.sh @@ -32,6 +32,8 @@ declare -a package_tree=( gzip python3 binutils + bison + flex ) usage() { diff --git a/packaging/installer/dependencies/centos.sh b/packaging/installer/dependencies/centos.sh index e2599f0f6..532a0a71e 100755 --- a/packaging/installer/dependencies/centos.sh +++ b/packaging/installer/dependencies/centos.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Package tree used for installing netdata on distribution: -# << CentOS: [7] [8] >> +# << CentOS: [7] [8] [9] >> set -e @@ -8,9 +8,12 @@ declare -a package_tree=( autoconf autoconf-archive automake + bison cmake + cmake3 curl elfutils-libelf-devel + flex findutils gcc gcc-c++ diff --git a/packaging/installer/dependencies/debian.sh b/packaging/installer/dependencies/debian.sh index 09a5d6338..692a71191 100755 --- a/packaging/installer/dependencies/debian.sh +++ b/packaging/installer/dependencies/debian.sh @@ -12,8 +12,10 @@ package_tree=" autoconf-archive autogen automake + bison cmake curl + flex g++ gcc git diff --git a/packaging/installer/dependencies/fedora.sh b/packaging/installer/dependencies/fedora.sh index a457ef2e2..fc30b6113 100755 --- a/packaging/installer/dependencies/fedora.sh +++ b/packaging/installer/dependencies/fedora.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Package tree used for installing netdata on distribution: -# << Fedora: [24->35] >> +# << Fedora: [24->38] >> set -e @@ -28,10 +28,12 @@ declare -a package_tree=( autoconf-archive autogen automake + bison cmake curl elfutils-libelf-devel findutils + flex gcc gcc-c++ git diff --git a/packaging/installer/dependencies/freebsd.sh b/packaging/installer/dependencies/freebsd.sh index 69a650a6e..eadbcfa98 100755 --- a/packaging/installer/dependencies/freebsd.sh +++ b/packaging/installer/dependencies/freebsd.sh @@ -26,6 +26,8 @@ package_tree=" liblz4 openssl python3 + bison + flex " prompt() { diff --git a/packaging/installer/dependencies/gentoo.sh b/packaging/installer/dependencies/gentoo.sh index cbe8c8e51..9cf7f281a 100755 --- a/packaging/installer/dependencies/gentoo.sh +++ b/packaging/installer/dependencies/gentoo.sh @@ -31,6 +31,8 @@ package_tree=" virtual/libelf dev-lang/python dev-libs/libuv + sys-devel/bison + sys-devel/flex " usage() { cat << EOF diff --git a/packaging/installer/dependencies/ol.sh b/packaging/installer/dependencies/ol.sh index f0d8341f1..2dc10cee5 100755 --- a/packaging/installer/dependencies/ol.sh +++ b/packaging/installer/dependencies/ol.sh @@ -12,9 +12,11 @@ declare -a package_tree=( autoconf-archive autogen automake + bison cmake curl elfutils-libelf-devel + flex gcc gcc-c++ git diff --git a/packaging/installer/dependencies/opensuse.sh b/packaging/installer/dependencies/opensuse.sh index 3f14d58be..ecf1268fc 100755 --- a/packaging/installer/dependencies/opensuse.sh +++ b/packaging/installer/dependencies/opensuse.sh @@ -14,8 +14,10 @@ declare -a package_tree=( autoconf-archive autogen automake + bison cmake curl + flex gcc gcc-c++ git diff --git a/packaging/installer/dependencies/rockylinux.sh b/packaging/installer/dependencies/rockylinux.sh index fc26c8beb..cc8d45204 100755 --- a/packaging/installer/dependencies/rockylinux.sh +++ b/packaging/installer/dependencies/rockylinux.sh @@ -12,10 +12,12 @@ declare -a package_tree=( autoconf-archive autogen automake + bison cmake curl elfutils-libelf-devel findutils + flex gcc gcc-c++ git diff --git a/packaging/installer/dependencies/ubuntu.sh b/packaging/installer/dependencies/ubuntu.sh index a5dc2eae5..e223ca384 100755 --- a/packaging/installer/dependencies/ubuntu.sh +++ b/packaging/installer/dependencies/ubuntu.sh @@ -12,8 +12,10 @@ package_tree=" autoconf-archive autogen automake + bison cmake curl + flex g++ gcc git diff --git a/packaging/installer/functions.sh b/packaging/installer/functions.sh index b12a9a585..dd3158d6d 100644 --- a/packaging/installer/functions.sh +++ b/packaging/installer/functions.sh @@ -905,8 +905,13 @@ create_netdata_conf() { fi if [ -z "$url" ]; then - echo "# netdata can generate its own config which is available at 'http://<netdata_ip>/netdata.conf'" > "${path}" - echo "# You can download it with command like: 'wget -O ${path} http://localhost:19999/netdata.conf'" >> "${path}" + cat << EOF > "${path}" +# netdata can generate its own config which is available at 'http://<IP>:19999/netdata.conf' +# You can download it using: +# curl -o ${path} http://localhost:19999/netdata.conf +# or +# wget -O ${path} http://localhost:19999/netdata.conf +EOF fi } diff --git a/packaging/installer/install-required-packages.sh b/packaging/installer/install-required-packages.sh index 807c30552..bdd529397 100755 --- a/packaging/installer/install-required-packages.sh +++ b/packaging/installer/install-required-packages.sh @@ -28,6 +28,7 @@ PACKAGES_UPDATE_IPSETS=${PACKAGES_UPDATE_IPSETS-0} PACKAGES_NETDATA_DEMO_SITE=${PACKAGES_NETDATA_DEMO_SITE-0} PACKAGES_NETDATA_SENSORS=${PACKAGES_NETDATA_SENSORS-0} PACKAGES_NETDATA_DATABASE=${PACKAGES_NETDATA_DATABASE-1} +PACKAGES_NETDATA_STREAMING_COMPRESSION=${PACKAGES_NETDATA_STREAMING_COMPRESSION-0} PACKAGES_NETDATA_EBPF=${PACKAGES_NETDATA_EBPF-1} # needed commands @@ -671,6 +672,28 @@ declare -A pkg_cmake=( ['default']="cmake" ) +# bison and flex are required by Fluent-Bit +declare -A pkg_bison=( + ['default']="bison" +) + +declare -A pkg_flex=( + ['default']="flex" +) + +# fts-dev is required by Fluent-Bit on Alpine +declare -A pkg_fts_dev=( + ['default']="NOTREQUIRED" + ['alpine']="musl-fts-dev" + ['alpine-3.16.9']="fts-dev" +) + +# cmake3 is required by Fluent-Bit on CentOS 7 +declare -A pkg_cmake3=( + ['default']="NOTREQUIRED" + ['centos-7']="cmake3" +) + declare -A pkg_json_c_dev=( ['alpine']="json-c-dev" ['arch']="json-c" @@ -1016,6 +1039,18 @@ declare -A pkg_lz4=( ['default']="lz4-devel" ) +declare -A pkg_zstd=( + ['alpine']="zstd-dev" + ['debian']="libzstd-dev" + ['ubuntu']="libzstd-dev" + ['gentoo']="app-arch/zstd" + ['clearlinux']="zstd-devel" + ['arch']="zstd" + ['macos']="zstd" + ['freebsd']="zstd" + ['default']="libzstd-devel" +) + declare -A pkg_libuv=( ['alpine']="libuv-dev" ['debian']="libuv1-dev" @@ -1209,6 +1244,7 @@ packages() { require_cmd automake || suitable_package automake require_cmd pkg-config || suitable_package pkg-config require_cmd cmake || suitable_package cmake + require_cmd cmake3 || suitable_package cmake3 # ------------------------------------------------------------------------- # debugging tools for development @@ -1231,6 +1267,8 @@ packages() { require_cmd tar || suitable_package tar require_cmd curl || suitable_package curl require_cmd gzip || suitable_package gzip + require_cmd bison || suitable_package bison + require_cmd flex || suitable_package flex fi # ------------------------------------------------------------------------- @@ -1262,6 +1300,7 @@ packages() { suitable_package libuuid-dev suitable_package libmnl-dev suitable_package json-c-dev + suitable_package fts-dev suitable_package libyaml-dev suitable_package libsystemd-dev fi @@ -1281,6 +1320,10 @@ packages() { suitable_package openssl fi + if [ "${PACKAGES_NETDATA_STREAMING_COMPRESSION}" -ne 0 ]; then + suitable_package zstd + fi + # ------------------------------------------------------------------------- # ebpf plugin if [ "${PACKAGES_NETDATA_EBPF}" -ne 0 ]; then @@ -1923,6 +1966,7 @@ while [ -n "${1}" ]; do PACKAGES_NETDATA_SENSORS=1 PACKAGES_NETDATA_DATABASE=1 PACKAGES_NETDATA_EBPF=1 + PACKAGES_NETDATA_STREAMING_COMPRESSION=1 ;; netdata) @@ -1930,6 +1974,7 @@ while [ -n "${1}" ]; do PACKAGES_NETDATA_PYTHON3=1 PACKAGES_NETDATA_DATABASE=1 PACKAGES_NETDATA_EBPF=1 + PACKAGES_NETDATA_STREAMING_COMPRESSION=1 ;; python | netdata-python) diff --git a/packaging/installer/kickstart.sh b/packaging/installer/kickstart.sh index 0ddd750be..f7c078274 100755 --- a/packaging/installer/kickstart.sh +++ b/packaging/installer/kickstart.sh @@ -2,7 +2,7 @@ # # SPDX-License-Identifier: GPL-3.0-or-later # -# Next unused error code: F0515 +# Next unused error code: F0516 # ====================================================================== # Constants @@ -38,6 +38,7 @@ FORUM_URL="https://community.netdata.cloud/" INSTALL_DOC_URL="https://learn.netdata.cloud/docs/install-the-netdata-agent/one-line-installer-for-all-linux-systems" PACKAGES_SCRIPT="https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/install-required-packages.sh" PUBLIC_CLOUD_URL="https://app.netdata.cloud" +RELEASE_INFO_URL="https://repo.netdata.cloud/releases" REPOCONFIG_DEB_URL_PREFIX="https://repo.netdata.cloud/repos/repoconfig" REPOCONFIG_RPM_URL_PREFIX="https://repo.netdata.cloud/repos/repoconfig" TELEMETRY_URL="https://us-east1-netdata-analytics-bi.cloudfunctions.net/ingest_agent_events" @@ -614,6 +615,22 @@ download() { fi } +get_actual_version() { + major="${1}" + channel="${2}" + url="${RELEASE_INFO_URL}/${channel}/${major}" + + if check_for_remote_file "${RELEASE_INFO_URL}"; then + if check_for_remote_file "${url}"; then + download "${url}" - + else + echo "NONE" + fi + else + echo "" + fi +} + get_redirect() { url="${1}" @@ -1321,24 +1338,20 @@ netdata_avail_check() { # Check for any distro-specific dependencies we know we need. check_special_native_deps() { - if [ "${DISTRO_COMPAT_NAME}" = "centos" ] && [ "${SYSVERSION}" = "7" ]; then - progress "Checking for libuv availability." - if ${pm_cmd} search --nogpgcheck -v libuv | grep -q "No matches found"; then - progress "libuv not found, checking for EPEL availability." - if ${pm_cmd} search --nogpgcheck -v epel-release | grep -q "No matches found"; then - warning "Unable to find a suitable source for libuv, cannot install using native packages on this system." - return 1 - else - progress "EPEL is available, attempting to install so that required dependencies are available." + if [ "${DISTRO_COMPAT_NAME}" = "centos" ] && [ "${SYSVERSION}" -gt 6 ]; then + progress "EPEL is required on this system, checking if it’s available." - # shellcheck disable=SC2086 - if ! run_as_root env ${env} ${pm_cmd} ${install_subcmd} ${pkg_install_opts} epel-release; then - warning "Failed to install EPEL, even though it is required to install native packages on this system." - return 1 - fi - fi + if ${pm_cmd} search --nogpgcheck -v epel-release | grep -q "No matches found"; then + warning "Unable to find a suitable source for libuv, cannot install using native packages on this system." + return 1 else - return 0 + progress "EPEL is available, attempting to install so that required dependencies are available." + + # shellcheck disable=SC2086 + if ! run_as_root env ${env} ${pm_cmd} ${install_subcmd} ${pkg_install_opts} epel-release; then + warning "Failed to install EPEL, even though it is required to install native packages on this system." + return 1 + fi fi fi } @@ -2025,6 +2038,40 @@ install_on_freebsd() { # ====================================================================== # Argument parsing code +handle_major_version() { + CONTINUE_INSTALL_PROMPT="Attempting to install will use the latest version available overall. Do you wish to continue the install?" + + if [ -z "${INSTALL_MAJOR_VERSION}" ]; then + return + fi + + actual_version="$(get_actual_version "v${INSTALL_MAJOR_VERSION}" "${RELEASE_CHANNEL}")" + + if [ -z "${actual_version}" ]; then + if [ "${INTERACTIVE}" -eq 0 ]; then + fatal "Could not determine the lastest releaase in channel '${RELEASE_CHANNEL}' with major version '${INSTALL_MAJOR_VERSION}'" F0517 + else + if confirm "Unable to determine the correct version to install for major version '${INSTALL_MAJOR_VERSION}'. ${CONTINUE_INSTALL_PROMPT}"; then + progress "User requested continuing the install with the latest version." + else + fatal "Cancelling installation at user request." F0518 + fi + fi + elif [ "${actual_version}" = 'NONE' ]; then + if [ "${INTERACTIVE}" -eq 0 ]; then + warning "No releases with major version '${INSTALL_MAJOR_VERSION}' have been published. Continuing the install with the latest version instead." + else + if confirm "No releases with major version '${INSTALL_MAJOR_VERSION}' have been published. ${CONTINUE_INSTALL_PROMPT}"; then + progress "User requested continuing the install with the latest version." + else + fatal "Cancelling installation at user request." F0519 + fi + fi + else + INSTALL_VERSION="${actual_version}" + fi +} + validate_args() { check_claim_opts @@ -2056,22 +2103,6 @@ validate_args() { esac fi - if [ -n "${NETDATA_OFFLINE_INSTALL_SOURCE}" ] && [ -n "${INSTALL_VERSION}" ]; then - fatal "Specifying an install version alongside an offline install source is not supported." F050A - fi - - if [ "${NETDATA_AUTO_UPDATES}" = "default" ]; then - if [ -n "${NETDATA_OFFLINE_INSTALL_SOURCE}" ] || [ -n "${INSTALL_VERSION}" ]; then - AUTO_UPDATE=0 - else - AUTO_UPDATE=1 - fi - elif [ "${NETDATA_AUTO_UPDATES}" = 1 ]; then - AUTO_UPDATE=1 - else - AUTO_UPDATE=0 - fi - if [ "${RELEASE_CHANNEL}" = "default" ]; then if [ -n "${NETDATA_OFFLINE_INSTALL_SOURCE}" ]; then SELECTED_RELEASE_CHANNEL="$(cat "${NETDATA_OFFLINE_INSTALL_SOURCE}/channel")" @@ -2089,6 +2120,31 @@ validate_args() { SELECTED_RELEASE_CHANNEL="${RELEASE_CHANNEL}" fi + + if [ -n "${INSTALL_MAJOR_VERSION}" ] && [ -n "${INSTALL_VERSION}" ]; then + fatal "Only one of --install-version or --install-major-version may be specified." F0515 + fi + + handle_major_version # Appropriately updates INSTALL_VERSION if INSTALL_MAJOR_VERSION is set. + + if [ -n "${NETDATA_OFFLINE_INSTALL_SOURCE}" ] && [ -n "${INSTALL_VERSION}" ]; then + fatal "Specifying an install version alongside an offline install source is not supported." F050A + fi + + if [ "${NETDATA_AUTO_UPDATES}" = "default" ]; then + if [ -n "${NETDATA_OFFLINE_INSTALL_SOURCE}" ] || [ -n "${INSTALL_VERSION}" ]; then + AUTO_UPDATE=0 + else + AUTO_UPDATE=1 + fi + elif [ "${NETDATA_INSTALL_MAJOR_VERSION}" ]; then + warning "Forcibly disabling auto updates as a specific major version was requested." + AUTO_UPDATE=0 + elif [ "${NETDATA_AUTO_UPDATES}" = 1 ]; then + AUTO_UPDATE=1 + else + AUTO_UPDATE=0 + fi } set_action() { @@ -2134,7 +2190,7 @@ parse_args() { "--claim-only") set_action 'claim' ;; "--no-updates") NETDATA_AUTO_UPDATES=0 ;; "--auto-update") NETDATA_AUTO_UPDATES="1" ;; - "--auto-update-method") + "--auto-update-type"|"--auto-update-method") NETDATA_AUTO_UPDATE_TYPE="$(echo "${2}" | tr '[:upper:]' '[:lower:]')" case "${NETDATA_AUTO_UPDATE_TYPE}" in systemd|interval|crontab) shift 1 ;; @@ -2168,6 +2224,10 @@ parse_args() { OLD_INSTALL_PREFIX="${2}" shift 1 ;; + "--install-major-version") + INSTALL_MAJOR_VERSION="${2}" + shift 1 + ;; "--install-version") INSTALL_VERSION="${2}" AUTO_UPDATE=0 diff --git a/packaging/installer/methods/kickstart.md b/packaging/installer/methods/kickstart.md index bdead412e..b21f4dde9 100644 --- a/packaging/installer/methods/kickstart.md +++ b/packaging/installer/methods/kickstart.md @@ -17,18 +17,31 @@ import TabItem from '@theme/TabItem'; ![](https://registry.my-netdata.io/api/v1/badge.svg?chart=web_log_nginx.requests_by_url_pattern&options=unaligned&dimensions=kickstart&group=sum&after=-3600&label=last+hour&units=kickstart%20downloads&value_color=orange&precision=0) ![](https://registry.my-netdata.io/api/v1/badge.svg?chart=web_log_nginx.requests_by_url_pattern&options=unaligned&dimensions=kickstart&group=sum&after=-86400&label=today&units=kickstart%20downloads&precision=0) -This page covers detailed instructions on using and configuring the automatic one-line installation script named -`kickstart.sh`. +`kickstart.sh` is the recommended way of installing Netdata. -The kickstart script works on all Linux distributions and macOS environments. By default, automatic nightly updates are enabled. If you are installing on macOS, make sure to check the [install documentation for macOS](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/macos.md) before continuing. +This script works on all Linux distributions and macOS environments, by detecting the optimal method of installing Netdata directly to the operating system (it will never install a docker image of Netdata - to run Netdata in a container [check Installing with Docker](https://learn.netdata.cloud/docs/installing/docker)). +If you are installing on macOS, make sure to check the [install documentation for macOS](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/macos.md) before continuing. + + +## Verify script integrity + +To use `md5sum` to verify the integrity of the `kickstart.sh` script you will download using the one-line command above, +run the following: + +```bash +[ "<checksum-will-be-added-in-documentation-processing>" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID" +``` + +If the script is valid, this command will return `OK, VALID`. + + +## Installation > :bulb: Tip > > If you are unsure whether you want nightly or stable releases, read the [installation guide](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md#nightly-vs-stable-releases). -> If you want to turn off [automatic updates](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md#automatic-updates), use the `--no-updates` option. You can find more installation options below. - To install Netdata, run the following as your normal user: <Tabs> @@ -49,6 +62,7 @@ To install Netdata, run the following as your normal user: > If you plan to also connect the node to Netdata Cloud, make sure to replace `YOUR_CLAIM_TOKEN` with the claim token of your space, > and `YOUR_ROOM_ID` with the ID of the room you are willing to connect the node to. + ## What does `kickstart.sh` do? The `kickstart.sh` script does the following after being downloaded and run using `sh`: @@ -68,48 +82,116 @@ The `kickstart.sh` script does the following after being downloaded and run usin The `kickstart.sh` script accepts a number of optional parameters to control how the installation process works: -- `--non-interactive`: Don’t prompt for anything and assume yes whenever possible, overriding any automatic detection of an interactive run. -- `--interactive`: Act as if running interactively, even if automatic detection indicates a run is non-interactive. -- `--dont-wait`: Synonym for `--non-interactive` -- `--dry-run`: Show what the installer would do, but don’t actually do any of it. -- `--dont-start-it`: Don’t auto-start the daemon after installing. This parameter is not guaranteed to work. -- `--release-channel`: Specify a particular release channel to install from. Currently supported release channels are: +### destination directory + +- `--install-prefix` + Specify an installation prefix for local builds (by default, we use a sane prefix based on the type of system). +- `--old-install-prefix` + Specify the custom local build's installation prefix that should be removed. + +### interactivity + +The script automatically detects if it is running interactively, on a user's terminal, or headless in a CI/CD environment. These are options related to overriding this behavior. + +- `--non-interactive` or `--dont-wait` + Don’t prompt for anything and assume yes whenever possible, overriding any automatic detection of an interactive run. Use this option when installing Netdata agent with a provisioning tool or in CI/CD. +- `--interactive` + Act as if running interactively, even if automatic detection indicates a run is non-interactive. + +### release channel + +By default, the script installs the nightly channel of Netdata, providing you with the most recent Netdata. For production systems where stability is more important than new features, we recommend using the stable channel. + +- `--release-channel` + Specify a particular release channel to install from. Currently supported release channels are: - `nightly`: Installs a nightly build (this is currently the default). - `stable`: Installs a stable release. - `default`: Explicitly request whatever the current default is. -- `--nightly-channel`: Synonym for `--release-channel nightly`. -- `--stable-channel`: Synonym for `--release-channel stable`. -- `--auto-update`: Enable automatic updates (this is the default). -- `--no-updates`: Disable automatic updates. -- `--disable-telemetry`: Disable anonymous statistics. -- `--native-only`: Only install if native binary packages are available. -- `--static-only`: Only install if a static build is available. -- `--build-only`: Only install using a local build. -- `--disable-cloud`: For local builds, don’t build any of the cloud code at all. For native packages and static builds, - use runtime configuration to disable cloud support. -- `--require-cloud`: Only install if Netdata Cloud can be enabled. Overrides `--disable-cloud`. -- `--install-prefix`: Specify an installation prefix for local builds (by default, we use a sane prefix based on the type of system). -- `--install-version`: Specify the version of Netdata to install. -- `--old-install-prefix`: Specify the custom local build's installation prefix that should be removed. -- `--local-build-options`: Specify additional options to pass to the installer code when building locally. Only valid if `--build-only` is also specified. -- `--static-install-options`: Specify additional options to pass to the static installer code. Only valid if --static-only is also specified. -- `--claim-token`: Specify a unique claiming token associated with your Space in Netdata Cloud to be used to connect to the node - after the install. -- `--claim-rooms`: Specify a comma-separated list of tokens for each War Room this node should appear in. -- `--claim-proxy`: Specify a proxy to use when connecting to the cloud in the form of `http://[user:pass@]host:ip` for an HTTP(S) proxy. - See [connecting through a proxy](https://github.com/netdata/netdata/blob/master/claim/README.md#connect-through-a-proxy) for details. -- `--claim-url`: Specify a URL to use when connecting to the cloud. Defaults to `https://app.netdata.cloud`. -- `--override-distro`: Override the distro detection logic and assume the system is using a specific Linux distribution and release. Takes a single argument consisting of the values of the `ID`, `VERSION_ID`, and `VERSION_CODENAME` fields from `/etc/os-release` for the desired distribution. - -The following options are mutually exclusive and specifiy special operations other than trying to install Netdata normally or update an existing install: - -- `--reinstall`: If there is an existing install, reinstall it instead of trying to update it. If there is not an existing install, install netdata normally. -- `--reinstall-even-if-unsafe`: If there is an existing install, reinstall it instead of trying to update it, even if doing so is known to potentially break things (for example, if we cannot detect what type of installation it is). If there is not an existing install, install Netdata normally. -- `--reinstall-clean`: If there is an existing install, uninstall it before trying to install Netdata. Fails if there is no existing install. -- `--uninstall`: Uninstall an existing installation of Netdata. Fails if there is no existing install. -- `--claim-only`: If there is an existing install, only try to claim it without attempting to update it. If there is no existing install, install and claim Netdata normally. -- `--repositories-only`: Only install repository configuration packages instead of doing a full install of Netdata. Automatically sets --native-only. -- `--prepare-offline-install-source`: Instead of insallling the agent, prepare a directory that can be used to install on another system without needing to download anything. See our [offline installation documentation](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/offline.md) for more info. +- `--nightly-channel` + Synonym for `--release-channel nightly`. +- `--stable-channel` + Synonym for `--release-channel stable`. +- `--install-version` + Specify the exact version of Netdata to install. + +### install type + +By default the script will prefer native builds when they are available, and then static builds. It will fallback to build from source when all others are not available. + +- `--native-only` + Only install if native binary packages are available. It fails otherwise. +- `--static-only` + Only install if a static build is available. It fails otherwise. + When installing a static build, the parameter `--static-install-options` can provide additional options to pass to the static installer code. +- `--build-only` + Only install using a local build. It fails otherwise. + When it builds from source, the parameter `--local-build-options` can be used to give additional build options. + +### automatic updates + +By default the script installs a cron job to automatically update Netdata to the latest version of the release channel used. + +- `--auto-update` + Enable automatic updates (this is the default). +- `--no-updates` + Disable automatic updates (not recommended). + +### Netdata Cloud related options + +By default, the kickstart script will provide a Netdata agent installation that can potentially communicate with Netdata Cloud, if of course the Netdata agent is further configured to do so. + +- `--claim-token` + Specify a unique claiming token associated with your Space in Netdata Cloud to be used to connect to the node after the install. This will enable, connect and claim the Netdata agent, to Netdata Cloud. +- `--claim-url` + Specify a URL to use when connecting to the cloud. Defaults to `https://app.netdata.cloud`. Use this option to change the Netdata Cloud URL to point to your Netdata Cloud installation. +- `--claim-rooms` + Specify a comma-separated list of tokens for each War Room this node should appear in. +- `--claim-proxy` + Specify a proxy to use when connecting to the cloud in the form of `http://[user:pass@]host:ip` for an HTTP(S) proxy. See [connecting through a proxy](https://github.com/netdata/netdata/blob/master/claim/README.md#connect-through-a-proxy) for details. +- `--claim-only` + If there is an existing install, only try to claim it without attempting to update it. If there is no existing install, install and claim Netdata normally. +- `--require-cloud` + Only install if Netdata Cloud can be enabled. +- `--disable-cloud` + For local builds, don’t build any of the Netdata Cloud code at all. For native packages and static builds, use runtime configuration to disable Netdata Cloud support. + +### anonymous telemetry + +By default, the agent is sending anonymous telemetry data to help us take identify the most common operating systems and the configurations Netdata agents run. We use this information to prioritize our efforts towards what is most commonly used by our community. + +- `--disable-telemetry` + Disable anonymous statistics. + +### reinstalling + +- `--reinstall` + If there is an existing install, reinstall it instead of trying to update it. If there is not an existing install, install netdata normally. +- `--reinstall-even-if-unsafe` + If there is an existing install, reinstall it instead of trying to update it, even if doing so is known to potentially break things (for example, if we cannot detect what type of installation it is). If there is not an existing install, install Netdata normally. +- `--reinstall-clean` + If there is an existing install, uninstall it before trying to install Netdata. Fails if there is no existing install. + +### uninstall + +- `--uninstall` + Uninstall an existing installation of Netdata. Fails if there is no existing install. + +### other options +- `--dry-run` + Show what the installer would do, but don’t actually do any of it. +- `--dont-start-it` + Don’t auto-start the daemon after installing. This parameter is not guaranteed to work. +- `--override-distro` + Override the distro detection logic and assume the system is using a specific Linux distribution and release. Takes a single argument consisting of the values of the `ID`, `VERSION_ID`, and `VERSION_CODENAME` fields from `/etc/os-release` for the desired distribution. + +The following options are mutually exclusive and specify special operations other than trying to install Netdata normally or update an existing install: + +- `--repositories-only` + Only install repository configuration packages instead of doing a full install of Netdata. Automatically sets --native-only. +- `--prepare-offline-install-source` + Instead of insallling the agent, prepare a directory that can be used to install on another system without needing to download anything. See our [offline installation documentation](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/offline.md) for more info. + +### environment variables Additionally, the following environment variables may be used to further customize how the script runs (most users should not need to use special values for any of these): @@ -123,7 +205,7 @@ should not need to use special values for any of these): - `DISABLE_TELEMETRY`: If set to a value other than 0, behave as if `--disable-telemetry` was specified. -### Native packages +## Native packages We publish official DEB/RPM packages for a number of common Linux distributions as part of our releases and nightly builds. These packages are available for 64-bit x86 systems. Depending on the distribution and release they may @@ -133,7 +215,7 @@ default installation method. This allows you to handle Netdata updates as part o If you want to enforce the usage of native packages and have the installer return a failure if they are not available, you can do so by adding `--native-only` to the options you pass to the installer. -### Static builds +## Static builds We publish pre-built static builds of Netdata for Linux systems. Currently, these are published for 64-bit x86, ARMv7, AArch64, and POWER8+ hardware. These static builds are able to operate in a mostly self-contained manner and only @@ -144,7 +226,7 @@ will be used by default for installation. If you want to enforce the usage of a static build and have the installer return a failure if one is not available, you can do so by adding `--static-only` to the options you pass to the installer. -### Local builds +## Local builds For systems which do not have available native packages or static builds, we support building Netdata locally on the system it will be installed on. When using this approach, the installer will attempt to install any required @@ -153,14 +235,3 @@ dependencies for building Netdata, though this may not always work correctly. If you want to enforce the usage of a local build (perhaps because you require a custom installation prefix, which is not supported with native packages or static builds), you can do so by adding `--build-only` to the options you pass to the installer. - -## Verify script integrity - -To use `md5sum` to verify the integrity of the `kickstart.sh` script you will download using the one-line command above, -run the following: - -```bash -[ "<checksum-will-be-added-in-documentation-processing>" = "$(curl -Ss https://my-netdata.io/kickstart.sh | md5sum | cut -d ' ' -f 1)" ] && echo "OK, VALID" || echo "FAILED, INVALID" -``` - -If the script is valid, this command will return `OK, VALID`. diff --git a/packaging/installer/methods/manual.md b/packaging/installer/methods/manual.md index 789f8aaab..269b67c1a 100644 --- a/packaging/installer/methods/manual.md +++ b/packaging/installer/methods/manual.md @@ -22,6 +22,9 @@ To install the latest git version of Netdata, please follow these 2 steps: ## Prepare your system +Before you begin, make sure that your repo and the repo's submodules are clean from any previous builds and up to date. +Otherwise, [perform a cleanup](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/manual.md#perform-a-cleanup-in-your-netdata-repo) + Use our automatic requirements installer (_no need to be `root`_), which attempts to find the packages that should be installed on your system to build and run Netdata. It supports a large variety of major Linux distributions and other operating systems and is regularly tested. You can find this tool [here](https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/install-required-packages.sh) or run it directly with `bash <(curl -sSL https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/install-required-packages.sh)`. Otherwise read on for how to get requires packages manually: @@ -226,3 +229,20 @@ Our current build process unfortunately has some issues when using certain confi If the installation fails with errors like `/bin/ld: externaldeps/libwebsockets/libwebsockets.a(context.c.o): relocation R_X86_64_32 against '.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC`, and you are trying to build with `clang` on Linux, you will need to build Netdata using GCC to get a fully functional install. In most cases, you can do this by running `CC=gcc ./netdata-installer.sh`. + + +### Perform a cleanup in your netdata repo + +The Netdata repo consist of the main git tree and it's submodules. Either working on a fork or on the main repo you need to make sure that there +are no "leftover" artifacts from previous builds and that your submodules are up to date to the **corresponding checkouts**. + +> #### Important: Make sure that you have commited any work in progress, before you proceed the with the clean up instruction below + + +```sh +git clean -dfx && git submodule foreach 'git clean -dfx' && git submodule update --recursive --init +``` + + +> Note: In previous builds, you may have created artifacts belonging to an another user (e.g root), so you may need to run +> each of the _git clean_ commands as sudoer. diff --git a/packaging/installer/methods/packages.md b/packaging/installer/methods/packages.md index 914920444..d49e21394 100644 --- a/packaging/installer/methods/packages.md +++ b/packaging/installer/methods/packages.md @@ -23,10 +23,10 @@ and fail if it cannot do so. > ### Note -> +> > In July 2022, we switched hosting of our native packages from Package Cloud to self-hosted repositories. -> We still maintain the Package cloud repositories, but they are not guaranteed to work and may be removed -> without prior warning. +> We still maintain the Package cloud repositories, but they are not guaranteed to work and may be removed +> without prior warning. > > When selecting a repository configuration package, note that the version 2 packages provide configuration for > our self-hosted repositories, and then version 1 packages provide configuration for Package Cloud. @@ -44,7 +44,9 @@ repositories at that top level: Within each top level group of repositories, there are directories for each supported group of distributions: -- `el`: Is for Red Hat Enterprise Linux and binary compatible distros, such as CentOS, Alma Linux, and Rocky Linux. +- `amazonlinux`: Is for Amazon Linux and binary compatible distros. +- `el`: Is for Red Hat Enterprise Linux and binary compatible distros that are not covered by other repos, such + as CentOS, Alma Linux, and Rocky Linux. - `fedora`: Is for Fedora and binary compatible distros. - `ol`: Is for Oracle Linux and binary compatible distros. - `opensuse`: Is for openSUSE and binary compatible distros. @@ -64,6 +66,13 @@ appropriate repository configuration package from https://repo.netdata.cloud/rep directly on the target system using the system package manager. This will ensure any packages needed to use the repository are also installed, and will help enable a seamless transition if we ever need to change our infrastructure. +> ### Note +> +> On RHEL and other systems that use the `el` repostiroies, some of the dependencies for Netdata can only be found +> in the EPEL repository, which is not enabled or installed by default on most of these systems. This additional +> repository _should_ be pulled in automatically by our repository config packages, but if it is not you may need +> to manually install `epel-release` to be able to successfully install the Netdata packages. + ## Manual setup of DEB packages. Netdata’s official DEB repositories are hosted at https://repo.netdata.cloud/repos. We provide four groups of @@ -99,3 +108,44 @@ If you are explicitly configuring a system to use our repositories, the recommen appropriate repository configuration package from https://repo.netdata.cloud/repos/repoconfig and install it directly on the target system using the system package manager. This will ensure any packages needed to use the repository are also installed, and will help enable a seamless transition if we ever need to change our infrastructure. + +## Local mirrors of the official Netdata repositories + +Local mirrors of our official repositories can be created in one of two ways: + +1. Using the standard tooling for mirroring the type of repository you want a local mirror of, such as Aptly for + APT repositories, or reposync for RPM repositories. For this approach, please consult the documentation for + the specific tool you are using for info on how to mirror the repositories. +2. Using a regular website mirroring tool, such as GNU wget’s `--mirror` option. For this approach, simply point + your mirroring tool at `https://repo.netdata.cloud/repos/`, and everything should just work. + +We do not provide official support for mirroring our repositories, +but we do have some tips for anyone looking to do so: + +- Our `robots.txt` file explicitly disallows indexing, so if you’re using a regular website mirroring tool, + you wil need to tell it to ignore `robots.txt` (for example, if using GNU wget, add `-e robots=off` to the + options you pass) to ensure that it actually retrieves everything. +- Excluding special cases of caching proxies (such as apt-cacher-ng), our repository configuration packages _DO NOT_ + work with custom local mirrors. Thus, you will need to manually configure your systems to use your local mirror. +- Packages are published as they are built, with 64-bit x86 packages being built first, followed by 32-bit x86, + and then non-x86 packages in alphabetical order of the CPU architecture. Because of the number of different + packages being built, this means that packages for a given nightly build or stable release are typically published + over the course of a few hours, usually starting about 15-20 minutes after the build or release is started. +- Repository metadata is updated every hour on the hour, and the process may take anywhere from a few seconds to + more than 20 minutes. Because of this, it makes little sense to sync your mirror more frequently than once an hour, + and it’s generally preferred to start syncing at least 30 minutes into the hour. +- A full mirror of all of our repositories currently requires up to 100 GB of storage space, though the exact + amount of space needed fluctuates over time. Because of this, users seeking to mirror our repositories are + encouraged to mirror only those repositories they actually need instead of mirroring everything. +- If syncing daily (or less frequently), some time between 05:00 and 08:00 UTC each day is usually the saftest + time to do so, as publishing nightly packages will almost always be done by this point, and publishing of stable + releases typically happens after that time window. +- If you intend to use our existing GPG signatures on the repository metadata and packages, you probably also want + a local copy of our public GPG key, which can be fetched from `https://repo.netdata.cloud/netdatabot.gpg.key`. + +## Public mirrors of the official Netdata repositories + +There are no official public mirrors of our repositories. + +If you wish to provide a public mirror of our official repositories, you are free to do so, but we kindly ask that +you make it clear to your users that your mirror is not an official mirror of our repositories. diff --git a/packaging/installer/methods/source.md b/packaging/installer/methods/source.md index 7b6b88d4b..8f34218a2 100644 --- a/packaging/installer/methods/source.md +++ b/packaging/installer/methods/source.md @@ -50,7 +50,11 @@ which the the build system will link statically into Netdata. These libraries and their header files must be copied into specific locations in the source tree to be used. +Before you begin, make sure that your repo and the repo's submodules are clean from any previous builds and up to date. +Otherwise, [perform a cleanup](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/manual.md#perform-a-cleanup-in-your-netdata-repo) + ### Netdata cloud + #### JSON-C Netdata requires the use of JSON-C for JSON parsing when using Netdata diff --git a/packaging/installer/netdata-uninstaller.sh b/packaging/installer/netdata-uninstaller.sh index 2f9fc91f9..4326ebe25 100755 --- a/packaging/installer/netdata-uninstaller.sh +++ b/packaging/installer/netdata-uninstaller.sh @@ -731,6 +731,7 @@ rm_file /etc/init.d/netdata rm_file /etc/periodic/daily/netdata-updater rm_file /etc/cron.daily/netdata-updater rm_file /etc/cron.d/netdata-updater +rm_file /etc/cron.d/netdata-updater-daily if [ -n "${NETDATA_PREFIX}" ] && [ -d "${NETDATA_PREFIX}" ] && [ "netdata" = "$(basename "$NETDATA_PREFIX")" ] ; then @@ -738,9 +739,11 @@ if [ -n "${NETDATA_PREFIX}" ] && [ -d "${NETDATA_PREFIX}" ] && [ "netdata" = "$( else rm_file "${NETDATA_PREFIX}/usr/sbin/netdata" rm_file "${NETDATA_PREFIX}/usr/sbin/netdatacli" + rm_file "${NETDATA_PREFIX}/usr/sbin/netdata-claim.sh" + rm_file "${NETDATA_PREFIX}/usr/sbin/log2journal" + rm_file "${NETDATA_PREFIX}/usr/sbin/systemd-cat-native" rm_file "/tmp/netdata-ipc" rm_file "/tmp/netdata-service-cmds" - rm_file "${NETDATA_PREFIX}/usr/sbin/netdata-claim.sh" rm_dir "${NETDATA_PREFIX}/usr/share/netdata" rm_dir "${NETDATA_PREFIX}/usr/libexec/netdata" rm_dir "${NETDATA_PREFIX}/var/lib/netdata" diff --git a/packaging/makeself/README.md b/packaging/makeself/README.md index 9219aefc5..1f2c746bf 100644 --- a/packaging/makeself/README.md +++ b/packaging/makeself/README.md @@ -19,13 +19,20 @@ will be used by default for installation. If you want to enforce the usage of a static build and have the installer return a failure if one is not available, you can do so by adding `--static-only` to the options you pass to the installer. +## Requirements + +- Container runtime tool (Docker or Podman) + ## Building a static binary package -To build the static binary 64-bit distribution package, run: +Before you begin, make sure that your repo and the repo's submodules are clean from any previous builds and up to date. +Otherwise, [perform a cleanup](https://github.com/netdata/netdata/blob/master/packaging/installer/methods/manual.md#perform-a-cleanup-in-your-netdata-repo) + + +To build the static binary 64-bit distribution package, into the root folder on the netdata repo, run: ```bash -cd /path/to/netdata.git -./packaging/makeself/build-x86_64-static.sh +./packaging/makeself/build-static.sh x86_64 ``` The program will: @@ -37,13 +44,15 @@ The program will: Once finished, a file named `netdata-vX.X.X-gGITHASH-x86_64-DATE-TIME.run` will be created in the current directory. This is the Netdata binary package that can be run to install Netdata on any other computer. +You can build static binaries for other architectures such as `armv7l`, `aarch64`, and `ppc64le`. + ## Building binaries with debug info To build Netdata binaries with debugging / tracing information in them, use: ```bash cd /path/to/netdata.git -./packaging/makeself/build-x86_64-static.sh debug +./packaging/makeself/build-static.sh x86_64 debug ``` These binaries are not optimized (they are a bit slower), they have certain features disables (like log flood protection), other features enables (like `debug flags`) and are not stripped (the binary files are bigger, since they now include source code tracing information). diff --git a/packaging/makeself/bundled-packages b/packaging/makeself/bundled-packages new file mode 100644 index 000000000..02ee4469d --- /dev/null +++ b/packaging/makeself/bundled-packages @@ -0,0 +1,16 @@ +# Source of truth for all the packages we bundle in static builds +PACKAGES=("OPENSSL" "CURL" "BASH" "IOPING" "LIBNETFILTER_ACT") +SOURCE_TYPES=("GH_REPO_CLONE" "GH_REPO_CLONE" "DW_TARBALL" "GH_REPO_SOURCE" "DW_TARBALL") +OPENSSL_VERSION="openssl-3.1.4" +OPENSSL_SOURCE="https://github.com/openssl/openssl" +CURL_VERSION="curl-8_4_0" +CURL_SOURCE="https://github.com/curl/curl" +BASH_VERSION="5.1.16" +BASH_ARTIFACT_SOURCE="http://ftp.gnu.org/gnu/bash" +BASH_ARTIFACT_SHA256="5bac17218d3911834520dad13cd1f85ab944e1c09ae1aba55906be1f8192f558" +IOPING_VERSION="1.3" +IOPING_SOURCE="https://github.com/koct9i/ioping" +IOPING_ARTIFACT_SHA256="7aa48e70aaa766bc112dea57ebbe56700626871052380709df3a26f46766e8c8" +LIBNETFILTER_ACT_VERSION="1.0.3" +LIBNETFILTER_ACT_SOURCE="https://www.netfilter.org/projects/libnetfilter_acct/files" +LIBNETFILTER_ACT_ARTIFACT_SHA256="4250ceef3efe2034f4ac05906c3ee427db31b9b0a2df41b2744f4bf79a959a1a" diff --git a/packaging/makeself/curl.version b/packaging/makeself/curl.version deleted file mode 100644 index 12ab47e6c..000000000 --- a/packaging/makeself/curl.version +++ /dev/null @@ -1 +0,0 @@ -curl-8_4_0 diff --git a/packaging/makeself/jobs/20-openssl.install.sh b/packaging/makeself/jobs/20-openssl.install.sh index 04ec2626f..1158a6330 100755 --- a/packaging/makeself/jobs/20-openssl.install.sh +++ b/packaging/makeself/jobs/20-openssl.install.sh @@ -3,12 +3,11 @@ # shellcheck source=packaging/makeself/functions.sh . "$(dirname "${0}")/../functions.sh" "${@}" || exit 1 - +# Source of truth for all the packages we bundle in static builds +. "$(dirname "${0}")/../bundled-packages" # shellcheck disable=SC2015 [ "${GITHUB_ACTIONS}" = "true" ] && echo "::group::Building OpenSSL" || true -version="$(cat "$(dirname "${0}")/../openssl.version")" - export CFLAGS='-fno-lto -pipe' export LDFLAGS='-static' export PKG_CONFIG="pkg-config --static" @@ -29,7 +28,7 @@ if [ -d "${cache}" ]; then CACHE_HIT=1 else echo "No cached copy of build directory for openssl found, fetching sources instead." - run git clone --branch "${version}" --single-branch --depth 1 https://github.com/openssl/openssl.git "${NETDATA_MAKESELF_PATH}/tmp/openssl" + run git clone --branch "${OPENSSL_VERSION}" --single-branch --depth 1 "${OPENSSL_SOURCE}" "${NETDATA_MAKESELF_PATH}/tmp/openssl" CACHE_HIT=0 fi diff --git a/packaging/makeself/jobs/50-bash-5.1.16.install.sh b/packaging/makeself/jobs/50-bash-5.1.16.install.sh index f75bb9d2e..7a302f2ee 100755 --- a/packaging/makeself/jobs/50-bash-5.1.16.install.sh +++ b/packaging/makeself/jobs/50-bash-5.1.16.install.sh @@ -3,14 +3,14 @@ # shellcheck source=packaging/makeself/functions.sh . "$(dirname "${0}")/../functions.sh" "${@}" || exit 1 - -version="5.1.16" +# Source of truth for all the packages we bundle in static builds +. "$(dirname "${0}")/../bundled-packages" # shellcheck disable=SC2015 [ "${GITHUB_ACTIONS}" = "true" ] && echo "::group::building bash" || true -fetch "bash-${version}" "http://ftp.gnu.org/gnu/bash/bash-${version}.tar.gz" \ - 5bac17218d3911834520dad13cd1f85ab944e1c09ae1aba55906be1f8192f558 bash +fetch "bash-${BASH_VERSION}" "${BASH_ARTIFACT_SOURCE}/bash-${BASH_VERSION}.tar.gz" \ + "${BASH_ARTIFACT_SHA256}" bash export CFLAGS="-pipe" export PKG_CONFIG_PATH="/openssl-static/lib64/pkgconfig" @@ -39,7 +39,7 @@ fi run make install -store_cache bash "${NETDATA_MAKESELF_PATH}/tmp/bash-${version}" +store_cache bash "${NETDATA_MAKESELF_PATH}/tmp/bash-${BASH_VERSION}" if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then run strip "${NETDATA_INSTALL_PATH}"/bin/bash diff --git a/packaging/makeself/jobs/50-curl.install.sh b/packaging/makeself/jobs/50-curl.install.sh index b3ed33186..824b30562 100755 --- a/packaging/makeself/jobs/50-curl.install.sh +++ b/packaging/makeself/jobs/50-curl.install.sh @@ -3,8 +3,8 @@ # shellcheck source=packaging/makeself/functions.sh . "$(dirname "${0}")/../functions.sh" "${@}" || exit 1 - -version="$(cat "$(dirname "${0}")/../curl.version")" +# Source of truth for all the packages we bundle in static builds +. "$(dirname "${0}")/../bundled-packages" # shellcheck disable=SC2015 [ "${GITHUB_ACTIONS}" = "true" ] && echo "::group::Building cURL" || true @@ -21,7 +21,7 @@ if [ -d "${cache}" ]; then CACHE_HIT=1 else echo "No cached copy of build directory for curl found, fetching sources instead." - run git clone --branch "${version}" --single-branch --depth 1 'https://github.com/curl/curl.git' "${NETDATA_MAKESELF_PATH}/tmp/curl" + run git clone --branch "${CURL_VERSION}" --single-branch --depth 1 "${CURL_SOURCE}" "${NETDATA_MAKESELF_PATH}/tmp/curl" CACHE_HIT=0 fi @@ -36,7 +36,7 @@ if [ "${CACHE_HIT:-0}" -eq 0 ]; then run autoreconf -fi run ./configure \ - --prefix="${NETDATA_INSTALL_PATH}" \ + --prefix="/curl-local" \ --enable-optimize \ --disable-shared \ --enable-static \ @@ -69,6 +69,7 @@ run make install store_cache curl "${NETDATA_MAKESELF_PATH}/tmp/curl" +cp /curl-local/bin/curl "${NETDATA_INSTALL_PATH}"/bin/curl if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then run strip "${NETDATA_INSTALL_PATH}"/bin/curl fi diff --git a/packaging/makeself/jobs/50-ioping-1.3.install.sh b/packaging/makeself/jobs/50-ioping-1.3.install.sh index 88309c3eb..6bd538e35 100755 --- a/packaging/makeself/jobs/50-ioping-1.3.install.sh +++ b/packaging/makeself/jobs/50-ioping-1.3.install.sh @@ -3,14 +3,14 @@ # shellcheck source=packaging/makeself/functions.sh . "$(dirname "${0}")/../functions.sh" "${@}" || exit 1 - -version='1.3' +# Source of truth for all the packages we bundle in static builds +. "$(dirname "${0}")/../bundled-packages" || exit 1 # shellcheck disable=SC2015 [ "${GITHUB_ACTIONS}" = "true" ] && echo "::group::Building ioping" || true -fetch "ioping-${version}" "https://github.com/koct9i/ioping/archive/v${version}.tar.gz" \ - 7aa48e70aaa766bc112dea57ebbe56700626871052380709df3a26f46766e8c8 ioping +fetch "ioping-${IOPING_VERSION}" "${IOPING_SOURCE}/archive/refs/tags/v${IOPING_VERSION}.tar.gz" \ + "${IOPING_ARTIFACT_SHA256}" ioping export CFLAGS="-static -pipe" @@ -22,7 +22,7 @@ fi run mkdir -p "${NETDATA_INSTALL_PATH}"/usr/libexec/netdata/plugins.d/ run install -o root -g root -m 4750 ioping "${NETDATA_INSTALL_PATH}"/usr/libexec/netdata/plugins.d/ -store_cache ioping "${NETDATA_MAKESELF_PATH}/tmp/ioping-${version}" +store_cache ioping "${NETDATA_MAKESELF_PATH}/tmp/ioping-${IOPING_VERSION}" if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then run strip "${NETDATA_INSTALL_PATH}"/usr/libexec/netdata/plugins.d/ioping diff --git a/packaging/makeself/jobs/50-libnetfilter_acct-1.0.3.install.sh b/packaging/makeself/jobs/50-libnetfilter_acct-1.0.3.install.sh index 2b7a761c5..829752178 100755 --- a/packaging/makeself/jobs/50-libnetfilter_acct-1.0.3.install.sh +++ b/packaging/makeself/jobs/50-libnetfilter_acct-1.0.3.install.sh @@ -2,12 +2,12 @@ # SPDX-License-Identifier: GPL-3.0-or-later # # Install the libnetfilter_acct and it's dependency libmnl -# + # shellcheck source=packaging/makeself/functions.sh . "$(dirname "${0}")/../functions.sh" "${@}" || exit 1 - -version="1.0.3" +# Source of truth for all the packages we bundle in static builds +. "$(dirname "${0}")/../bundled-packages" || exit 1 # shellcheck disable=SC2015 [ "${GITHUB_ACTIONS}" = "true" ] && echo "::group::building libnetfilter_acct" || true @@ -17,8 +17,8 @@ export LDFLAGS="-static -L/usr/lib -lmnl" export PKG_CONFIG="pkg-config --static" export PKG_CONFIG_PATH="/usr/lib/pkgconfig" -fetch "libnetfilter_acct-${version}" "https://www.netfilter.org/projects/libnetfilter_acct/files/libnetfilter_acct-${version}.tar.bz2" \ - 4250ceef3efe2034f4ac05906c3ee427db31b9b0a2df41b2744f4bf79a959a1a libnetfilter_acct +fetch "libnetfilter_acct-${LIBNETFILTER_ACT_VERSION}" "${LIBNETFILTER_ACT_SOURCE}/libnetfilter_acct-${LIBNETFILTER_ACT_VERSION}.tar.bz2" \ + "${LIBNETFILTER_ACT_ARTIFACT_SHA256}" libnetfilter_acct if [ "${CACHE_HIT:-0}" -eq 0 ]; then @@ -32,7 +32,7 @@ fi run make install -store_cache libnetfilter_acct "${NETDATA_MAKESELF_PATH}/tmp/libnetfilter_acct-${version}" +store_cache libnetfilter_acct "${NETDATA_MAKESELF_PATH}/tmp/libnetfilter_acct-${LIBNETFILTER_ACT_VERSION}" # shellcheck disable=SC2015 diff --git a/packaging/makeself/jobs/70-netdata-git.install.sh b/packaging/makeself/jobs/70-netdata-git.install.sh index 62f0c822d..83d28bf87 100755 --- a/packaging/makeself/jobs/70-netdata-git.install.sh +++ b/packaging/makeself/jobs/70-netdata-git.install.sh @@ -7,12 +7,12 @@ cd "${NETDATA_SOURCE_PATH}" || exit 1 if [ "${NETDATA_BUILD_WITH_DEBUG}" -eq 0 ]; then - export CFLAGS="-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl -pipe" + export CFLAGS="-ffunction-sections -fdata-sections -static -O2 -funroll-loops -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/curl-local/include/curl -I/usr/include/libmnl -pipe" else - export CFLAGS="-static -O1 -pipe -ggdb -Wall -Wextra -Wformat-signedness -DNETDATA_INTERNAL_CHECKS=1 -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/usr/include/libmnl" + export CFLAGS="-static -O1 -pipe -ggdb -Wall -Wextra -Wformat-signedness -DNETDATA_INTERNAL_CHECKS=1 -I/openssl-static/include -I/libnetfilter-acct-static/include/libnetfilter_acct -I/curl-local/include/curl -I/usr/include/libmnl" fi -export LDFLAGS="-Wl,--gc-sections -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl" +export LDFLAGS="-Wl,--gc-sections -static -L/openssl-static/lib64 -L/libnetfilter-acct-static/lib -lnetfilter_acct -L/usr/lib -lmnl -L/usr/lib -lzstd -L/curl-local/lib" # We export this to 'yes', installer sets this to .environment. # The updater consumes this one, so that it can tell whether it should update a static install or a non-static one @@ -21,7 +21,7 @@ export IS_NETDATA_STATIC_BINARY="yes" # Set eBPF LIBC to "static" to bundle the `-static` variant of the kernel-collector export EBPF_LIBC="static" export PKG_CONFIG="pkg-config --static" -export PKG_CONFIG_PATH="/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig" +export PKG_CONFIG_PATH="/openssl-static/lib64/pkgconfig:/libnetfilter-acct-static/lib/pkgconfig:/usr/lib/pkgconfig:/curl-local/lib/pkgconfig" # Set correct CMake flags for building against non-System OpenSSL # See: https://github.com/warmcat/libwebsockets/blob/master/READMEs/README.build.md diff --git a/packaging/makeself/openssl.version b/packaging/makeself/openssl.version deleted file mode 100644 index c4d4ea136..000000000 --- a/packaging/makeself/openssl.version +++ /dev/null @@ -1 +0,0 @@ -openssl-3.1.3 diff --git a/packaging/repoconfig/debian/changelog b/packaging/repoconfig/debian/changelog index 02eedfc36..d056fa43b 100644 --- a/packaging/repoconfig/debian/changelog +++ b/packaging/repoconfig/debian/changelog @@ -1,3 +1,9 @@ +netdata-repo (2-2) unstable; urgency=medium + + * Version bump to keep in sync with RPM repo packages + + -- Netdata Builder <bot@netdata.cloud> Mon, 13 Nov 2023 11:15:00 -0500 + netdata-repo (2-1) unstable; urgency=medium * Switched to new package hosting infrastructure diff --git a/packaging/repoconfig/netdata-repo.spec b/packaging/repoconfig/netdata-repo.spec index 6139e52bb..242178ba7 100644 --- a/packaging/repoconfig/netdata-repo.spec +++ b/packaging/repoconfig/netdata-repo.spec @@ -2,7 +2,7 @@ Name: netdata-repo Version: 2 -Release: 1 +Release: 2 Summary: Netdata stable repositories configuration. Group: System Environment/Base @@ -25,6 +25,10 @@ BuildArch: noarch Requires: yum-plugin-priorities %endif +%if 0%{?centos_ver} && 0%{!?amazon_linux:1} && 0%{!?oraclelinux:1} +Requires: epel-release +%endif + # Overlapping file installs Conflicts: netdata-repo-edge @@ -104,6 +108,8 @@ This package contains the official Netdata package repository configuration for %endif %changelog +* Mon Nov 13 2023 Austin Hemmelgarn <austin@netdata.cloud> 2-2 +- Add EPEL requirement for RHEL packages. * Wed Dec 7 2022 Austin Hemmelgarn <austin@netdata.cloud> 2-1 - Switch to new hosting at repo.netdata.cloud. * Mon Jun 6 2022 Austin Hemmelgarn <austin@netdata.cloud> 1-2 diff --git a/packaging/version b/packaging/version index 9c55840dd..0ade2bb9e 100644 --- a/packaging/version +++ b/packaging/version @@ -1 +1 @@ -v1.43.2 +v1.44.3 |