diff options
Diffstat (limited to '')
| -rw-r--r-- | .travis/README.md | 149 |
1 files changed, 149 insertions, 0 deletions
diff --git a/.travis/README.md b/.travis/README.md new file mode 100644 index 000000000..8927dd4c5 --- /dev/null +++ b/.travis/README.md @@ -0,0 +1,149 @@ +<!-- +--- +title: "Description of CI build configuration" +custom_edit_url: https://github.com/netdata/netdata/edit/master/.travis/README.md +--- +--> + +# Description of CI build configuration + +## Variables needed by travis + +- GITHUB_TOKEN - GitHub token with push access to repository +- DOCKER_USERNAME - Username (netdatabot) with write access to docker hub repository +- DOCKER_PWD - Password to docker hub +- encrypted_8daf19481253_key - key needed by openssl to decrypt GCS credentials file +- encrypted_8daf19481253_iv - IV needed by openssl to decrypt GCS credentials file +- COVERITY_SCAN_TOKEN - Token to allow coverity test analysis uploads +- SLACK_USERNAME - This is required for the slack notifications triggered by travis pipeline +- SLACK_CHANNEL - This is the channel that Travis will be posting messages +- SLACK_NOTIFY_WEBHOOK_URL - This is the incoming URL webhook as provided by slack integration. Visit Apps integration in slack to generate the required hook +- SLACK_BOT_NAME - This is the name your bot will appear with on slack + +## CI workflow details +Our CI pipeline is designed to help us identify and mitigate risks at all stages of implementation. +To accommodate this need, we used [Travis CI](http://www.travis-ci.com) as our CI/CD tool. +Our main areas of concern are: +1) Only push code that is working. That means fail fast so that we can improve before we reach the public + +2) Reduce the time to market to minimum, by streamlining the release process. + That means a lot of testing, a lot of consistency checks, a lot of validations + +3) Generated artifacts consistency. We should not allow broken software to reach the public. + When this happens, it's embarrassing and we struggle to eliminate it. + +4) We are an innovative company, so we love to automate :) + + +Having said that, here's a brief introduction to Netdata's improved CI/CD pipeline with Travis. +Our CI/CD lifecycle contains three different execution entry points: +1) A user opens a pull request to netdata/master: Travis will run a pipeline on the branch under that PR +2) A merge or commit happens on netdata/master. This will trigger travis to run, but we have two distinct cases in this scenario: + a) A user merges a pull request to netdata/master: Travis will run on master, after the merge. + b) A user runs a commit/merge with a special keyword (mentioned later). + This triggers a release for either minor, major or release candidate versions, depending the keyword +3) A scheduled job runs on master once per day: Travis will run on master at the scheduled interval + +To accommodate all three entry points our CI/CD workflow has a set of steps that run on all three entry points. +Once all these steps are successful, then our pipeline executes another subset of steps for entry points 2 and 3. +In travis terms the "steps" are "Stages" and within each stage we execute a set of activities called "jobs" in travis. + +### Always run: Stages that running on all three execution entry points + +## Code quality, linting, syntax, code style +At this early stage we iterate through a set of basic quality control checks: +- Shell checking: Run linters for our various BASH scripts +- Checksum validators: Run validators to ensure our installers and documentation are in sync +- Dashboard validator: We provide a pre-generated dashboard.js script file that we need to make sure its up to date. We validate that. + +## Build process +At this stage, basically, we build :-) +We do a baseline check of our build artifacts to guarantee they are not broken +Briefly our activities include: +- Verify docker builds successfully +- Run the standard Netdata installer, to make sure we build & run properly +- Do the same through 'make dist', as this is our stable channel for our kickstart files + +## Artifacts validation +At this point we know our software is building, we need to go through the a set of checks, to guarantee +that our product meets certain expectations. At the current stage, we are focusing on basic capabilities +like installing in different distributions, running the full lifecycle of install-run-update-install and so on. +We are still working on enriching this with more and more use cases, to get us closer to achieving full stability of our software. +Briefly we currently evaluate the following activities: +- Basic software unit testing (only run when changes happen that require it) +- Non containerized build and install on ubuntu 14.04 +- Non containerized build and install on ubuntu 18.04 +- Running the full Netdata lifecycle (install, update, uninstall) on ubuntu 18.04 +- Build and install on CentOS 7 +(More to come) + +### Nightly operations: Stages that run daily under cronjob +The nightly stages are related to the daily nightly activities, that produce our daily latest releases. +We also maintain a couple of cronjobs that run during the night to provide us with deeper insights, +like for example coverity scanning or extended kickstart checksum checks + +## Nightly operations +At this stage we run scheduled jobs and execute the nightly changelog generator, coverity scans, +labeler for our issues and extended kickstart files checksum validations. + +## Nightly release +During this stage we are building and publishing latest docker images, prepare the nightly artifacts +and deploy them (the artifacts) to our google cloud service provider. + + +### Publishing +Publishing is responsible for executing the major/minor/patch releases and is separated +in two stages: packaging preparation process and publishing. + +## Packaging for release +During packaging we are preparing the release changelog information and run the labeler. + +## Publish for release +The publishing stage is the most complex part in publishing. This is the stage were we generate and publish docker images, +prepare the release artifacts and get ready with the release draft. + +### Package Management workflows +As part of our goal to provide the best support to our customers, we have created a set of CI workflows to automatically produce +DEB and RPM for multiple distributions. These workflows are implemented under the templated stages '_DEB_TEMPLATE' and '_RPM_TEMPLATE'. +We currently plan to actively support the following Operating Systems, with a plan to further expand this list following our users needs. + +### Operating systems supported +The following distributions are supported +- Debian versions + - Buster (TBD - not released yet, check [debian releases](https://www.debian.org/releases/) for details) + - Stretch + - Jessie + - Wheezy + +- Ubuntu versions + - Disco + - Cosmic + - Bionic + - artful + +- Enterprise Linux versions (Covers Red Hat, CentOS, and Amazon Linux with version 6) + - Version 8 (TBD) + - Version 7 + - Version 6 + +- Fedora versions + - Version 31 (TBD) + - Version 30 + - Version 29 + - Version 28 + +- openSUSE versions + - 15.1 + - 15.0 + +- Gentoo distributions + - TBD + +### Architectures supported +We plan to support amd64, x86 and arm64 architectures. As of June 2019 only amd64 and x86 will become available, as we are still working on solving issues with the architecture. + +The Package deployment can be triggered manually by executing an empty commit with the following message pattern: `[Package PACKAGE_TYPE PACKAGE_ARCH] DESCRIBE_THE_REASONING_HERE`. +Travis Yaml configuration allows the user to combine package type and architecture as necessary to regenerate the current stable release (For example tag v1.15.0 as of 4th of May 2019) +Sample patterns to trigger building of packages for all amd64 supported architecture: +- '[Package amd64 RPM]': Build & publish all amd64 available RPM packages +- '[Package amd64 DEB]': Build & publish all amd64 available DEB packages |