diff options
Diffstat (limited to '')
| -rw-r--r-- | src/go/collectors/go.d.plugin/modules/weblog/integrations/web_server_log_files.md | 375 |
1 files changed, 375 insertions, 0 deletions
diff --git a/src/go/collectors/go.d.plugin/modules/weblog/integrations/web_server_log_files.md b/src/go/collectors/go.d.plugin/modules/weblog/integrations/web_server_log_files.md new file mode 100644 index 000000000..0feb6a706 --- /dev/null +++ b/src/go/collectors/go.d.plugin/modules/weblog/integrations/web_server_log_files.md @@ -0,0 +1,375 @@ +<!--startmeta +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/go/collectors/go.d.plugin/modules/weblog/README.md" +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/go/collectors/go.d.plugin/modules/weblog/metadata.yaml" +sidebar_label: "Web server log files" +learn_status: "Published" +learn_rel_path: "Collecting Metrics/Web Servers and Web Proxies" +most_popular: False +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" +endmeta--> + +# Web server log files + + +<img src="https://netdata.cloud/img/webservers.svg" width="150"/> + + +Plugin: go.d.plugin +Module: web_log + +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> + +## Overview + +This collector monitors web servers by parsing their log files. + + + + +This collector is supported on all platforms. + +This collector supports collecting metrics from multiple instances of this integration, including remote instances. + + +### Default Behavior + +#### Auto-Detection + +It automatically detects log files of web servers running on localhost. + + +#### Limits + +The default configuration for this integration does not impose any limits on data collection. + +#### Performance Impact + +The default configuration for this integration is not expected to impose a significant performance impact on the system. + + +## Metrics + +Metrics grouped by *scope*. + +The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. + + + +### Per Web server log files instance + +These metrics refer to the entire monitored application. + +This scope has no labels. + +Metrics: + +| Metric | Dimensions | Unit | +|:------|:----------|:----| +| web_log.requests | requests | requests/s | +| web_log.excluded_requests | unmatched | requests/s | +| web_log.type_requests | success, bad, redirect, error | requests/s | +| web_log.status_code_class_responses | 1xx, 2xx, 3xx, 4xx, 5xx | responses/s | +| web_log.status_code_class_1xx_responses | a dimension per 1xx code | responses/s | +| web_log.status_code_class_2xx_responses | a dimension per 2xx code | responses/s | +| web_log.status_code_class_3xx_responses | a dimension per 3xx code | responses/s | +| web_log.status_code_class_4xx_responses | a dimension per 4xx code | responses/s | +| web_log.status_code_class_5xx_responses | a dimension per 5xx code | responses/s | +| web_log.bandwidth | received, sent | kilobits/s | +| web_log.request_processing_time | min, max, avg | milliseconds | +| web_log.requests_processing_time_histogram | a dimension per bucket | requests/s | +| web_log.upstream_response_time | min, max, avg | milliseconds | +| web_log.upstream_responses_time_histogram | a dimension per bucket | requests/s | +| web_log.current_poll_uniq_clients | ipv4, ipv6 | clients | +| web_log.vhost_requests | a dimension per vhost | requests/s | +| web_log.port_requests | a dimension per port | requests/s | +| web_log.scheme_requests | http, https | requests/s | +| web_log.http_method_requests | a dimension per HTTP method | requests/s | +| web_log.http_version_requests | a dimension per HTTP version | requests/s | +| web_log.ip_proto_requests | ipv4, ipv6 | requests/s | +| web_log.ssl_proto_requests | a dimension per SSL protocol | requests/s | +| web_log.ssl_cipher_suite_requests | a dimension per SSL cipher suite | requests/s | +| web_log.url_pattern_requests | a dimension per URL pattern | requests/s | +| web_log.custom_field_pattern_requests | a dimension per custom field pattern | requests/s | + +### Per custom time field + +TBD + +This scope has no labels. + +Metrics: + +| Metric | Dimensions | Unit | +|:------|:----------|:----| +| web_log.custom_time_field_summary | min, max, avg | milliseconds | +| web_log.custom_time_field_histogram | a dimension per bucket | observations | + +### Per custom numeric field + +TBD + +This scope has no labels. + +Metrics: + +| Metric | Dimensions | Unit | +|:------|:----------|:----| +| web_log.custom_numeric_field_{{field_name}}_summary | min, max, avg | {{units}} | + +### Per URL pattern + +TBD + +This scope has no labels. + +Metrics: + +| Metric | Dimensions | Unit | +|:------|:----------|:----| +| web_log.url_pattern_status_code_responses | a dimension per pattern | responses/s | +| web_log.url_pattern_http_method_requests | a dimension per HTTP method | requests/s | +| web_log.url_pattern_bandwidth | received, sent | kilobits/s | +| web_log.url_pattern_request_processing_time | min, max, avg | milliseconds | + + + +## Alerts + + +The following alerts are available: + +| Alert name | On metric | Description | +|:------------|:----------|:------------| +| [ web_log_1m_unmatched ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.excluded_requests | percentage of unparsed log lines over the last minute | +| [ web_log_1m_requests ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.type_requests | ratio of successful HTTP requests over the last minute (1xx, 2xx, 304, 401) | +| [ web_log_1m_redirects ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.type_requests | ratio of redirection HTTP requests over the last minute (3xx except 304) | +| [ web_log_1m_bad_requests ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.type_requests | ratio of client error HTTP requests over the last minute (4xx except 401) | +| [ web_log_1m_internal_errors ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.type_requests | ratio of server error HTTP requests over the last minute (5xx) | +| [ web_log_web_slow ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.request_processing_time | average HTTP response time over the last 1 minute | +| [ web_log_5m_requests_ratio ](https://github.com/netdata/netdata/blob/master/src/health/health.d/web_log.conf) | web_log.type_requests | ratio of successful HTTP requests over over the last 5 minutes, compared with the previous 5 minutes | + + +## Setup + +### Prerequisites + +No action required. + +### Configuration + +#### File + +The configuration file name for this integration is `go.d/web_log.conf`. + + +You can edit the configuration file using the `edit-config` script from the +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration.md#the-netdata-config-directory). + +```bash +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata +sudo ./edit-config go.d/web_log.conf +``` +#### Options + +Weblog is aware of how to parse and interpret the following fields (**known fields**): + +> [nginx](https://nginx.org/en/docs/varindex.html) +> +> [apache](https://httpd.apache.org/docs/current/mod/mod_log_config.html) + +| nginx | apache | description | +|-------------------------|----------|------------------------------------------------------------------------------------------| +| $host ($http_host) | %v | Name of the server which accepted a request. | +| $server_port | %p | Port of the server which accepted a request. | +| $scheme | - | Request scheme. "http" or "https". | +| $remote_addr | %a (%h) | Client address. | +| $request | %r | Full original request line. The line is "$request_method $request_uri $server_protocol". | +| $request_method | %m | Request method. Usually "GET" or "POST". | +| $request_uri | %U | Full original request URI. | +| $server_protocol | %H | Request protocol. Usually "HTTP/1.0", "HTTP/1.1", or "HTTP/2.0". | +| $status | %s (%>s) | Response status code. | +| $request_length | %I | Bytes received from a client, including request and headers. | +| $bytes_sent | %O | Bytes sent to a client, including request and headers. | +| $body_bytes_sent | %B (%b) | Bytes sent to a client, not counting the response header. | +| $request_time | %D | Request processing time. | +| $upstream_response_time | - | Time spent on receiving the response from the upstream server. | +| $ssl_protocol | - | Protocol of an established SSL connection. | +| $ssl_cipher | - | String of ciphers used for an established SSL connection. | + +Notes: + +- Apache `%h` logs the IP address if [HostnameLookups](https://httpd.apache.org/docs/2.4/mod/core.html#hostnamelookups) is Off. The web log collector counts hostnames as IPv4 addresses. We recommend either to disable HostnameLookups or use `%a` instead of `%h`. +- Since httpd 2.0, unlike 1.3, the `%b` and `%B` format strings do not represent the number of bytes sent to the client, but simply the size in bytes of the HTTP response. It will differ, for instance, if the connection is aborted, or if SSL is used. The `%O` format provided by [`mod_logio`](https://httpd.apache.org/docs/2.4/mod/mod_logio.html) will log the actual number of bytes sent over the network. +- To get `%I` and `%O` working you need to enable `mod_logio` on Apache. +- NGINX logs URI with query parameters, Apache doesnt. +- `$request` is parsed into `$request_method`, `$request_uri` and `$server_protocol`. If you have `$request` in your log format, there is no sense to have others. +- Don't use both `$bytes_sent` and `$body_bytes_sent` (`%O` and `%B` or `%b`). The module does not distinguish between these parameters. + + +<details><summary>Config options</summary> + +| Name | Description | Default | Required | +|:----|:-----------|:-------|:--------:| +| update_every | Data collection frequency. | 1 | no | +| autodetection_retry | Recheck interval in seconds. Zero means no recheck will be scheduled. | 0 | no | +| path | Path to the web server log file. | | yes | +| exclude_path | Path to exclude. | *.gz | no | +| url_patterns | List of URL patterns. | [] | no | +| url_patterns.name | Used as a dimension name. | | yes | +| url_patterns.pattern | Used to match against full original request URI. Pattern syntax in [matcher](https://github.com/netdata/netdata/tree/master/src/go/collectors/go.d.plugin/pkg/matcher#supported-format). | | yes | +| parser | Log parser configuration. | | no | +| parser.log_type | Log parser type. | auto | no | +| parser.csv_config | CSV log parser config. | | no | +| parser.csv_config.delimiter | CSV field delimiter. | , | no | +| parser.csv_config.format | CSV log format. | | no | +| parser.ltsv_config | LTSV log parser config. | | no | +| parser.ltsv_config.field_delimiter | LTSV field delimiter. | \t | no | +| parser.ltsv_config.value_delimiter | LTSV value delimiter. | : | no | +| parser.ltsv_config.mapping | LTSV fields mapping to **known fields**. | | yes | +| parser.json_config | JSON log parser config. | | no | +| parser.json_config.mapping | JSON fields mapping to **known fields**. | | yes | +| parser.regexp_config | RegExp log parser config. | | no | +| parser.regexp_config.pattern | RegExp pattern with named groups. | | yes | + +##### url_patterns + +"URL pattern" scope metrics will be collected for each URL pattern. + +Option syntax: + +```yaml +url_patterns: + - name: name1 + pattern: pattern1 + - name: name2 + pattern: pattern2 +``` + + +##### parser.log_type + +Weblog supports 5 different log parsers: + +| Parser type | Description | +|-------------|-------------------------------------------| +| auto | Use CSV and auto-detect format | +| csv | A comma-separated values | +| json | [JSON](https://www.json.org/json-en.html) | +| ltsv | [LTSV](http://ltsv.org/) | +| regexp | Regular expression with named groups | + +Syntax: + +```yaml +parser: + log_type: auto +``` + +If `log_type` parameter set to `auto` (which is default), weblog will try to auto-detect appropriate log parser and log format using the last line of the log file. + +- checks if format is `CSV` (using regexp). +- checks if format is `JSON` (using regexp). +- assumes format is `CSV` and tries to find appropriate `CSV` log format using predefined list of formats. It tries to parse the line using each of them in the following order (the first one matches is used later): + + ```sh + $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time + $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time + $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time + $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time + $host:$server_port $remote_addr - - [$time_local] "$request" $status $body_bytes_sent + $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time $upstream_response_time + $remote_addr - - [$time_local] "$request" $status $body_bytes_sent - - $request_length $request_time + $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time $upstream_response_time + $remote_addr - - [$time_local] "$request" $status $body_bytes_sent $request_length $request_time + $remote_addr - - [$time_local] "$request" $status $body_bytes_sent + ``` + + If you're using the default Apache/NGINX log format, auto-detect will work for you. If it doesn't work you need to set the format manually. + + +##### parser.csv_config.format + + + +##### parser.ltsv_config.mapping + +The mapping is a dictionary where the key is a field, as in logs, and the value is the corresponding **known field**. + +> **Note**: don't use `$` and `%` prefixes for mapped field names. + +```yaml +parser: + log_type: ltsv + ltsv_config: + mapping: + label1: field1 + label2: field2 +``` + + +##### parser.json_config.mapping + +The mapping is a dictionary where the key is a field, as in logs, and the value is the corresponding **known field**. + +> **Note**: don't use `$` and `%` prefixes for mapped field names. + +```yaml +parser: + log_type: json + json_config: + mapping: + label1: field1 + label2: field2 +``` + + +##### parser.regexp_config.pattern + +Use pattern with subexpressions names. These names should be **known fields**. + +> **Note**: don't use `$` and `%` prefixes for mapped field names. + +Syntax: + +```yaml +parser: + log_type: regexp + regexp_config: + pattern: PATTERN +``` + + +</details> + +#### Examples +There are no configuration examples. + + + +## Troubleshooting + +### Debug Mode + +To troubleshoot issues with the `web_log` collector, run the `go.d.plugin` with the debug option enabled. The output +should give you clues as to why the collector isn't working. + +- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on + your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. + + ```bash + cd /usr/libexec/netdata/plugins.d/ + ``` + +- Switch to the `netdata` user. + + ```bash + sudo -u netdata -s + ``` + +- Run the `go.d.plugin` to debug the collector: + + ```bash + ./go.d.plugin -d -m web_log + ``` + + |