1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
<!--
title: "Monitor Nginx or Apache web server log files with Netdata"
custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/collect-apache-nginx-web-logs.md
-->
# Monitor Nginx or Apache web server log files with Netdata
Log files have been a critical resource for developers and system administrators who want to understand the health and
performance of their web servers, and Netdata is taking important steps to make them even more valuable.
By parsing web server log files with Netdata, and seeing the volume of redirects, requests, or server errors over time,
you can better understand what's happening on your infrastructure. Too many bad requests? Maybe a recent deploy missed a
few small SVG icons. Too many requests? Time to batten down the hatches—it's a DDoS.
You can use the [LTSV log format](http://ltsv.org/), track TLS and cipher usage, and the whole parser is faster than
ever. In one test on a system with SSD storage, the collector consistently parsed the logs for 200,000 requests in
200ms, using ~30% of a single core.
The [web_log](https://github.com/netdata/go.d.plugin/blob/master/modules/weblog/README.md) collector is currently compatible
with [Nginx](https://nginx.org/en/) and [Apache](https://httpd.apache.org/).
This guide will walk you through using the new Go-based web log collector to turn the logs these web servers
constantly write to into real-time insights into your infrastructure.
## Set up your web servers
As with all data sources, Netdata can auto-detect Nginx or Apache servers if you installed them using their standard
installation procedures.
Almost all web server installations will need _no_ configuration to start collecting metrics. As long as your web server
has readable access log file, you can configure the web log plugin to access and parse it.
## Custom configuration of the web log collector
The web log collector's default configuration comes with a few example jobs that should cover most Linux distributions
and their default locations for log files:
```yaml
# [ JOBS ]
jobs:
# NGINX
# debian, arch
- name: nginx
path: /var/log/nginx/access.log
# gentoo
- name: nginx
path: /var/log/nginx/localhost.access_log
# APACHE
# debian
- name: apache
path: /var/log/apache2/access.log
# gentoo
- name: apache
path: /var/log/apache2/access_log
# arch
- name: apache
path: /var/log/httpd/access_log
# debian
- name: apache_vhosts
path: /var/log/apache2/other_vhosts_access.log
# GUNICORN
- name: gunicorn
path: /var/log/gunicorn/access.log
- name: gunicorn
path: /var/log/gunicorn/gunicorn-access.log
```
However, if your log files were not auto-detected, it might be because they are in a different location. Try the default
`web_log.conf` file.
```bash
./edit-config go.d/web_log.conf
```
To create a new custom configuration, you need to set the `path` parameter to point to your web server's access log
file. You can give it a `name` as well, and set the `log_type` to `auto`.
```yaml
jobs:
- name: example
path: /path/to/file.log
log_type: auto
```
Restart Netdata with `sudo systemctl restart netdata`, or the [appropriate
method](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for your system. Netdata should pick up your web server's access log and
begin showing real-time charts!
### Custom log formats and fields
The web log collector is capable of parsing custom Nginx and Apache log formats and presenting them as charts, but we'll
leave that topic for a separate guide.
We do have [extensive
documentation](https://github.com/netdata/go.d.plugin/blob/master/modules/weblog/README.md#custom-log-format) on how
to build custom parsing for Nginx and Apache logs.
## Tweak web log collector alarms
Over time, we've created some default alarms for web log monitoring. These alarms are designed to work only when your
web server is receiving more than 120 requests per minute. Otherwise, there's simply not enough data to make conclusions
about what is "too few" or "too many."
- [web log alarms](https://raw.githubusercontent.com/netdata/netdata/master/health/health.d/web_log.conf).
You can also edit this file directly with `edit-config`:
```bash
./edit-config health.d/weblog.conf
```
For more information about editing the defaults or writing new alarm entities, see our [health monitoring
documentation](https://github.com/netdata/netdata/blob/master/health/README.md).
## What's next?
Now that you have web log collection up and running, we recommend you take a look at the collector's [documentation](https://github.com/netdata/go.d.plugin/blob/master/modules/weblog/README.md) for some ideas of how you can turn these rather "boring" logs into powerful real-time tools for keeping your servers happy.
Don't forget to give GitHub user [Wing924](https://github.com/Wing924) a big 👍 for his hard work in starting up the Go
refactoring effort.
|
