1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
|
<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/exporting/pubsub/README.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/exporting/pubsub/metadata.yaml"
sidebar_label: "Google Cloud Pub Sub"
learn_status: "Published"
learn_rel_path: "Exporting"
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE EXPORTER'S metadata.yaml FILE"
endmeta-->
# Google Cloud Pub Sub
<img src="https://netdata.cloud/img/pubsub.png" width="150"/>
Export metrics to Google Cloud Pub/Sub Service
<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
## Setup
### Prerequisites
####
- First [install](https://github.com/googleapis/google-cloud-cpp/) install Google Cloud Platform C++ Client Libraries
- Pub/Sub support is also dependent on the dependencies of those libraries, like `protobuf`, `protoc`, and `grpc`
- Next, Netdata should be re-installed from the source. The installer will detect that the required libraries are now available.
### Configuration
#### File
The configuration file name for this integration is `exporting.conf`.
You can edit the configuration file using the `edit-config` script from the
Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory).
```bash
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config exporting.conf
```
#### Options
The following options can be defined for this exporter.
<details><summary>Config options</summary>
| Name | Description | Default | Required |
|:----|:-----------|:-------|:--------:|
| enabled | Enables or disables an exporting connector instance (yes/no). | no | True |
| destination | Accepts a space separated list of hostnames, IPs (IPv4 and IPv6) and ports to connect to. Netdata will use the first available to send the metrics. | pubsub.googleapis.com | True |
| username | Username for HTTP authentication | my_username | False |
| password | Password for HTTP authentication | my_password | False |
| data source | Selects the kind of data that will be sent to the external database. (as collected/average/sum) | | False |
| hostname | The hostname to be used for sending data to the external database server. | [global].hostname | False |
| prefix | The prefix to add to all metrics. | Netdata | False |
| update every | Frequency of sending sending data to the external database, in seconds. | 10 | False |
| buffer on failures | The number of iterations (`update every` seconds) to buffer data, when the external database server is not available. | 10 | False |
| timeout ms | The timeout in milliseconds to wait for the external database server to process the data. | 2 * update_every * 1000 | False |
| send hosts matching | Hosts filter. Determines which hosts will be sent to the external database. The syntax is [simple patterns](https://github.com/netdata/netdata/tree/master/libnetdata/simple_pattern#simple-patterns). | localhost * | False |
| send charts matching | One or more space separated patterns (use * as wildcard) checked against both chart id and chart name. | * | False |
| send names instead of ids | Controls the metric names Netdata should send to the external database (yes/no). | | False |
| send configured labels | Controls if host labels defined in the `[host labels]` section in `netdata.conf` should be sent to the external database (yes/no). | | False |
| send automatic labels | Controls if automatically created labels, like `_os_name` or `_architecture` should be sent to the external database (yes/no). | | False |
##### destination
The format of each item in this list, is: [PROTOCOL:]IP[:PORT].
- PROTOCOL can be udp or tcp. tcp is the default and only supported by the current exporting engine.
- IP can be XX.XX.XX.XX (IPv4), or [XX:XX...XX:XX] (IPv6). For IPv6 you can to enclose the IP in [] to separate it from the port.
- PORT can be a number of a service name. If omitted, the default port for the exporting connector will be used.
Example IPv4:
```yaml
destination = pubsub.googleapis.com
```
When multiple servers are defined, Netdata will try the next one when the previous one fails.
##### update every
Netdata will add some randomness to this number, to prevent stressing the external server when many Netdata servers
send data to the same database. This randomness does not affect the quality of the data, only the time they are sent.
##### buffer on failures
If the server fails to receive the data after that many failures, data loss on the connector instance is expected (Netdata will also log it).
##### send hosts matching
Includes one or more space separated patterns, using * as wildcard (any number of times within each pattern).
The patterns are checked against the hostname (the localhost is always checked as localhost), allowing us to
filter which hosts will be sent to the external database when this Netdata is a central Netdata aggregating multiple hosts.
A pattern starting with `!` gives a negative match. So to match all hosts named `*db*` except hosts containing `*child*`,
use `!*child* *db*` (so, the order is important: the first pattern matching the hostname will be used - positive or negative).
##### send charts matching
A pattern starting with ! gives a negative match. So to match all charts named apps.* except charts ending in *reads,
use !*reads apps.* (so, the order is important: the first pattern matching the chart id or the chart name will be used,
positive or negative). There is also a URL parameter filter that can be used while querying allmetrics. The URL parameter
has a higher priority than the configuration option.
##### send names instead of ids
Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and names
are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several cases they are
different : disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
</details>
#### Examples
##### Basic configuration
- Set the destination option to a Pub/Sub service endpoint. pubsub.googleapis.com is the default one.
- Create the credentials JSON file by following Google Cloud's authentication guide.
- The user running the Agent (typically netdata) needs read access to google_cloud_credentials.json, which you can set
`chmod 400 google_cloud_credentials.json; chown netdata google_cloud_credentials.json`
- Set the credentials file option to the full path of the file.
```yaml
[pubsub:my_instance]
enabled = yes
destination = pubsub.googleapis.com
credentials file = /etc/netdata/google_cloud_credentials.json
project id = my_project
topic id = my_topic
```
|
