Using fluentd with Search Guard
Fluentd connects to Elasticsearch on the REST layer, just like a browser or curl. To use fluentd with a Search Guard secured cluster:
- set up a fluentd user with permissions to read and write to the fluentd index
- configure fluentd to use HTTPS instead of HTTP (optional, only applicable if you enabled HTTPS on the REST layer)
- configure fluentd to provide HTTP Basic Authentication credentials when connecting to Elasticsearch / Search Guard
Setting up the fluentd user and role
For fluentd being able to write to Elasticsearch, set up a role first that has full access to the fluentd index. Let’s assume you use a daily rolling index in fluentd like:
index_name fluentd-%Y%m%d
You then would set up a Search Guard role that has access to all indices starting with fluentd-.
sg_roles.yml:
sg_fluentd:
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS_RO
indices:
'fluentd-*':
'*':
- UNLIMITED
If you use the Search Guard internal user database, set up a fluentd user.
sg_internal_users.yml:
fluentd:
hash: $2y$12$pcoEhYWjbiMqQldLgK/dnezy9DXzi/wahiADmiYVPvNmzoGWiKoVi
Last, map the fluentd user to the sg_fluentd Search Guard role:
sg_roles_mapping.yml:
sg_fluentd:
users:
- fluentd
Configuring the Elasticsearch output
In your td-agent.conf make sure you provide the username and password of the fluentd user you have configured above.
If you configured Search Guard to use HTTPS instead of HTTP, make sure you set the scheme to https.
If you use self-signed certificates, set ssl_verify to none.
<match apache.access>
@type elasticsearch
host sgssl-0.example.com
scheme https
ssl_verify false
user fluentd
password fluentd
port 9200
index_name fluentd-%Y%m%d
type_name _doc
include_timestamp true
utc_index true
flush_interval 1s
buffer_chunk_limit 1M
buffer_queue_limit 512
<buffer>
flush_interval 1s
buffer_chunk_limit 1M
buffer_queue_limit 512
</buffer>
</match>