Version: 6.x-23
Community

Using fluentd with Search Guard

Fluentd connects to Elasticsearch on the REST layer, just like a browser or curl. To use fluentd with a Search Guard secured cluster:

  • set up a fluentd user with permissions to read and write to the fluentd index
  • configure fluentd to use HTTPS instead of HTTP (optional, only applicable if you enabled HTTPS on the REST layer)
  • configure fluentd to provide HTTP Basic Authentication credentials when connecting to Elasticsearch / Search Guard

Setting up the fluentd user and role

For fluentd being able to write to Elasticsearch, set up a role first that has full access to the fluentd index. Let’s assume you use a daily rolling index in fluentd like:

index_name fluentd-%Y%m%d

You then would set up a Search Guard role that has access to all indices starting with fluentd-.

sg_roles.yml:

sg_fluentd:
  cluster:
    - CLUSTER_MONITOR  
    - CLUSTER_COMPOSITE_OPS_RO
  indices:
    'fluentd-*':
      '*':
        - UNLIMITED

If you use the Search Guard internal user database, set up a fluentd user.

sg_internal_users.yml:

fluentd:
  hash: $2y$12$pcoEhYWjbiMqQldLgK/dnezy9DXzi/wahiADmiYVPvNmzoGWiKoVi

Last, map the fluentd user to the sg_fluentd Search Guard role:

sg_roles_mapping.yml:

sg_fluentd:
  users:
    - fluentd

Configuring the Elasticsearch output

In your td-agent.conf make sure you provide the username and password of the fluentd user you have configured above.

If you configured Search Guard to use HTTPS instead of HTTP, make sure you set the scheme to https.

If you use self-signed certificates, set ssl_verify to none.

<match apache.access>
 @type             elasticsearch
 host              sgssl-0.example.com
 scheme            https
 ssl_verify        false
 user              fluentd
 password          fluentd
 port              9200
 index_name        fluentd-%Y%m%d
 type_name         _doc 
 include_timestamp true
 utc_index         true
 flush_interval 1s
 buffer_chunk_limit 1M
 buffer_queue_limit 512
 <buffer>
   flush_interval 1s
   buffer_chunk_limit 1M
   buffer_queue_limit 512
 </buffer>
</match>

Not what you were looking for? Try the search.