Using Search Guard with ElastAlert for Elasticsearch
Search Guard provides free Alerting for Elasticsearch, perfectly integrated with all our security features. Signals Alerting comes bundled with Search Guard as a single install. You do not need to maintain a separate service for Alerting, as with ElastAlert.
What is ElastAlert?
ElastAlert, under active development by Yelp
is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. ElastAlert works with all versions of Elasticsearch. […] It works by combining Elasticsearch with two types of components, rule types and alerts. Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match.
In the following description, we assume that you have already installed Elasticsearch and Search Guard.
How do you set up ElastAlert?
ElastAlert is a Python application and is platform independent. For an in-depth description on installing and configuring ElastAlert, please follow the official documentation. In this document, we mainly show how to configure ElastAlert for Search Guard.
- ISO8601 or Unix timestamped data
- Python 2.7
- pip, see requirements.txt
- Packages on Ubuntu 14.x: python-pip python-dev libffi-dev libssl-dev
To install ElastAlert, simply run
$ pip install elastalert
Depending on the version of Elasticsearch, you may need to manually install the correct version of elasticsearch-py.
$ pip install "elasticsearch>=5.0.0"
Is Elasticsearch alerting free?
ElastAlert is free and licensed under the Apache2 license. Since ElastAlert is a Python application that runs outside Elasticsearch, you need to maintain and monitor ElastAlert seperately. Search Guard provides free Alerting for Elasticsearch as well which runs as an Elasticsearch plugin and is automatically installed when you install Search Guard.
Configuring ElastAlert for Search Guard
All configuration options for ElastAlert reside in
config.yaml. You can find a sample configuration file in the official ElastAlert repository for a quick start.
The Search Guard relevant configuration settings are as follows:
es_host: elastic.example.com es_port: 9200 use_ssl: True verify_certs: True ca_certs: '/path/to/chain.pem' es_username: elastalert es_password: elastalert writeback_index: elastalert_status
|es_host||String, Hostname of your Elasticsearch node|
|es_port||String, HTTP(S) port of your Elasticsearch node|
|use_ssl||Boolean, if set to true, ElastAlert will connect via HTTPS, if set to false HTTP is used|
|verify_certs||Boolean, whether to verify the certificate used by Elasticsearch/Search Guard or not. If you use self-signed certificates, you need to either set this to false, or configure the Root CA via the
|ca_certs||String, path to the Root certificate (and intermediate certificates if any) in PEM format. If verify_certs is set to true, and you use self-sgned certificates, this entry is mandatory.|
|es_username||String, the username that ElastAlert uses when connecting to Elasticsearch|
|es_password||String, the password that ElastAlert uses when connecting to Elasticsearch|
|writeback_index||String, the name of the index in which ElastAlert will store data|
If you’re working with self-signed certificates, we recommend to provide ElastAlert with the certificate chain containing the root certificate and all intermediate certificates. You can extract this certificate chain from a
truststore.jks by using the
keytool -list -keystore truststore.jks -rfc > chain.pem
Configuring the ElastAlert user
ElastAlert connects to Elasticsearch on the REST layer, and provides the configured
HTTP Basic Authentication header. If this user does not exist in your Search Guard setup, add it.
The required permissions of this user depend on the indices you want to run your alerts on. As a rule of thumb:
- The ElastAlert user needs full permissions on the configured
writeback_indexindex. This is where ElastAlert will store its data.
- The ElastAlert user needs at least READ permissions on the indices against which the rules and alerts should be executed
For example, if you collect audit events with the Search Guard audit log module in an index called
auditlog, and your
writeback_index is called
elastalert_status, the corresponding role definition looks like:
sg_elastalert: cluster_permissions: - SGS_CLUSTER_COMPOSITE_OPS - "indices:data/write/bulk" index_permissions: - index_patterns: - 'auditlog' allowed_actions: - SGS_READ - index_patterns: - 'elastalert_status' allowed_actions: - SGS_UNLIMITED
Create the ElastAlert index
Next we set up the
writeback_index index and the corresponding mapping. ElastAlert can do that automatically for you:
Testing the connection settings
Now we can test if the connection to Elasticsearch works by simply starting ElastAlert:
If you configured everything correctly, ElastAlert should start up without errors.
Rules and Multi-Cluster monitoring
Rules are what actually powers ElastAlert:
Each rule defines a query to perform, parameters on what triggers a match, and a list of alerts to fire for each match.
Source: ElastAlert Documentation
Each rule can be executed against a different cluster if necessary, so it’s possible to monitor several clusters at once, and collect the results in one central cluster.
Each rule is defined in it’s own file, and all rules files are placed in the rules folder configured in
Each rule can overwrite any setting in
config.yaml, including connection settings like
Changes in the rules folder are detected automatically, so you can add, remove and change any rule in real time.
You can read more about how to set up rules and alerts in the ElastAlert documentation.
Each rule can have one or more alerts attached to it. They define where the actual alerts are shipped to. ElastAlert supports a wide range of endpoints, for example:
- MS Teams
- RabbitMQ / ActiveMQ
And of course you can execute an arbitrary command, or provide your own implementation.
The following sample rule will query for FAILED_LOGIN events in the Search Guard audit log cluster, and will output a message when more than 5 attempts within the last minute are detected.
name: Monitor Login Attempts type: frequency index: auditlog num_events: 5 timeframe: minutes: 1 filter: - query: query_string: query: "audit_category: FAILED_LOGIN" timestamp_field: "audit_utc_timestamp" alert: - "debug"
You can test your rules without actually firing an alert by executing: