Setting the log level to debug
For troubleshooting any problem with Search Guard, it is recommended to set the log level to at least
Add the following lines in
config/log4j2.properties and restart your node:
logger.searchguard.name = com.floragunn logger.searchguard.level = debug
This will already print out a lot if helpful information in your log file. If this information is not sufficient, you can also set the log level to
Validate your elasticsearch.yml
The Elasticsearch configuration is in yaml format, and so is the Search Guard configuration. A quick way of checking the validity of any yml file is to use the Yaml Lint web service:
Just copy and paste the content of your yaml file there and check for any errors.
Note: You can of course use http://www.yamllint.com/ to also validate any other Search Guard configuration file.
Viewing the contents of your Key- and Truststore
In order to view information about the certificates stored in your keystore or truststore, use the
keytool command like:
keytool -list -v -keystore keystore.jks
keytool will prompt for the password of the keystore and list all entries with detailed information. For example you can use this output to check for the correctness of the SAN and EKU settings.
If you rather like to work with a GUI, we recommend KeyStore Explorer:
KeyStore Explorer is an open source GUI replacement for the Java command-line utilities keytool and jarsigner. KeyStore Explorer presents their functionality, and more, via an intuitive graphical user interface.
You can use it to examine the contents of locally stored files, but you can also retrieve and inspect certificates from a server (or Elasticsearch cluster) directly.
Viewing the contents of PEM certificates
The content of PEM certificates can either be displayed by using OpenSSL or by the diagnose function of the Search Guard TLS tool.
openssl x509 -in node1.pem -text -noout
TLS diagnose tool:
./sgtlsdiag.sh -ca root-ca.pem -crt node1.pem
The TLS diagnose tool will also check the validity of the certificate chain.
Checking the main attributes of a certificate
The main attributes of an entry in the keystore look like:
Alias name: node-0 Entry type: PrivateKeyEntry Certificate chain length: 3 Certificate: Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
|Alias name||The alias for this entry in the keystore|
|Entry type||Type of the entry, either
|Certificate chain length||Length of the certificate chain|
|Certificate[n]||Number of the certificate in the chain|
|Owner||DN of the owner of the certificate|
|Issue||DN of the issuer / signer of the certificate|
Checking the configured alias
If you have multiple entries in the keystore and you are using aliases to refer to them, make sure that the configured alias in
elasticsearch.yml matches the alias name in the keystore.
In the example above example, you’d need to set:
If there is only one entry in the keystore, you do not need to configure an alias.
Checking the type of the certificate
The relevant certificate types for Search Guard are:
- This type of entry holds a cryptographic PrivateKey, which is optionally stored in a protected format to prevent unauthorized access. It is also accompanied by a certificate chain for the corresponding public key.
- This type of entry contains a single public key Certificate belonging to another party. It is called a trusted certificate because the keystore owner trusts that the public key in the certificate indeed belongs to the identity identified by the subject (owner) of the certificate.
For Client-, Admin- and Node certificates the type is
PrivateKeyEntry. Root and intermediate certificates have type
Checking the DN of the certificate
If you are unsure what the DN of your certificate looks like, check the
Owner field of keytool output. In the example above the DN is:
Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
The corresponding configuration in elasticsearch.yml is:
searchguard.nodes_dn: - 'CN=node-0.example.com,OU=SSL,O=Test,L=Test,C=DE'
Checking for special characters in DNs
Search Guard uses the String Representation of Distinguished Names (RFC1779) when validating node certificates.
If parts of your DN contain special characters, for example a comma, make sure it is escaped properly in your configuration, for example:
searchguard.nodes_dn: - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'
Omit whitespaces between the individual parts of the DN. Instead of:
searchguard.nodes_dn: - 'CN=node-0.example.com, OU=SSL,O=My\, Test, L=Test, C=DE'
searchguard.nodes_dn: - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'
Checking the IP Addresses of the certificate
Sometimes the IP address contained in your certificate is not the one communicating with the cluster. This can happen if your node has multiple interfaces, or is running dual stack (IPv6 + IPv4).
When this happens, you would see the following in the node’s elasticsearch log:
SSL Problem Received fatal alert: certificate_unknown javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
And the following message in your cluster’s master log when the new node tries to join the cluster:
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 10.0.0.42 found
Check the IP address in the certificate:
In this example, the node tries to join the cluster with the IPv4
10.0.0.42, but it’s the IPv6 address
2001:db8:0:1:22.214.171.124 that is contained in the certificate.
Validating the certificate chain
TLS certificates are organized in a certificate chain:
A certificate chain is an ordered list of certificates, containing an SSL Certificate and Certificate Authority (CA) Certificates, that enable the receiver to verify that the sender and all CA’s are trustworthy. The chain or path begins with the SSL certificate, and each certificate in the chain is signed by the entity identified by the next certificate in the chain.
You can check with keytool that the certificate chain is correct by inspecting the owner and the issuer of each certificate. If you used the demo installation script that ships with Search Guard, the chain looks like:
Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
Intermediate / Signing certificate
Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com
Owner: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com
From the entries you can see that the node certificate was signed by the intermediate certificate. The intermediate certificate was signed by the root certificate. The root certificate was signed by itself, hence the name root or self signed certificate. If you’re using separate key- and truststore files, your root CA can most likely be found in the truststore.
As a rule of thumb:
- The keystore contains the client or node certificate with its private key, and all intermediate certificates
- The truststore contains the root certificate
Checking the SAN hostnames and IP addresses
The valid hostnames and IP addresses of a TLS certificates are stored as
SAN entries. Check that the hostname and IP entries in the
SAN section are correct, especially when you use hostname verification:
Certificate: Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE ... Extensions: ... #5: ObjectId: 126.96.36.199 Criticality=false SubjectAlternativeName [ DNSName: node-0.example.com DNSName: localhost IPAddress: 127.0.0.1 ... ]
Node certificates: Checking the OID
If you are using OIDs to denote valid node certificates, check that the
SAN extension for your node certificate contains the correct
Certificate: Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE ... Extensions: ... #5: ObjectId: 188.8.131.52 Criticality=false SubjectAlternativeName [ ... OIDName: 184.108.40.206.5.5 ]
Node certificates: Checking the EKU field
Node certificates need to have both
clientAuth set in the extended key usage field:
#3: ObjectId: 220.127.116.11 Criticality=false ExtendedKeyUsages [ serverAuth clientAuth ]
Search Guard disables
TLSv1 by default, because it is outdated, unsecure and vulnerable. If you need to use
TLSv1 and you know what you are doing, you can re-enable it in
searchguard.ssl.http.enabled_protocols: - "TLSv1" - "TLSv1.1" - "TLSv1.2"
TLS relies on the server and client negotiating a common cipher suite. Depending on your system, the available ciphers will vary. They depend on the JDK or OpenSSL version you’re using, and whether or not the
JCE Unlimited Strength Jurisdiction Policy Files are installed.
For legal reasons, the JDK does not include strong ciphers like AES256. In order to use strong ciphers you need to download and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. If you don’t have them installed, you will see an info message on startup like:
On startup, Search Guard will print out the available ciphers for the REST- and transport layer:
[INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit. That is not an issue, it just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files'
Search Guard will still work and fall back to weaker cipher suites.
Search Guard will also print out all available cipher suites on startup:
[INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider: JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, ...]
curl is not always curl when it comes to SSL/TLS. Sometimes it depends against which SSL/TLS implementation your curl version was compiled against. And there are a few including OpenSSL, GnuTLS and NSS. We made the experience that NSS and GnuTLS are very often problematic and so we assume (in our docs and scripts) your curl version is compiled against OpenSSL. You can check your curl version with
This will print out something like:
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 OpenSSL/1.0.2g zlib/1.2.11 nghttp2/1.24.0 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy
To make curl work best with Search Guard we recommend you use a curl binary with
- Version 7.50 or later
- Compiled against OpenSSL 1.0.2 or later
- Protocol needs to include
- Features should include
GSS-API Kerberos SPNEGO NTLMif you like to use Kerberos
If curl is compiled against NSS you may need to use explicitly a
./ for a relative or a
/ for an absolute paths like
curl --insecure --cert ./chain.pem --key ./kirk.key.pem "<API Endpoint>"
We already had a lot of issues, here are the most relevant ones:
- Is it possible to configure elasticsearch/search guard to not make curl try to look for NSS certs?
- curl -XGET request on elasticsearch integrated with search-guard throws SSLException
- Curl with admin ssl certificate not working
- Unable to Use REST API
- enabling kerberos
Encrypted PKCS#8 key does not work
When you experience a
File does not contain valid private key or
data isn't an object ID exception while using encrypted PKCS#8 keys (pem keys with a password)
you might have hit a JDK bug where “SunJCE support of password-based encryption scheme 2 params (PBES2) is not working”. There is also an related issue in the elastic github repo. The solution is to use the v1 encryption scheme. With openssl use the
-v1 flag in your command like
openssl pkcs8 -in key.pem -topk8 -out enckey.pem -v1 PBE-SHA1-3DES. Please also refer to Search Guard issue #524 and OpenSSL pkcs8 documentation for more details.