Version: 6.x-21
This is an older version of Search Guard. Switch to Latest version
Compliance

Write History Audit Logging

To use the compliance features please install the Search Guard Compliance Edition

Search Guard can monitor write access to sensitive data in Elasticsearch, and produce an audit trail of all write activity. It uses the Audit Logging storage engine to ship the emitted audit events to one or more storage endpoints.

Search Guard tracks

  • insertion and deletion of documents
  • changes on field level

Changes on field level are written in JSON patch format:

[
  {
    "op": "remove",
    "path": "/date"
  },
  {
    "op": "replace",
    "path": "/revenue",
    "value": 67000
  },
  {
    "op": "remove",
    "path": "/customers"
  },
  {
    "op": "add",
    "path": "/remarks",
    "value": "none"
  }
]

The write history can be used to track changes to PII or otherwise sensitive data. The audit trail will contain date of access, the username, the document id, and a list of the changes on JSON patch format.

By tracking the insertion and the deletion of documens you can prove when PII data was created and also deleted. This makes it extremely easy to implement GDPR, HIPAA, PCI or SOX compliance.

Audit logging and also the compliance features are statically configured in elasticsearch.yml and cannot be changed at runtime.

Audit Log Category

Events generated by the read history audit trail are logged in the COMPLIANCE_DOC_WRITE category.

Configuring the indices to watch

To enable the write history audit trail, list the indices to watch in elasticsearch.yml:

searchguard.compliance.history.write.watched_indices:
  - indexname1
  - indexname2

For example:

searchguard.compliance.history.write.watched_indices:
  - finance

In the example above, any write or delete operation by any user to the finance index will generate an audit event. Wildcards ares supported for index names.

Excluding users

You can exclude users from the write history by listing them in elasticsearch.yml:

searchguard.compliance.history.read.ignore_users:
  - admin

Configuring the event content

By default Search Guard logs

  • the complete document content, in case it is newly created
  • a diff in JSON patch format, in case a document was changed

You can control the level of detail by the following configuration settings in elasticsearch.yml:

Name Description
searchguard.compliance.history.write.metadata_only boolean, if set to true Search Guard will not log any document content, only meta data. Enable this if you need to know when a document was created, changed or delete, but you are not interested in the actual content. Default is false.
searchguard.compliance.history.write.log_diffs boolean, if set to true Search Guard will log diffs for document updates. Default is false.

Field reference

Events in the COMPLIANCE_DOC_WRITE category have the following attributes:

Format, timestamp and category attributes

Name Description
audit_format_version Audit log message format version, current: 3
audit_utc_timestamp UTC timestamp when the event was generated
audit_category Audit log category, COMPLIANCE_DOC_WRITE for all events

Cluster and node attributes

Name Description
audit_cluster_name Name of the cluster this event was emitted on.
audit_node_id The ID of the node where the event was generated.
audit_node_name The name of the node where the event was generated.
audit_node_host_address The host address of the node where the event was generated.
audit_node_host_name The host address of the node where the event was generated.
audit_trace_shard_id id of the shard

Request attributes

Name Description
audit_request_origin The layer from which the event originated. One if TRANSPORT or REST.
audit_request_remote_address The adress where the request came from.

User attributes

Name Description
audit_request_effective_user The username of the user that has accessed watched fields

Index attributes

Name Description
audit_trace_indices Array, the index name(s) as contained in the request. Can contain wildcards, date patterns and aliases.
audit_trace_resolved_indices Array, the resolved, concrete index name(s) affected by this request. Only logged if resolve_indices is true. Optional.

Document and fields attributes

Name Description
audit_compliance_operation The operation on the document, can be one of CREATE, UPDATE or DELETE.
audit_trace_doc_id Id of the document containing the watched fields.
audit_trace_doc_version The version of the document that has been inserted, changed or deleted.
audit_request_body The diff of the old and new version of the document in JSON patch format. Only logged for UPDATE.
audit_request_body The content of newly created documents. Only logged for CREATE, and only if searchguard.compliance.history.write.diffs_only is false.

Example

{
  "audit_compliance_operation": "UPDATE",
  "audit_cluster_name": "searchguard",
  "audit_node_name": "wfAmfPn",
  "audit_category": "COMPLIANCE_DOC_WRITE",
  "audit_request_origin": "REST",
  "audit_compliance_doc_version": 8,
  "audit_node_id": "wfAmfPncS_62cJ2kPcC0GQ",
  "audit_compliance_diff_is_noop": false,
  "audit_format_version": 3,
  "audit_utc_timestamp": "2018-01-23T11:21:10.898+00:00",
  "audit_request_remote_address": "172.16.0.254",
  "audit_trace_doc_types": [
    "revenue"
  ],
  "audit_trace_doc_id": "11",
  "audit_compliance_diff_content": "[{\"op\":\"remove\",\"path\":\"/date\"},{\"op\":\"replace\",\"path\":\"/revenue\",\"value\":12345},{\"op\":\"remove\",\"path\":\"/customers\"},{\"op\":\"add\",\"path\":\"/remarks\",\"value\":\"none\"}]",
  "audit_node_host_address": "172.16.0.2",
  "audit_request_effective_user": "finance_employee",
  "audit_trace_shard_id": 4,
  "audit_trace_indices": [
    "finance"
  ],
  "audit_trace_resolved_indices": [
    "finance"
  ],
  "audit_node_host_name": "sgssl-1.example.com"
}

Performance considerations

Use diffs only on relevant indices

Logging the difference between to versions of a document has a performance impact. It should be used only on relevant data, and bulk updates should be minimized. Performance will improve by leveraging new features in the upcoming Elasticsearch 6.3 release.

Keeping the watched indices at a minumum

The more indices you watch, the more events are possibly created. Consider only watching indices that you are required to monitor.

Use a high-volume storage type

The write history can emit a lot of events in a short time. Consider using a storage type or cache that can handle a high volume of events, like Kafka, Redis or AWS Kinesis.