Write History Audit Logging
Content
Search Guard can monitor write access to sensitive data in Elasticsearch, and produce an audit trail of all write activity. It uses the Audit Logging storage engine to ship the emitted audit events to one or more storage endpoints.
Search Guard tracks
- insertion and deletion of documents
- changes on field level
Changes on field level are written in JSON patch format:
[
{
"op": "remove",
"path": "/date"
},
{
"op": "replace",
"path": "/revenue",
"value": 67000
},
{
"op": "remove",
"path": "/customers"
},
{
"op": "add",
"path": "/remarks",
"value": "none"
}
]
The write history can be used to track changes to PII or otherwise sensitive data. The audit trail will contain date of access, the username, the document id, and a list of the changes on JSON patch format.
By tracking the insertion and the deletion of documents you can prove when PII data was created and also deleted. This makes it extremely easy to implement GDPR, HIPAA, PCI or SOX compliance.
Audit logging and also the compliance features are statically configured in elasticsearch.yml
and cannot be changed at runtime.
Audit Log Category
Events generated by the write history audit trail are logged in the COMPLIANCE_DOC_WRITE
category.
Configuring the indices to watch
To enable the write history audit trail, list the indices to watch in elasticsearch.yml:
searchguard.compliance.history.write.watched_indices:
- indexname1
- indexname2
For example:
searchguard.compliance.history.write.watched_indices:
- finance
In the example above, any write or delete operation by any user to the finance
index will generate an audit event. Wildcards ares supported for index names.
Excluding users
You can exclude users from the write history by listing them in elasticsearch.yml
:
searchguard.compliance.history.write.ignore_users:
- admin
Configuring the event content
By default Search Guard logs
- the complete document content, in case it is newly created
- a diff in JSON patch format, in case a document was changed
You can control the level of detail by the following configuration settings in elasticsearch.yml:
Name | Description |
---|---|
searchguard.compliance.history.write.metadata_only | boolean, if set to true Search Guard will not log any document content, only meta data. Enable this if you need to know when a document was created, changed or delete, but you are not interested in the actual content. Default is false. |
searchguard.compliance.history.write.log_diffs | boolean, if set to true Search Guard will log diffs for document updates. Default is false. |
Field reference
Events in the COMPLIANCE_DOC_WRITE
category have the following attributes:
Format, timestamp and category attributes
Name | Description |
---|---|
audit_format_version | Audit log message format version, current: 4 |
@timestamp | UTC timestamp when the event was generated |
audit_category | Audit log category, COMPLIANCE_DOC_WRITE for all events |
Cluster and node attributes
Name | Description |
---|---|
audit_cluster_name | Name of the cluster this event was emitted on. |
audit_node_id | The ID of the node where the event was generated. |
audit_node_name | The name of the node where the event was generated. |
audit_node_elasticsearch_version | The Elasticsearch version of the node where the event was generated. |
audit_node_host_address | The host address of the node where the event was generated. |
audit_node_host_name | The host address of the node where the event was generated. |
audit_trace_shard_id | id of the shard |
Request attributes
Name | Description |
---|---|
audit_request_origin | The layer from which the event originated. One of TRANSPORT or REST . |
audit_request_remote_address | The address where the request came from. |
User attributes
Name | Description |
---|---|
audit_request_effective_user | The username of the user that has accessed watched fields |
audit_request_effective_user_auth_domain | The domain that authenticated the user, as defined in sg_authc.yml . E.g. “ldap”, “jwt” |
Index attributes
Name | Description |
---|---|
audit_trace_indices | Array, the index name(s) as contained in the request. Can contain wildcards, date patterns and aliases. |
audit_trace_resolved_indices | Array, the resolved, concrete index name(s) affected by this request. Only logged if resolve_indices is true. Optional. |
Document and fields attributes
Name | Description |
---|---|
audit_compliance_operation | The operation on the document, can be one of CREATE , UPDATE or DELETE . |
audit_trace_doc_id | Id of the document containing the watched fields. |
audit_compliance_doc_version | The version of the document that has been inserted, changed or deleted. |
audit_compliance_diff_content | The diff of the old and new version of the document in JSON patch format. Only logged for UPDATE . |
audit_compliance_diff_is_noop | Boolean. False if any of the document fields have been modified, otherwise true. Only logged for UPDATE . |
audit_request_body | The content of newly created or updated document. Only logged if searchguard.compliance.history.write.metadata_only is false . |
Example
{
"audit_compliance_operation": "UPDATE",
"audit_cluster_name": "searchguard",
"audit_node_name": "wfAmfPn",
"audit_node_elasticsearch_version": "7.17.11",
"audit_category": "COMPLIANCE_DOC_WRITE",
"audit_request_origin": "REST",
"audit_compliance_doc_version": 8,
"audit_node_id": "wfAmfPncS_62cJ2kPcC0GQ",
"audit_compliance_diff_is_noop": false,
"audit_format_version": 3,
"@timestamp": "2018-01-23T11:21:10.898+00:00",
"audit_request_remote_address": "172.16.0.254",
"audit_trace_doc_types": [
"revenue"
],
"audit_trace_doc_id": "11",
"audit_compliance_diff_content": "[{\"op\":\"remove\",\"path\":\"/date\"},{\"op\":\"replace\",\"path\":\"/revenue\",\"value\":12345},{\"op\":\"remove\",\"path\":\"/customers\"},{\"op\":\"add\",\"path\":\"/remarks\",\"value\":\"none\"}]",
"audit_node_host_address": "172.16.0.2",
"audit_request_effective_user": "finance_employee",
"audit_trace_shard_id": 4,
"audit_trace_indices": [
"finance"
],
"audit_trace_resolved_indices": [
"finance"
],
"audit_node_host_name": "sgssl-1.example.com"
}
Performance considerations
Use diffs only on relevant indices
Logging the difference between two versions of a document has a performance impact. It should be used only on relevant data, and bulk updates should be minimized.
Keeping the watched indices at a minimum
The more indices you watch, the more events are possibly created. Consider only watching indices that you are required to monitor.
Use a high-volume storage type
The write history can emit a lot of events in a short time. Consider using a storage type or cache that can handle a high volume of events, like Kafka, Redis or AWS Kinesis.
Additional resources