# Elasticsearch Configuration Options

## EF\_OUTPUT\_ELASTICSEARCH\_ENABLE

Specifies whether the Elasticsearch output is enabled.

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_ECS\_ENABLE

Specifies whether the data will be sent using Elastic Common Schema (ECS).

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_BATCH\_DEADLINE

The maximum time, in milliseconds, to wait for a batch of records to fill before being sent to the Elasticsearch bulk API.

* Default
  * `2000`

## EF\_OUTPUT\_ELASTICSEARCH\_BATCH\_MAX\_BYTES

The maximum size, in bytes, for a batch of records being sent to the Elasticsearch bulk API.

* Default
  * `8388608`

## EF\_OUTPUT\_ELASTICSEARCH\_TIMESTAMP\_SOURCE

Determines the timestamp source to be used to set the `@timestamp` field. Usually `end` would be the best setting. However, in the case of poorly behaving or misconfigured devices, `collect` may be the better option.

{% hint style="info" %}
Beginning with ElastiFlow `6.3.0` the Elasticsearch output's default value for this setting was changed to `collect`. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the old default setting of `end`, you should ensure that it is specifically set in your configuration.
{% endhint %}

* Valid Values
  * `start` - Use the timestamp from `flow.start.timestamp`. The flow start time indicated in the flow.
  * `end` - Use the timestamp from `flow.end.timestamp`. The flow end time (or last reported time).
  * `export` - Use the timestamp from `flow.export.timestamp`. The time from the flow record header.
  * `collect` - Use the timestamp from `flow.collect.timestamp`. The time that the collector processed the flow record.
* Default
  * `collect`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_PERIOD

{% hint style="info" %}
Beginning with ElastiFlow `6.3.0` the Elasticsearch output's default value for this setting was changed to `rollover`. This will enable the use of Index Lifecycle Management (ILM) to manage retention of ElastiFlow indices. If you wish to continue to use the old default setting of `daily`, you should ensure that it is specifically set in your configuration.
{% endhint %}

* Valid Values
  * `daily` - New indices will be created each day. The format of the time period suffix will be `-yyyy.MM.dd`.
  * `weekly` - New indices will be created each week. The format of the time period suffix will be `-yyyy.'w'ww`.
  * `monthly` - New indices will be created each month. The format of the time period suffix will be `-yyyy.MM`.
  * `rollover` - Index Lifecycle Management (ILM) will be used to handle the creation and deletion of indices. If no ILM policy is defined, a default policy will be created. The rollover alias is determined automatically. If the rollover alias does not exist, an initial write index will be bootstrapped with the rollover alias.
* Default
  * `rollover`

## EF\_OUTPUT\_ELASTICSEARCH\_TSDS\_ENABLE

{% hint style="info" %}
Only supported in NetObserv Flow and NetObserv SNMP. For NetObserv SNMP, support was introduced in 7.13.0
{% endhint %}

Elasticsearch 8.7 introduced TSDS (time series data stream) feature. Storing flow and telemetry data using TSDS can result in a storage savings of 50-70% depending on the content of the flow records. TSDS also supports downsampling (initially for bytes and packets fields) which can result in even less storage capacity needed for historical data. Enabling TSDS does increase the ingest-related CPU load for Elasticsearch. Setting this value to `true` will enable TSDS support.

{% hint style="info" %}
Enabling TSDS requires NetObserv to push new index templates. Please shut down the collector and remove ElastiFlow index templates before you enable TSDS. After you restart the collector you should see new index templates being created.
{% endhint %}

* Valid Values
  * `true`, `false`
* Default
  * `false`

If set to `true`:

* the option `EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD` will be set to `rollover`, overriding any user-provided value.
* the option `EF_OUTPUT_ELASTICSEARCH_STORAGE_OPTIMIZATION_ENABLE` will be set to `false`, overriding any user-provided value.

### TSDS Metric Support Note

For NetObserv SNMP, only some fields will be marked as a '[metric](https://www.elastic.co/docs/manage-data/data-store/data-streams/time-series-data-stream-tsds#time-series-metric)' to Elasticsearch, which impacts downsampling. More specifically, the `time_series_metric` field is only set for a subset of fields in the Elasticsearch templates that NetObserv SNMP creates when it starts.

Initially, only IETF MIB objects will have TSDS metric support. ElastiFlow leverages the [snmp definition files](https://github.com/elastiflow/snmp/tree/main) (which ship with the product) to control which fields are supported as a TSDS metric. If you see the field "metric" assigned to a field (see [IP-MIB](https://github.com/elastiflow/snmp/blob/main/objects/ietf/IP-MIB.yml) for an example), it means that those fields will have time\_series\_metric defined in the Elaticsearch templates that NetObserv SNMP creates.

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_SUFFIX

It can sometimes be useful to have separate indices for different environments, locations or other organizational units. This setting allows you to specify a suffix that will be added to the index for such purposes.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_ENABLE

Specifies whether the output should attempt to add the required index template to Elasticsearch.

* Valid Values
  * `true`, `false`
* Default
  * `true`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_OVERWRITE

If the output is configured to add the index template to Elasticsearch (`EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ENABLE` is `true`), this setting determines whether the index template should be overwritten if it already exists.

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_SHARDS

The number of shards with which the index should be created. As a general rule, additional shards increase ingest performance, assuming there are sufficient data nodes across which the shards can be distributed.

* Recommended
  * Equal to the number of Elasticsearch data nodes to which data will be indexed.
* Default
  * `3`

{% hint style="info" %}
This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.
{% endhint %}

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_REPLICAS

The number of replicas that should be created for each shard. If using a multi-node cluster and data redundancy is desired, this value must be at least `1`.

In general, additional replicas will increase query performance, assuming there are sufficient data nodes across which the replicas can be distributed.

* Recommended
  * `1` if indexing data to a multi-node cluster.
  * `0` for a single-node.
* Default
  * `1`

{% hint style="info" %}
This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.
{% endhint %}

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_REFRESH\_INTERVAL

Specifies the period for the refresh interval. The refresh interval is the time window in which newly ingested documents are added to a segment, prior to the segment being added to the index. Only after the refresh interval has ended and the segment has been added to the index do the documents become searchable.

* Recommended
  * `5s` - If the data needs to become available for queries more quickly. However shorter refresh intervals will negatively impact ingest performance.
  * `30s` - (or longer) If maximizing ingest performance is the highest priority. Longer refresh intervals negatively impact the real-time accessibility of new records.
  * `10s` or `20s` - This is a reasonable compromise between ingest performance and data accessibility for most network traffic analytics use-cases.
* Default
  * `20s`

{% hint style="info" %}
This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.
{% endhint %}

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_CODEC

The setting determines the level of compression used for stored values.

* Valid Values
  * `default` - Stored values are compressed using LZ4.
  * `best_compression` - Stored values are compressed using DEFLATE. This reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.
* Default
  * `best_compression`

{% hint style="info" %}
This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.
{% endhint %}

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_ILM\_LIFECYCLE

If data is being stored to an Elasticsearch cluster with Index Lifecycle Management (ILM) features enabled, this setting specifies the name of the ILM Lifecycle that should be applied to the indices.

{% hint style="info" %}
The ILM Lifecycle itself MUST be configured separately in Elasticsearch.
{% endhint %}

* Default
  * `elastiflow`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_PIPELINE\_DEFAULT

If it is desired to process the incoming with an Elasticsearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the default pipeline.

* Default
  * `_none`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_PIPELINE\_FINAL

If it is desired to process the incoming with an Elasticsearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the final pipeline.

* Default
  * `_none`

## EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_SYNTHETIC\_SOURCE\_ENABLE

If set to `true`, this setting will configure the index template to use the synthetic `_source` field. This will improve storage efficiency, but may reduce the performance of some queries.

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_ADDRESSES

This setting specifies the Elasticsearch servers to which the output should connect. It is a comma-separated list of Elasticsearch nodes, including port number. We recommend specifying a maximum of 6 to 8 Elasticsearch servers here. If you require sending data directly to more than 6 to 8 servers, consider using an Elastic "coordinating node".

{% hint style="danger" %}
Do **NOT** include `http://` or `https://` in the provided value. TLS communications is enabled/disabled using `EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE`.
{% endhint %}

* Default
  * `127.0.0.1:9200`

## EF\_OUTPUT\_ELASTICSEARCH\_USERNAME

The username to use when connecting to Elasticsearch.

* Default
  * `elastic`

## EF\_OUTPUT\_ELASTICSEARCH\_PASSWORD

The password to use when connecting to Elasticsearch.

* Default
  * `changeme`

## EF\_OUTPUT\_ELASTICSEARCH\_CLOUD\_ID

The URI for the Elastic Cloud endpoint to which the output should connect. If set, this value overrides `EF_OUTPUT_ELASTICSEARCH_ADDRESSES`.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_API\_KEY

The base64-encoded token to use for authorization.

Elasticsearch provides [Security APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api.html#security-api-keys) to:

* [Create API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html)
* [Get API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-get-api-key.html)
* [Invalidate API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-invalidate-api-key.html)
* [Clear API key cache](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-clear-api-key-cache.html)
* [Grant API key](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-grant-api-key.html)

If set, this value overrides `EF_OUTPUT_ELASTICSEARCH_USERNAME` and `EF_OUTPUT_ELASTICSEARCH_PASSWORD`.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_CLIENT\_CA\_CERT\_FILEPATH

The path to the Certificate Authority (CA) certificate to use for client PKI authentication.

* Default
  * `''`

To use PKI authentication, your Elasticsearch cluster needs to be configured for such a setup. Here is a [guide](https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication). The collectors require the use of a [role mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-put-role-mapping.html) with a "superuser" role. Alternatively, a custom role that can contain the privileges allowing the collectors to read/create/update/delete indices, and read/create/update/delete data to/from indices.

## EF\_OUTPUT\_ELASTICSEARCH\_CLIENT\_CERT\_FILEPATH

The path to the client certificate to use for client PKI authentication.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_CLIENT\_KEY\_FILEPATH

The path to the client key to use for client PKI authentication.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_TLS\_ENABLE

This setting is used to enable/disable TLS connections to Elasticsearch.

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_TLS\_SKIP\_VERIFICATION

This setting is used to enable/disable TLS verification of the Elasticsearch server to which the output is attempting to connect.

* Valid Values
  * `true`, `false`
* Default
  * `false`

## EF\_OUTPUT\_ELASTICSEARCH\_TLS\_CA\_CERT\_FILEPATH

The path to the Certificate Authority (CA) certificate to use for verification of the Elasticsearch server to which the output is attempting to connect.

* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_RETRY\_ENABLE

Specifies whether to retry connecting to Elasticsearch after a connection has failed.

* Valid Values
  * `true`, `false`
* Default
  * `true`

## EF\_OUTPUT\_ELASTICSEARCH\_RETRY\_ON\_TIMEOUT\_ENABLE

Specifies whether to retry bulk indexing requests which have timed-out.

* Valid Values
  * `true`, `false`
* Default
  * `true`

## EF\_OUTPUT\_ELASTICSEARCH\_MAX\_RETRIES

Specifies the number of times to retry bulk indexing requests which have timed-out.

* Default
  * `3`

## EF\_OUTPUT\_ELASTICSEARCH\_RETRY\_BACKOFF

If set, this value specifies the quantity of milliseconds that the output should "backoff" prior to retrying a failed bulk request.

* Default
  * `1000`

## EF\_OUTPUT\_ELASTICSEARCH\_STORAGE\_OPTIMIZATION\_ENABLE

This setting is used to enable/disable the storage optimization feature. This feature will configure the index sort setting in the index template to optimize the storage of the data.

{% hint style="info" %}
`EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD` will be overridden to `rollover` when this setting is enabled.
{% endhint %}

* Valid Values
  * `true`, `false`
* Default
  * `true`

## EF\_OUTPUT\_ELASTICSEARCH\_DROP\_FIELDS

This setting allows for a comma-separated list of fields that are to be removed from all records.

{% hint style="info" %}
Fields are dropped after any output specific fields have been added and after any schema conversion. This means that you should use the field names as you see them in the user interface.
{% endhint %}

* Valid Values
  * any field names related to the enabled schema, comma-separated
* Example
  * `flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num`
* Default
  * `''`

## EF\_OUTPUT\_ELASTICSEARCH\_ALLOWED\_RECORD\_TYPES

This setting allows for a comma-separated list of record types that the output will send will emit. This is particularly useful when used with multiple namespaced outputs, e.g. sending flow records to one datastore and telemetry to another.

* Valid Values
  * `as_path_hop`, `flow_option`, `flow`, `ifa_hop`, `telemetry`, `metric`
* Default
  * `'as_path_hop,flow_option,flow,ifa_hop,telemetry,metric'`


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/trapcoll/configuration/outputs/output_elasticsearch/elasticsearch-configuration-options.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.