Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

OpenSearch Configuration Options

EF_OUTPUT_OPENSEARCH_ENABLE

Specifies whether the OpenSearch output is enabled.

  • Valid Values

    • true, false

  • Default

    • false

EF_OUTPUT_OPENSEARCH_ECS_ENABLE

Specifies whether the data will be sent using Elastic Common Schema (ECS).

  • Valid Values

    • true, false

  • Default

    • false

EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE

The maximum time, in milliseconds, to wait for a batch of records to fill before being sent to the OpenSearch bulk API.

  • Default

    • 2000

EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES

The maximum size, in bytes, for a batch of records being sent to the OpenSearch bulk API.

  • Default

    • 8388608

EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE

Determines the timestamp source to be used to set the @timestamp field. Usually end would be the best setting. However, in the case of poorly behaving or misconfigured devices, collect may be the better option.

Beginning with ElastiFlow 6.3.0 the OpenSearch output's default value for this setting was changed to collect. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the old default setting of end, you should ensure that it is specifically set in your configuration.

  • Valid Values

    • start - Use the timestamp from flow.start.timestamp. The flow start time indicated in the flow.

    • end - Use the timestamp from flow.end.timestamp. The flow end time (or last reported time).

    • export - Use the timestamp from flow.export.timestamp. The time from the flow record header.

    • collect - Use the timestamp from flow.collect.timestamp. The time that the collector processed the flow record.

  • Default

    • collect

EF_OUTPUT_OPENSEARCH_INDEX_PERIOD

Beginning with ElastiFlow 7.11.0 the OpenSearch output's default value for this setting will be changed to rollover. This will enable the use of Index State Management (ISM) to manage retention of ElastiFlow indices. If you wish to continue to use the old default setting of daily, you should ensure that it is specifically set in your configuration.

  • Valid Values

    • daily - New indices will be created each day. The format of the time period suffix will be -yyyy.MM.dd.

    • weekly - New indices will be created each week. The format of the time period suffix will be -yyyy.'w'ww.

    • monthly - New indices will be created each month. The format of the time period suffix will be -yyyy.MM.

    • rollover - Index State Management will be used to handle the creation and deletion of indices.

  • Default

    • rollover

EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX

It can sometimes be useful to have separate indices for different environments, locations or other organizational units. This setting allows you to specify a suffix that will be added to the index for such purposes.

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE

Specifies whether the output should attempt to add the required index template to OpenSearch.

  • Valid Values

    • true, false

  • Default

    • true

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE

If the output is configured to add the index template to OpenSearch (EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE is true), this setting determines whether the index template should be overwritten if it already exists.

  • Valid Values

    • true, false

  • Default

    • false

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS

The number of shards with which the index should be created. As a general rule, additional shards increase ingest performance, assuming there are sufficient data nodes across which the shards can be distributed.

  • Recommended

    • Equal to the number of OpenSearch data nodes to which data will be indexed.

  • Default

    • 3

This setting configures the index template sent to OpenSearch. It does NOT change any existing indices.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS

The number of replicas that should be created for each shard. If using a multi-node cluster and data redundancy is desired, this value must be at least 1.

In general, additional replicas will increase query performance, assuming there are sufficient data nodes across which the replicas can be distributed.

  • Recommended

    • 1 if indexing data to a multi-node cluster.

    • 0 for a single-node.

  • Default

    • 1

This setting configures the index template sent to OpenSearch. It does NOT change any existing indices.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL

Specifies the period for the refresh interval. The refresh interval is the time window in which newly ingested documents are added to a segment, prior to the segment being added to the index. Only after the refresh interval has ended and the segment has been added to the index do the documents become searchable.

  • Recommended

    • 5s - If the data needs to become available for queries more quickly. However, shorter refresh intervals will negatively impact ingest performance.

    • 30s - (or longer) If maximizing ingest performance is the highest priority. Longer refresh intervals negatively impact the real-time accessibility of new records.

    • 10s or 20s - This is a reasonable compromise between ingest performance and data accessibility for most network traffic analytics use-cases.

  • Default

    • 20s

This setting configures the index template sent to OpenSearch. It does NOT change any existing indices.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC

The setting determines the level of compression used for stored values.

  • Valid Values

    • default - Stored values are compressed using LZ4.

    • best_compression - Stored values are compressed using DEFLATE. This reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.

  • Default

    • best_compression

This setting configures the index template sent to OpenSearch. It does NOT change any existing indices.

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY

If data is being stored to an Open Distro for OpenSearch cluster, this setting specifies the Index State Management (ISM) Policy ID that should be applied to the indices.

The ISM Policy itself MUST be configured separately in OpenSearch. If EF_OUTPUT_OPENSEARCH_INDEX_PERIOD is set to rollover a default ISM policy will be added automatically with the name defined in this setting.

  • Default

    • elastiflow

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT

If it is desired to process the incoming with an OpenSearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the default pipeline.

  • Default

    • _none

EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL

If it is desired to process the incoming with an OpenSearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the final pipeline.

  • Default

    • _none

EF_OUTPUT_OPENSEARCH_ADDRESSES

This setting specifies the OpenSearch servers to which the output should connect. It is a comma-separated list of OpenSearch nodes, including port number.

Do NOT include http:// or https:// in the provided value. TLS communications is enabled/disabled using EF_OUTPUT_OPENSEARCH_TLS_ENABLE.

  • Default

    • 127.0.0.1:9200

EF_OUTPUT_OPENSEARCH_USERNAME

The username to use when connecting to OpenSearch.

  • Default

    • admin

EF_OUTPUT_OPENSEARCH_PASSWORD

The password to use when connecting to OpenSearch.

  • Default

    • admin

EF_OUTPUT_OPENSEARCH_AWS_ACCESS_KEY

The AWS Access key to use when connecting to OpenSearch via Auth Sig V4

EF_OUTPUT_OPENSEARCH_AWS_SECRET_KEY

The AWS Secret key to use when connecting to OpenSearch via Auth Sig V4

EF_OUTPUT_OPENSEARCH_AWS_REGION

The AWS Region to use when connecting to OpenSearch via Auth Sig V4

EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH

The path to the Certificate Authority (CA) certificate to use for client PKI authentication.

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH

The path to the client certificate to use for client PKI authentication.

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH

The path to the client key to use for client PKI authentication.

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_TLS_ENABLE

This setting is used to enable/disable TLS connections to OpenSearch.

  • Valid Values

    • true, false

  • Default

    • false

EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION

This setting is used to enable/disable TLS verification of the OpenSearch server to which the output is attempting to connect.

  • Valid Values

    • true, false

  • Default

    • false

EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH

The path to the Certificate Authority (CA) certificate to use for verification of the OpenSearch server to which the output is attempting to connect.

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_RETRY_ENABLE

Specifies whether to retry connecting to OpenSearch after a connection has failed.

  • Valid Values

    • true, false

  • Default

    • true

EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE

Specifies whether to retry bulk indexing requests which have timed-out.

  • Valid Values

    • true, false

  • Default

    • true

EF_OUTPUT_OPENSEARCH_MAX_RETRIES

Specifies the number of times to retry bulk indexing requests which have timed-out.

  • Default

    • 3

EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF

If set, this value specifies the quantity of milliseconds that the output should "backoff" prior to retrying a failed bulk request.

  • Default

    • 1000

EF_OUTPUT_OPENSEARCH_STORAGE_OPTIMIZATION_ENABLE

This setting is used to enable/disable the storage optimization feature. This feature will configure the index sort setting in the index template to optimize the storage of the data.

EF_OUTPUT_OPENSEARCH_INDEX_PERIOD will be overridden to rollover when this setting is enabled.

  • Valid Values

    • true, false

  • Default

    • true

EF_OUTPUT_OPENSEARCH_DROP_FIELDS

This setting allows for a comma-separated list of fields that are to be removed from all records.

Fields are dropped after any output specific fields have been added and after any schema conversion. This means that you should use the field names as you see them in the user interface.

  • Valid Values

    • any field names related to the enabled schema, comma-separated

  • Example

    • flow.export.sysuptime,flow.export.version.ver,flow.start.sysuptime,flow.end.sysuptime,flow.seq_num

  • Default

    • ''

EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES

This setting allows for a comma-separated list of record types that the output will send will emit. This is particularly useful when used with multiple namespaced outputs, e.g. sending flow records to one datastore and telemetry to another.

  • Valid Values

    • as_path_hop, flow_option, flow, ifa_hop, telemetry, metric, log

  • Default

    • 'as_path_hop,flow_option,flow,ifa_hop,telemetry,metric,log'

PreviousOpenSearchNextCribl

Last updated

Was this helpful?