Skip to main content

Configuration Reference

License#

EF_FLOW_ACCOUNT_ID#

License keys are generated per account. This field must contain the Account ID for the Licence Key specified in EF_FLOW_LICENSE_KEY.

EF_FLOW_LICENSE_KEY#

This field contains the License Key issued for this instance of the Unified Flow Collector. EF_FLOW_ACCOUNT_ID must also contain the Account ID to which this key belongs.

EF_FLOW_LICENSED_CORES#

The ElastiFlowâ„¢ Unified Flow Collector is licensed by cores. By default the number of cores will be set based on the provided license key. However the number of cores to be used by as instance can be configured manually. This is usually done when it is desired to use multiple instances of the collector. For example, a subscription for 8 licensed cores can be split into 2 instances, of 4 cores each, by setting EF_FLOW_LICENSED_CORES: 4 for each instance. If set to a value greater than allowed by the license key, the instances will be started with the number of cores from the license key.

The default values of EF_FLOW_SERVER_UDP_PACKET_STREAM_MAX_SIZE, EF_FLOW_RECORD_STREAM_MAX_SIZE, EF_FLOW_OUTPUT_ELASTICSEARCH_POOL_SIZE and EF_FLOW_OUTPUT_RISKIQ_POOL_SIZE are based on the number of licensed cores. However, these settings may also be set manually to override the defaults as needed for a given environment.

The default value of 1 is the number of cores allowed under the community and basic subscription tiers. See https://www.elastiflow.com/subscriptions for more details about subscription options.

  • Default
    • 1

Logging#

EF_FLOW_LOGGER_LEVEL#

Specifies the output level for logging.

  • Valid Values
    • debug, info, warn, error, panic, fatal
  • Default
    • info

EF_FLOW_LOGGER_ENCODING#

Specifies the output format of the produced logs.

  • Valid Values
    • console, json
  • Default
    • json

EF_FLOW_LOGGER_FILE_LOG_ENABLE#

Set to true to enable writing logs to a file.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_LOGGER_FILE_LOG_DIR#

If logging to files is enabled (EF_FLOW_LOGGER_FILE_LOG_ENABLE is true) this option specifies the path to the directory where the logs will be written.

  • Default
    • /var/log/elastiflow/flowcoll

EF_FLOW_LOGGER_FILE_LOG_COUNT#

This option species the number of log files that till be kept.

  • Default
    • 4

EF_FLOW_LOGGER_FILE_LOG_INTERVAL#

This option specifies the time period to which a single log will be written.

  • Valid Values
    • daily, monthly, yearly
  • Default
    • daily

EF_FLOW_LOGGER_FILE_LOG_SIZE#

This option specifies the maximum size to which a single log will allowed to grow.

  • Valid Values
    • 100, 100kB, 100MB, 100GB
  • Default
    • 100MB

UDP Server#

The ElastiFlowâ„¢ Unified Flow Collector receives network flow records over UDP.

EF_FLOW_SERVER_UDP_IP#

  • Valid Values
    • 0.0.0.0 or any valid IP address to which the UDP socket can be bound.
  • Default
    • 0.0.0.0 (listen on all interfaces)

EF_FLOW_SERVER_UDP_PORT#

  • Valid Values
    • Any valid port number. Common values include:
      • 2055: the standard port for Netflow
      • 4739: the standard port for IPFIX
      • 6343: the standard port for sFlow
      • 9995-9998: commonly use port numbers
  • Default
    • 9995

EF_FLOW_SERVER_UDP_PACKET_STREAM_MAX_SIZE#

Received UDP PDUs are queued prior to being processed by an available decoder. This value specifies the size of the queue as a quantity of PDUs.

  • Default
    • 4096 * EF_FLOW_LICENSED_CORES

EF_FLOW_SERVER_UDP_READ_BUFFER_MAX_SIZE#

The size, in bytes, of the UDP receive buffer that the UDP server will request be created by the operating system kernel when the socket is created. If this value exceeds the maximum allowed buffer size (net.core.rmem_max on Linux), the maximum allowed size is used.

  • Default
    • 33554432

Decoder#

EF_FLOW_DECODER_SETTINGS_PATH#

The path where any files used by the collector's decoder functions are loacted.

  • Default
    • /etc/elastiflow

EF_FLOW_DECODER_IPFIX_ENABLE#

Set to true to enable decoding of IPFIX records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_NETFLOW5_ENABLE#

Set to true to enable decoding of Netflow v5 records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_NETFLOW9_ENABLE#

Set to true to enable decoding of Netflow v9 records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_SFLOW5_ENABLE#

Set to true to enable decoding of sFlow v5 records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_SFLOW_FLOWS_ENABLE#

Set to true to enable decoding of sFlow flow_sample and flow_sample_expanded records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_SFLOW_FLOWS_KEEP_SAMPLES#

When set to true, the packet data from an sFlow sampled_header record will be stored in l2.section.sample as a hex-encoded string.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_SFLOW_COUNTERS_ENABLE#

Set to true to enable decoding of sFlow counters_sample and counters_sample_expanded records.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_TRANSLATE_KEEP_IDS#

Specifies which identifier values will be included in the final dataset.

  • Valid Values
    • none - All identifiers are removed from the final dataset.
    • default - Most identifiers are removed from the final dataset. However some identifiers which are required for common use-cases (e.g. raw protocol port values) are included.
    • all - All identifiers are included in the final dataset.
  • Default
    • default

EF_FLOW_DECODER_ENRICH_ASN_PREF#

If enrichment with autonomous system attributes is enabled, but the autonomous system is already indicated directly in the flow record data, this setting specifies which source is prefered. If the preferred source is not available for a given record, the decoder will fall-back to the alternate option.

  • Valid Values
    • lookup - prefer the autonomous system determined by lookup.
    • flow - prefer the autonomous system indicated directly in the flow record data.
  • Default
    • lookup

EF_FLOW_DECODER_ENRICH_JOIN_ASN#

Some features require that related values from separate fields are stored as an array in a single field. Such a "join" of autonomous system related fields is enabled when this setting is true.

important

If records are being output to Elasticsearch this setting should be set to true.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_JOIN_GEOIP#

Some features require that related values from separate fields are stored as an array in a single field. Such a "join" of GeoIP related fields is enabled when this setting is true.

important

If records are being output to Elasticsearch this setting should be set to true.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_JOIN_NETATTR#

Some features require that related values from separate fields are stored as an array in a single field. Such a "join" of network attribute related fields is enabled when this setting is true.

important

If records are being output to Elasticsearch this setting should be set to true.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_JOIN_SEC#

Some features require that related values from separate fields are stored as an array in a single field. Such a "join" of security attribute related fields is enabled when this setting is true.

important

If records are being output to Elasticsearch this setting should be set to true.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_DURATION_PRECISION#

The desired precision of duration-related values. Values received at a different precision than specified will be converted to the desired precision.

  • Valid Values
    • sec - seconds
    • ds - deciseconds
    • cs - centiseconds
    • ms - millseconds
    • us - microseconds
    • ns - nanoseconds
  • Default
    • ms
tip

For most data sources this should millseconds (ms)

EF_FLOW_DECODER_TIMESTAMP_PRECISION#

The desired precision of timestamp values. Values received at a different precision than specified will be converted to the desired precision.

  • Valid Values
    • sec - seconds
    • ds - deciseconds
    • cs - centiseconds
    • ms - millseconds
    • us - microseconds
    • ns - nanoseconds
  • Default
    • ms
tip

For most data stores, e.g. Elasticsearch, this should millseconds (ms)

EF_FLOW_DECODER_PERCENT_NORM#

The desired representation of percentages. Values received with a different representation than specified will be converted to the desired representation.

  • Valid Values
    • 1 - values will be based on a scale of 0-1.
    • 100 - values will be based on a scale of 0-100.
  • Default
    • 100

EF_FLOW_DECODER_ENRICH_EXPAND_CLISRV#

The collector will infer the client/server relationship of two source/destination endpoints. The is setting determines whether such inferrence is enabled or not.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_KEEP_CPU_TICKS#

For telemetry sources which provide CPU usage as timeticks, utilization percentages will be calculated. If this setting is set false the timetick values will be removed from the final dataset. If true they will be kept, in addition to the utilization values.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_RECORD_STREAM_MAX_SIZE#

Processed records are queued prior to being processed by an available output instance. This value specifies the size of the queue as a quantity of records. As a single PDU typically contains multiple flow records, this value will typically be a multiple of EF_FLOW_SERVER_UDP_PACKET_STREAM_MAX_SIZE.

  • Default
    • 16 * EF_FLOW_SERVER_UDP_PACKET_STREAM_MAX_SIZE

Name Resolution#

EF_FLOW_DECODER_ENRICH_DNS_ENABLE#

This setting enables DNS reverse lookups of IP addresses found in the received flow records.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_DNS_CACHE_SIZE#

The ElastiFlowâ„¢ Unified Flow Collector will cache the result of DNS reverse lookups, including failures. This reduces the overall number of DNS queries, and increases throughput.

This setting specifies the maximum number of IPs which will be held in the cache.

  • Default
    • 524288

EF_FLOW_DECODER_ENRICH_DNS_RESOLVE_EXPORTER#

If DNS resolution is enabled (EF_FLOW_DECODER_ENRICH_DNS_ENABLE is true) this option specifies whether flow exporter IP addresses will be resolved to hostnames.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_DNS_RESOLVE_PRIVATE#

If DNS resolution is enabled (EF_FLOW_DECODER_ENRICH_DNS_ENABLE is true) this option specifies whether private IP addresses will be resolved to hostnames.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_DNS_RESOLVE_PUBLIC#

If DNS resolution is enabled (EF_FLOW_DECODER_ENRICH_DNS_ENABLE is true) this option specifies whether public IP addresses will be resolved to hostnames.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_DNS_USERDEF_ENABLE#

Instead of using DNS to determine hostnames from IP addresses, a file of static IP to hostname mappings can be provided. DNS queries will not be issued for IP addresses in this file.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_DNS_USERDEF_PATH#

If user-defined IP to hostname mappings are enabled (EF_FLOW_DECODER_ENRICH_DNS_USERDEF_ENABLE is true) this setting specifies the path to this file.

If the value of the path begins with a / this path will be interpreted as an absolute file system path. Otherwise it will be interpreted as relative to the value of EF_FLOW_DECODER_SETTINGS_PATH.

An example of the format of this file is:

'192.0.2.1': 'host1'
'192.0.2.2': 'host2'
  • Default
    • settings/hostnames_user_defined.yml

Network Interface Attributes#

EF_FLOW_DECODER_ENRICH_NETIF_GET_ATTRS#

Flow records generally include the index of ingress and egress interfaces by which the network traffic traversed the exporting device. The ElastiFlowâ„¢ Unified Flow Collector will attempt to determine the names, and additional attributes, of these interfaces as learned from Netflow v9 or IPFIX option records, or determined by polling the exporting device using SNMP.

Setting this value to false will disable the enrichment of records with interface atributes.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_NETIF_CACHE_SIZE#

Interface attributes, learned from either option data or SNMP, will be cached to reduce possible SNMP polls and improve performance.

This setting specifies the maximum number of interfaces which will be held in the cache.

  • Default
    • 262144

EF_FLOW_DECODER_ENRICH_SNMP_ENABLE#

If the enrichment with network interface attributes is enabled (EF_FLOW_DECODER_ENRICH_NETIF_GET_ATTRS is true), this setting determined whether SNMP polls will be used to gather these attributes.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_SNMP_PORT#

If SNMP polling of attributes is enabled (EF_FLOW_DECODER_ENRICH_SNMP_ENABLE is true), this setting specifies the UDP port that is used for such polls.

  • Default
    • 161 (the default SNMP port number)

EF_FLOW_DECODER_ENRICH_SNMP_VERSION#

If SNMP polling of attributes is enabled (EF_FLOW_DECODER_ENRICH_SNMP_ENABLE is true), this setting specifies the SNMP version that is used for such polls.

important

All network devices which may be polled MUST support this version of SNMP.

  • Valid Values
    • 1 - use SNMPv1
    • 2 - use SNMPv2c
  • Default
    • 2

EF_FLOW_DECODER_ENRICH_SNMP_COMMUNITY#

If SNMP polling of attributes is enabled (EF_FLOW_DECODER_ENRICH_SNMP_ENABLE is true), this setting specifies the SNMP community string that is used for such polls.

important

All network devices which may be polled MUST be configured to all visibility of collected attributes using this community. It may be necessary to specify a view associated with this community. The documentation for your devices should contain the necessary information that you will need for determining the correct configuration steps.

  • Default
    • public

EF_FLOW_DECODER_ENRICH_SNMP_TIMEOUT#

If SNMP polling of attributes is enabled (EF_FLOW_DECODER_ENRICH_SNMP_ENABLE is true), this setting specifies the number of seconds to wait for the polled device to respond.

  • Default
    • 2

EF_FLOW_DECODER_ENRICH_SNMP_RETRIES#

If SNMP polling of attributes is enabled (EF_FLOW_DECODER_ENRICH_SNMP_ENABLE is true), this setting specifies the number of retries to attempt after the initial poll has timed out or otherwise fail. The timeout period will be doubled for each retry.

  • Default
    • 1

Application Attributes#

EF_FLOW_DECODER_ENRICH_APP_CACHE_SIZE#

The ElastiFlowâ„¢ Unified Flow Collector will cache application attribues learned from option data. This setting specifies the maximum number of device specific application IDs which will be held in the cache.

  • Default
    • 262144

EF_FLOW_DECODER_ENRICH_APP_USERDEF_ENABLE#

While various flow record sources send the mapping of application IDs to applications names as option data. In cases where no application identity technology is available, applications can be statically specified by IP address and port number. The application name specified will be used to populate the app.name (for the default CODEX schema) or network.application (if using the optional ECS schema) field.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_APP_USERDEF_PRIVATE#

If user-defined application name are enabled (EF_FLOW_DECODER_ENRICH_APP_USERDEF_ENABLE is true) this option specifies whether application names will be checked for private IP addresses.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_APP_USERDEF_PUBLIC#

If user-defined application name are enabled (EF_FLOW_DECODER_ENRICH_APP_USERDEF_ENABLE is true) this option specifies whether application names will be checked for public IP addresses.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_APP_USERDEF_PATH#

If user-defined IP/port to application mappings are enabled (EF_FLOW_DECODER_ENRICH_APP_USERDEF_ENABLE is true) this setting specifies the path to this file.

note

If the value of the path begins with a / this path will be interpreted as an absolute file system path. Otherwise it will be interpreted as relative to the value of EF_FLOW_DECODER_SETTINGS_PATH.

An example of the format of this file is:

'192.0.2.11':
5601: 'kibana'
9200: 'elasticsearch'
9300: 'elasticsearch_transport'
'192.0.2.12':
2181: 'zookeeper'
2888: 'zookeeper_leader'
3888: 'zookeeper_election'
9092: 'kafka'
  • Default
    • settings/apps_user_defined.yml

RiskIQ#

For the RiskIQ Integration to function fully, both the RiskIQ output as well as the enrichment option MUST be enabled. Only information about traffic to/from public IP addresses is transmitted to RiskIQ. No internal/private IP addresses are transmitted.

EF_FLOW_OUTPUT_RISKIQ_ENABLE#

This setting specifies whether the RiskIQ is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_RISKIQ_HOST#

This setting specifies hostname of the RiskIQ service to which data is sent to RiskIQ for analysis. The value for this setting will be provided to you in your RiskIQ PassiveTotal account settings.

  • Default
    • ''

EF_FLOW_OUTPUT_RISKIQ_PORT#

This setting specifies port number of the RiskIQ service to which data is sent to RiskIQ for analysis. The value for this setting will be provided to you in your RiskIQ PassiveTotal account settings.

  • Default
    • ``

EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_UUID#

This setting specifies the user-specific UUID required by the RiskIQ service to associate the data with your account.

  • Default
    • ''

EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_ENCRYPTION_KEY#

This setting specifies the user-specific encryption key required to transmit data securely to the RiskIQ service.

  • Default
    • ''

EF_FLOW_OUTPUT_RISKIQ_POOL_SIZE#

The number of RiskIQ output workers.

  • Default
    • 0.5 * EF_FLOW_LICENSED_CORES (rounded up)

EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE#

This setting specifies whether enrichment with autonomous system attributes from the RiskIQ service is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENDPOINT#

If RiskIQ ASN enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE is true) this setting specifies the endpoint of the RiskIQ enrichment API to query.

warning

Do NOT change this value unless directed by ElastiFlowâ„¢ support.

  • Default
    • https://api.passivetotal.org/v2/netflow/as/download

EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_REFRESH_INTERVAL#

If RiskIQ ASN enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE is true) this setting specifies the interval, in minutes, at which the RiskIQ enrichment API will be queried to refresh the dataset.

note

60 minutes is the minimum refresh interval. The collector will fail with an error if this value is less than 60.

  • Default
    • 1440

EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE#

This setting specifies whether enrichment with threat attributes from the RiskIQ service is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENDPOINT#

If RiskIQ threat enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE is true) this setting specifies the endpoint of the RiskIQ enrichment API to query.

warning

Do NOT change this value unless directed by ElastiFlowâ„¢ support.

  • Default
    • https://api.passivetotal.org/v2/netflow/blocklist/download

EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_REFRESH_INTERVAL#

If RiskIQ threat enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE is true) this setting specifies the interval, in minutes, at which the RiskIQ enrichment API will be queried to refresh the dataset.

note

60 minutes is the minimum refresh interval. The collector will fail with an error if this value is less than 60.

  • Default
    • 1440

EF_FLOW_DECODER_ENRICH_RISKIQ_API_USER#

If RiskIQ enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE or EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE is true) this setting specifies the API user from the PassiveTotal account that will be used to access the RiskIQ enrichment API.

  • Default
    • ''

EF_FLOW_DECODER_ENRICH_RISKIQ_API_KEY#

If RiskIQ enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE or EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE is true) this setting specifies the API key from the PassiveTotal account that will be used to access the RiskIQ enrichment API.

  • Default
    • ''

EF_FLOW_DECODER_ENRICH_RISKIQ_API_TIMEOUT#

If RiskIQ enrichment is enabled (EF_FLOW_DECODER_ENRICH_RISKIQ_ASN_ENABLE or EF_FLOW_DECODER_ENRICH_RISKIQ_THREAT_ENABLE is true) this setting specifies the timeout duration, in seconds, for API queries.

  • Default
    • 30

Maxmind#

EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE#

The ElastiFlowâ„¢ Unified Flow Collector will attempt to determine attributes associated with the autonomous system to which a public IP address belongs. This setting determines whether this feature is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_CACHE_SIZE#

If enrichment with autonomous system attributes is enabled (EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE is true), attributes determined by lookup will be cached to improve performance. This setting specifies the maximum number of IP address for which attributes will be held in the cache.

  • Default
    • 262144

EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_PATH#

If enrichment with autonomous system attributes is enabled using lookups in a Maxmind database (EF_FLOW_DECODER_ENRICH_MAXMIND_ASN_ENABLE is true), this setting specifies the path to the Maxmind database.

note

If the value of the path begins with a / this path will be interpreted as an absolute file system path. Otherwise it will be interpreted as relative to the value of EF_FLOW_DECODER_SETTINGS_PATH.

  • Default
    • maxmind/GeoLite2-ASN.mmdb

EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE#

The ElastiFlowâ„¢ Unified Flow Collector will attempt to determine GeoIP attributes associated with a public IP address. This setting determines whether this feature is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_CACHE_SIZE#

If enrichment with GeoIP attributes is enabled (EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE is true), attributes determined by lookup will be cached to improve performance. This setting specifies the maximum number of IP address for which attributes will be held in the cache.

  • Default
    • 262144

EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_PATH#

If enrichment with GeoIP attributes is enabled using lookups in a Maxmind database (EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE is true), this setting specifies the path to the Maxmind database.

note

If the value of the path begins with a / this path will be interpreted as an absolute file system path. Otherwise it will be interpreted as relative to the value of EF_FLOW_DECODER_SETTINGS_PATH.

  • Default
    • maxmind/GeoLite2-City.mmdb

EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_VALUES#

If enrichment with GeoIP attributes is enabled using lookups in a Maxmind database (EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE is true), this setting specifies the GeoIP attributes from the Maxmind database to be included in the resulting record.

  • Valid Values
    • city, continent, continent_code, country, country_code, location, timezone
  • Default
    • city,country,country_code,location,timezone

EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_LANG#

If enrichment with GeoIP attributes is enabled using lookups in a Maxmind database (EF_FLOW_DECODER_ENRICH_MAXMIND_GEOIP_ENABLE is true), this setting specifies the language which should be used for any language-specifc values.

  • Valid Values
    • de - German
    • en - English
    • es - Spanish
    • fr - French
    • ja - Japanese
    • pt-BR - Brazilian Portuguese
    • ru - Russian
    • zh-CN - Simplified Chinese
  • Default
    • en

Sample Rate#

Devices may sample packets to reduce the overall volume of traffic metered for flow accounting.

EF_FLOW_DECODER_ENRICH_SAMPLERATE_CACHE_SIZE#

The collector must adjust the calculation of bytes and packets based on the sampling rate used. Usually devices will inform the collector of the sampling rate either within the flow record itself, or as option data sent periodically by the device. This setting specifies the size of the cache to be used to hold sample rate information learned from option data.

  • Default
    • 32768

EF_FLOW_DECODER_ENRICH_SAMPLERATE_USERDEF_ENABLE#

In some cases a device may not transmit information about the sampling rate for which it is configured. In this case it is possible to statically define the sampling rate in file provided to the collector. This setting is used to enable that feature.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_DECODER_ENRICH_SAMPLERATE_USERDEF_PATH#

If static sample rates are configured for devices in a file, this setting specifies the path from where that file can be loaded.

note

If the value of the path begins with a / this path will be interpreted as an absolute file system path. Otherwise it will be interpreted as relative to the value of EF_FLOW_DECODER_SETTINGS_PATH.

An example of the format of this file is:

'192.0.2.1': 1024
'192.0.2.2': 512
  • Default
    • settings/sample_rate.yml

Community ID#

EF_FLOW_DECODER_ENRICH_COMMUNITYID_ENABLE#

Specifies whether flow records should be enriched with a Community ID value.

note

For more information on community IDs see https://github.com/corelight/community-id-spec.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_COMMUNITYID_SEED#

A 16-bit value used as the seed for determining the Community ID of a flow record.

  • Default
    • 0

Conversation ID#

EF_FLOW_DECODER_ENRICH_CONVERSATIONID_ENABLE#

Specifies whether flow records should be enriched with a Conversation ID value. This value is similar to a community ID (see... EF_FLOW_DECODER_ENRICH_COMMUNITYID_ENABLE). However rather than being based on the src/dst relationship of two endpoints, it is based on the client/server perspective. While two related unidirectional flows, e.g. an HTTP request and the corresponding HTTP response, will have different community IDs. Both of these flows will have the same conversation ID. This provides greater flexibility when exploring a complex flow dataset.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_DECODER_ENRICH_CONVERSATIONID_SEED#

A 16-bit value used as the seed for determining the Conversation ID of a flow record.

  • Default
    • 0

stdout Output#

EF_FLOW_OUTPUT_STDOUT_ENABLE#

Specifies whether the stdout output is enabled.

note

At anything more than a few flow records per second the data will scroll too fast to be useful. For this reason the stdout output should be used primarily for manual testing.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_STDOUT_FORMAT#

  • Valid Values
    • json - Output as a single JSON-formatted record per line.
    • json_pretty - Output each record as a "pretty" formatted JSON document.
  • Default
    • json_pretty

Monitor Output#

EF_FLOW_OUTPUT_MONITOR_ENABLE#

The monitor output generates a log message containing the rate of records received and decoded by the collector over the past interval specified in EF_FLOW_OUTPUT_MONITOR_INTERVAL. This can be useful for sizing or troubleshooting. This setting specifies whether the monitor output is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_MONITOR_INTERVAL#

Specifies the interval, in seconds, after which the rate of records will be calculated and logged.

  • Default
    • 300 (5 minutes)

Elasticsearch Output#

EF_FLOW_OUTPUT_ELASTICSEARCH_ENABLE#

Specifies whether the Elasticsearch output is enabled.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_ELASTICSEARCH_ECS_ENABLE#

Specifies whether the data will be sent using Elastic Common Schema (ECS).

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_ELASTICSEARCH_BATCH_DEADLINE#

The maximum time, in milliseconds, to wait for a batch of records to fill before being sent to the Elasticsearch bulk API.

  • Default
    • 2000

EF_FLOW_OUTPUT_ELASTICSEARCH_BATCH_MAX_BYTES#

The maximum size, in bytes, for a batch of records being sent to the Elasticsearch bulk API.

  • Default
    • 8388608

EF_FLOW_OUTPUT_ELASTICSEARCH_POOL_SIZE#

The number of Elasticsearch output workers.

  • Default
    • 1 * EF_FLOW_LICENSED_CORES

EF_FLOW_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE#

Determines the timestamp source to be used to set the @timestamp field. Usually end would be the best setting. However, in the case of poorly behaving or misconfigured devices, collect may be the better option.

  • Valid Values
    • start - Use the timestamp from flow.start.timestamp. The flow start time indicated in the flow.
    • end - Use the timestamp from flow.end.timestamp. The flow end time (or last reported time).
    • export - Use the timestamp from flow.export.timestamp. The time from the flow record header.
    • collect - Use the timestamp from flow.collect.timestamp. The time that the collector processed the flow record.
  • Default
    • end

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_PERIOD#

  • Valid Values
    • daily - New indices will be created each day. The format of the time period suffix will be -yyyy.MM.dd.
    • weekly - New indices will be created each week. The format of the time period suffix will be -yyyy.'w'ww.
    • monthly - New indices will be created each month. The format of the time period suffix will be -yyyy.MM.
    • ilm - Index Lifecycle Management will be used to handle the creation and deletion of indices.
  • Default
    • daily

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_SUFFIX#

It can sometimes be useful to have separate indices for different environments, locations or other organizational unit. This settings allow you to specify a suffix that will be added to the index for such purposes.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ENABLE#

Specifies whether the output should attempt to add the required index template to Elasticsearch.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_OVERWRITE#

If the output is configured to add the index template to Elasticsearch (EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ENABLE is true), this setting determines whether the index template should be overwritten if it already exists.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS#

The number of shards with which the index should be created. As a general rule, additional shards increases ingest performance, assuming there are sufficient data nodes across which the shards can be distributed.

  • Recommended
    • 2 times the number of Elasticsearch data nodes to which data will be indexed.
  • Default
    • 3
note

This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS#

The number of replicas that should be created for each shard. If using a multi-node cluster and data redundancy is desired, this value must be at least 1.

In general, additional replicas will increase query performance, assuming there are sufficient data nodes across which the replicas can be distributed.

  • Recommended
    • 1 times the number of Elasticsearch data nodes to which data will be indexed.
  • Default
    • 1
note

This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL#

Specifies the period for the refresh interval. The refresh interval is the time window in which newly ingested documents are added to a segment, prior to the segment being added to the index. Only after the refresh interval has ended and the segment has been added to the index do the documents become searchable.

  • Recommended
    • 5s - If the data needs to become available for queries more quickly. However shorter refresh intervals will negatively impact ingest performance.
    • 30s - (or longer) If maximizing ingest performance is the highest priority. Longer refresh intervals negatively impact the real-time accessibility of new records.
    • 10s or 15s - This is a reasonable compromise between ingest performance and data accessibility for most network traffic analytics use-cases.
  • Default
    • 10s
note

This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_CODEC#

The setting determines the level of compression used for stored values.

  • Valid Values
    • default - Stored values are compressed using LZ4.
    • best_compression - Stored values are compressed using DEFLATE. This reduces disk capacity requirements with the trade-off of slightly higher CPU utilization.
  • Default
    • best_compression
note

This setting configures the index template sent to Elasticsearch. It does NOT change any existing indices.

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE X-Pack#

If data is being stored to an Elasticsearch cluster with Index Lifecycle Management (ILM) features enabled, this setting species the name of the ILM Lifecycle that should be applied to the indices.

note

The ILM Lifecycle itself MUST be configured separately in Elasticsearch.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_ROLLOVER_ALIAS X-Pack#

If data is being stored to an Elasticsearch cluster with Index Lifecycle Management (ILM) features enabled, this setting species the name of the ILM Lifecycle Rollover Alias that should be applied to the indices.

note

The ILM Lifecycle itself MUST be configured separately in Elasticsearch.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ISM_POLICY Open Distro#

If data is being stored to an Open Distro for Elasticsearch cluster, this setting species Index State Management (ISM) Policy ID that should be applied to the indices.

note

The ISM Policy itself MUST be configured separately in Elasticsearch.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT#

If it is desireed to process the incoming with an Elasticsearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the default pipeline.

  • Default
    • _none

EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL#

If it is desireed to process the incoming with an Elasticsearch Ingest Pipeline prior to it being indexed, this setting specifies the name of the final pipeline.

  • Default
    • _none

EF_FLOW_OUTPUT_ELASTICSEARCH_ADDRESSES#

This setting specifies the Elasticsearch servers to which the output should connect. It is a comma-separated list of Elasticsearch nodes, including port number.

warning

Do NOT include http:// or https:// in the provided value. TLS communications is enabled/disabled using EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_ENABLE.

  • Default
    • 127.0.0.1:9200

EF_FLOW_OUTPUT_ELASTICSEARCH_USERNAME#

The username to use when connecting to Elasticsearch.

  • Default
    • elastic

EF_FLOW_OUTPUT_ELASTICSEARCH_PASSWORD#

The password to use when connecting to Elasticsearch.

  • Default
    • changeme

EF_FLOW_OUTPUT_ELASTICSEARCH_CLOUD_ID#

The URI for the Elastic Cloud endpoint to which the output should connect. If set, this value overrides EF_FLOW_OUTPUT_ELASTICSEARCH_ADDRESSES.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_API_KEY X-Pack#

The base64-encoded token to use for authorization.

Elasticsearch provides Security APIs to:

If set, this value overrides EF_FLOW_OUTPUT_ELASTICSEARCH_USERNAME and EF_FLOW_OUTPUT_ELASTICSEARCH_PASSWORD.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_ENABLE#

This setting is used to enable/disable TLS connections to Elasticsearch.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION#

This setting is used to enable/disable TLS verification of the Elasticsearch server to which the output is attempting to connect.

  • Valid Values
    • true, false
  • Default
    • false

EF_FLOW_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH#

The path to the Certificate Authority (CA) certificate to use for verification of the Elasticsearch server to which the output is attempting to connect.

  • Default
    • ''

EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_ENABLE#

Specifies whether to retry connecting to Elasticsearch after a connection has failed.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_ON_TIMEOUT_ENABLE#

Specifies whether to retry bulk indexing requests which have timed-out.

  • Valid Values
    • true, false
  • Default
    • true

EF_FLOW_OUTPUT_ELASTICSEARCH_MAX_RETRIES#

Specifies the number of times to retry bulk indexing requests which have timed-out.

  • Default
    • 3

EF_FLOW_OUTPUT_ELASTICSEARCH_RETRY_BACKOFF#

If set, this value specifies the quantity of milliseconds that the output should "backoff" prior to retrying a failed bulk request.

  • Default
    • 1000
Edit this page