Changelog

Latest Version: 6.4.4

Release History

6.4.4

Fixes

• Elasticsearch and OpenSearch Outputs - Components templates have been added for Path and Telemetry indices for ECS fields that were not being indexed correctly.
• Elasticsearch and OpenSearch Outputs - Corrected the ECS schema issue where "agent.version" incorrectly displayed as 1.2.3.
• Support Bundle - Corrected an issue where support bundles were checking for license keys to be produced.

Updates

• IPFIX IEs - Updated Calix IEs for AXOS R24.x
• Configuration - To prevent configuration conflicts on new installations, /etc/systemd/system/flowcoll.service.d/flowcoll.conf is now empty. Please use /etc/elastiflow/flowcoll.yml to configure the collector. Note: This will not affect installations with configurations already made in /etc/systemd/system/flowcoll.service.d/flowcoll.conf.
• App IDs - Added support for Versa Networks App ID mappings. Mappings are provided in /etc/elastiflow/app/versa.yml
• AWS Transit Gateway support - Support was added for AWS Transit Gateway flow logs. For more information on how to set up and collect flow logs from Transit Gateway, please see AWS VPC Flow logs and Setting up Transit Gateway flow logs.
• Packaging - Added the ca-certificates package to the collector Docker image, since it is now a prerequisite dependency. For .deb and .rpm installs, refer to the linux installation documentation for more information on ensuring that ca-certificates is installed.

6.4.3

Fixes

• Fixed bug where checking for license expiration caused a system panic for community licenses.

6.4.2

Fixes

• Flow Processor: Juniper IFA - Added hop.src.ifa.device.id and hop.dst.ifa.device.id fields to IFA hop records.
• Flow Processor: Juniper IFA - Fixed an issue that caused flows from IFA records to be written to the wrong Elasticsearch/OpenSearch index when IFA metadata was disabled in the collector configuration.
• Elasticsearch and OpenSearch Outputs - Index templates update to include Geo fields for flow.export. (CODEX) and host. (ECS). These fields are also now included in CODEX to ECS conversion.
• Elasticsearch Output - Fixed a panic condition related to processing flows for ESP traffic when TSDS is enabled.

Updates

• Flow Processor: Juniper IFA - Refactored IFA hop records to facilitate better dashboards.
• App IDs - Updated Fortinet AppIDs in fortinet.yml
• IPFIX IEs - Added new VMware Antrea IEs
• IPFIX IEs - Added Juniper SSR (formerly 128 Technology) IEs
• IPFIX IEs - Added new Gigamon RADIUS IEs
• IPFIX IEs - Added Flowmon IE for VXLAN NVI
• IPFIX IEs - Added Allegro Packets IEs

Security

• Updated net to resolve CVE-2023-39325

6.4.1

Fixes

• Elasticsearch/OpenSearch Outputs - Fixed incorrect log for write index creation.
• Support Bundler - Fixed default directory path for config files.
• Fixed an issue which caused the collector to panic when the configuration file was not provided.

Updates

• IPFIX IEs - Added NetQuest OSPF-related IEs

6.4.0

New Features

• Elasticsearch Output: support for TSDS - TSDS output for Elasticsearch is now a fully supported feature and out of Technology Preview. Enabling Time Series Data Streams (TSDS), introduced in Elasticsearch 8.7, can result in storage savings of 50-70% depending on the content of flow records. Enabling TSDS does increase the ingest-related CPU load for Elasticsearch, which can be largely mitigated by the ingest CPU optimizations introduced in Elasticsearch 8.8. How to enable TSDS:
  1. In Kibana, delete the 3 existing ElastiFlow index templates, as new ones will automatically be created once TSDS is enabled.
  2. Stop your the Unified Flow Collector instance.
  3. Open flowcoll.conf and set EF_OUTPUT_ELASTICSEARCH_TSDS_ENABLE to true.
  4. Restart your the Unified Flow Collector instance.

info

Note: Enabling TSDS will not affect any existing data already in Elasticsearch. All dashboards will visualize data both before and after TSDS is enabled.

6.3.7

Updates

• Packaging - Sign the rpm package using a FIPS-compliant GPG key, and provide a FIPS-compliant GPG public key for package signature verification.

6.3.6

Fixes

• Flow Processor - Fixed a panic condition when translating MPLS Route Distinguisher values.
• Flow Processor - Fixed an issue which caused the VRF name not be saved from an option record in some scenarios.
• AWS VPC Flow Logs Input - Fixed a panic condition when setting skip tls verification

6.3.5

New Features

• Support Bundler - Added endpoint and command-line interface to retrieve a support bundle. Support Bundler will collect logs, configs, and metrics for troubleshooting or analysis. See Generating A Support Bundle for more details.

Updates

• OpenSearch Output - The OpenSearch output will automatically bootstrap the initial write index and add the rollover alias when EF_FLOW_OUTPUT_OPENSEARCH_INDEX_PERIOD is set to rollover. If the ISM policy configured in EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY (default is elastiflow) is not found in OpenSearch, a default policy will be created which deletes data after 7 days. This policy can be changed later using the OpenSearch Dashboards UI or OpenSearch API.
• AWS VPC Flow Logs Input - Added additional options to configure TLS when using user-provided certificates.

Fixes

• Metrics - Fixed an issue where the collector could panic due to mishandling the parsing of metrics.
• Elasticsearch Output - Fixed an issue with the upload of index templates when TSDS was enabled along with ECS.

Deprecations

• **Default value of EF_OUTPUT_OPENSEARCH_INDEX_PERIOD - In a future release, the OpenSearch output's default value for this setting will be changed to rollover. This will enable the use of Index State Management (ISM) to manage the retention of ElastiFlow indices. If you wish to continue to use the old default setting of daily, you should ensure that it is specifically set in your configuration.

6.3.4

Fixes

• Logger - Fixed an issue where the configuration options for logging are not recognized when using YAML for configuration. This resulted in the logs not being written.
• OpenSearch Output and Splunk Output - Fixed an issue which caused auto-scaling of the output worker pool not function properly. This could result in a reduction of throughput unless the pool size was set manually.
• 6.3.3 - Fixed an issue which prevented the collector from running on operating systems based on Debian 11 and earlier (e.g. Ubuntu 20.04).

6.3.3

Fixes

• Elasticsearch Output and OpenSearch Output - Fixed an issue with the index template for Path indices when ECS is enabled. This caused path hop records to be incorrectly indexed.

Updates

• Various security updates based on ElastiFlow's Static Application Security Testing (SAST) and Software Composition Analysis (SCA) scanning processes.
• Elasticsearch Output - Added the managed and managed_by attributes to the _meta section of the Index Templates. This allows Kibana to indicate that they are managed by an external process (The Unified Flow Collector) and not user-defined.

6.3.2

Fixes

• Elasticsearch Output - Fixed an issue related to index naming specific to partner-specific builds.

info

This change only affects behavior specific to certain ElastiFlow partners. Non-partner users and ElastiFlow customers are unaffected by these changes and can continue to use 6.3.1.

6.3.1

Fixes

• Flow Processor - Fixed an issue which caused Netscaler flow records to be incorrectly identified as telemetry.

Updates

• IPFIX IEs - Added NetQuest DTLS-related IEs

6.3.0

Breaking Changes

• Elasticsearch Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the Elasticsearch output have been changed as follows.

Option	Old Value	New Value
EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE	end	collect
EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD	daily	rollover

• Kafka Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the Kafka output have been changed as follows. Performance testing has shown that this change can improve throughput.

Option	Old Value	New Value
EF_OUTPUT_KAFKA_PRODUCER_COMPRESSION	0 (none)	3 (LZ4)
EF_OUTPUT_KAFKA_PRODUCER_FLUSH_FREQUENCY	1000	500
EF_OUTPUT_KAFKA_FLAT_RECORD_ENABLE	false	true
EF_OUTPUT_KAFKA_TIMESTAMP_SOURCE	end	collect

• OpenSearch Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the OpenSearch output have been changed as follows.

Option	Old Value	New Value
EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE	end	collect

New Features

• Elasticsearch Output: support for TSDS (TECHNOLOGY PREVIEW) - Support has been added to the Elasticsearch output for Time Series Data Streams (TSDS), introduced in Elasticsearch 8.7. Storing flow data using TSDS can result in a storage savings of 30-50% depending on the content of the flow records. TSDS also supports downsampling (initially for bytes and packets fields) which can result in even less storage capacity needed for historical data. Enabling TSDS does increase the ingest-related CPU load for Elasticsearch.
• OpenSearch Output: support for AWS Sig v4 - Support has been added for authentication via Sig v4. This is required when connecting to the AWS OpenSearch Serverless Service.
• Flow Processor: Juniper IFA - Support has been added for Juniper IFA records. The resulting IFA hop details are stored in the path index.
• YAML Configuration - The collector can now be configured via YAML files in addition to environment variables. The YAML file to be used can be specified using the -c or --config arguments. When both YAML and environment variables are set, environment variables will override the values from the YAML files.

Fixes

• Flow Processor - Fixed a regression introduced in 6.2.2 which caused sample rates learned from option records to be ignored.
• Flow Processor - Fixed an issues which can cause a panic when a Netflow v9 packet contains excessive padding.
• Elasticsearch Output - Telemetry index templates are now created with the correct rollover alias.
• IPFIX IEs - Fixed Ixia AppID/Name values.
• HTTP-based Outputs - All HTTP-based outputs now set the Host header, as is required by some environments.

Updates

• Flow UDP Input - Added 2055, 4739 and 6343 to default ports on which the input will listen.
• Flow Processor - Unsupported PEN-specific sFlow structures are now gracefully ignored, rather than rejecting the entire record.
• Flow Processor - Enrichment of network interface index values now supports SNMPv3.
• Flow Processor - Added ntop nDPI AppIDs to statically defined attribute values.
• Flow Processor - Added Viptela AppIDs to statically defined attribute values.
• IPFIX IEs - Added Versa Networks IEs
• IPFIX IEs - Added NetQuest SIP-related IEs
• IPFIX IEs - Added Ixia GTP-related IEs

Deprecations

• While we have added support for configuration via YAML files in 6.3.0, the default method of configuration remains the use of environment variables set in the systemd unit file for the collector daemon. For example, /etc/systemd/system/flowcoll.service.d/flowcoll.conf for the Unified Flow Collector binary flowcoll.

  In a future release, the default configuration method will be via YAML files, as described here.

6.2.2

Fixes

• Flow Processor - Sample rates are properly applied to sFlow records when user-defined sample rates are enabled

Updates

• IPFIX IEs - Added new NetQuest packet counter IEs
• IPFIX IEs - Added IPv6-related IEs from Sonicwall devices

6.2.1

New Features

• Kafka Output: support for ECS - An option has been added to output records in Elastic Common Schema (ECS).

Fixes

• Ixia IPFIX IEs - Fixed an issue which caused a panic whenever IE 202 was present.

Updates

• IPFIX IEs - Expanded support for AMD/Pensando.

6.2.0

Breaking Changes

• Telemetry Index Name Change - The telemetry index created by the Unified Flow Collector has been changed from elastiflow-telemetry-[schema]... to elastiflow-telemetry_flow-[schema].... The Unified SNMP Collector will index its data to elastiflow-telemetry_snmp-[schema].... Using separate indices allows the index mappings to be kept more manageable. Both indices can queried using a Data View/Index Pattern of elastiflow-telemetry_*-[schema]-*. You may need to modify any dashboards that you have created which use the previous index name. This can usually be achieved by exporting the relevant saved objects. Modifying the name via global replacement, and re-importing the objects.

New Features

• Kafka Output: optional flattened field names - An option has been added to use flattened, rather than nested, field names in the JSON records produced to Kafka.
• Netflow/Ipfix Decoder: max records per packet - The maximum flow records allowed per packet is now configurable via the option EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET. This improves support for records sent over networks with an MTU greater than 1500, while still providing malformed packet detection.
• Flow Benchmark Input - The flow benchmark input replays in a loop a variety of packets through the collector as if they were received from network devices. This allows the end-to-end performance of the environment to be evaluated, for both the collector and platform to which records are sent. This is very useful prior to the "go live" of a deployment to ensure that the expected volume of records can be handled.
• Flow Evaluator - The flow evaluator (floweval) is a standalone tool to assess the volume of flow records being sent by network devices. It decodes enough of the incoming packets to count the number of flow records they contain and log the observed record rates.
• API (formerly Metrics) Server - Added support for basic authentication to secure the API's HTTP Server.
• Elasticsearch/OpenSearch Dashboards - Added new BGP AS-Hop and Graph dashboards.

Deprecations

• Default value of EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE - Beginning with ElastiFlow 6.3.0 the Elasticsearch output's default value for EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE will be changed to collect. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the current default setting of end, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
• Default value of EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD - Beginning with ElastiFlow 6.3.0 the Elasticsearch output's default value for EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD will be changed to rollover. This will enable the use of Index Lifecycle Management (ILM) to manage retention of ElastiFlow indices. If you wish to continue to use the current default setting of daily, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
• Default value of EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE - Beginning with ElastiFlow 6.3.0 the OpenSearch output's default value for EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE will be changed to collect. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the current default setting of end, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
• Kafka output default values - Performance testing has shown that the current default values can be modified for improved throughput. Beginning with ElastiFlow 6.3.0 the default values of various Kafka output configuration options will be changed as in the table below. If you wish to continue to use the current default settings, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.

Option	6.2.x and earlier	planned for 6.3.0
EF_OUTPUT_KAFKA_PRODUCER_COMPRESSION	0 (none)	3 (LZ4)
EF_OUTPUT_KAFKA_PRODUCER_FLUSH_FREQUENCY	500ms	1000ms
EF_OUTPUT_KAFKA_FLAT_RECORD_ENABLE	false	true

Updates

• The "technology preview" of the SNMP polling capability has ended with the launch of the new ElastiFlow Unified SNMP Collector
• IPFIX IEs - added support for NetQuest QUIC-related IEs.
• IPFIX IEs - Expanded support for Ixia.
• Logging - logs have been improved for improved structure and readability.
• Packet Parser - added support for decoding MACSEC headers.
• Elasticsearch Output - Bulk index errors returned from Elasticsearch/OpenSearch are now logged.
• Kafka Output - producer pool has been improved for increased performance.

Fixes

• Application Enricher - A EF_PROCESSOR_ENRICH_APP_REFRESH_RATE value of 0 will no longer cause an error.
• Packet Parser - Updated to verify the EtherType indicated IP version matches that in the IP header, as well as to validate the IP header size. This prevents packets from certain tunneled traffic protocols from causing a panic.

6.1.3

Updates

• Added the EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET option. Corrupt packets can cause issues with the decoding of records. One way this is handled is by limiting the number of records that will be decoded from a packet. The default value is 64. When the network between the device and collector has an MTU larger than 1500, the default value may be exceeded by normal packets. This new configuration option allows the threshold to be increased when necessary.

Fixes

• App enrichment: Fixed an issue which caused the app enrichment YAML files to be continually reloaded. This could cause significantly increased CPU load.

6.1.2

Updates

• SNMP Input: Added support for syntax values of EnumBitmap and EnumIntegerKeepID and EnumObjectIdentifierKeepOID.
• SNMP Input: Added support for index values of type MacAddress.
• SNMP Input: Updated SNMP object, object group and device group definitions.

Fixes

• Calix IPFIX: Fixed a regression introduced in 6.0.0 which caused calix.aid.type to no longer be populated.
• Kafka Output: Fixed an issue where the default output worker pool size was not being properly set. This prevented the output from connecting to Kafka unless it was specifically configured.
• NetQuest IPFIX: Corrected swapped src/dst values for BGP IEs.

6.1.1

Fixes

note

If you are using 6.0.0 to collect Netflow v9 records it is HIGHLY RECOMMENDED that you upgrade IMMEDIATELY to 6.1.1 to fix the issue described below.

• Netflow v9: Fixed a regression introduced in 6.0.0 which could cause Netflow v9 flowsets to be decoded incorrectly.

Updates

• SNMP Input: Added support for syntax values of CounterBasedGauge64 and ZeroBasedCounter64 from HCNUM-TC.

6.1.0

New Features

• TECHNOLOGY PREVIEW: We have added a new input for collecting metrics using SNMP. Please note that we will be adding device support over time. The initial out-of-the-box definitions can be found in a public GitHub repository at https://github.com/elastiflow/snmp, and are also included in the provided packages.

6.0.1

Breaking Changes

If you are migrating to 6.0.x from a previous version of the ElastiFlow Unified Collector, please see Breaking Changes for 6.0.0 below.

Fixes

• Fixed a panic condition when exporterIPv4Address or exporterIPv6Address was included in the flow record.
• Fixed a panic condition that related to the license level check of sFlow records.

6.0.0

Breaking Changes

danger

IMPORTANT! In preparation for new features and solutions which will be available in the near future, many of the configuration option names have been changed since 5.6.x and 6.0.0-rc.1. It will be necessary to modify your previous configuration for 6.0.0. Please refer to Upgrading to 6.0.0 for more details. ElastiFlow customers can contact support for assistance with this upgrade. Community users can ask for assistance in the ElastiFlow Community Slack.

• The JSON structure for records sent to Elasticsearch and OpenSearch has been flattened. This has no effect on the function of dashboards, and features such Elasticsearch ML jobs and alerts. It is also possible to seamlessly combine the 5.x (nested) and 6.0.0 (flat) indices. This is because Elasticsearch flattens field names for indexing. However if you have been extracting the raw record JSON from Elasticsearch to send to other applications, this change may affect such processes.
• Removed Logz.io Output - We have decided not to proceed with the technology preview of the Logz.io output, and it has been removed. We may revisit support for Logz.io in the future.
• Non-Flow Record Types - The addition of non-flow indices (see below) may require some user-created tools or processes to be modified to access these new indices.

New Features

• AWS VPC Flow Logs - AWS VPC Flow Logs are now supported via collection from S3. This includes support for all fields from VPC Flow Log versions 2 thru 5.
• ElastiFlow Splunk App - The ElastiFlow Netflow Analytics for Splunk App is now available on Splunkbase. We will continue to update this app over the coming weeks and months.
• Bi-Directional Flows - Bi-directional records, as sent by Velocloud SD-WAN and certain Cisco and other devices, are now split into two uni-directional records, enabling the full ElastiFlow feature set to be applied to both directions represented in the original record.
• Performance Improvements - A variety of performance enhancements provide a throughput increase of up to 250% at the same CPU utilization.
• Collector Statistics - A Prometheus endpoint provides statistics for various internal collector components.
• Liveness & Readiness - Liveness and Readiness endpoints have been added to improve the ability to monitor the state of the collector when running in Kubernetes.
• Graceful Shutdown - When the collector is stopped, any buffered messages will be processed prior to the collector exiting.
• Improved Application Enrichment - User-defined Application enrichment now supports any combination of IP addresses, CIDR blocks, IP ranges, ports and port ranges. Additionally it is now possible to enrich records with more than only app.name, including user-defined metadata. Finally, vendor-specific AppID to App attributes mappings may be added for devices which send AppIDs without option records.
• Non-Flow Record Types - The collector will now create separate indices for non-flow record types. Currently supported are flow, telemetry (such as sFlow counter samples and Calix IPFIX telemetry records), and AS-Path hop records.
• Generic HTTP Output - We have added an output which can be used to send records to an HTTP endpoint such as the http_endpoint input of Elastic's Filebeat, or the http input of Elastic's Logstash.

Updates

• Added Extreme Networks IEs for userName and appGroupName.
• Added packet parser support for TCP sequence (tcp.seq_num) and acknowledge numbers (tcp.ack_num).
• Client/Server inference for protocols without layer-4 ports will now be based on IP order, where the lower IP address is the server. This improves the functionality of dashboards for many use-cases. The configuration option EF_ENRICH_EXPAND_CLISRV_NO_L4_PORTS can be used to disable this change.
• The packetparser, used to decode sFlow, IFA, and other sampled headers, now provides the IPv6 Flow Label value.

Fixes

• The data type and translation for Calix the bin-duration IE has been fixed.