Skip to main content
Version: 6.2

Changelog

Latest Version: 6.2.2

Release History​

6.2.2​

Fixes​

  • Flow Processor - Sample rates are properly applied to sFlow records when user-defined sample rates are enabled

Updates​

  • IPFIX IEs - Added new NetQuest packet counter IEs
  • IPFIX IEs - Added IPv6-related IEs from Sonicwall devices

6.2.1​

New Features​

  • Kafka Output: support for ECS - An option has been added to output records in Elastic Common Schema (ECS).

Fixes​

  • Ixia IPFIX IEs - Fixed an issue which caused a panic whenever IE 202 was present.

Updates​

  • IPFIX IEs - Expanded support for AMD/Pensando.

6.2.0​

Breaking Changes​

  • Telemetry Index Name Change - The telemetry index created by the Unified Flow Collector has been changed from elastiflow-telemetry-[schema]... to elastiflow-telemetry_flow-[schema].... The Unified SNMP Collector will index its data to elastiflow-telemetry_snmp-[schema].... Using separate indices allows the index mappings to be kept more manageable. Both indices can queried using a Data View/Index Pattern of elastiflow-telemetry_*-[schema]-*. You may need to modify any dashboards that you have created which use the previous index name. This can usually be achieved by exporting the relevant saved objects. Modifying the name via global replacement, and re-importing the objects.

New Features​

  • Kafka Output: optional flattened field names - An option has been added to use flattened, rather than nested, field names in the JSON records produced to Kafka.
  • Netflow/Ipfix Decoder: max records per packet - The maximum flow records allowed per packet is now configurable via the option EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET. This improves support for records sent over networks with an MTU greater than 1500, while still providing malformed packet detection.
  • Flow Benchmark Input - The flow benchmark input replays in a loop a variety of packets through the collector as if they were received from network devices. This allows the end-to-end performance of the environment to be evaluated, for both the collector and platform to which records are sent. This is very useful prior to the "go live" of a deployment to ensure that the expected volume of records can be handled.
  • Flow Evaluator - The flow evaluator (floweval) is a standalone tool to assess the volume of flow records being sent by network devices. It decodes enough of the incoming packets to count the number of flow records they contain and log the observed record rates.
  • API (formerly Metrics) Server - Added support for basic authentication to secure the API's HTTP Server.
  • Elasticsearch/OpenSearch Dashboards - Added new BGP AS-Hop and Graph dashboards.

Deprecations​

  • Default value of EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE - Beginning with ElastiFlow 6.3.0 the Elasticsearch output's default value for EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE will be changed to collect. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the current default setting of end, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
  • Default value of EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD - Beginning with ElastiFlow 6.3.0 the Elasticsearch output's default value for EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD will be changed to rollover. This will enable the use of Index Lifecycle Management (ILM) to manage retention of ElastiFlow indices. If you wish to continue to use the current default setting of daily, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
  • Default value of EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE - Beginning with ElastiFlow 6.3.0 the OpenSearch output's default value for EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE will be changed to collect. This will allow the collector to handle a wider variety of situations without additional configuration. If you wish to continue to use the current default setting of end, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
  • Kafka output default values - Performance testing has shown that the current default values can be modified for improved throughput. Beginning with ElastiFlow 6.3.0 the default values of various Kafka output configuration options will be changed as in the table below. If you wish to continue to use the current default settings, you should ensure that it is specifically set in your configuration prior to the release and deployment of 6.3.0.
Option6.2.x and earlierplanned for 6.3.0
EF_OUTPUT_KAFKA_PRODUCER_COMPRESSION0 (none)3 (LZ4)
EF_OUTPUT_KAFKA_PRODUCER_FLUSH_FREQUENCY500ms1000ms
EF_OUTPUT_KAFKA_FLAT_RECORD_ENABLEfalsetrue

Updates​

  • The "technology preview" of the SNMP polling capability has ended with the launch of the new ElastiFlow Unified SNMP Collector
  • IPFIX IEs - added support for NetQuest QUIC-related IEs.
  • IPFIX IEs - Expanded support for Ixia.
  • Logging - logs have been improved for improved structure and readability.
  • Packet Parser - added support for decoding MACSEC headers.
  • Elasticsearch Output - Bulk index errors returned from Elasticsearch/OpenSearch are now logged.
  • Kafka Output - producer pool has been improved for increased performance.

Fixes​

  • Application Enricher - A EF_PROCESSOR_ENRICH_APP_REFRESH_RATE value of 0 will no longer cause an error.
  • Packet Parser - Updated to verify the EtherType indicated IP version matches that in the IP header, as well as to validate the IP header size. This prevents packets from certain tunneled traffic protocols from causing a panic.

6.1.3​

Updates​

  • Added the EF_PROCESSOR_DECODE_MAX_RECORDS_PER_PACKET option. Corrupt packets can cause issues with the decoding of records. One way this is handled is by limiting the number of records that will be decoded from a packet. The default value is 64. When the network between the device and collector has an MTU larger than 1500, the default value may be exceeded by normal packets. This new configuration option allows the threshold to be increased when necessary.

Fixes​

  • App enrichment: Fixed an issue which caused the app enrichment YAML files to be continually reloaded. This could cause significantly increased CPU load.

6.1.2​

Updates​

  • SNMP Input: Added support for syntax values of EnumBitmap and EnumIntegerKeepID and EnumObjectIdentifierKeepOID.
  • SNMP Input: Added support for index values of type MacAddress.
  • SNMP Input: Updated SNMP object, object group and device group definitions.

Fixes​

  • Calix IPFIX: Fixed a regression introduced in 6.0.0 which caused calix.aid.type to no longer be populated.
  • Kafka Output: Fixed an issue where the default output worker pool size was not being properly set. This prevented the output from connecting to Kafka unless it was specifically configured.
  • NetQuest IPFIX: Corrected swapped src/dst values for BGP IEs.

6.1.1​

Fixes​

note

If you are using 6.0.0 to collect Netflow v9 records it is HIGHLY RECOMMENDED that you upgrade IMMEDIATELY to 6.1.1 to fix the issue described below.

  • Netflow v9: Fixed a regression introduced in 6.0.0 which could cause Netflow v9 flowsets to be decoded incorrectly.

Updates​

  • SNMP Input: Added support for syntax values of CounterBasedGauge64 and ZeroBasedCounter64 from HCNUM-TC.

6.1.0​

New Features​

  • TECHNOLOGY PREVIEW: We have added a new input for collecting metrics using SNMP. Please note that we will be adding device support over time. The initial out-of-the-box definitions can be found in a public GitHub repository at https://github.com/elastiflow/snmp, and are also included in the provided packages.

6.0.1​

Breaking Changes​

If you are migrating to 6.0.x from a previous version of the ElastiFlow Unified Collector, please see Breaking Changes for 6.0.0 below.

Fixes​

  • Fixed a panic condition when exporterIPv4Address or exporterIPv6Address was included in the flow record.
  • Fixed a panic condition that related to the license level check of sFlow records.

6.0.0​

Breaking Changes​

danger

IMPORTANT! In preparation for new features and solutions which will be available in the near future, many of the configuration option names have been changed since 5.6.x and 6.0.0-rc.1. It will be necessary to modify your previous configuration for 6.0.0. Please refer to Upgrading to 6.0.0 for more details. ElastiFlow customers can contact support for assistance with this upgrade. Community users can ask for assistance in the ElastiFlow Community Slack.

  • The JSON structure for records sent to Elasticsearch and OpenSearch has been flattened. This has no effect on the function of dashboards, and features such Elasticsearch ML jobs and alerts. It is also possible to seamlessly combine the 5.x (nested) and 6.0.0 (flat) indices. This is because Elasticsearch flattens field names for indexing. However if you have been extracting the raw record JSON from Elasticsearch to send to other applications, this change may affect such processes.
  • Removed Logz.io Output - We have decided not to proceed with the technology preview of the Logz.io output, and it has been removed. We may revisit support for Logz.io in the future.
  • Non-Flow Record Types - The addition of non-flow indices (see below) may require some user-created tools or processes to be modified to access these new indices.

New Features​

  • AWS VPC Flow Logs - AWS VPC Flow Logs are now supported via collection from S3. This includes support for all fields from VPC Flow Log versions 2 thru 5.
  • ElastiFlow Splunk App - The ElastiFlow Netflow Analytics for Splunk App is now available on Splunkbase. We will continue to update this app over the coming weeks and months.
  • Bi-Directional Flows - Bi-directional records, as sent by Velocloud SD-WAN and certain Cisco and other devices, are now split into two uni-directional records, enabling the full ElastiFlow feature set to be applied to both directions represented in the original record.
  • Performance Improvements - A variety of performance enhancements provide a throughput increase of up to 250% at the same CPU utilization.
  • Collector Statistics - A Prometheus endpoint provides statistics for various internal collector components.
  • Liveness & Readiness - Liveness and Readiness endpoints have been added to improve the ability to monitor the state of the collector when running in Kubernetes.
  • Graceful Shutdown - When the collector is stopped, any buffered messages will be processed prior to the collector exiting.
  • Improved Application Enrichment - User-defined Application enrichment now supports any combination of IP addresses, CIDR blocks, IP ranges, ports and port ranges. Additionally it is now possible to enrich records with more than only app.name, including user-defined metadata. Finally, vendor-specific AppID to App attributes mappings may be added for devices which send AppIDs without option records.
  • Non-Flow Record Types - The collector will now create separate indices for non-flow record types. Currently supported are flow, telemetry (such as sFlow counter samples and Calix IPFIX telemetry records), and AS-Path hop records.
  • Generic HTTP Output - We have added an output which can be used to send records to an HTTP endpoint such as the http_endpoint input of Elastic's Filebeat, or the http input of Elastic's Logstash.

Updates​

  • Added Extreme Networks IEs for userName and appGroupName.
  • Added packet parser support for TCP sequence (tcp.seq_num) and acknowledge numbers (tcp.ack_num).
  • Client/Server inference for protocols without layer-4 ports will now be based on IP order, where the lower IP address is the server. This improves the functionality of dashboards for many use-cases. The configuration option EF_ENRICH_EXPAND_CLISRV_NO_L4_PORTS can be used to disable this change.
  • The packetparser, used to decode sFlow, IFA, and other sampled headers, now provides the IPv6 Flow Label value.

Fixes​

  • The data type and translation for Calix the bin-duration IE has been fixed.