Skip to main content
Version: 5.6

Changelog

Latest Version: 5.6.1

Release History​

5.6.1​

Fixes​

  • The Kafka output will now attempt to reconnect after receiving an EOF error. This condition can occur after TLS certificates have been changed without restarting the Kafka brokers.

5.6.0​

Breaking Changes​

  • Elasticsearch and OpenSearch Outputs: The mappings for vendor-specific and less common standard fields is now dynamic rather than a fixed property. This greatly reduces the number of fields displayed in various places in Kibana when the fields are not actually populated in the underlying indices. This isn't really a "breaking change", but we wanted to call it out, as it may mean that you need to refresh the Kibana index pattern after these dynamically mapped fields first appear in the Elasticsearch/OpenSearch index.

New Features​

  • The parser for sFlow sampled_header flow samples has been replaced with our all-new packet parser. The new packet parser supports many more protocol headers and can more flexibly supports the various contents and orders of header structures. The most exciting enhancement is support for tunnel and encapsulation technologies such VXLAN, GRE, PPTP, 4in4, 4in6, 6in4 and 6in6. Tunnel and encapsulating headers are now assigned to their own objects, tunnel and encap, where the inner most headers and payload are assigned to the flow object. For example, prior to 5.6.0 the flow object would contain attributes from the VXLAN header, with no visibility into the traffic within the tunnel. In 5.6.0 the VXLAN header attributes would assigned to tunnel. The parser then continues decoding the packet, assigning the attributes of the tunneled traffic to flow.
info

The additional fields will increase the record size of sFlow records for tunneled or encapsulated flows.

  • Global and Namespace scoped output configurations have been introduced. Global scope, which has been the standard behavior of the collector, allows one instance of an output to be run. Namespaces allow for multiple instances of an output to be run. This is useful when it is necessary to send data to two separate platforms of the same type, e.g. two Elasticsearch clusters, with each having different configurations. See the output configuration documentation HERE for more details.

Updates​

  • Added new NetQuest Information Elements for DTLS, DNS, HTTP, BGP and SSH.
  • Added new Juniper Networks Information Elements for Inband Flow Analyzer (IFA).
note

IFA Metadata is provided only as a payload field. In a future release we will generate telemetry records from this payload to provide insights into hop-by-hop network latency.

Fixes​

  • Fixed a condition where network interface enrichment with attributes from flow option records could cause an extra index field to be added to the record.

5.5.2​

Fixes​

  • Fixed issue with Elasticsearch Output that caused EF_FLOW_OUTPUT_ELASTICSEARCH_DROP_FIELDS to be ignored.

5.5.1​

Fixes​

  • Fixed issue with OpenSearch Output that prevented successful authentication.
  • Fixed the issue that stops the collector when there is a failure creating an ILM Policy.

5.5.0​

New Features​

  • The Elasticsearch output will automatically bootstrap the initial write index and add the rollover alias when EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_PERIOD is set to rollover. If the ILM lifecycle policy configured in EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE (default is elastiflow) is not found in Elasticsearch, a default policy will be created which deletes data after 7 days. This policy can be changed later using the Kibana UI or Elasticsearch API.

Updates​

  • Log messages have been improved, including logging the configuration of each enabled output (sensitive field values are redacted).
  • The Elasticsearch output forces uses of TLS when a Cloud ID is configured.
  • Improved handling of Calix IEs.
  • Translation values for Cisco's FW_EXT_EVENT have been improved.

Fixes​

  • Fixed an issue with the Elasticsearch output that prevented some component templates from being loaded.
  • A bug that would cause an "expired license" error when using the Community License (which applies when no license key is provided in the configuration) has been fixed.

5.4.1​

Updates​

  • Added new Vmware Tanzu Antrea Information Elements.

Fixes​

  • Fixed authentication error upon inserting index template when using API keys for the Elasticsearch output.
  • Added fix to help relieve 503 errors with Elasticsearch and OpenSearch upon inserting component templates.
  • Added infinite retry upon failure of creating component templates.
  • Fixed a condition in which interface name enrichment was not working when only values from option records were available.

5.4.0​

Breaking Changes​

  • Various configuration options related to network interface enrichment have been changed. See the configuration documentation HERE for more details.
  • When sending data to Elasticsearch, OpenSearch or Logz.io the field flow.client.l4.port.id has been changed from a keyword to an integer, which is consistent with other port ID fields. As the 5.4.0 collector will write data to new indices (1.4 schema rather than 1.3) this will not create an issue indexing data. However querying flow.client.l4.port.id will result in an error unless the older data is first re-indexed to also convert this field. An explanation of the reindexing procedure can be found HERE. Customers can contact ElastiFlow support for assistance.

New Features​

  • Licensing model has been changed from "per core" to "per FPU (Flow Pack Unit)", where one FPU equals 4000 flows/s. Contact sales@elastiflow.com for more details.
    • The number of decoder "workers" is now independent of the license and can be configured using EF_FLOW_DECODER_POOL_SIZE. This allows the collector to be configured for greater concurrency resulting in better utilization of older multi-core hardware while also minimizing the throughput impact of high-latency enrichment (e.g. DNS reverse lookups). The default pool size is 4 x licensed units.
    • It is possible to burst over the licensed flow rate for up to 20 secs. After a burst period throughput will be limited to the licensed rate for 30 minutes.
  • User-Defined Metadata Enrichment for Network Interface. For more details see: User-Defined Metadata Enrichment.
  • Dedicated OpenSearch output. Previously the Elasticsearch output was used for both Elasticsearch and OpenSearch. There is now a dedicated output for OpenSearch which include only those configuration options specific to OpenSearch. The OpenSearch-related configuration options have been removed from the Elasticsearch output.
  • Cached network interface enrichment features for flow option data, SNMP and User-Defined Metadata have been combined into an all-new combined enrichment module.
    • The contents of the cache are now expired using a configurable time-to-live (TTL).
    • The enrichment features which read external files can reload those files, refreshing values, without having to restart the collector.
  • Dropped fields can now be configured per output (Cribl, Elasticsearch, Kafka, Splunk, OpenSearch, & Logzio) in addition to the ability to drop fields at the decoder level (i.e. globally for all outputs).
  • The sFlow decoder now parses ICMP headers to provide ICMP Type and ICMP Code.
  • New dashboards for Core Network Service Health (DNS, DHCP, RADIUS, LDAP and NTP) and Threat Hunting (DDoS TCP, DDoS Flood, RECON and Brute Force) are provided for Kibana, OpenSearch Dashboards, and Logz.io.
  • New Anomaly Detection jobs, 110 in total, are now available to automate the analysis of network traffic with Elastic's Machine Learning features. These jobs cover the areas of Availability, Performance and Network Security. More documentation for these ML jobs will follow.

Updates​

  • Added new NetQuest TLS/SSL Information Elements.
  • Added new Information Elements for Calix AXOS R21.x
  • The Elasticsearch output now uses Component Index Templates rather than legacy templates.
  • Decoder has been optimized to improve the way that it uses memory.

Fixes​

  • Fixed a possible panic condition when flushing cache entries that exceed the TTL.
  • As mention in Breaking Changes above, the field flow.client.l4.port.id has been changed from a keyword to an integer when sending data to Elasticsearch, OpenSearch or Logz.io.
  • An issue with handling application IDs larger than 8 bytes has been corrected.
  • Empty TCP Flags (value of 0) are no longer discarded for TCP flows, allowing null Attacks to be detected.

5.3.5​

Updates​

  • Updated naming of RADIUS services.

Fixes​

  • Fixed client/server scoring for DHCP traffic.
  • 3GPP timezone values as sent by Nokia equipment are now properly decoded.

5.3.4​

Fixes​

  • IP Enrichment: Final fix for a race condition which could occur when a very large number of IP addresses expired past the time-to-live (TTL) within the same cache maintenance interval.

5.3.3​

New Features​

  • Added support for additional option data types: Cisco TrustSec SGT names

Updates​

  • Client/Server inference more accurately identifies BitTorrent traffic

Fixes​

  • IP Enrichment: concurrency safety improvements

5.3.2​

Fixes​

  • IP Enrichment: Fixed a possible panic condition.

5.3.1​

Updates​

  • Cribl Output: Added TLS verification, and the option to specify a CA certificate.
  • Kafka Output: Added all config options to default files.

Fixes​

  • Splunk Output: fixed missing fields when CIM schema enabled.
  • Splunk Output: fixed a condition where the connection to the HEC could fail when TLS is enabled.
  • sFlow: Records with a sample_length that is not aligned on a 4-byte boundary no longer cause a parsing error.
  • sFlow: Incorrect IP header size no longer causes a panic.
  • IP Enrichment: A possible panic condition has been fixed.

5.3.0​

Breaking Changes​

  • Various configuration related to logging to file have been changed. See the configuration documentation HERE for more details.
  • The field device.vendor has been changed to device.vendor.name. If receiving flow records from Cisco AnyConnect NVM clients (nvzFlow) or some F5 sources this may cause a mapping conflict between pre-5.3.0 and 5.3.0 indices. Customers can contact ElastiFlow support for assistance.

New Features​

Updates​

  • Client/Server inference has been enhanced for better accuracy.
  • User-defined metadata and other IP address-related fields are fully sync'd between source/destination and client/server objects.
  • The Splunk HEC, Cribl and Logz.io outputs have been enhanced to support multiple endpoints. This provides better throughput and resiliency when writing to clustered endpoints. NOTE: the server and port are now combined into a single configuration option. Please refer to the output's documentation for more information.

Fixes​

  • The log to file mechanism has been completely replaced, fixing the issue of logs not being reliably written. Refer to the configuration changes related to this fix.
  • flow.bytes and flow.packets is now correctly set for flow records which include only initiator/responder bytes and packets.

5.2.1​

New Features​

  • TECHNICAL PREVIEW: Cribl LogStream output. The Cribl output can be used to send records to the Cribl LogStream HTTP/S (Bulk API).
  • Splunk Output: Added the option to output records in Splunk's Common Information Model (CIM).
  • Supported IEs: added NetQuest JA3-related fields.

Updates​

  • Kafka Output: Switch to async producer for greater throughput.
  • Splunk Output: Added TLS verification and the option specify a CA certificate.

Fixes​

  • Corrected a scenario where a TCP RST flag could cause inaccurate client/server inference.
  • Correctly handle zero sized variable-length IEs.
  • IP Address Enrichment: User-Defined metadata values assigned to a key which ends in .ip.addr are converted from strings to IP address type to prevent a panic condition.

5.2.0​

Breaking Changes​

  • Various configuration options related to the enrichment of IP addresses have changed. See the configuration documentation HERE for more details.

New Features​

  • User-Defined Metadata Enrichment for IP addresses. For more details see: User-Defined Metadata Enrichment
    • User-Defined Metadata Enrichment can be used to override IP address and hostname values to obfuscate individual IPs and hostnames where such privacy is required. An example is provided HERE (see the note).
  • Cached enrichment features for DNS/hostnames, Maxmind ASN and GeoIP, RiskIQ ASN and Threats, and User-Defined Metadata have been combined into an all-new combined enrichment module.
    • Hostname/DNS, RiskIQ Threat/IP Reputation and Maxmind GeoIP enrichment features can be scoped to a subset of IP addresses by specifying specific Autonomous Systems or CIDRs. For more details see: Scoping Enrichment with Include/Exclude
    • Better performance is achieved by fetching multiple enrichment attributes concurrently.
    • Cache maintenance tasks are handled asynchronously. This eliminates the throughput impact of cache purges, especially when a high number of IP addresses are cached.
    • The contents of the cache are now expired using a configurable time-to-live (TTL).
    • The enrichment features which read external files can reload those files, refreshing values, without having to restart the collector.
  • Kafka Output: topic name is now configurable by setting the EF_FLOW_OUTPUT_KAFKA_TOPIC option. The default value is elastiflow-flow-codex.
  • Kafka Output: now allows a partition key to be specified by setting EF_FLOW_OUTPUT_KAFKA_PARTITION_KEY. The default is set to flow.export.ip.addr.

Updates​

  • Splunk Output: now sends flattened key-value pairs rather than nested JSON objects.
  • Splunk Output: batching of records and retries for failed bulk requests have been improved.
  • If the autonomous system cannot be determined or enrichment is not enabled, the AS values will be set as PUBLIC or PRIVATE. This improves the function of various visualizations.
  • Updated from 1.15 to 1.17. Our benchmark tests show this provides a minor performance increase.

Fixes​

  • Kafka Output: settings have been modified to provide significantly better throughput (but there are more performance enhancements coming).
  • Splunk Output: it is now possible to connect to HEC port values below 1024, which is necessary to connect to Splunk Cloud.
  • RiskIQ ASN Enricher: the most specific AS is now returned, rather than the first match, which may have been an aggregate route.
  • Fixed an incorrect data type for Cisco SGT IDs.

5.1.10​

New Features​

info

The Splunk HEC and Kafka outputs are currently a technology preview. The design and implementation are less mature than stable features and subject to change.

Updates​

  • Logz.io output improved to more efficiently reuse HTTP sessions.

Fixes​

  • A condition has been fixed when file logging is enabled, that would cause a short write error under heavy load.

5.1.9​

New Features​

  • Network interface bandwidth (ifSpeed/ifHighSpeed) is now fetched via SNMP for enrichment.

Fixes​

  • A condition has been fixed where processing of certain option data may have caused a panic.

5.1.8​

New Features​

  • TECHNICAL PREVIEW: Logz.io output. Enables sending data to the Logz.io service via HTTP(S).
  • Elasticsearch output now supports PKI authentication.
info

The Logz.io output is currently a technology preview. The design and implementation are less mature than stable features and subject to change.

Updates​

  • Support for additional Gigamon HTTP-related fields.

5.1.7​

Breaking Changes​

  • BGP schemas have been updated. We do not expect the changes to cause any issues. However we want to mention it in case anyone is collecting BGP-related IEs and encounters any issues.

Updates​

  • Handling of sFlow the extended_gateway structure has been improved to provide AS path, communities and local preference.
  • AS details, used when processing ASNs received in a flow record, have been updated.
  • Additional Antrea IEs are now supported.

Fixes​

  • Handling of IPFIX and Netflow v9 option records has been updated to prevent a possible panic condition.
  • Fixed a situation where exporters IPs were not resolved unless private or public IP resolution was also enabled.
  • A condition that caused logs not to be written has been fixed.

5.1.6​

Fixes​

  • Elasticsearch output configuration options for which 0 is a valid value will no longer cause an invalid configuration error, which prevented the collector fro starting.

5.1.5​

Breaking Changes​

  • None

New Features​

  • The collector can now listen on multiple UDP ports. Simply set EF_FLOW_SERVER_UDP_PORT to a comma-separated list of ports, e.g. 2055,4739,6343.
  • In addition to the default behavior of using the host OS's name resolution to resolve IP addresses to hostnames. It is now possible to specify a nameserver using EF_FLOW_DECODER_ENRICH_DNS_NAMESERVER_IP.
  • Added support for Netflow v1, v6 and v7.
  • Added support for additional option data types: VRF names, Flow metering classes, and Cisco SD-WAN drop causes, TLOC attributes, Extended Firewall Events and Firewall Zone Pairs.

Fixes​

  • RiskIQ ASN enricher will now respect EF_FLOW_DECODER_ENRICH_ASN_PREF.
  • Fixed a possible panic related to corrupt packet detection where the entire payload must be discarded.
  • Fixed an issue where length field was ignored for sampled_ethernet, sampled_ipv4 and sampled_ipv6 structures, which caused incorrectly decoded records.
  • Fixed a bug in the decoding of Netflow v9 flowsets containing multiple option data templates. This fixes excessive "Template not yet received" errors related to option data records.
  • Fixed a condition where sample rates may not have been learned from IPFIX option data.

5.1.4​

Breaking Changes​

  • None

Updates​

  • sFlow parsing of UDP sampled headers supports VXLAN and VXLAN-GPE header attributes.
  • Interpretation of TCP Flags has been modified to improve inference of layer-4 session establishment and client/server sides of a flow.

Fixes​

  • initiator/responder bytes and packets are now properly adjusted based on sample rate.
  • Exporter IPs are now properly resolved to user-defined hostnames.
  • Default cache sizes have been increased to better handle environments with a very high number of unique IP addresses.

5.1.3​

Breaking Changes​

  • None

Fixes​

  • Fixed an issue with validation of configuration options which contain file paths.

5.1.2​

Breaking Changes​

  • None

Updates​

  • Added initial support for IPFIX records from Pensando.

Fixes​

  • Modified bad packet detection to prevent triggering when records contain a large number of valid fields.
  • Fixed condition that could cause short write errors when logging to files is enabled.
  • Fixed a possible panic when weekly log file rotation is used.
  • Timestamps in logs are now ISO8601 format.
  • Improved validation of configuration for UDP input and all outputs.

5.1.1​

Breaking Changes​

  • None

Updates​

  • sFlow sample_header decoding now supports PPPoE headers.
  • Logic for detecting and logging bad packets has been improved. In many scenarios the guilty PDU will be logged as well, to simplify troubleshooting and support.

Fixes​

  • DNS reverse lookup for exporter IPs now works when when only exporter lookups are enabled.
  • Juniper flows with truncated records no longer cause a panic.

5.1.0​

Breaking Changes​

  • If you are receiving flow records from Citrix Netscaler environments there have been a few minor updates to the netscaler-specific fields, which may cause field type conflicts. Users with an active support agreement should contact support@elastiflow.com to discuss migration options prior to upgrading to 5.1.0.

New Features​

  • The RiskIQ integration to enrich flow records with threat details and autonomous system attributes is now generally available, and can be used in large scale production environments.
  • The ability to configure index.lifecycle.rollover_alias has been added for the Elasticsearch output, when it is used with Elastic's X-Pack ILM rollover features. The configuration option is EF_FLOW_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_ROLLOVER_ALIAS.

Updates​

  • Support for additional IEs from the Antrea Kubernetes CNI have been added, and is current for Antrea v1.0.0.
  • Support for additional IEs from Citrix Netscaler has been added. This is the start of a larger Citrix Netscaler update.

v5.0.2​

Breaking Changes​

  • The default log file interval (EF_FLOW_LOGGER_FILE_LOG_INTERVAL) has been changed to daily. You may need to remove all existing log files from the log directory.

New Features​

  • Added the initial release of session establishment inference which populates the field l4.session.established session established field.
  • Jobs for Elastic's Machine Learning features are now provided in the elasitflow_for_elasticsearch repository.
  • License details are now logged when the collector starts.

Updates​

  • Added option to install via RPM package.

Fixes​

  • Fixed a scenario where flow.locality can be private and should be public.
  • Fix an issue with the AppID cache that could cause a panic.
  • Initiator/Responder bytes and packets value will now populate flow.bytes and flow.packets when in/out bytes and packets are not available (fixes as issue with flow records from Cisco ASA).
  • SNMP enrichment of interface attributes now supports the polled device responding with a different source IP.

v5.0.1​

Breaking Changes​

  • While the documentation explained that the RiskIQ output must be enabled whenever the enricher is enabled, this is now enforced properly at runtime. If EF_FLOW_DECODER_ENRICH_RISKIQ_ENABLE is true, EF_FLOW_OUTPUT_RISKIQ_ENABLE must also be true. EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_UUID and EF_FLOW_OUTPUT_RISKIQ_CUSTOMER_ENCRYPTION_KEY must also be set. If you were previously using the RiskIQ enrichment feature without the RiskIQ output, you must enable and configure the output, or disable enrichment.

New Features​

  • Added the option EF_FLOW_DECODER_ENRICH_RISKIQ_CACHE_FAILED_REQS which when set to true (the default) improves performance when the RiskIQ service is unavailable.

Updates​

  • Default value of EF_FLOW_DECODER_ENRICH_RISKIQ_API_TIMEOUT changed from 5 to 3.

Fixes​

  • Corrected handling of signed integers from sFlow counters.
  • When interface attributes are learned from option data, and no ifName value is provide, fallback to ifDescr now works correctly.

v5.0.0​

Breaking Changes​

  • The GA version of the collector now enforces the license. If no license key is used, the collector will run as a Community tier subscription. A Basic tier license can be requested from the ElastiFlow website, as we a 30-day trial of the full Premium tier features. The beta collector will no longer run after 31 March 2021. All users who requested access to the beta will be proactively sent a Basic tier license to the email address which they provided.

Fixes​

  • Fixed an issue with sFlow sample rate that caused bytes and packets to be overstated.
  • Improved rejection of malformed packets.
  • Fixed incorrect timestamps for some Netflow v9 Option Data.
  • Added RiskIQ IP reputation and routed block data to client and server objects.
  • Fixed decoding of microsecond and nanosecond timestamps.
  • Interface name enrichment will fallback to ifDescr when ifName is unavailable.
  • Improved handling of malformed (non-RFC6759 compliant) Application IDs
  • Fixed CA certificate validation for Elasticsearch output when TLS is enabled.
  • Improved normalization of bytes and packets when flow direction is unknown.

Updates​

  • ASN to Organization mappings have been updated.

v5.0.0-beta.3​

Breaking Changes​

  • Environment variables for configuring the flow collector have changed. Please review the installation documentation for a detailed description of all configuration options.
  • Elasticsearch Index names and templates have been changed to be schema-specific (CODEX vs ECS). This prevents some conflicts where schemas could be inadvertently mixed due to misconfiguration.
Migration Steps​

As we are still in a beta phase, we recommend the installation of v5.0.0-beta.3 in a clean environment that does not contain data of the v5.0.0-beta.2, v5.0.0-beta.1 or previous ElastiFlow (v4.x) installations.

Release Highlights​

  • Integration with RiskIQ PassiveTotal for enrichment and flow data analysis.
  • Expanded options for handling sampled flow records.
  • Ability to define Applications per IP address and port number.
  • Elasticsearch Output support for Index Management and Ingest Pipelines

New Features​

  • Integration with RiskIQ PassiveTotal.
  • Sampling rates can be learned from option data records.
  • Sampling rates can be statically defined per flow exporter IP address.
  • Applications can be statically defined per IP address and port number.
  • Hostnames for reverse IP lookups can be statically defined.
  • Elasticsearch Ingest Pipelines can now be specified.
  • An Elasticsearch ILM Lifecycle can now be specified.
  • Open Distro for Elasticsearch ISM Policy can be specified.
  • Logging to file with log rotation.

Updates​

  • Refactored reverse name lookups for better performance.

Fixes​

  • Fixed issue where flow.export.host.name was not being set using the sFlow agent IP address.
  • Fixed a condition where the timestamp was not normalized properly result in indices created in the past.
  • Fixed issue where Maxmind ASN and GeoIP cache sizes where not set as configured.
  • Enabling both the Elasticsearch and stdout outputs simultaneously no longer causes the collector to exit.

v5.0.0-beta.2​

Breaking Changes​

  • Environment variables for configuring unicolld have changed.
  • Changes to Elasticsearch templates.
Migration Steps​

As we are still in a beta environment, we recommend the installation of v5.0.0-beta.2 in a new environment that does not contain data of the v5.0.0-beta.1 or previous ElastiFlow (v4.x) installations.

Release Highlights​

  • TLS support on the Elasticsearch output.
  • ElastiFlow's documentation site.
  • Support for Elastic Common Schema (ECS).
  • General bugs and performance updates.

New Features​

  • Adds support for TLS configuration of Elasticsearch output.
  • Adds support for configuring the UDP server's kernel buffer size.
  • Adds support for enabling the ECS output on the Elasticsearch output.
  • Created ECS-based Kibana dashboards.

Updates​

  • Configuration changes for licensing and core allowances.

Fixes​

  • Populates *.host.name with IP when DNS disabled.
  • Ensures all timestamps are normalized.
  • Prevents error that occurred when the TCP header size is too small.
  • Prevents record duplication issue.