Skip to main content
Version: 5.6

Upgrading to 5.4

Configuration Changes​

Due to the changes made in the method of enriching network interfaces, it is necessary to modify your collector's configuration when upgrading from 5.3.x and earlier to 5.4. The following configuration changes should be reviewed and the relevant guidance followed.

note

If upgrading from 5.2.x or earlier please also review the configuration changes introduced in 5.2.0 and configuration changes introduced in 5.3.0.

Network Interface Enrichment Options​

OptionStatusNotes for 5.3
EF_FLOW_DECODER_ENRICH_NETIF_GET_ATTRSRENAMEDEF_FLOW_DECODER_ENRICH_NETIF_FLOW_OPTIONS_ENABLE
EF_FLOW_DECODER_ENRICH_NETIF_CACHE_SIZE✕REMOVED.
EF_FLOW_DECODER_ENRICH_SNMP_ENABLERENAMEDEF_FLOW_DECODER_ENRICH_NETIF_SNMP_ENABLE
EF_FLOW_DECODER_ENRICH_SNMP_PORTRENAMEDEF_FLOW_DECODER_ENRICH_NETIF_SNMP_PORT
EF_FLOW_DECODER_ENRICH_SNMP_VERSIONRENAMEDEF_FLOW_DECODER_ENRICH_NETIF_SNMP_VERSION
EF_FLOW_DECODER_ENRICH_SNMP_COMMUNITY✕REMOVED. Replaced by EF_FLOW_DECODER_ENRICH_NETIF_SNMP_COMMUNITIES
EF_FLOW_DECODER_ENRICH_SNMP_TIMEOUTRENAMEDEF_FLOW_DECODER_ENRICH_NETIF_SNMP_TIMEOUT
EF_FLOW_DECODER_ENRICH_SNMP_RETRIESRENAMEDEF_FLOW_DECODER_ENRICH_NETIF_SNMP_RETRIES
EF_FLOW_DECODER_ENRICH_NETIF_METADATA_ENABLENEWAdded in 5.4
EF_FLOW_DECODER_ENRICH_NETIF_METADATA_USERDEF_PATHNEWAdded in 5.4
EF_FLOW_DECODER_ENRICH_NETIF_METADATA_REFRESH_RATENEWAdded in 5.4
EF_FLOW_DECODER_ENRICH_NETIF_SNMP_COMMUNITIESNEWAdded in 5.4

Reindexing Pre-5.4 Indices​

When sending data to Elasticsearch, OpenSearch or Logz.io using the CODEX schema the field flow.client.l4.port.id has been changed from a keyword to an integer, which is consistent with other port ID fields. As the 5.4.0 collector will write data to new indices (1.4 schema rather than 1.3) this will not create an issue indexing data. However querying flow.client.l4.port.id will result in errors unless the older data is first reindexed to also convert this field. Customers can contact ElastiFlow support for assistance.

field type conflict

danger

While reindexing data is a safe process, it can be resource intensive. Special attention should be paid to the available storage capacity, as you will need space to store the new index prior deleting the existing data. It is recommended that you reindex your data ONLY if you experience query errors.

Reindexing​

info

To correctly create the new indices, the new index template must already be available in Elasticsearch. The 5.4 collector will upload the updated index template. So you should NOT begin reindex data until AFTER you are successfully ingesting data via the 5.4 collector.

The _reindex API is used to reindex indices. For example:

POST _reindex
{
  "source": {
    "index": "elastiflow-flow-codex-1.3-2022.04.22"
  },
  "dest": {
    "index": "elastiflow-flow-codex-1.4-2022.04.22"
  }
}

You can then proceed to reindex each index one-by-one. After an existing index has been reindexed. The old index can be deleted:

DELETE elastiflow-flow-codex-1.3-2022.04.22

Reindexing When Using ILM Rollover​

The official Elasticsearch documentation describes the process to reindex and ILM managed index. A summary of the steps follow:

  1. Bootstrap an initial write index. If you manually setup rollover for pre-5.4 indices. You will need to do the same for 5.4 indices. The steps to manually setup ILM rollover for ElastiFlow can be found HERE.

  2. Reduce the ILM poll interval to ensure that the index doesn’t grow too large while waiting for the rollover check. By default, ILM checks to see what actions need to be taken every 10 minutes.

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "1m"
  }
}
  1. Reindex the data using the _reindex API. We recommend reindexing each index one at a time.
POST _reindex
{
  "source": {
    "index": "elastiflow-flow-codex-1.3-rollover-000001"
  },
  "dest": {
    "index": "elastiflow-flow-codex-1.4-rollover"
  }
}
info

When reindexing ILM-managed rollover indices, the destination index MUST be the rollover alias as seen in the above example.

  1. When reindexing is complete, set the ILM poll interval back to its default value to prevent unnecessary load on the master node.
PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": null
  }
}
  1. Once you have verified that all of the reindexed data is available in the new managed indices, you can safely remove the old indices.
DELETE elastiflow-flow-codex-1.3-rollover-000001

After the old indices have been reindexed and removed the field type conflict will be resolved.

field type conflict resolved