Due to the changes made in the method of enriching network interfaces, it is necessary to modify your collector's configuration when upgrading from 5.3.x and earlier to 5.4. The following configuration changes should be reviewed and the relevant guidance followed.
|Option||Status||Notes for 5.3|
|EF_FLOW_DECODER_ENRICH_SNMP_COMMUNITY||✕||REMOVED. Replaced by |
|EF_FLOW_DECODER_ENRICH_NETIF_METADATA_ENABLE||NEW||Added in 5.4|
|EF_FLOW_DECODER_ENRICH_NETIF_METADATA_USERDEF_PATH||NEW||Added in 5.4|
|EF_FLOW_DECODER_ENRICH_NETIF_METADATA_REFRESH_RATE||NEW||Added in 5.4|
|EF_FLOW_DECODER_ENRICH_NETIF_SNMP_COMMUNITIES||NEW||Added in 5.4|
When sending data to Elasticsearch, OpenSearch or Logz.io using the CODEX schema the field
flow.client.l4.port.id has been changed from a
keyword to an
integer, which is consistent with other port ID fields. As the 5.4.0 collector will write data to new indices (
1.4 schema rather than
1.3) this will not create an issue indexing data. However querying
flow.client.l4.port.id will result in errors unless the older data is first reindexed to also convert this field. Customers can contact ElastiFlow support for assistance.
While reindexing data is a safe process, it can be resource intensive. Special attention should be paid to the available storage capacity, as you will need space to store the new index prior deleting the existing data. It is recommended that you reindex your data ONLY if you experience query errors.
To correctly create the new indices, the new index template must already be available in Elasticsearch. The
5.4 collector will upload the updated index template. So you should NOT begin reindex data until AFTER you are successfully ingesting data via the
_reindex API is used to reindex indices. For example:
You can then proceed to reindex each index one-by-one. After an existing index has been reindexed. The old index can be deleted:
The official Elasticsearch documentation describes the process to reindex and ILM managed index. A summary of the steps follow:
Bootstrap an initial write index. If you manually setup rollover for pre-5.4 indices. You will need to do the same for 5.4 indices. The steps to manually setup ILM rollover for ElastiFlow can be found HERE.
Reduce the ILM poll interval to ensure that the index doesn’t grow too large while waiting for the rollover check. By default, ILM checks to see what actions need to be taken every 10 minutes.
- Reindex the data using the
_reindexAPI. We recommend reindexing each index one at a time.
When reindexing ILM-managed rollover indices, the destination index MUST be the rollover alias as seen in the above example.
- When reindexing is complete, set the ILM poll interval back to its default value to prevent unnecessary load on the master node.
- Once you have verified that all of the reindexed data is available in the new managed indices, you can safely remove the old indices.
After the old indices have been reindexed and removed the field type conflict will be resolved.