Skip to main content
Version: 7.11

How to set up storage optimzation

NetObserv 7.11 introduces storage optimization for Elasticsearch and OpenSearch​

In 2023 we introduced TSDS (Time-series data streams) support for Elasticsearch, reducing the storage requirements to store NetObserv flow data by up to 70%. Even though enabling TSDS increased the CPU requirements on the ingest node slightly, it was quickly adopted by the majority of teams using NetObserv with Elastic.

With the release of 8.17 of Elastic, the synthetic _source feature (which is an important part of how TSDS works) was moved out of the free version of Elastic and only available with an Elastic Enterprise license, leaving many ElastiFlow users without access to these storage savings.

With NetObserv 7.11 we’re not only bringing back the majority of the storage savings of TSDS, but many other benefits on top of that. While the storage savings are comparable to TSDS, this new storage optimization reduces query times significantly (up to 30% in our tests). And since synthetic _source is not required, it is available to all Elastic users, no matter if they are on a free or a paid tier.

Storage optimization is also available for OpenSearch, bringing both reduced storage (up to 60%) and reduced query times (up to 30%) to all OpenSearch users.

How to get started with Storage Optimization​

First of all, storage optimization is only available for NetObserv Flow right now, but storage optimization for NetObserv SNMP (both polling and traps) will be added at a later time.

  1. If you are an OpenSearch user
    Just update to NetObserv 7.11 and see the storage savings begin to ramp up. Storage optimization is enabled by default. Please review the changelog for NetObserv 7.11 for more details.

  2. If you are an Elastic user and currently don’t have TSDS enabled
    The same things are true as outlined above. Storage optimization is enabled by default. Let the storage savings begin. Please review the changelog for NetObserv 7.11 for more details.

  3. If you are an Elastic user and currently have TSDS enabled
    Storage optimization is not enabled by default, but we still highly recommend you enable it to gain some extra storage savings, since storage optimization works even better when synthetic _source is enabled.
    Here are the steps to enable both storage optimization and synthetic _source after upgrading to 7.11 or higher:

    1. Stop your NetObserv instance.
    2. In Kibana, delete the existing ElastiFlow data streams
    3. In Kibana, delete the existing ElastiFlow index templates, as new ones will automatically be created once TSDS is disabled.
    4. Open flowcoll.yaml and set:
      1. EF_OUTPUT_ELASTICSEARCH_TSDS_ENABLE to false.
      2. EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SYNTHETIC_SOURCE_ENABLE to true.
    5. Restart your NetObserv instance.

What will Change​

NetObserv already uses the API of Elasticsearch or OpenSearch to create indexes when NetObserv starts up.

With this storage optimization feature enabled, NetObserv Flow will now change how it configures indexes in the downstream data store so that it uses disk storage more efficiently.

Note: EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD will automatically be set to ‘rollover’ when storage optimization is enabled, no matter what is configured by a user. This is necessary for storage optimization to work properly.

When to Expect Improvements​

Full storage savings will be realized after the rolled-over indexes are fully "segment merged" by the ILM/ISM policies. There will be storage savings when the index is being actively written to, however, not as much as after the final segment merge.