# How to set up storage optimization

#### Storage optimization for Elasticsearch and OpenSearch

The following table shows the minimum NetObserv versions required for storage optimization support across different NetObserv products:

| NetObserv Product | Supported Version |
| ----------------- | ----------------- |
| Flow              | 7.11              |
| SNMP              | 7.12              |
| SNMP Trap         | Unsupported       |

In 2023 we introduced TSDS (Time-series data streams) support for Elasticsearch, reducing the storage requirements to store NetObserv flow data by up to 70%. Even though enabling TSDS increased the CPU requirements on the ingest node slightly, it was quickly adopted by the majority of teams using NetObserv with Elastic.

With the release of 8.17 of Elastic, the *synthetic \_source* feature (which is an important part of how TSDS works) was moved out of the free version of Elastic and only available with an Elastic Enterprise license, leaving many ElastiFlow users without access to these storage savings.

With NetObserv 7.11 we’re not only bringing back the majority of the storage savings of TSDS, but many other benefits on top of that. While the storage savings are comparable to TSDS, this new storage optimization reduces query times significantly (up to 30% in our tests). And since *synthetic \_source* is not required, it is available to all Elastic users, no matter if they are on a free or a paid tier.

Storage optimization is also available for OpenSearch, bringing both reduced storage (up to 60%) and reduced query times (up to 30%) to all OpenSearch users.

#### How to get started with Storage Optimization

1. **If you are an OpenSearch user**\
   Just update to NetObserv 7.11 and see the storage savings begin to ramp up. Storage optimization is enabled by default. Please review the changelog for NetObserv 7.11 for more details.
2. **If you are an Elastic user and currently don’t have TSDS enabled**\
   The same things are true as outlined above. Storage optimization is enabled by default. Let the storage savings begin. Please review the changelog for NetObserv 7.11 for more details.
3. **If you are an Elastic user and currently have TSDS enabled**\
   Storage optimization is *not* enabled by default, but we still highly recommend you enable it to gain some extra storage savings, since storage optimization works even better when *synthetic \_source* is enable&#x64;*.* <mark style="color:purple;">Using</mark> <mark style="color:purple;"></mark>*<mark style="color:purple;">synthetic \_source</mark>* <mark style="color:purple;"></mark><mark style="color:purple;">in Elastic 8.17 or higher requires an Enterprise tier subscription.</mark>\
   \
   Here are the steps to enable both storage optimization and *synthetic \_source* after upgrading to 7.11 or higher:
   1. Stop your NetObserv instance.
   2. In Kibana, delete the existing ElastiFlow data streams
   3. In Kibana, delete the existing ElastiFlow index templates, as new ones will automatically be created once TSDS is disabled.
   4. Open flowcoll.yaml and set:
      1. `EF_OUTPUT_ELASTICSEARCH_TSDS_ENABLE` to `false`.
      2. EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_TEMPLATE\_SYNTHETIC\_SOURCE\_ENABLE to `true.`
   5. Restart your NetObserv instance.

#### What will Change

NetObserv already uses the API of Elasticsearch or OpenSearch to create indexes when NetObserv starts up.

With this storage optimization feature enabled, NetObserv Flow will now change how it configures indexes in the downstream data store so that it uses disk storage more efficiently.

Note: EF\_OUTPUT\_ELASTICSEARCH\_INDEX\_PERIOD will automatically be set to ‘rollover’ when storage optimization is enabled, no matter what is configured by a user. This is necessary for storage optimization to work properly.

#### When to Expect Improvements

Full storage savings will be realized after the rolled-over indexes are fully "segment merged" by the ILM/ISM policies. There will be storage savings when the index is being actively written to, however, not as much as after the final segment merge.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/additional-resources-reference-articles/faq/storage_optimization.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.