# Elasticsearch The Elasticsearch output can be used to send records to Elasticsearch and Elastic Cloud. ### Download Kibana Objects The Kibana dashboards and related configuration artifacts can be easily imported. You must first download the relevant import file, depending on the configuration of your environment. #### Kibana Objects for Network Flow Data | Version | Schema | Saved Objects | | ---------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 8.14.0 and later | CODEX | [kibana-8.14.x-flow-codex.ndjson](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-8.14.x-flow-codex.ndjson) | | 8.14.0 and later | ECS | [kibana-8.14.x-flow-ecs.ndjson](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-8.14.x-flow-ecs.ndjson) | | 8.2.0 - 8.13.x | CODEX | [kibana-8.2.x-flow-codex.ndjson](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-8.2.x-flow-codex.ndjson) | | 8.2.0 - 8.13.x | ECS | [kibana-8.2.x-flow-ecs.ndjson](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-8.2.x-flow-ecs.ndjson) | | 7.17.x - 8.1.x (unsupported) | CODEX | [kibana-7.17.x-flow-codex.ndjson (unsupported)](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-7.17.x-flow-codex.ndjson) | | 7.17.x - 8.1.x (unsupported) | ECS | [kibana-7.17.x-flow-ecs.ndjson (unsupported)](https://raw.githubusercontent.com/elastiflow/elastiflow_for_elasticsearch/master/kibana/flow/kibana-7.17.x-flow-ecs.ndjson) | ### Import Kibana Objects #### Importing via the User Interface To import the configuration, in Kibana go to *Stack Management* --> *Saved Objects* and click *Import* in the upper right corner. ![Saved Objects Before](https://user-images.githubusercontent.com/10326954/109737687-c132b900-7bc6-11eb-8ec7-843e8d48dcdc.png) A sidebar will appear. Again click *Import* at the top of the sidebar. ![Import](https://user-images.githubusercontent.com/10326954/109737831-05be5480-7bc7-11eb-942f-33e3958ee44b.png) Select the file which you downloaded, and click the *Import* button at the bottom of the sidebar. The configuration will be imported, and you will see all of the imported objects. ![Imported](https://user-images.githubusercontent.com/10326954/109738035-651c6480-7bc7-11eb-929d-d42ce9263c1d.png) Close the sidebar. You will also see all of the imported objects in the *Saved Objects* list. ![Saved Objects After](https://user-images.githubusercontent.com/10326954/109738146-972dc680-7bc7-11eb-946c-156799f3487e.png) #### Importing via the API ```shell curl -XPOST "https://username:password@IPORHOSTOFKIBANA:5601/api/saved_objects/_import?overwrite=true" -k -H "kbn-xsrf: true" -H "securitytenant: global" --form file=@kibana-7.14.x-ecs-light.ndjson ``` #### Troubleshooting Import Problems When the Kibana import fails, neither the import UI nor the Kibana logs will provide any useful information. However, attempting to import via `curl` will usually provide more detail. The most common issue is related to the relative large size of the Kibana Saved Objects file. This can cause the import to fail unless the maximum allowed payload size is increased. The Kibana setting is `server.maxPayloadBytes` (or `SERVER_MAXPAYLOADBYTES` if using Docker), which should also be set to `8388608`. If you have a reverse proxy in front of Kibana, you may have to modify your proxy settings as well. Many reverse proxies also have relatively low values for the maximum body size. For example, the NGINX default is only 1MB. This can be increased by setting `client_max_body_size` to `8388608`. Refer to the documentation for your proxy software to similarly modify its behavior. ### Recommended Kibana Advanced Settings #### Settings for All Kibana Distributions You may find that modifying a few of the Kibana advanced settings will produce a more user-friendly experience while using ElastiFlow. These settings are made in Kibana, under `Stack Management -> Kibana -> Advanced Settings`. | Advanced Setting | Value | Why make the change? | | ----------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | filters:pinnedByDefault | `true` | Pinning a filter allows it to persist when you are changing dashboards. This is very useful when drilling-down into something of interest and you want to change dashboards for a different perspective of the same data. This is the #1 setting we recommend changing. | | defaultRoute | *see description* | If your primary or only use-case for Kibana is ElastiFlow, set this the URL path for the dashboard to which you which to load immediately after logging in, or when returning to "home". The format of this value is `/app/dashboards#/view/4a608bc0-3d3e-11eb-bc2c-c5758316d788`. | | doc\_table:highlight | `false` | There be a query performance penalty that comes with using the highlighting feature. As it isn't very useful for this use-case, it is better to just turn it off. | | state:storeInSessionStorage | `true` | Kibana URLs can get pretty large. Especially when working with Vega visualizations. This will likely result in error messages for users of Internet Explorer. Using in-session storage will fix this issue for these users. | | theme:darkMode | `true` or `false` | Enable dark mode for the Kibana UI. This setting should match the `ndjson` import file discussed above. | | timepicker:timeDefaults | *see below* | The Time Picker Quick Range to use when Kibana is started without one. | | timepicker:quickRanges | *see below* | The default options in the Time Picker are less than optimal, for most logging and monitoring use-cases. Fortunately Kibana now allows you to customize the time picker. Our recommended settings can be found below. | | format:number:defaultPattern | `0,0.[00]` | Default numeral format for the "number" format. | | format:percent:defaultPattern | `0,0.[00]%` | Default numeral format for the "percent" format. | **Recommended Time Picker Time Defaults (timepicker:timeDefaults)** We find that the following Time Picker Time Default provides more useful views of the data for network flow-related use-cases. ```json { "from": "now-1h/m", "to": "now" } ``` **Recommended Time Picker Quick Ranges (timepicker:quickRanges)** We find that the following set of Time Picker Quick Ranges provides more useful views of the data for network flow-related use-cases. ```json [ { "from": "now-15m/m", "to": "now/m", "display": "Last 15 minutes" }, { "from": "now-30m/m", "to": "now/m", "display": "Last 30 minutes" }, { "from": "now-1h/m", "to": "now/m", "display": "Last 1 hour" }, { "from": "now-2h/m", "to": "now/m", "display": "Last 2 hours" }, { "from": "now-4h/m", "to": "now/m", "display": "Last 4 hours" }, { "from": "now-12h/m", "to": "now/m", "display": "Last 12 hours" }, { "from": "now-24h/m", "to": "now/m", "display": "Last 24 hours" }, { "from": "now-48h/m", "to": "now/m", "display": "Last 48 hours" }, { "from": "now-7d/m", "to": "now/m", "display": "Last 7 days" }, { "from": "now-30d/m", "to": "now/m", "display": "Last 30 days" }, { "from": "now-60d/m", "to": "now/m", "display": "Last 60 days" }, { "from": "now-90d/m", "to": "now/m", "display": "Last 90 days" } ] ``` #### Settings for Elastic's X-Pack Features The following settings require that NetObserv Flow is used with an Elastic Stack deployment licensed for X-Pack Basic or higher. The collector's Elasticsearch output must also be configured with ECS-support enabled (set `EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE` to `true`). | Advanced Setting | Value | Why make the change? | | ---------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | securitySolution:ipReputationLinks | *see below* | Modifying these settings provides a more seamless integration with the ElastiFlow NetIntel service while using Kibana's Security app. | | securitySolution:defaultIndex | add `elastiflow-flow-ecs-*` | Add the ElastiFlow index pattern, with support for ECS, to populate the Kibana's Security app with data network flow records from ElastiFlow. | | securitySolution:timeDefaults | *see below* | For the best experience set this value similar to `timepicker:timeDefaults`. | **Recommended IP Reputation Links (securitySolution:ipReputationLinks)** ```json [ { "name": "ElastiFlow NetIntel", "url_template": "https://elastiflow.com/netintel/search?ip={{ip}}" }, { "name": "VirusTotal", "url_template": "https://www.virustotal.com/gui/search/{{ip}}" } ] ``` **Recommended Security Solution Time Defaults (securitySolution:timeDefaults)** We find that the following Security Solution Time Default provides more useful views of the data for network flow-related use-cases. ```json { "from": "now-1h/m", "to": "now" } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.elastiflow.com/flowcoll/configuration/outputs/output_elasticsearch.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.