> For the complete documentation index, see [llms.txt](https://docs.elastiflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.elastiflow.com/flowcoll/docker-compose-quickstart-guide-for-elastiflow-with-elasticsearch.md). # Docker Compose Quickstart Guide for ElastiFlow with Elasticsearch ### Getting Started Follow this guide to get immediate insights into your network. This is an opinionated, non-production installation option that gets you set up with NetObserv and Elasticsearch on a single server using an installation and docker compose scripts. Refer to the [Installation section](/flowcoll/installation.md) for production-grade deployments and setup. {% hint style="info" %} This is the quickstart guide for installing the ElastiFlow NetObserv flow collector, Elasticsearch, and Kibana dashboards on **a SINGLE server** using Docker compose on a linux server. This is designed to get you up and running quickly. For production deployments, consider installing NetObserv flow collector and Elasticsearch on dedicated infrastructure using Docker or native packages. {% endhint %} #### Technical Recommendations POC Recommended Server specs: | | CPU | RAM | STORAGE | | ------ | -------- | ----- | ------- | | Good | 16 Cores | 64GB | 1TB SSD | | Better | 24 Cores | 128GB | 2TB SSD | Linux distributions: * debian-based: debian, ubuntu, etc. * redhat-based: rocky, alma, fedora, centos, etc. * to reduce resource overhead, please use server-based install options (headless, without a GUI desktop). ### Quickstart Guide The following general procedure should get you going in less than 30 minutes! A video overview of this quickstart installation is available here: [Installation example video](https://youtu.be/2XU1eLSoOAk) #### Step 1 - Download the docker compose installation zip file The docker compose script is part of a zip file. This zip file includes the docker-compose.yml, .env, and initial\_installation.sh scripts. Download the zip fle here: {% file src="/files/iZSz5NPrQx7toUqg8EwR" %} #### Step 2 - Copy the zip file to your server Copy the Elastiflow docker zip file to your server using secure copy (or other method): ``` scp elastiflow_elasticsearch_certs.zip @:~ ``` #### Step 3 - ssh to server and verify docker is installed ssh into the server: ``` ssh @ ``` Check to see if docker/docker compose is installed: ``` sudo docker compose version ``` **If there was an error because docker or docker compose was not found:** * Refer to the official Docker documentation to install docker: [Install Docker Engine on Linux](https://docs.docker.com/engine/install) * Please follow the instructions to install docker and docker compose on your version of Linux. * Once docker and docker compose are installed, proceed to Step 4. #### Step 4 - Unzip and setup the .env configuration file Unzip the Elastiflow docker zip file: ``` unzip elastiflow_elasticsearch_certs.zip ``` If unzip is not found, install it: * debian-based installs: `sudo apt install unzip` * rehat-based installs: `sudo dnf install unzip` Change into the new directory: ``` cd elastiflow_elasticsearch_certs/ ``` Verify the IP address of the server with the ip addr command: ``` ip addr 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host noprefixroute         valid_lft forever preferred_lft forever 2: ens18: mtu 1500 qdisc fq_codel state UP group default qlen 1000     link/ether bc:24:11:3e:06:20 brd ff:ff:ff:ff:ff:ff     altname enp0s18     altname enxbc24113e0620     inet 192.168.0.226/24 brd 192.168.0.255 scope global dynamic noprefixroute ens18        valid_lft 3138sec preferred_lft 2638sec     inet6 fe80::8536:a8b8:bdf7:f8b6/64 scope link         valid_lft forever preferred_lft forever 3: docker0: mtu 1500 qdisc noqueue state DOWN group default      link/ether ee:97:b8:e3:5a:66 brd ff:ff:ff:ff:ff:ff     inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0        valid_lft forever preferred_lft forever ``` Edit the .env file and set variables for install. * Set the SERVER\_IP with the IP address you verified above. * Set the ES\_DATA\_DIRECTORY to a directory that maps to your largest storage partition. * view the readme.txt file in this directory for more details. ``` vi .env or nano .env # assigned ip address of the server SERVER_IP=192.168.0.226 # elastic user's password ES_ELASTIC_PASSWORD=elastic1234 # kibana_system's password ES_KIBANA_PASSWORD=kibana1234 # version of elasticsearch and kibana to install ES_STACK_VERSION=9.3.1 # elasticsearch persistant local data store on the host ES_DATA_DIR=/var/lib/elasticsearch # version of ElastiFlow flow collector to install EF_FLOWCOLL_VERSION=7.25.0 # DNS server for elastiflow enrichment. keep blank to use OS defined DNS server(s) EF_DNS= # Add the account ID and license key provided to you via email. EF_ACCOUNT_ID= EF_LICENSE_KEY= ``` #### Step 5 - Run install script and import dashboards Run the initial install as root to start the installation. ``` sudo ./initial_install.sh ``` This script will take care of the initial installation and detect when all 3 systems (ElastiFlow flow collector, Elasticsearch, and Kibana) are up and ready to serve. It may take several minutes to complete. When complete the details to access the server will be provided (see example below). Open a new web browser tab to the provided ip address, and login with the given credentials: ``` rich@docker$ sudo ./initial_install.sh [sudo] password for rich: creating elasticsearch data directory on host. creating elastiflow system directory on host. starting up docker containers. [+] up 5/5 ✔ Network elastiflow_elasticsearch_certs_default Created 0.2sss ✔ Container elasticsearch Healthy 136.6s ✔ Container es_setup Started 3.3sss ✔ Container flow-collector Started 137.9s ✔ Container kibana Started 138.0s waiting for kibana to come online. ................................................ done! -> open a web browser to https://192.168.0.226:5601 -> username: elastic -> password: elastic1234 -> configure your network devices to send netflow/ipfix flows to: -> 192.168.123.58 to any of these udp ports: 2055, 4739, 6343, or 9995 -> You do not need to use this install script again, please manage -> the elastiflow stack using standard 'docker' and 'docker compose' commands. -> see readme.txt file for more details. rich@docker:~/elastiflow_elasticsearch_certs$ ``` Open a second web browser tab, and download the kibana dashboards for elasticsearch [Kibana Dashboards for Elasticsearch](https://docs.elastiflow.com/flowcoll/configuration/outputs/output_elasticsearch) Please select the **codex** version of the dashboards. Important: Right click the link and Save Link As... to your local computer.
From Elasticsearch, click on the 3 line (hamburger) menu at the top left, the scroll down the mento and click on 'Stack Management' Under the 'Kibana' section, click on 'Saved Objects'
Click 'Import' at the top right. Click the Import box to select the dashboard file you just saved on your computer. Once that is done, click 'Import' at the bottom right to import all of the dashboards.
Under Advanced Settings, search for "pin" and enable pinned filters by default and click save.
Navigate to the 3 line (hamburger) menu. Under the Analytics section, click on Dashboards And search for "overview". Click on the "ElastiFlow (flow): Overview" dashboard.
#### Step 6 - Configure your devices to send Netflow/IPFIX data to collector You should now configure network devices to send flow data to the flow collector using the server IP address to any of the following UDP ports: * 2055, 4739, 6343, 9995 Please reference the vendor's documentation for specific device configuration for fllow data output. You can find a limited collection of example configurations in the [ElastiFlow Knowledge Base](/additional-resources-reference-articles/guides/configuring-flow-logs-on-devices.md). Once this is complete, the dashboards will populate with data after a few minutes.
### Troubleshooting **Cannot connect to the Dashboard at port 5601 on my web browser** * Make sure you are connecting via https and not http: https\://\:5601 * The local firewall on the server may be blocking access from your host. To **temporarily** disable the firewall to verify if that is the problem: * debian-based: `sudo systemctl stop ufw` * rehat-based: `sudo systemctl stop firewalld` * If disabling the firewall allows connection, you will need to create firewall rules to permit traffic to port 5601 or disable the firewall permanently. * Check status of Kibana for more details: `sudo docker compose logs kibana -f` **How do I restart the the stack or a service in the stack?** * change in the elastiflow\_elasticsearch\_certs/ folder, where the docker-compose.yml file is located * to restart the entire stack: `sudo docker compose down` and then: `sudo docker compose up -d` * to restart a specific service: `sudo docker stop ` and then: `sudo docker start ` * replace \ with any of the following: flow-collector, elasticsearch, kibana **How do I view the logs to troubleshoot an issue?** * cd into the elastiflow\_elasticsearch\_certs/ directory, where the docker-compose.yml file is located * To view and follow the logs for a service: `sudo docker compose logs -f` * To view the last 50 log lines: `sudo docker compose logs --tail=50` * replace \ with any of the following: flow-collector, elasticsearch, kibana **How do I modify the ElastiFlow configuration and enable changes?** * cd into the elastiflow\_elasticsearch\_certs/ directory, where the docker-compose.yml file is located * Since we will be editing the docker compose file, first make a back up: * `cp docker-compose.yml docker-compose.yml_backup` * Edit the docker-compose.yml file with vi, nano, or some other editor * At the end of the file, under the flow-collector service, find the environment: section * These are the environmental variables that you can set to modify the flow collector * These variables begin with 'EF\_' * Uncomment lines, to enable them, by removing the # character at the beginning of the line * Comment lines, to disable them, by adding a # character at the beginning of the line * Add new lines anywhere in the environment section * Make sure that spacing is lined up with the existing lines, or the change could generate an error! * Save the file when you complete the changes * Restart the flow-collector service with the new settings: * `sudo docker stop flow-collector` and then: `sudo docker compose up -d` **I changed the password for the 'elastic' user in the web interface, and now the flow collector is not working/processing flows.** * In this docker installation, the flow collector uses the elastic user account to update the elasticsearch back-end and it uses the password stored in the .env docker compose file. If you change the elastic password in the elasticsearch gui, the flow collector will now use the older, now incorrect, password. Therefore for this POC docker install, changes to the elastic(and kibana\_system) user, should be made directly to the .env file, and then restart the entire docker stack which will automatically update the passwords and enable the flow collector to stay in sync. * Edit the .env file with vi or nano, and update the line 'ES\_ELASTIC\_PASSWORD=' removing the old elastic user password and updating it with the new one. If you wish to change the kibana\_system password, you can modify the 'ES\_KIBANA\_PASSWORD' line. * Bring down the stack: `sudo docker compose down` * And then bring it back up: `sudo docker compose up -d` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.elastiflow.com/flowcoll/docker-compose-quickstart-guide-for-elastiflow-with-elasticsearch.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.