Docker Compose Quickstart Guide for ElastiFlow with Elasticsearch

Getting Started

Follow this guide to get immediate insights into your network. This is an opinionated, non-production installation option that gets you set up with NetObserv and Elasticsearch on a single server using an installation and docker compose scripts. Refer to the Installation section for production-grade deployments and setup.

This is the quickstart guide for installing the ElastiFlow NetObserv flow collector, Elasticsearch, and Kibana dashboards on a SINGLE server using Docker compose on a linux server. This is designed to get you up and running quickly. For production deployments, consider installing NetObserv flow collector and Elasticsearch on dedicated infrastructure using Docker or native packages.

Technical Recommendations

POC Recommended Server specs:

	CPU	RAM	STORAGE
Good	16 Cores	64GB	1TB SSD
Better	24 Cores	128GB	2TB SSD

Linux distributions:

• debian-based: debian, ubuntu, etc.

• redhat-based: rocky, alma, fedora, centos, etc.

• to reduce resource overhead, please use server-based install options (headless, without a GUI desktop).

Quickstart Guide

The following general procedure should get you going in less than 30 minutes!

A video overview of this quickstart installation is available here: Installation example video

Step 1 - Download the docker compose installation zip file

The docker compose script is part of a zip file. This zip file includes the docker-compose.yml, .env, and initial_installation.sh scripts.

Download the zip fle here:

7KB

elastiflow_elasticsearch_certs.zip

archive

Open

Step 2 - Copy the zip file to your server

Copy the Elastiflow docker zip file to your server using secure copy (or other method):

scp elastiflow_elasticsearch_certs.zip <username>@<server ip or hostname>:~

Step 3 - ssh to server and verify docker is installed

ssh into the server:

ssh <username>@<server ip or hostname>

Check to see if docker/docker compose is installed:

sudo docker compose version

If there was an error because docker or docker compose was not found:

• Refer to the official Docker documentation to install docker: Install Docker Engine on Linux

• Please follow the instructions to install docker and docker compose on your version of Linux.

• Once docker and docker compose are installed, proceed to Step 4.

Step 4 - Unzip and setup the .env configuration file

Unzip the Elastiflow docker zip file:

unzip elastiflow_elasticsearch_certs.zip 

If unzip is not found, install it:

• debian-based installs: sudo apt install unzip

• rehat-based installs: sudo dnf install unzip

Change into the new directory:

cd elastiflow_elasticsearch_certs/

Verify the IP address of the server with the ip addr command:

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:3e:06:20 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    altname enxbc24113e0620
    inet 192.168.0.226/24 brd 192.168.0.255 scope global dynamic noprefixroute ens18
       valid_lft 3138sec preferred_lft 2638sec
    inet6 fe80::8536:a8b8:bdf7:f8b6/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether ee:97:b8:e3:5a:66 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
        valid_lft forever preferred_lft forever

Edit the .env file and set variables for install.

• Set the SERVER_IP with the IP address you verified above.

• Set the ES_DATA_DIRECTORY to a directory that maps to your largest storage partition.

• view the readme.txt file in this directory for more details.

vi .env or nano .env

# assigned ip address of the server
SERVER_IP=192.168.0.226 

# elastic user's password
ES_ELASTIC_PASSWORD=elastic1234

# kibana_system's password
ES_KIBANA_PASSWORD=kibana1234

# version of elasticsearch and kibana to install
ES_STACK_VERSION=9.3.1

# elasticsearch persistant local data store on the host
ES_DATA_DIR=/var/lib/elasticsearch

# version of ElastiFlow flow collector to install
EF_FLOWCOLL_VERSION=7.25.0

# DNS server for elastiflow enrichment. keep blank to use OS defined DNS server(s)
EF_DNS=

# Add the account ID and license key provided to you via email.
EF_ACCOUNT_ID= 
EF_LICENSE_KEY=

Step 5 - Run install script and import dashboards

Run the initial install as root to start the installation.

sudo ./initial_install.sh

This script will take care of the initial installation and detect when all 3 systems (ElastiFlow flow collector, Elasticsearch, and Kibana) are up and ready to serve. It may take several minutes to complete.

When complete the details to access the server will be provided (see example below). Open a new web browser tab to the provided ip address, and login with the given credentials:

rich@docker$ sudo ./initial_install.sh 
[sudo] password for rich: 

creating elasticsearch data directory on host.
creating elastiflow system directory on host.
starting up docker containers.
[+] up 5/5
 ✔ Network elastiflow_elasticsearch_certs_default Created                                                         0.2sss
 ✔ Container elasticsearch                        Healthy                                                         136.6s
 ✔ Container es_setup                             Started                                                         3.3sss
 ✔ Container flow-collector                       Started                                                         137.9s
 ✔ Container kibana                               Started                                                         138.0s
waiting for kibana to come online.
................................................

done!

-> open a web browser to https://192.168.0.226:5601
-> username: elastic
-> password: elastic1234

-> configure your network devices to send netflow/ipfix flows to:
-> 192.168.123.58 to any of these udp ports: 2055, 4739, 6343, or 9995

-> You do not need to use this install script again, please manage
-> the elastiflow stack using standard 'docker' and 'docker compose' commands.
-> see readme.txt file for more details.
rich@docker:~/elastiflow_elasticsearch_certs$

Open a second web browser tab, and download the kibana dashboards for elasticsearch

Kibana Dashboards for Elasticsearch

Please select the codex version of the dashboards.

Important: Right click the link and Save Link As... to your local computer.

From Elasticsearch, click on the 3 line (hamburger) menu at the top left, the scroll down the mento and click on 'Stack Management'

Under the 'Kibana' section, click on 'Saved Objects'

Click 'Import' at the top right.

Click the Import box to select the dashboard file you just saved on your computer.

Once that is done, click 'Import' at the bottom right to import all of the dashboards.

Under Advanced Settings, search for "pin" and enable pinned filters by default and click save.

Navigate to the 3 line (hamburger) menu.

Under the Analytics section, click on Dashboards

And search for "overview".

Click on the "ElastiFlow (flow): Overview" dashboard.

Step 6 - Configure your devices to send Netflow/IPFIX data to collector

You should now configure network devices to send flow data to the flow collector using the server IP address to any of the following UDP ports:

• 2055, 4739, 6343, 9995

Please reference the vendor's documentation for specific device configuration for fllow data output.

You can find a limited collection of example configurations in the ElastiFlow Knowledge Base.

Once this is complete, the dashboards will populate with data after a few minutes.

Troubleshooting

Cannot connect to the Dashboard at port 5601 on my web browser

• Make sure you are connecting via https and not http: https://<server ip>:5601

• The local firewall on the server may be blocking access from your host. To temporarily disable the firewall to verify if that is the problem:

  • debian-based: sudo systemctl stop ufw

  • rehat-based: sudo systemctl stop firewalld

  • If disabling the firewall allows connection, you will need to create firewall rules to permit traffic to port 5601 or disable the firewall permanently.

• Check status of Kibana for more details: sudo docker compose logs kibana -f

How do I restart the the stack or a service in the stack?

• change in the elastiflow_elasticsearch_certs/ folder, where the docker-compose.yml file is located

• to restart the entire stack: sudo docker compose down and then: sudo docker compose up -d

• to restart a specific service: sudo docker stop <service_name> and then: sudo docker start <service_name>

  • replace <service_name> with any of the following: flow-collector, elasticsearch, kibana

How do I view the logs to troubleshoot an issue?

• cd into the elastiflow_elasticsearch_certs/ directory, where the docker-compose.yml file is located

• To view and follow the logs for a service: sudo docker compose logs <service_name> -f

• To view the last 50 log lines: sudo docker compose logs <service_name> --tail=50

  • replace <service_name> with any of the following: flow-collector, elasticsearch, kibana

How do I modify the ElastiFlow configuration and enable changes?

• cd into the elastiflow_elasticsearch_certs/ directory, where the docker-compose.yml file is located

• Since we will be editing the docker compose file, first make a back up:

  • cp docker-compose.yml docker-compose.yml_backup

• Edit the docker-compose.yml file with vi, nano, or some other editor

  • At the end of the file, under the flow-collector service, find the environment: section

  • These are the environmental variables that you can set to modify the flow collector

    • These variables begin with 'EF_'

  • Uncomment lines, to enable them, by removing the # character at the beginning of the line

  • Comment lines, to disable them, by adding a # character at the beginning of the line

  • Add new lines anywhere in the environment section

  • Make sure that spacing is lined up with the existing lines, or the change could generate an error!

  • Save the file when you complete the changes

• Restart the flow-collector service with the new settings:

  • sudo docker stop flow-collector and then: sudo docker compose up -d

I changed the password for the 'elastic' user in the web interface, and now the flow collector is not working/processing flows.

• In this docker installation, the flow collector uses the elastic user account to update the elasticsearch back-end and it uses the password stored in the .env docker compose file. If you change the elastic password in the elasticsearch gui, the flow collector will now use the older, now incorrect, password. Therefore for this POC docker install, changes to the elastic(and kibana_system) user, should be made directly to the .env file, and then restart the entire docker stack which will automatically update the passwords and enable the flow collector to stay in sync.

• Edit the .env file with vi or nano, and update the line 'ES_ELASTIC_PASSWORD=' removing the old elastic user password and updating it with the new one. If you wish to change the kibana_system password, you can modify the 'ES_KIBANA_PASSWORD' line.

• Bring down the stack: sudo docker compose down

• And then bring it back up: sudo docker compose up -d