RHEL/AlmaLinux
Elastic Stack Installation
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. NetObserv Flow can be configured to store the collected, processed and enriched records in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis happens.
This document describes in detail the installation of NetObserv Flow and the Elastic Stack (Elasticsearch and Kibana) on a single server running RHEL 8. These steps should work similarly for other RedHat-based Linux distributions.
Sizing
Elasticsearch can be deployed as a single-mode server or multi-node cluster. The latter provides for horizontal scaling to handle very high ingest rates and longer retention periods. For more information on properly sizing an Elasticsearch cluster, see Sizing.
Environment
Resource | Information |
---|---|
Hostname | myhost |
IP Address | 192.168.56.101 |
CPU Cores | 4 |
Memory | 32 GB |
Storage | 1 TB |
OS | RHEL 8 |
Elasticsearch | 8.14.0 |
Kibana | 8.14.0 |
ES Features | TLS, RBAC |
ElastiFlow UFC | 7.5.3 |
The hostname and IP address above are for examples only. You MUST replace these values with those of your own server when executing any commands or editing any files.
Tune the Linux Kernel
1. Add Parameters Required by Elasticsearch
Elasticsearch uses a mmapfs
directory by default to store its indices. The Linux default limits on mmaps is usually too low, which can result in out-of-memory exceptions. This limit should be raised to 262144
.
Run the following command to add the file /etc/sysctl.d/70-elasticsearch.conf
with the attribute vm.max_map_count=262144
:
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/70-elasticsearch.conf > /dev/null
2. Tune Network Parameters
The default Linux network parameters are not optimal for high throughput applications, in particular a high volume of ingress UDP packets. This can result in dropped packets and lost data. Linux network performance for ElastiFlow can optimized by changing the parameters below.
Run the following command to add the file /etc/sysctl.d/60-net.conf
with the recommended changes.
echo -e "net.core.netdev_max_backlog=4096\nnet.core.rmem_default=262144\nnet.core.rmem_max=67108864\nnet.ipv4.udp_rmem_min=131072\nnet.ipv4.udp_mem=2097152 4194304 8388608" | sudo tee /etc/sysctl.d/60-net.conf > /dev/null
3. Apply Changes
For changes to the above parameters to take effect the system can be restarted. Alternatively the following commands can be run to apply the changes without a reboot:
sudo sysctl -w vm.max_map_count=262144 && \
sudo sysctl -w net.core.netdev_max_backlog=4096 && \
sudo sysctl -w net.core.rmem_default=262144 && \
sudo sysctl -w net.core.rmem_max=67108864 && \
sudo sysctl -w net.ipv4.udp_rmem_min=131072 && \
sudo sysctl -w net.ipv4.udp_mem='2097152 4194304 8388608'
Disable the Firewall
The easiest way to get started is to disable the Linux firewall. Alternatively the firewall can be configured to allow access to any required ports. Details of configuring the Linux firewall are beyond the scope of this document. However if enabled, you will need to allow access to the following ports:
Application | Port |
---|---|
Elasticsearch | TCP/9200 |
Kibana | TCP/5601 |
NetObserv Flow | UDP 9995 or other port(s) configured by EF_FLOW_SERVER_UDP_PORT |
sudo systemctl stop firewalld.service && sudo systemctl disable firewalld.service
Install Prerequisite Packages
Run the following commands to install required packages.
sudo yum install -y unzip
Install Elasticsearch
1. Add Elastic PGP Key
Elastic signs all of their packages with the Elasticsearch Signing Key (PGP key D88E42B4
, available from https://pgp.mit.edu) with fingerprint: 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
Download and install the public signing key.
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
2. Add the Elastic Repository
Add the Elastic repository definition to /etc/yum.repos.d/elasticsearch.repo
by running the following command.
echo -e "[elasticsearch]\nname=Elasticsearch repository for 8.x packages\nbaseurl=https://artifacts.elastic.co/packages/8.x/yum\ngpgcheck=1\ngpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch\nenabled=0\nautorefresh=1\ntype=rpm-md" | sudo tee /etc/yum.repos.d/elasticsearch.repo > /dev/null
3. Install Elasticsearch using yum
Run the following commands to install the Elasticsearch package.
sudo yum install --enablerepo=elasticsearch -y elasticsearch
4. Configure JVM Heap Size
If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason it’s best to start the JVM with the initial and maximum heap sizes set to equal values.
Add the file heap.options
to /etc/elasticsearch/jvm.options.d
and set -Xms
and -Xmx
to about one third of the system memory, but do not exceed 31g
. For this example we will use 12GB of the available 32GB of memory for JVM heap.
echo -e "-Xms12g\n-Xmx12g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null
5. Increase System Limits
Increased system limits should be specified in a systemd
attributes file for the elasticsearch
service.
sudo mkdir /etc/systemd/system/elasticsearch.service.d && \
echo -e "[Service]\nLimitNOFILE=131072\nLimitNPROC=8192\nLimitMEMLOCK=infinity\nLimitFSIZE=infinity\nLimitAS=infinity" | \
sudo tee /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf > /dev/null
6. Generate CA and Certificates
There are numerous ways to generate certificates that can be used to secure communications using TLS. To simplify the process Elastic provides the elasticsearch-certutil
tool. For more details about this tool, refer to Elastic's documentation.
It is first necessary to generate a certificate authority (CA) by running the following command.
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem
When you see Please enter the desired output file [elastic-stack-ca.zip]:
press enter
to accept the default.
The resulting file will be placed in /usr/share/elasticsearch
. To unzip and move the CA key and cert to /etc/elasticsearch/certs
run the following commands.
sudo mkdir /etc/elasticsearch/certs && \
sudo unzip /usr/share/elasticsearch/elastic-stack-ca.zip -d /etc/elasticsearch/certs
To generate certificates for the Elasticsearch node, create a file named /usr/share/elasticsearch/instances.yml
similar to the following. Replace the values with those appropriate for your environment.
instances:
- name: "myhost"
ip:
- "192.0.2.1"
dns:
- "myhost.mydomain.com"
For example, in the system used for this guide, the name of the server is myhost
, the IP address is 192.168.56.101
and there is no name configured in DNS. The instance would contain:
instances:
- name: "myhost"
ip:
- "192.168.56.101"
Use elasticsearch-certutil
to generate the certificates and keys from the CA and instances file.
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in instances.yml --out certs.zip --pem --ca-cert /etc/elasticsearch/certs/ca/ca.crt --ca-key /etc/elasticsearch/certs/ca/ca.key
The resulting file will be placed in /usr/share/elasticsearch
. To unzip and move the node keys and certs to /etc/elasticsearch/certs
run the following commands.
sudo unzip /usr/share/elasticsearch/certs.zip -d /etc/elasticsearch/certs
7. Edit elasticsearch.yml
Edit the Elasticsearch configuration file, /etc/elasticsearch/elasticsearch.yml
, replacing the contents of the file with the following configuration. Edit as necessary for your environment.
cluster.name: elastiflow
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.type: 'single-node'
indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
action.destructive_requires_name: 'true'
xpack.security.http.ssl.enabled: 'true'
xpack.security.http.ssl.verification_mode: 'none'
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt
xpack.security.http.ssl.key: /etc/elasticsearch/certs/myhost/myhost.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/myhost/myhost.crt
xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s
xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'
If you want Elasticsearch data to be stored on a different mount point, you must first create the directory and assign permissions to the elasticsearch
. For example, to store data on /mnt/data0
, run sudo mkdir /mnt/data0/elasticsearch && sudo chown -R elasticsearch:elasticsearch /mnt/data0/elasticsearch
. The edit the path.data
option in elasticsearch.yml
specifying this path.
8. Enable and Start Elasticsearch
Execute the following commands to start Elasticsearch and enable it run automatically when the server boots:
sudo systemctl daemon-reload && \
sudo systemctl enable elasticsearch && \
sudo systemctl start elasticsearch
Confirm Elasticsearch started successfully by executing:
sudo systemctl status elasticsearch
9. Set Passwords for Elasticsearch Built-in Accounts
Execute the following command for to setup passwords for the various built-in accounts:
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
The following will be displayed:
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]
Answer y
, then enter and confirm passwords for the built-in Elasticsearch accounts.
10. Verify Elasticsearch
Ensure that the Elasticsearch REST API is available by running the following:
curl -XGET -k "https://elastic:PASSWORD@127.0.0.1:9200"
The output should be similar to the following:
{
"name" : "myhost",
"cluster_name" : "elastiflow",
"cluster_uuid" : "S5Y3Z2USSq2sR2TyOkLe3A",
"version" : {
"number" : "8.14.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
"build_date" : "2021-08-26T09:01:05.390870785Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Install Kibana
1. Install Kibana using yum
Run the following commands to install the Kibana package.
sudo yum install --enablerepo=elasticsearch -y kibana
2. Copy CA and Certificates
Kibana will also require access to the CA, certificates and keys. To use the same files that were created for Elasticsearch, copy them from /etc/elasticsearch
to /etc/kibana
.
sudo cp -r /etc/elasticsearch/certs /etc/kibana
3. Edit kibana.yml
Edit the Kibana configuration file /etc/kibana/kibana.yml
, replacing the contents of the file with the following configuration. Edit as necessary for your environment (especially elasticsearch.password
).
telemetry.enabled: false
telemetry.optIn: false
newsfeed.enabled: false
server.host: '0.0.0.0'
server.port: 5601
server.maxPayload: 8388608
server.publicBaseUrl: 'https://192.168.56.101:5601'
server.ssl.enabled: true
server.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
server.ssl.key: /etc/kibana/certs/myhost/myhost.key
server.ssl.certificate: /etc/kibana/certs/myhost/myhost.crt
elasticsearch.hosts: ['https://192.168.56.101:9200']
elasticsearch.username: 'kibana_system'
elasticsearch.password: 'PASSWORD'
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
elasticsearch.ssl.key: /etc/kibana/certs/myhost/myhost.key
elasticsearch.ssl.certificate: /etc/kibana/certs/myhost/myhost.crt
elasticsearch.ssl.verificationMode: 'certificate'
elasticsearch.requestTimeout: 132000
elasticsearch.shardTimeout: 120000
kibana.autocompleteTimeout: 2000
kibana.autocompleteTerminateAfter: 500000
monitoring.enabled: true
monitoring.kibana.collection.enabled: true
monitoring.kibana.collection.interval: 30000
monitoring.ui.enabled: true
monitoring.ui.min_interval_seconds: 20
xpack.maps.showMapVisualizationTypes: true
xpack.security.enabled: true
xpack.security.audit.enabled: false
xpack.encryptedSavedObjects.encryptionKey: 'ElastiFlow_0123456789_0123456789_0123456789'
4. Enable and Start Kibana
Execute the following commands:
sudo systemctl daemon-reload && \
sudo systemctl enable kibana && \
sudo systemctl start kibana
Confirm Kibana started successfully by executing:
sudo systemctl status kibana
You should now be able to access Kibana at https://IP_OF_KIBANA_HOST:5601
. Since this HTTPS connection is using a self-signed certificate, you may see an error similar to the following.
Chrome:
Firefox:
Safari:
You need to either create an exception in your browser, or import and trust the CA certificate on the system running the browser. This can usually be achieved by downloading the ca.crt
file from the server. Double-clicking it will usually prompt you to import the certificate. On MacOS the certificate should appear as follows in the keychain application after it is configured to be trusted.
You should now be able to connect to Kibana after allowing an exception. To login use the user elastic
and the password you defined earlier for this user.
Install NetObserv Flow
NetObserv Flow can be installed natively on Ubuntu and Debian Linux. The instructions are available here. In this section we will cover the primary configuration options for the Elasticsearch output.
The NetObserv Flow options are configured using YAML. To configure the collector edit the file /etc/flowcoll.yml
. For details on the configuration options, please refer to the Configuration Reference.
1. Request a Basic or Trial License
Without a license key NetObserv Flow runs with a Community tier license. The Basic tier is also available at no cost and supports additional standard information elements. A license can be requested on the ElastiFlow website. Alternatively a 30-day Premium trial may be requested, which increases the scalability of the collector and enables all supported vendor and standard information elements.
After requesting a license it can take up to 30 minutes for the email to arrive.
License keys are generated per account. EF_ACCOUNT_ID
must contain the Account ID for the License Key specified in EF_FLOW_LICENSE_KEY
. The number of licensed units will be 1
for a Basic license, and up to 64 for a 30-day Trial. The ElastiFlow EULA must also be accepted to use the software.
Environment="EF_LICENSE_ACCEPTED=true"
Environment="EF_ACCOUNT_ID=FROM_THE_EMAIL"
Environment="EF_FLOW_LICENSE_KEY=FROM_THE_EMAIL"
Environment="EF_FLOW_LICENSED_UNITS=1"
2. Copy CA Certificate
The NetObserv Flow will require access to the CA certificate to verify the Elasticsearch node. Copy the CA certificate from /etc/elasticsearch/certs/ca/ca.crt
to /etc/elastiflow/ca/ca.crt
.
sudo mkdir /etc/elastiflow/ca && \
sudo cp /etc/elasticsearch/certs/ca/ca.crt /etc/elastiflow/ca
3. Enable the Elasticsearch Output
Set EF_OUTPUT_ELASTICSEARCH_ENABLE
to true
to enable the Elasticsearch output.
Environment="EF_OUTPUT_ELASTICSEARCH_ENABLE=true"
4. Specify a Schema
The NetObserv Flow outputs data using ElastiFlow's CODEX schema. Optionally you can choose to output data in Elastic Common Schema (ECS). To do so, set EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE
to true
.
Environment="EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE=true"
5. Source of @timestamp
There are multiple possible sources to set the value of @timestamp
field, which is the primary timestamp field used by Kibana. They supported options are:
Value | Field Used | Description |
---|---|---|
start | flow.start.timestamp | The flow start time indicated in the flow. |
end | flow.end.timestamp | The flow end time (or last reported time). |
export | flow.export.timestamp | The time from the flow record header. |
collect | flow.collect.timestamp | The time that the collector processed the flow record. |
Usually end
would be the best setting. However, in the case of poorly behaving or misconfigured devices, collect
may be the better option. The actual timestamp used may be different than configured depending on the content of the received records. If end
is not available the collector will fall back to export
. If export
is not available the collector will fall back to collect
.
Environment="EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE=collect"
6. Index Shards and Replicas
For this small single node install set the number of shards to 1
and replicas to 0
.
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS=1"
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS=0"
The optimum value for these settings will depend on a number of factors. The number of shards should be at least 1 for each Elasticsearch data node in a cluster. Larger nodes (16+ CPU cores) and higher ingest rates can benefit from 2 shards per node. The largest nodes (64 CPU cores, 8 memory channels and multiple SSD drives) can even benefit from 3 or 4 shards per node. In a multi-node cluster 1 or more replicas may be specified for redundancy.
7. Index Lifecycle Management (ILM)
Index Lifecycle Management (ILM) can be used to rollover the indices which store the ElastiFlow data, preventing issues that can occur when shards become too large. Enable rollover by setting, EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD
to rollover
. When enabled the collector will automatically bootstrap the initial index and write alias.
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD=rollover"
The default Index Lifecycle Management (ILM) lifecycle is elastiflow
. If this lifecycle doesn't exist a basic lifecycle will be added which will remove data after 7 days. This lifecycle can be edited later via Kibana or the Elasticsearch ILM API.
8. Elasticsearch Server and Credentials
Define the Elasticsearch node to which the collector should connect and the credentials for which the password was defined during the Elasticsearch installation.
Environment="EF_OUTPUT_ELASTICSEARCH_ADDRESSES=192.168.56.101:9200"
Environment="EF_OUTPUT_ELASTICSEARCH_USERNAME=elastic"
Environment="EF_OUTPUT_ELASTICSEARCH_PASSWORD=changeme"
9. Encrypted Communications with TLS
Enable TLS and specify the path to the CA certificate.
Environment="EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE=true"
Environment="EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH=/etc/elastiflow/ca/ca.crt"
10. Enable and Start the NetObserv Flow
Execute the following commands:
sudo systemctl daemon-reload && \
sudo systemctl enable flowcoll && \
sudo systemctl start flowcoll
Confirm the service started successfully by executing:
sudo systemctl status flowcoll
The collector is now ready to receive flow records from the network infrastructure.
Import Kibana Objects
The last step is to import the Kibana saved objects and apply the recommended advanced settings. Follow the instructions in the Kibana section of the documentation for detailed instructions.