# RHEL/AlmaLinux

### Elastic Stack Installation

Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. NetObserv Flow can be configured to store the collected, processed, and enriched records in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis happens.

This document describes in detail the installation of NetObserv Flow and the Elastic Stack (Elasticsearch and Kibana) on a single server running RHEL 8. These steps should work similarly for other RedHat-based Linux distributions.

### Sizing

Elasticsearch can be deployed as a single-mode server or multi-node cluster. The latter provides for horizontal scaling to handle very high ingest rates and longer retention periods. For more information on properly sizing an Elasticsearch cluster, see [Sizing](/data_platforms/elastic/elasticsearch.md#sizing).

### Environment

| Resource      | Information                                                 |
| ------------- | ----------------------------------------------------------- |
| Hostname      | `myhost`                                                    |
| IP Address    | `192.168.56.101`                                            |
| CPU Cores     | 4                                                           |
| Memory        | 32 GB                                                       |
| Storage       | 1 TB                                                        |
| OS            | RHEL 8                                                      |
| Elasticsearch | 8.14.0                                                      |
| Kibana        | 8.14.0                                                      |
| ES Features   | TLS, RBAC                                                   |
| NetObserv     | <code class="expression">space.vars.current\_version</code> |

{% hint style="info" %}
The hostname and IP address above are for examples only. You \*\*MUST\*\* replace these values with those of your own server when executing any commands or editing any files.
{% endhint %}

### Tune the Linux Kernel

#### 1. Add Parameters Required by Elasticsearch

Elasticsearch uses a `mmapfs` directory by default to store its indices. The Linux default limit on mmaps is usually too low, which can result in out-of-memory exceptions. This limit should be raised to `262144`.

Run the following command to add the file `/etc/sysctl.d/70-elasticsearch.conf` with the attribute `vm.max_map_count=262144`:

```shell
echo "vm.max_map_count=262144" | sudo tee /etc/sysctl.d/70-elasticsearch.conf > /dev/null
```

#### 2. Tune Network Parameters

The default Linux network parameters are not optimal for high throughput applications, in particular a high volume of ingress UDP packets. This can result in dropped packets and lost data. Linux network performance for NetObserv can be optimized by changing the parameters below.

Run the following command to add the file `/etc/sysctl.d/60-net.conf` with the recommended changes.

```shell
echo -e "net.core.netdev_max_backlog=4096\nnet.core.rmem_default=262144\nnet.core.rmem_max=67108864\nnet.ipv4.udp_rmem_min=131072\nnet.ipv4.udp_mem=2097152 4194304 8388608" | sudo tee /etc/sysctl.d/60-net.conf > /dev/null
```

#### 3. Apply Changes

For changes to the above parameters to take effect, the system can be restarted. Alternatively, the following commands can be run to apply the changes without a reboot:

```shell
sudo sysctl -w vm.max_map_count=262144 && \
  sudo sysctl -w net.core.netdev_max_backlog=4096 && \
  sudo sysctl -w net.core.rmem_default=262144 && \
  sudo sysctl -w net.core.rmem_max=67108864 && \
  sudo sysctl -w net.ipv4.udp_rmem_min=131072 && \
  sudo sysctl -w net.ipv4.udp_mem='2097152 4194304 8388608'
```

### Disable the Firewall

The easiest way to get started is to disable the Linux firewall. Alternatively, the firewall can be configured to allow access to any required ports. Details of configuring the Linux firewall are beyond the scope of this document. However, if enabled, you will need to allow access to the following ports:

| Application    | Port                                                                                                                                       |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Elasticsearch  | TCP/9200                                                                                                                                   |
| Kibana         | TCP/5601                                                                                                                                   |
| NetObserv Flow | UDP 9995 or other port(s) configured by [EF\_FLOW\_SERVER\_UDP\_PORT](/flowcoll/configuration/inputs/input_udp.md#ef_flow_server_udp_port) |

```shell
sudo systemctl stop firewalld.service && sudo systemctl disable firewalld.service
```

### Install Prerequisite Packages

Run the following commands to install required packages.

```shell
sudo yum install -y unzip
```

### Install Elasticsearch

#### 1. Add Elastic PGP Key

Elastic signs all of their packages with the Elasticsearch Signing Key (PGP key `D88E42B4`, available from <https://pgp.mit.edu>) with fingerprint: `4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4`

Download and install the public signing key.

```shell
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
```

#### 2. Add the Elastic Repository

Add the Elastic repository definition to `/etc/yum.repos.d/elasticsearch.repo` by running the following command.

```shell
echo -e "[elasticsearch]\nname=Elasticsearch repository for 8.x packages\nbaseurl=https://artifacts.elastic.co/packages/8.x/yum\ngpgcheck=1\ngpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch\nenabled=0\nautorefresh=1\ntype=rpm-md" | sudo tee /etc/yum.repos.d/elasticsearch.repo > /dev/null
```

#### 3. Install Elasticsearch using `yum`

Run the following commands to install the Elasticsearch package.

```shell
sudo yum install --enablerepo=elasticsearch -y elasticsearch
```

#### 4. Configure JVM Heap Size

If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason, it’s best to start the JVM with the initial and maximum heap sizes set to equal values.

Add the file `heap.options` to `/etc/elasticsearch/jvm.options.d` and set `-Xms` and `-Xmx` to about one third of the system memory, but do not exceed `31g`. For this example, we will use 12GB of the available 32GB of memory for JVM heap.

```shell
echo -e "-Xms12g\n-Xmx12g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null
```

#### 5. Increase System Limits

Increased system limits should be specified in a `systemd` attributes file for the `elasticsearch` service.

```shell
sudo mkdir /etc/systemd/system/elasticsearch.service.d && \
  echo -e "[Service]\nLimitNOFILE=131072\nLimitNPROC=8192\nLimitMEMLOCK=infinity\nLimitFSIZE=infinity\nLimitAS=infinity" | \
  sudo tee /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf > /dev/null
```

#### 6. Generate CA and Certificates

There are numerous ways to generate certificates that can be used to secure communications using TLS. To simplify the process, Elastic provides the `elasticsearch-certutil` tool. For more details about this tool, refer to [Elastic's documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html).

It is first necessary to generate a certificate authority (CA) by running the following command.

```shell
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem
```

When you see `Please enter the desired output file [elastic-stack-ca.zip]:` press `enter` to accept the default.

The resulting file will be placed in `/usr/share/elasticsearch`. To unzip and move the CA key and cert to `/etc/elasticsearch/certs` run the following commands.

```shell
sudo mkdir /etc/elasticsearch/certs && \
  sudo unzip /usr/share/elasticsearch/elastic-stack-ca.zip -d /etc/elasticsearch/certs
```

To generate certificates for the Elasticsearch node, create a file named `/usr/share/elasticsearch/instances.yml` similar to the following. Replace the values with those appropriate for your environment.

```yaml
instances:
  - name: "myhost"
    ip:
      - "192.0.2.1"
    dns:
      - "myhost.mydomain.com"
```

For example, in the system used for this guide, the name of the server is `myhost`, the IP address is `192.168.56.101` and there is no name configured in DNS. The instance would contain:

```yaml
instances:
  - name: "myhost"
    ip:
      - "192.168.56.101"
```

Use `elasticsearch-certutil` to generate the certificates and keys from the CA and instances file.

```shell
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in instances.yml --out certs.zip --pem --ca-cert /etc/elasticsearch/certs/ca/ca.crt --ca-key /etc/elasticsearch/certs/ca/ca.key
```

The resulting file will be placed in `/usr/share/elasticsearch`. To unzip and move the node keys and certs to `/etc/elasticsearch/certs` run the following commands.

```shell
sudo unzip /usr/share/elasticsearch/certs.zip -d /etc/elasticsearch/certs
```

#### 7. Edit elasticsearch.yml

Edit the Elasticsearch configuration file, `/etc/elasticsearch/elasticsearch.yml`, replacing the contents of the file with the following configuration. Edit as necessary for your environment.

```yaml
cluster.name: elastiflow

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 0.0.0.0
http.port: 9200

discovery.type: 'single-node'

indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000

action.destructive_requires_name: 'true'

xpack.security.http.ssl.enabled: 'true'
xpack.security.http.ssl.verification_mode: 'none'
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt
xpack.security.http.ssl.key: /etc/elasticsearch/certs/myhost/myhost.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/myhost/myhost.crt

xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s

xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'
```

{% hint style="info" %}
If you want Elasticsearch data to be stored on a different mount point, you must first create the directory and assign permissions to the \`elasticsearch\`. For example, to store data on \`/mnt/data0\`, run \`sudo mkdir /mnt/data0/elasticsearch && sudo chown -R elasticsearch:elasticsearch /mnt/data0/elasticsearch\`. The edit the \`path.data\` option in \`elasticsearch.yml\` specifying this path.
{% endhint %}

#### 8. Enable and Start Elasticsearch

Execute the following commands to start Elasticsearch and enable it run automatically when the server boots:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable elasticsearch && \
  sudo systemctl start elasticsearch
```

Confirm Elasticsearch started successfully by executing:

```shell
sudo systemctl status elasticsearch
```

#### 9. Set Passwords for Elasticsearch Built-in Accounts

Execute the following command to set up passwords for the various built-in accounts:

```shell
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
```

The following will be displayed:

```shell
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]
```

Answer `y`, then enter and confirm passwords for the built-in Elasticsearch accounts.

#### 10. Verify Elasticsearch

Ensure that the Elasticsearch REST API is available by running the following:

```shell
curl -XGET -k "https://elastic:PASSWORD@127.0.0.1:9200"
```

The output should be similar to the following:

```json
{
  "name" : "myhost",
  "cluster_name" : "elastiflow",
  "cluster_uuid" : "S5Y3Z2USSq2sR2TyOkLe3A",
  "version" : {
    "number" : "8.14.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```

### Install Kibana

#### 1. Install Kibana using `yum`

Run the following commands to install the Kibana package.

```shell
sudo yum install --enablerepo=elasticsearch -y kibana
```

#### 2. Copy CA and Certificates

Kibana will also require access to the CA, certificates and keys. To use the same files that were created for Elasticsearch, copy them from `/etc/elasticsearch` to `/etc/kibana`.

```shell
sudo cp -r /etc/elasticsearch/certs /etc/kibana
```

#### 3. Edit kibana.yml

Edit the Kibana configuration file `/etc/kibana/kibana.yml`, replacing the contents of the file with the following configuration. Edit as necessary for your environment (especially `elasticsearch.password`).

```yaml
telemetry.enabled: false
telemetry.optIn: false
newsfeed.enabled: false

server.host: '0.0.0.0'
server.port: 5601
server.maxPayload: 8388608
server.publicBaseUrl: 'https://192.168.56.101:5601'

server.ssl.enabled: true
server.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
server.ssl.key: /etc/kibana/certs/myhost/myhost.key
server.ssl.certificate: /etc/kibana/certs/myhost/myhost.crt

elasticsearch.hosts: ['https://192.168.56.101:9200']
elasticsearch.username: 'kibana_system'
elasticsearch.password: 'PASSWORD'
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
elasticsearch.ssl.key: /etc/kibana/certs/myhost/myhost.key
elasticsearch.ssl.certificate: /etc/kibana/certs/myhost/myhost.crt
elasticsearch.ssl.verificationMode: 'certificate'

elasticsearch.requestTimeout: 132000
elasticsearch.shardTimeout: 120000

kibana.autocompleteTimeout: 2000
kibana.autocompleteTerminateAfter: 500000

monitoring.enabled: true
monitoring.kibana.collection.enabled: true
monitoring.kibana.collection.interval: 30000

monitoring.ui.enabled: true
monitoring.ui.min_interval_seconds: 20

xpack.maps.showMapVisualizationTypes: true

xpack.security.enabled: true
xpack.security.audit.enabled: false

xpack.encryptedSavedObjects.encryptionKey: 'ElastiFlow_0123456789_0123456789_0123456789'
```

#### 4. Enable and Start Kibana

Execute the following commands:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable kibana && \
  sudo systemctl start kibana
```

Confirm Kibana started successfully by executing:

```shell
sudo systemctl status kibana
```

You should now be able to access Kibana at `https://IP_OF_KIBANA_HOST:5601`. Since this HTTPS connection is using a self-signed certificate, you may see an error similar to the following.

**Chrome**:

![cert\_error\_chrome](https://user-images.githubusercontent.com/10326954/132236899-ee58db1f-f46f-4ede-8ad0-f13348913a2b.png)

**Firefox**:

![cert\_error\_firefox](https://user-images.githubusercontent.com/10326954/132237367-06635dce-b23f-41e3-a912-5186a2520bd8.png)

**Safari**:

![cert\_error\_safari](https://user-images.githubusercontent.com/10326954/132237646-e15b7a6d-1335-4344-a928-eb72251cd359.png)

You need to either create an exception in your browser, or import and trust the CA certificate on the system running the browser. This can usually be achieved by downloading the `ca.crt` file from the server. Double-clicking it will usually prompt you to import the certificate. On macOS the certificate should appear as follows in the keychain application after it is configured to be trusted.

![cert\_macos](https://user-images.githubusercontent.com/10326954/132238142-46a29075-8e9c-4b4b-990c-c4945111aeea.png)

You should now be able to connect to Kibana after allowing an exception. To login, use the user `elastic` and the password you defined earlier for this user.

### Install NetObserv Flow

NetObserv Flow can be installed natively on RedHat-based Linux distributions. The instructions are available [here](/flowcoll/installation/install_linux.md#redhatalmalinux-installation-rpm). In this section, we will cover the primary configuration options for the Elasticsearch output.

The NetObserv Flow options are configured using YAML. To configure the collector, edit the file `/etc/elastiflow/flowcoll.yml`. For details on the configuration options, please refer to the [Configuration Reference](https://github.com/elastiflow/poc-public-docs/blob/main/docs/data_platforms/elastic/broken-reference/README.md).

#### 1. Request a Basic or Trial License

Without a license key, NetObserv Flow runs with a Community tier license. The Basic tier is also available at no cost and supports additional standard information elements. A license can be [requested on the ElastiFlow website](https://www.elastiflow.com/basic-license). Alternatively [a 30-day Premium trial may be requested](https://www.elastiflow.com/get-started), which increases the scalability of the collector and enables all supported vendor and standard information elements.

{% hint style="info" %}
After requesting a license it can take up to 30 minutes for the email to arrive.
{% endhint %}

License keys are generated per account. `EF_ACCOUNT_ID` must contain the Account ID for the License Key specified in `EF_LICENSE_KEY`. The ElastiFlow EULA must also be accepted to use the software.

```conf
Environment="EF_LICENSE_ACCEPTED=true"
Environment="EF_ACCOUNT_ID=FROM_THE_EMAIL"
Environment="EF_LICENSE_KEY=FROM_THE_EMAIL"
```

#### 2. Copy CA Certificate

The NetObserv Flow will require access to the CA certificate to verify the Elasticsearch node. Copy the CA certificate from `/etc/elasticsearch/certs/ca/ca.crt` to `/etc/elastiflow/ca/ca.crt`.

```shell
sudo mkdir /etc/elastiflow/ca && \
  sudo cp /etc/elasticsearch/certs/ca/ca.crt /etc/elastiflow/ca
```

#### 3. Enable the Elasticsearch Output

Set `EF_OUTPUT_ELASTICSEARCH_ENABLE` to `true` to enable the Elasticsearch output.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_ENABLE=true"
```

#### 4. Specify a Schema

The NetObserv Flow outputs data using ElastiFlow's CODEX schema. Optionally, you can choose to output data in Elastic Common Schema (ECS). To do so, set `EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE` to `true`.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE=true"
```

#### 5. Source of `@timestamp`

There are multiple possible sources to set the value of `@timestamp` field, which is the primary timestamp field used by Kibana. They supported options are:

| Value   | Field Used               | Description                                            |
| ------- | ------------------------ | ------------------------------------------------------ |
| start   | `flow.start.timestamp`   | The flow start time indicated in the flow.             |
| end     | `flow.end.timestamp`     | The flow end time (or last reported time).             |
| export  | `flow.export.timestamp`  | The time from the flow record header.                  |
| collect | `flow.collect.timestamp` | The time that the collector processed the flow record. |

Usually `end` would be the best setting. However, in the case of poorly behaving or misconfigured devices, `collect` may be the better option. The actual timestamp used may be different than configured depending on the content of the received records. If `end` is not available the collector will fall back to `export`. If `export` is not available the collector will fall back to `collect`.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE=collect"
```

#### 6. Index Shards and Replicas

For this small single node install set the number of shards to `1` and replicas to `0`.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS=1"
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS=0"
```

{% hint style="info" %}
The optimum value for these settings will depend on a number of factors. The number of shards should be at least 1 for each Elasticsearch data node in a cluster. Larger nodes (16+ CPU cores) and higher ingest rates can benefit from 2 shards per node. The largest nodes (64 CPU cores, 8 memory channels and multiple SSD drives) can even benefit from 3 or 4 shards per node. In a multi-node cluster, 1 or more replicas may be specified for redundancy.
{% endhint %}

#### 7. Index Lifecycle Management (ILM)

Index Lifecycle Management (ILM) can be used to rollover the indices which store the ElastiFlow data, preventing issues that can occur when shards become too large. Enable rollover by setting, `EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD` to `rollover`. When enabled, the collector will automatically bootstrap the initial index and write alias.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD=rollover"
```

The default Index Lifecycle Management (ILM) lifecycle is `elastiflow`. If this lifecycle doesn't exist, a basic lifecycle will be added which will remove data after 7 days. This lifecycle can be edited later via Kibana or the Elasticsearch ILM API.

#### 8. Elasticsearch Server and Credentials

Define the Elasticsearch node to which the collector should connect and the credentials for which the password was defined during the Elasticsearch installation.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_ADDRESSES=192.168.56.101:9200"
Environment="EF_OUTPUT_ELASTICSEARCH_USERNAME=elastic"
Environment="EF_OUTPUT_ELASTICSEARCH_PASSWORD=changeme"
```

#### 9. Encrypted Communications with TLS

Enable TLS and specify the path to the CA certificate.

```conf
Environment="EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE=true"
Environment="EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH=/etc/elastiflow/ca/ca.crt"
```

#### 10. Enable and Start the NetObserv Flow

Execute the following commands:

```shell
sudo systemctl daemon-reload && \
  sudo systemctl enable flowcoll && \
  sudo systemctl start flowcoll
```

Confirm the service started successfully by executing:

```shell
sudo systemctl status flowcoll
```

The collector is now ready to receive flow records from the network infrastructure.

### Import Kibana Objects

The last step is to import the Kibana saved objects and apply the recommended advanced settings. Follow the instructions in the [Kibana section of the documentation](https://github.com/elastiflow/poc-public-docs/blob/main/docs/data_platforms/elastic/broken-reference/README.md) for detailed instructions.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/data_platforms/elastic/install_redhat.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.