System Requirements
Compute Resources
The resources required to collect, decode and process flow records is dependent on the type of record (Netflow, IPFIX or sFlow), and the specific contents of those records.
CPU Cores
The decoding and processing flow records is primarily a CPU-centric load. While there are various factors that will affect throughput, the following table provides guidance on the expected throughput per core on current server-class processors.
| Flow Type | Flows/sec. |
|---|---|
| Netflow v5 | 3500-4000 |
| Netflow v9 | 4500-5000 |
| IPFIX | 4500-5000 |
| sFlow v5 | 3250-3750 |
As mentioned, the above values are an approximate range of records per second. Depending on the specific content of flow records in a given environment,and the actually performnance of the CPU, actual throughput may be less (or more) than the indicated range.
Memory
The collector will cache various pieces of information, such an Netflow v9 and IPFIX templates, DNS names, Interfaces names, and more. This cached data is held in memory. Memory usage is generally less than a 1-2GB. However environments which observe a high number of public IP addresses, for which Maxmind or RiskIQ enrichment options are enabled, will have higher memory requirements.
Recommended Kernel Tuning
The high volume of UDP packets experienced in many environments, combined with less than optimal default Linux kernel network parameters, can result in kernel buffer overflows and dropped packets. To minimize the chance of such data loss, especially during sudden peaks in packet volume, the Linux kernel network parameters should be modified as shown in the following examples.
The recommended settings should be added to a file in /etc/sysctl.d so that they are applied automatically when the system is booted.
For light to moderate ingest rates (less than 75000 flows per second):
net.core.netdev_max_backlog=4096
net.core.rmem_default=262144
net.core.rmem_max=67108864
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=2097152 4194304 8388608
For heavy ingest rates (more than 75000 flows per second):
net.core.netdev_max_backlog=8192
net.core.rmem_default=262144
net.core.rmem_max=134217728
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=4194304 8388608 16777216
Network Connectivity
Depending on the configured options, the ElastiFlow solution will require various TCP and UDP ports to receive flow records, retrieve data for enrichment and store data in the choosen data platform. Any host or network firewalls through which such traffic must pass, will need to be configured to allow communication on these ports.
The UDP and TCP ports used by many systems is often configurable. The following tables of port numbers refer to the default ports.
Listening for Flow Data
The Unified Flow Collector can be configured to listen for incoming flow record packets on one or more UDP ports. The default and other common ports are listed in the following table.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| UDP | 9995 | in | ElastiFlow default port |
| UDP | 2055 | in | Netflow standard port |
| UDP | 4739 | in | IPFIX standard port |
| UDP | 6343 | in | sFlow standard port |
| UDP | 9996-9998 | in | additional common ports |
While a variety of ports can be used to listen for flow record packets, the specific ports which must allowed are those for which the collector is configured using EF_FLOW_SERVER_UDP_PORT.
Accessing Enrichment Data
The Unified Flow Collector can enrich flow records with various additional information. Depending on the configured enrichment options, communication on the following ports must be allowed.
DNS
Required when EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| UDP | 53 | out | DNS |
SNMP
Required when EF_PROCESSOR_ENRICH_NETIF_SNMP_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| UDP | 161 | out | Network interface attributes via SNMP |
RiskIQ
Required when EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_ASN_ENABLE or EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_THREAT_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 443 | out | RiskIQ ASN and Threat data |
| UDP | 20000 | out | RiskIQ ASN and Threat data |
Storing Data
The Unified Flow Collector supports sending the collected and processed flow records to a variety of data platforms. The ports used by each supported platform is provided in the following tables.
Elastic Stack
Required when EF_OUTPUT_ELASTICSEARCH_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 9200 | out | Elasticsearch REST API |
| TCP | 5601 | out | Kibana UI and API |
OpenSearch
Required when EF_OUTPUT_ELASTICSEARCH_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 9200 | out | OpenSearch REST API |
| TCP | 5601 | out | OpenSearch Dashboards UI and API |
Currently the Elasticsearch output is used for storing data in both the Elasticsearch and OpenSearch. It is expected that the APIs of the applications will diverge over time. For this reason a dedicated OpenSearch-specific output will be available in a future release.
Splunk
Required when EF_OUTPUT_SPLUNK_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 8088 | out | HTTP Event Collector (HEC) |
| TCP | 8000 | out | Splunk UI |
Kafka
Required when EF_OUTPUT_KAFKA_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 9092 | out | Kafka broker |
Cribl
Required when EF_OUTPUT_CRIBL_ENABLE is true.
| Protocol | Port | Direction | Description |
|---|---|---|---|
| TCP | 8088 | out | Cribl Worker |