NetIntel Enrichment Troubleshooting

NetIntel with NetObserv 7.0+ FAQ

With NetIntel enabled, NetObserv users may need to configure specific settings, especially in environments with unique network restrictions. Here’s a guide to common scenarios, troubleshooting tips, and necessary configurations to ensure smooth operation.

---

1. Firewall Permissions

To properly enrich your flow data, NetObserv needs to make HTTP requests to retrieve the required datasets. In air-gapped or firewall-restricted environments, configure permissions as follows to enable secure data access:

Protocol	Port	Direction	Description
TCP	443	Outbound	NetIntel API: query.netintel.elastiflow.com
TCP	443	Outbound	CloudFlare IP Ranges: https://www.cloudflare.com/en-gb/ips/ API access for flow enrichment and storage.
TCP	443	Outbound	Google Storage API: storage.googleapis.com

---

2. Too Many Requests (429 Errors)

Frequent system restarts or high request volumes may lead to 429 errors (Too Many Requests), which appear in your logs as:

2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}

Solution:

• Stop the running instance of the collectors for at least 30 minutes, allowing the system to reset before retrying.

• Upgrade to version 7.5+, and follow our upgrade guide here: Upgrade Documentation.

---

3. Timeout Errors

In low-bandwidth environments, network delays may result in timeout errors, with logs like:

2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}

Solution: Extend the HTTP client’s default timeout to accommodate slower network speeds:

1. Update flowcoll.yml to adjust timeout as needed:

   EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT: 120

2. For Docker instances using environment variables:

   export EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT=120

Increasing the timeout allows for longer data-fetching intervals, reducing the chance of timeouts.

---

4. Air-Gapped Environments

For standard and premium customers, you can still enrich your datasets in air gapped environments by downloading the dataset manually

CLI Tool Installation

Download the appropriate CLI tool for your system to facilitate dataset management:

• Ubuntu/Debian:

  wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv_7.24.0_linux_amd64.deb

• RedHat/AlmaLinux:

  wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv-7.24.0-1.x86_64.rpm

Download Dataset

To download the NetIntel dataset, use the following command, ensuring all environment variables are set correctly:

EF_ACCOUNT_ID="{your account id}" \
EF_FLOW_LICENSED_UNITS=0 \
EF_FLOW_LICENSE_KEY="{your license key}" \
EF_LICENSE_ACCEPTED="true" \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH=/etc/elastiflow/netintel/ipdb.pb \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH=/etc/elastiflow/netintel/threat_collection.pb \
/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel

Alternatively, run:

/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel --config={path to flowcoll.yml}

Note: Ensure the environment variables are correctly defined in your configuration file if using the flowcoll.yml method. This command must run in a directory with write permissions to store the downloaded dataset files.

Use Dataset

After downloading, provide paths to the dataset files in your configuration to enable NetIntel data processing:

• Valid Values:

  • EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH: /etc/elastiflow/netintel/ipdb.pb

  • EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH: /etc/elastiflow/netintel/threat_collection.pb

• Defaults: Leave empty if using default storage paths.

These settings allow NetObserv to load locally stored datasets, enabling secure, air-gapped operation.