Skip to main content
Version: 7.11

NetIntel Enrichment Troubleshooting

NetIntel with NetObserv 7.0+ FAQ​

With NetIntel enabled, NetObserv users may need to configure specific settings, especially in environments with unique network restrictions. Here’s a guide to common scenarios, troubleshooting tips, and necessary configurations to ensure smooth operation.

1. Firewall Permissions​

To properly enrich your flow data, NetObserv needs to make HTTP requests to retrieve the required datasets. In air-gapped or firewall-restricted environments, configure permissions as follows to enable secure data access:

ProtocolPortDirectionDescription
TCP443OutboundNetIntel API: query.netintel.elastiflow.com
TCP443OutboundCloudFlare IP Ranges: https://www.cloudflare.com/en-gb/ips/ API access for flow enrichment and storage.
TCP443OutboundGoogle Storage API: storage.googleapis.com

2. Too Many Requests (429 Errors)​

Frequent system restarts or high request volumes may lead to 429 errors (Too Many Requests), which appear in your logs as:

2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}

Solution:

  • Stop the running instance of the collectors for at least 30 minutes, allowing the system to reset before retrying.
  • Upgrade to version 7.5+, and follow our upgrade guide here: Upgrade Documentation.

3. Timeout Errors​

In low-bandwidth environments, network delays may result in timeout errors, with logs like:

2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}

Solution:
Extend the HTTP client’s default timeout to accommodate slower network speeds:

  1. Update flowcol.yml to adjust timeout as needed:

    EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT: 120

  2. For Docker instances using environment variables:

    export EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT=120

Increasing the timeout allows for longer data-fetching intervals, reducing the chance of timeouts.

4. Air-Gapped Environments​

For standard and premium customers, you can still enrich your datasets in air gapped environments by downloading the dataset manually

CLI Tool Installation​

Download the appropriate CLI tool for your system to facilitate dataset management:

  • Ubuntu/Debian:

    wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv_7.5.0_linux_amd64.deb

  • RedHat/AlmaLinux:

    wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv-7.5.0-1.x86_64.rpm

Download Dataset​

To download the NetIntel dataset, use the following command, ensuring all environment variables are set correctly:

EF_ACCOUNT_ID="{your account id}" \
EF_FLOW_LICENSED_UNITS=0 \
EF_FLOW_LICENSE_KEY="{your license key}" \
EF_LICENSE_ACCEPTED="true" \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH=/etc/elastiflow/netintel/ipdb.pb \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH=/etc/elastiflow/netintel/threat_collection.pb \
./netobserv pull dataset --source=netintel

Alternatively, run:

./netobserv pull dataset --source=netintel --config={path to flowcol.yml}

Note: Ensure the environment variables are correctly defined in your configuration file if using the flowcol.yml method. This command must run in a directory with write permissions to store the downloaded dataset files.

Use Dataset​

After downloading, provide paths to the dataset files in your configuration to enable NetIntel data processing:

  • Valid Values:

    • EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH: /etc/elastiflow/netintel/ipdb.pb
    • EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH: /etc/elastiflow/netintel/threat_collection.pb

  • Defaults: Leave empty if using default storage paths.

These settings allow NetObserv to load locally stored datasets, enabling secure, air-gapped operation.