> For the complete documentation index, see [llms.txt](https://docs.elastiflow.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.elastiflow.com/additional-resources-reference-articles/faq/netintel_enrichment.md).

# NetIntel Enrichment Troubleshooting

### NetIntel with NetObserv 7.0+ FAQ

With NetIntel enabled, NetObserv users may need to configure specific settings, especially in environments with unique network restrictions. Here’s a guide to common scenarios, troubleshooting tips, and necessary configurations to ensure smooth operation.

***

### 1. Firewall Permissions

To properly enrich your flow data, NetObserv needs to make HTTP requests to retrieve the required datasets. In air-gapped or firewall-restricted environments, configure permissions as follows to enable secure data access:

| Protocol | Port | Direction | Description                                                                                                   |
| -------- | ---- | --------- | ------------------------------------------------------------------------------------------------------------- |
| TCP      | 443  | Outbound  | **NetIntel API**: `query.netintel.elastiflow.com`                                                             |
| TCP      | 443  | Outbound  | **CloudFlare IP Ranges**: <https://www.cloudflare.com/en-gb/ips/> API access for flow enrichment and storage. |
| TCP      | 443  | Outbound  | **Google Storage API**: `storage.googleapis.com`                                                              |

***

### 2. Too Many Requests (429 Errors)

Frequent system restarts or high request volumes may lead to 429 errors (Too Many Requests), which appear in your logs as:

```yaml
2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}
```

**Solution:**

* Stop the running instance of the collectors for at least 30 minutes, allowing the system to reset before retrying.
* Upgrade to version 7.5+, and follow our upgrade guide here: [Upgrade Documentation](https://github.com/elastiflow/documentation/blob/main/docs/kb/config/docs/kb/install/upgrade_7.5.md).

***

### 3. Timeout Errors

In low-bandwidth environments, network delays may result in timeout errors, with logs like:

```yaml
2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}
```

**Solution:**\
Extend the HTTP client’s default timeout to accommodate slower network speeds:

1. Update `flowcoll.yml` to adjust timeout as needed:

   ```yaml
   EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT: 120
   ```
2. For Docker instances using environment variables:

   ```yaml
   export EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT=120
   ```

Increasing the timeout allows for longer data-fetching intervals, reducing the chance of timeouts.

***

### 4. Air-Gapped Environments

For standard and premium customers, you can still enrich your datasets in air gapped environments by downloading the dataset manually

#### CLI Tool Installation

Download the appropriate CLI tool for your system to facilitate dataset management:

* **Ubuntu/Debian:**

  <pre class="language-yaml"><code class="lang-yaml">wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv_<code class="expression">space.vars.current_version</code>_linux_amd64.deb
  </code></pre>
* **RedHat/AlmaLinux:**

  <pre class="language-yaml"><code class="lang-yaml">wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv-<code class="expression">space.vars.current_version</code>-1.x86_64.rpm
  </code></pre>

#### Download Dataset

To download the NetIntel dataset, use the following command, ensuring all environment variables are set correctly:

```sh
EF_ACCOUNT_ID="{your account id}" \
EF_FLOW_LICENSED_UNITS=0 \
EF_FLOW_LICENSE_KEY="{your license key}" \
EF_LICENSE_ACCEPTED="true" \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH=/etc/elastiflow/netintel/ipdb.pb \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH=/etc/elastiflow/netintel/threat_collection.pb \
/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel
```

Alternatively, run:

```yaml
/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel --config={path to flowcoll.yml}
```

**Note:** Ensure the environment variables are correctly defined in your configuration file if using the `flowcoll.yml` method. This command must run in a directory with write permissions to store the downloaded dataset files.

#### Use Dataset

After downloading, provide paths to the dataset files in your configuration to enable NetIntel data processing:

* **Valid Values:**
  * `EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH`: `/etc/elastiflow/netintel/ipdb.pb`
  * `EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH`: `/etc/elastiflow/netintel/threat_collection.pb`
* **Defaults:** Leave empty if using default storage paths.

These settings allow NetObserv to load locally stored datasets, enabling secure, air-gapped operation.
