# NetIntel Enrichment Troubleshooting

### NetIntel with NetObserv 7.0+ FAQ

With NetIntel enabled, NetObserv users may need to configure specific settings, especially in environments with unique network restrictions. Here’s a guide to common scenarios, troubleshooting tips, and necessary configurations to ensure smooth operation.

***

### 1. Firewall Permissions

To properly enrich your flow data, NetObserv needs to make HTTP requests to retrieve the required datasets. In air-gapped or firewall-restricted environments, configure permissions as follows to enable secure data access:

| Protocol | Port | Direction | Description                                                                                                   |
| -------- | ---- | --------- | ------------------------------------------------------------------------------------------------------------- |
| TCP      | 443  | Outbound  | **NetIntel API**: `query.netintel.elastiflow.com`                                                             |
| TCP      | 443  | Outbound  | **CloudFlare IP Ranges**: <https://www.cloudflare.com/en-gb/ips/> API access for flow enrichment and storage. |
| TCP      | 443  | Outbound  | **Google Storage API**: `storage.googleapis.com`                                                              |

***

### 2. Too Many Requests (429 Errors)

Frequent system restarts or high request volumes may lead to 429 errors (Too Many Requests), which appear in your logs as:

```yaml
2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}
```

**Solution:**

* Stop the running instance of the collectors for at least 30 minutes, allowing the system to reset before retrying.
* Upgrade to version 7.5+, and follow our upgrade guide here: [Upgrade Documentation](https://github.com/elastiflow/documentation/blob/main/docs/kb/config/docs/kb/install/upgrade_7.5.md).

***

### 3. Timeout Errors

In low-bandwidth environments, network delays may result in timeout errors, with logs like:

```yaml
2024-11-05T10:55:53.879-0400 error ipaddr_enricher.netintel_threats netintel/enricher.go:104 error initializing cidr tree & threat collection {"error": "unavailable: 429 Too Many Requests"}
```

**Solution:**\
Extend the HTTP client’s default timeout to accommodate slower network speeds:

1. Update `flowcoll.yml` to adjust timeout as needed:

   ```yaml
   EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT: 120
   ```
2. For Docker instances using environment variables:

   ```yaml
   export EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT=120
   ```

Increasing the timeout allows for longer data-fetching intervals, reducing the chance of timeouts.

***

### 4. Air-Gapped Environments

For standard and premium customers, you can still enrich your datasets in air gapped environments by downloading the dataset manually

#### CLI Tool Installation

Download the appropriate CLI tool for your system to facilitate dataset management:

* **Ubuntu/Debian:**

  <pre class="language-yaml"><code class="lang-yaml">wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv_<code class="expression">space.vars.current_version</code>_linux_amd64.deb
  </code></pre>
* **RedHat/AlmaLinux:**

  <pre class="language-yaml"><code class="lang-yaml">wget https://elastiflow-releases.s3.us-east-2.amazonaws.com/netobserv/netobserv-<code class="expression">space.vars.current_version</code>-1.x86_64.rpm
  </code></pre>

#### Download Dataset

To download the NetIntel dataset, use the following command, ensuring all environment variables are set correctly:

```sh
EF_ACCOUNT_ID="{your account id}" \
EF_FLOW_LICENSED_UNITS=0 \
EF_FLOW_LICENSE_KEY="{your license key}" \
EF_LICENSE_ACCEPTED="true" \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH=/etc/elastiflow/netintel/ipdb.pb \
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH=/etc/elastiflow/netintel/threat_collection.pb \
/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel
```

Alternatively, run:

```yaml
/usr/share/elastiflow/bin/netobserv pull dataset --source=netintel --config={path to flowcoll.yml}
```

**Note:** Ensure the environment variables are correctly defined in your configuration file if using the `flowcoll.yml` method. This command must run in a directory with write permissions to store the downloaded dataset files.

#### Use Dataset

After downloading, provide paths to the dataset files in your configuration to enable NetIntel data processing:

* **Valid Values:**
  * `EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_IP_DB_PATH`: `/etc/elastiflow/netintel/ipdb.pb`
  * `EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_THREAT_COLLECTION_PATH`: `/etc/elastiflow/netintel/threat_collection.pb`
* **Defaults:** Leave empty if using default storage paths.

These settings allow NetObserv to load locally stored datasets, enabling secure, air-gapped operation.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/additional-resources-reference-articles/faq/netintel_enrichment.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.