# Prometheus & Grafana

### Prerequisites

1. [The ElastiFlow Unified Collector is running and the metrics endpoint is reachable. (Reachable @ http://0.0.0.0:8080/metrics)](#elastiflow-metrics-endpoint)
2. [Prometheus is installed & running and can reach the NetObserv Flow Metrics endpoint. (Prometheus default @ http://0.0.0.0:9090)](#prometheus-installation-guide)
3. [Grafana is installed & running and can reach Prometheus. (Grafana default @ http://0.0.0.0:3000)](#grafana-installation-guide)

### ElastiFlow Metrics Endpoint

* Verify the ElastiFlow Unified Collector & Metrics Endpoint have successfully started (logs in /var/log/elastiflow/flowcoll/flowcoll.log):

```conf
2022-10-21T08:31:23.078-0700  info  pipeline/appserver.go:44  metrics exposed at http://0.0.0.0:8080/metrics
```

{% hint style="info" %}
If another service is already using port `8080`, you can change the port on which the collector listens by setting [`EF_API_PORT`](https://github.com/elastiflow/documentation/blob/main/docs/config_ref/common/api/README.md#ef_api_port).
{% endhint %}

### Grafana Installation Guide

* OS specific Grafana installation guides: <https://grafana.com/docs/grafana/latest/setup-grafana/installation/>
* Debian/Ubuntu installation guide: <https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/>
* RPM-based installation guide: <https://grafana.com/docs/grafana/latest/setup-grafana/installation/rpm/>

### Prometheus Installation Guide

* Main Installation guides: <https://prometheus.io/docs/prometheus/latest/installation/>
* Precompiled binaries: <https://prometheus.io/download/>

#### Prometheus Configuration

1. Once Prometheus is installed edit `prometheus.yml` and add a job pointing to the ElastiFlow Unified Collector as in the following example:

Example default Prometheus configuration:

```yaml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any time series scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
```

An ElastiFlow-specific Job:

```yaml
    job_name: elastiflow-metrics-1
    scrape_interval: 15s
    metrics_path: /metrics

    scheme: http
    static_configs:
      - targets: ['192.168.3.18:8080']
```

### Connect Grafana To Prometheus

1. Once Grafana is up and running; from the "Welcome To Grafana" page click the cogwheel in the bottom left corner, then click "Data Sources", and finally click "Add Data Source": ![image](https://user-images.githubusercontent.com/43585378/198896237-86b1b14e-7927-4e6e-a027-15d8b2773bc3.png)
2. Select Prometheus ![Image](https://user-images.githubusercontent.com/43585378/198896308-9705b414-d7f3-4536-86b9-0a6b7594bdf1.png)
3. Name this Prometheus Datasource, type your Prometheus endpoint IP address and port in the "URL" field, and configure the authentication and "scrape interval" you have set up:

&#x20;![Iamge](https://user-images.githubusercontent.com/43585378/198896472-60632d65-aed7-43d9-a69d-fcf14333776b.png)&#x20;

4. Click "Save & Test"

&#x20;![Image](https://user-images.githubusercontent.com/43585378/198896476-b428b194-387c-4128-9916-fca7a529ae10.png)

### Import The ElastiFlow Metrics Dashboard

The ElastiFlow Metrics Dashboard can be found by searching Grafana Labs Dashboards: <https://grafana.com/grafana/dashboards/>

Direct Link: <https://grafana.com/grafana/dashboards/17306-elastiflow-metrics/>

![Image](https://user-images.githubusercontent.com/43585378/198896571-e33b4167-1c31-4f1e-a71c-6049219abb1d.png) ![Image](https://user-images.githubusercontent.com/43585378/199298518-5c5dd192-c335-41c4-80b3-922eaa84eaa5.png)

1. Import the Dashboard by entering the dashboard ID (17306) found after searching ElastiFlow Metrics in the Grafana Labs Dashboards search by going to Dashboards -> Import and clicking "Load":

![Image](https://user-images.githubusercontent.com/43585378/198896760-1eb827c7-50ed-49f4-a550-be1a70e1cc01.png) ![Image](https://user-images.githubusercontent.com/43585378/198905376-bbec59ee-fbd9-4f83-8b44-0db108961eb9.png)&#x20;

2. Accept the default or give the dashboard a name and place it in the folder you'd like it to reside in. After this point to the Prometheus Datasource you previously connected above click "Import" and you are done.

![Image](https://user-images.githubusercontent.com/43585378/198905499-7e724ba8-b7ae-4b30-86de-a340ae6d94e3.png)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/flowcoll/overview/monitoring/prom_grafana.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
