# Trace

### Protocol Support

The OTLP output can operate in two modes:

* **gRPC** - Uses the OTLP/gRPC protocol (recommended for production). Provides efficient binary communication with built-in features like streaming, load balancing, and multiplexing.
* **HTTP** - Uses the OTLP/HTTP protocol with RESTful endpoints. Supports Protocol Buffers format for compatibility with various backend systems.

{% hint style="info" %}
**Namespace Support**\
The OTLP output supports multiple isolated namespaces, allowing you to run separate instances of the output with different configurations. To create additional namespaces, use the pattern `EF_OUTPUT_OTLP_<NAMESPACE>_` where `<NAMESPACE>` is your namespace identifier (e.g., `EF_OUTPUT_OTLP_INTERNAL_`, `EF_OUTPUT_OTLP_PARTNER_`).
{% endhint %}

* [Server Configuration](#server-configuration)
* [gRPC Health Check Server](#grpc-health-check-server)[gRPC Health Check Server](#grpc-health-check-server)
* [Config Examples](#configuration-examples)

Determines how many dedicated workers are processing the output data

*Default*: 3

### EF\_OUTPUT\_OTLP\_TRACE\_BUFFER

Determines the channels buffer size

*Default*: `128`

### EF\_OUTPUT\_OTLP\_TRACE\_ALLOWED\_RECORD\_TYPES

Determines which record types are allowed to be exported&#x20;

*Default*: `trace,flow`

## Server Configuration

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_ENABLE

Enable or disable the OTLP trace output.

*Valid Values*: `true`, `false`

*Default*: `false`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_TYPE

Specifies the protocol type for exporting traces.

*Valid Values*: `grpc`, `http`

*Default*: `grpc`

**Details:**

* When set to `grpc`, traces are exported using the OTLP/gRPC protocol on port 4317 (default)
* When set to `http`, traces are exported to the HTTP endpoint (default: `/v1/traces`) on port 4318 (default)

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_ADDR\_HOST

The hostname or IP address of the OTLP backend server to send traces to.

*Default*: `0.0.0.0`

**Examples:**

* `localhost` - Local backend
* `otlp-collector.example.com` - Remote backend
* `10.0.1.50` - IP address

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_ADDR\_PORT

The port number of the OTLP backend server.

*Default*: `4317`

Standard Port&#x73;**:**

* `4317` - Standard OTLP/gRPC port
* `4318` - Standard OTLP/HTTP port

### EF\_OUTPUT\_OTLP\_TRACE\_WORKERS

Specifies the number of concurrent worker goroutines to handle trace export operations. Increasing the number of workers allows the output to handle higher throughput when sending traces to the backend.

*Default*: `3`

### EF\_OUTPUT\_OTLP\_TRACE\_BUFFER\_SIZE

The size of the internal channel buffer that holds trace records before they are sent to the backend. A larger buffer can help handle traffic bursts but will consume more memory.

*Default*: `128`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_BATCH\_TIMEOUT

The maximum duration to wait before sending a batch of traces to the backend. This helps optimize throughput by grouping multiple traces together.

*Default*: `5s`

*Valid Values*: Any valid Go duration string (e.g., `1s`, `500ms`, `10s`) or any integer value which will be interpreted as milliseconds (e.g., `5000` = 5 seconds)

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_COMPRESSOR

Specifies the compression algorithm to use when sending traces to the backend.&#x20;

*Valid Values*: `none`, `gzip`

*Default*: `none`

{% hint style="info" %}
Compression is only availble when using the `EF_OUTPUT_OTLP_TRACE_SERVER_TYPE` = `grpc`
{% endhint %}

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_HTTP\_ENDPOINT

The HTTP endpoint path for sending traces when using HTTP protocol.

*Default*: `/v1/traces`

{% hint style="info" %}
Only applicable when `EF_OUTPUT_OTLP_TRACE_SERVER_TYPE=http`&#x20;
{% endhint %}

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_TLS\_ENABLE

Enable or disable TLS encryption for the connection to the OTLP backend.

*Valid Values*: `true`, `false`

*Default*: `false`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_TLS\_CERT\_FILE

The path to the TLS certificate file for client authentication.

{% hint style="info" %}
Required when the backend requires mutual TLS (mTLS) authentication.
{% endhint %}

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_FAILURE\_INIT\_INTERVAL

The initial wait time before the first retry attempt after a failed export.

*Valid Values*: Any valid Go duration string (e.g., `500ms`, `2s`, `5s`) or any integer value which will be interpreted as milliseconds (e.g., `1000` = 1 second)

*Default*: `1s`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_FAILURE\_MAX\_RETRY\_INTERVAL

The maximum wait time between retry attempts. The retry interval increases exponentially up to this maximum.

*Valid Values*: Any valid Go duration string (e.g., `500ms`, `2s`, `5s`) or any integer value which will be interpreted as milliseconds (e.g., `1000` = 1 second)

Default: `5s`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_FAILURE\_MAX\_ELAPSED\_TIME

The maximum total time to spend retrying failed exports before giving up.

*Valid Values*: Any valid Go duration string (e.g., `500ms`, `2s`, `5s`) or any integer value which will be interpreted as milliseconds (e.g., `1000` = 1 second)

*Default*: `1h`

### EF\_OUTPUT\_OTLP\_TRACE\_SERVER\_HEADERS

Custom HTTP headers to include with each trace export request. This is useful for authentication tokens, API keys, or other metadata required by the backend.

Format: comma delimited list of key value pairs

**Example**:

```bash
EF_OUTPUT_OTLP_TRACE_SERVER_HEADERS='x-greptime-pipeline-name=greptime_trace_v1'
```

### gRPC Health Check Server

A separate health check server is provided for Kubernetes liveness and readiness probes. The health check server implements the gRPC Health Checking Protocol (`grpc.health.v1.Health`).

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_ADDR\_HOST

The IP address to bind the health check server to.

*Default*: `0.0.0.0`

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_ADDR\_PORT

The port number for the health check server to listen on.

*Default*: `9412`

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_TLS\_ENABLE

Enable or disable TLS encryption for the health check server.

*Valid Values*: `true`, `false`

*Default*: `false`

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_TLS\_CERT\_FILE

The path to the TLS certificate file for the health check server.

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_TLS\_KEY\_FILE

The path to the TLS private key file for the health check server.

### EF\_OUTPUT\_OTLP\_HEALTH\_SERVER\_TLS\_VERIFICATION\_TYPE

The client certificate verification mode for the health check server.

*Valid Values:*

* `NoClientCert` (or `0`) - No client certificate required
* `RequestClientCert` (or `1`) - Request client certificate but don't verify
* `RequireAnyClientCert` (or `2`) - Require client certificate but don't verify CA
* `VerifyClientCertIfGiven` (or `3`) - Verify client certificate if provided
* `RequireAndVerifyClientCert` (or `4`) - Require and verify client certificate

*Default*: `NoClientCert`

## Configuration Examples

### Basic gRPC Configuration

Export traces to a local OTLP collector using gRPC:

```yaml
EF_OUTPUT_OTLP_TRACE_SERVER_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TYPE: grpc
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_HOST: localhost
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_PORT: 4317
EF_OUTPUT_OTLP_TRACE_WORKERS: 3
EF_OUTPUT_OTLP_TRACE_BUFFER_SIZE: 128
```

### gRPC with TLS and Custom Headers

Secure configuration with TLS encryption and API key authentication:

```yaml
EF_OUTPUT_OTLP_TRACE_SERVER_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TYPE: grpc
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_HOST: otlp.example.com
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_PORT: 4317
EF_OUTPUT_OTLP_TRACE_WORKERS: 5
EF_OUTPUT_OTLP_TRACE_BUFFER_SIZE: 256
EF_OUTPUT_OTLP_TRACE_SERVER_BATCH_TIMEOUT: 10s
EF_OUTPUT_OTLP_TRACE_SERVER_COMPRESSOR: gzip

# TLS Configuration
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_CERT_FILE: /etc/certs/client.crt
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_KEY_FILE: /etc/certs/client.key

# Custom Headers for Authentication
EF_OUTPUT_OTLP_TRACE_SERVER_HEADERS: '[{"key":"X-API-Key","value":"your-secret-api-key"}]'

# Health Check Server
EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_HOST: 0.0.0.0
EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_PORT: 9412
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.elastiflow.com/flowcoll/configuration/outputs/otel/trace.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.