Trace

The OTEL output enables the collector to export trace data to OpenTelemetry-compliant backends.

Protocol Support

The OTLP output can operate in two modes:

• gRPC - Uses the OTLP/gRPC protocol (recommended for production). Provides efficient binary communication with built-in features like streaming, load balancing, and multiplexing.

• HTTP - Uses the OTLP/HTTP protocol with RESTful endpoints. Supports Protocol Buffers format for compatibility with various backend systems.

Namespace Support The OTLP output supports multiple isolated namespaces, allowing you to run separate instances of the output with different configurations. To create additional namespaces, use the pattern EF_OUTPUT_OTLP_<NAMESPACE>_ where <NAMESPACE> is your namespace identifier (e.g., EF_OUTPUT_OTLP_INTERNAL_, EF_OUTPUT_OTLP_PARTNER_).

• Server Configuration

• gRPC Health Check ServergRPC Health Check Server

• Config Examples

Determines how many dedicated workers are processing the output data

Default: 3

EF_OUTPUT_OTLP_TRACE_BUFFER

Determines the channels buffer size

Default: 128

EF_OUTPUT_OTLP_TRACE_ALLOWED_RECORD_TYPES

Determines which record types are allowed to be exported

Default: trace,flow

Server Configuration

EF_OUTPUT_OTLP_TRACE_SERVER_ENABLE

Enable or disable the OTLP trace output.

Valid Values: true, false

Default: false

EF_OUTPUT_OTLP_TRACE_SERVER_TYPE

Specifies the protocol type for exporting traces.

Valid Values: grpc, http

Default: grpc

Details:

• When set to grpc, traces are exported using the OTLP/gRPC protocol on port 4317 (default)

• When set to http, traces are exported to the HTTP endpoint (default: /v1/traces) on port 4318 (default)

EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_HOST

The hostname or IP address of the OTLP backend server to send traces to.

Default: 0.0.0.0

Examples:

• localhost - Local backend

• otlp-collector.example.com - Remote backend

• 10.0.1.50 - IP address

EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_PORT

The port number of the OTLP backend server.

Default: 4317

Standard Ports:

• 4317 - Standard OTLP/gRPC port

• 4318 - Standard OTLP/HTTP port

EF_OUTPUT_OTLP_TRACE_WORKERS

Specifies the number of concurrent worker goroutines to handle trace export operations. Increasing the number of workers allows the output to handle higher throughput when sending traces to the backend.

Default: 3

EF_OUTPUT_OTLP_TRACE_BUFFER_SIZE

The size of the internal channel buffer that holds trace records before they are sent to the backend. A larger buffer can help handle traffic bursts but will consume more memory.

Default: 128

EF_OUTPUT_OTLP_TRACE_SERVER_BATCH_TIMEOUT

The maximum duration to wait before sending a batch of traces to the backend. This helps optimize throughput by grouping multiple traces together.

Default: 5s

Valid Values: Any valid Go duration string (e.g., 1s, 500ms, 10s) or any integer value which will be interpreted as milliseconds (e.g., 5000 = 5 seconds)

EF_OUTPUT_OTLP_TRACE_SERVER_COMPRESSOR

Specifies the compression algorithm to use when sending traces to the backend.

Valid Values: none, gzip

Default: none

Compression is only availble when using the EF_OUTPUT_OTLP_TRACE_SERVER_TYPE = grpc

EF_OUTPUT_OTLP_TRACE_SERVER_HTTP_ENDPOINT

The HTTP endpoint path for sending traces when using HTTP protocol.

Default: /v1/traces

Only applicable when EF_OUTPUT_OTLP_TRACE_SERVER_TYPE=http

EF_OUTPUT_OTLP_TRACE_SERVER_TLS_ENABLE

Enable or disable TLS encryption for the connection to the OTLP backend.

Valid Values: true, false

Default: false

EF_OUTPUT_OTLP_TRACE_SERVER_TLS_CERT_FILE

The path to the TLS certificate file for client authentication.

Required when the backend requires mutual TLS (mTLS) authentication.

EF_OUTPUT_OTLP_TRACE_SERVER_FAILURE_INIT_INTERVAL

The initial wait time before the first retry attempt after a failed export.

Valid Values: Any valid Go duration string (e.g., 500ms, 2s, 5s) or any integer value which will be interpreted as milliseconds (e.g., 1000 = 1 second)

Default: 1s

EF_OUTPUT_OTLP_TRACE_SERVER_FAILURE_MAX_RETRY_INTERVAL

The maximum wait time between retry attempts. The retry interval increases exponentially up to this maximum.

Valid Values: Any valid Go duration string (e.g., 500ms, 2s, 5s) or any integer value which will be interpreted as milliseconds (e.g., 1000 = 1 second)

Default: 5s

EF_OUTPUT_OTLP_TRACE_SERVER_FAILURE_MAX_ELAPSED_TIME

The maximum total time to spend retrying failed exports before giving up.

Valid Values: Any valid Go duration string (e.g., 500ms, 2s, 5s) or any integer value which will be interpreted as milliseconds (e.g., 1000 = 1 second)

Default: 1h

EF_OUTPUT_OTLP_TRACE_SERVER_HEADERS

Custom HTTP headers to include with each trace export request. This is useful for authentication tokens, API keys, or other metadata required by the backend.

Format: comma delimited list of key value pairs

Example:

EF_OUTPUT_OTLP_TRACE_SERVER_HEADERS='x-greptime-pipeline-name=greptime_trace_v1'

gRPC Health Check Server

A separate health check server is provided for Kubernetes liveness and readiness probes. The health check server implements the gRPC Health Checking Protocol (grpc.health.v1.Health).

EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_HOST

The IP address to bind the health check server to.

Default: 0.0.0.0

EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_PORT

The port number for the health check server to listen on.

Default: 9412

EF_OUTPUT_OTLP_HEALTH_SERVER_TLS_ENABLE

Enable or disable TLS encryption for the health check server.

Valid Values: true, false

Default: false

EF_OUTPUT_OTLP_HEALTH_SERVER_TLS_CERT_FILE

The path to the TLS certificate file for the health check server.

EF_OUTPUT_OTLP_HEALTH_SERVER_TLS_KEY_FILE

The path to the TLS private key file for the health check server.

EF_OUTPUT_OTLP_HEALTH_SERVER_TLS_VERIFICATION_TYPE

The client certificate verification mode for the health check server.

Valid Values:

• NoClientCert (or 0) - No client certificate required

• RequestClientCert (or 1) - Request client certificate but don't verify

• RequireAnyClientCert (or 2) - Require client certificate but don't verify CA

• VerifyClientCertIfGiven (or 3) - Verify client certificate if provided

• RequireAndVerifyClientCert (or 4) - Require and verify client certificate

Default: NoClientCert

Configuration Examples

Basic gRPC Configuration

Export traces to a local OTLP collector using gRPC:

EF_OUTPUT_OTLP_TRACE_SERVER_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TYPE: grpc
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_HOST: localhost
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_PORT: 4317
EF_OUTPUT_OTLP_TRACE_WORKERS: 3
EF_OUTPUT_OTLP_TRACE_BUFFER_SIZE: 128

gRPC with TLS and Custom Headers

Secure configuration with TLS encryption and API key authentication:

EF_OUTPUT_OTLP_TRACE_SERVER_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TYPE: grpc
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_HOST: otlp.example.com
EF_OUTPUT_OTLP_TRACE_SERVER_ADDR_PORT: 4317
EF_OUTPUT_OTLP_TRACE_WORKERS: 5
EF_OUTPUT_OTLP_TRACE_BUFFER_SIZE: 256
EF_OUTPUT_OTLP_TRACE_SERVER_BATCH_TIMEOUT: 10s
EF_OUTPUT_OTLP_TRACE_SERVER_COMPRESSOR: gzip

# TLS Configuration
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_ENABLE: true
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_CERT_FILE: /etc/certs/client.crt
EF_OUTPUT_OTLP_TRACE_SERVER_TLS_KEY_FILE: /etc/certs/client.key

# Custom Headers for Authentication
EF_OUTPUT_OTLP_TRACE_SERVER_HEADERS: '[{"key":"X-API-Key","value":"your-secret-api-key"}]'

# Health Check Server
EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_HOST: 0.0.0.0
EF_OUTPUT_OTLP_HEALTH_SERVER_ADDR_PORT: 9412