Skip to content

feat(tracing): correlate distributed traces with metrics via OTel exemplars #9798

@waveywaves

Description

@waveywaves

Feature request

Enable OTel exemplar collection so that metrics (histograms, counters) carry trace_id/span_id references. This bridges tracing and metrics at zero cardinality cost.

Use case

When a metric spike occurs (e.g., pipelinerun_duration_seconds P99 jumps), operators need to navigate directly to the trace that caused it. OTel exemplars attach trace context to metric data points, enabling one-click metric-to-trace navigation in Grafana/Tempo.

Implementation hints

MeterProvider config — pkg/reconciler/taskrun/metrics.go and pipelinerun/metrics.go:

The MeterProvider is created without sdkmetric.WithExemplarFilter(). Enable exemplars:

sdkmetric.WithExemplarFilter(exemplar.TraceBasedFilter)

The Prometheus exporter (vendor/.../exporters/prometheus/exporter.go) already has exemplar serialization code — no exporter changes needed.

Context threading: Ensure the ctx passed to DurationAndCount() and RecordPodLatency() carries the active span from initTracing(). The traced context may not be threaded into metrics recording today.

Vendored exemplar support: vendor/go.opentelemetry.io/otel/sdk/metric/exemplar/ is already present.

Part of umbrella: #9701

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions