Grafana Cloud

If you use a Grafana Cloud stack, your cluster can forward your applications’ OpenTelemetry traces and metrics to it. Your applications send to an endpoint inside the cluster, and the cluster forwards to Grafana Cloud. The Grafana Cloud tokens stay out of your applications, and you choose which services get forwarded. Your in-cluster Prometheus, Loki, Grafana and Tempo keep running as before; Grafana Cloud is an extra destination, not a replacement.

How it works

Traces and metrics each go through their own Grafana Alloy gateway in the observability namespace:

Both accept OTLP over gRPC on port 4317 and over HTTP on port 4318.

Each gateway only forwards the services you list, matched on the service.name your application reports. Everything else is dropped. An empty list forwards nothing, so turning the feature on sends no data until you add a service. That keeps every new service’s Grafana Cloud cost a deliberate choice.

The separate metrics gateway means a burst of metrics can’t push traces out, and the other way around.

This page covers what your applications push over OpenTelemetry. The cluster’s own metrics from Prometheus go to Grafana Cloud through Prometheus remote write instead; ask us if you need that.

What you need from Grafana Cloud

From your Grafana Cloud stack’s details page:

• For traces: the Tempo instance ID and its OTLP gRPC endpoint (host:port, for example tempo-prod-10-prod-eu-west-2.grafana.net:443).
• For metrics: the instance ID and the OTLP endpoint from the OpenTelemetry tile (for example https://otlp-gateway-prod-eu-west-2.grafana.net/otlp). This instance ID is not the Tempo one.
• An access policy token per signal: one with the traces:write scope, one with the metrics:write scope. Separate tokens keep each gateway to the one thing it needs.

Each token goes into the cluster definition as a tokenPayload, KMS-encrypted with the context k8s_stack=secrets, as for other secrets in your cluster definition.

Send traces

spec:
  observability:
    tracing:
      grafanaCloud:
        enabled: true
        endpoint: tempo-prod-10-prod-eu-west-2.grafana.net:443
        username: "123456"
        tokenPayload: a-kms-encrypted-payload
        services:
          - checkout
          - payments

When the in-cluster Tempo is on as well, it keeps receiving every span; only the listed services also reach Grafana Cloud.

Send metrics

spec:
  observability:
    metrics:
      grafanaCloud:
        enabled: true
        endpoint: https://otlp-gateway-prod-eu-west-2.grafana.net/otlp
        username: "654321"
        tokenPayload: a-kms-encrypted-payload
        services:
          - checkout
        dropMetrics:
          - v8js.memory.heap.limit

Leave /v1/metrics off the endpoint: the gateway adds it.

dropMetrics removes whole metrics before they leave the cluster, so you don’t pay for metrics you never look at. Use the name your OpenTelemetry SDK emits (http.server.request.duration), not the name Grafana Cloud shows after converting it to Prometheus style (http_server_request_duration_seconds). Dropping single attributes (labels) isn’t possible yet; if one attribute drives your cost, let us know.

Configure your application

Your SDK needs a separate endpoint per signal, since traces and metrics go to different gateways. The standard OpenTelemetry environment variables cover that. With gRPC:

env:
  - name: OTEL_SERVICE_NAME
    value: checkout
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: grpc
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: http://alloy.observability.svc.cluster.local:4317
  - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
    value: http://alloy-metrics.observability.svc.cluster.local:4317

With OTLP over HTTP (OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf), use port 4318 and include the path, because the SDK uses a signal-specific endpoint exactly as given:

  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: http://alloy.observability.svc.cluster.local:4318/v1/traces
  - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
    value: http://alloy-metrics.observability.svc.cluster.local:4318/v1/metrics

OTEL_SERVICE_NAME must match an entry in services exactly, including case. The Tempo guide covers instrumenting your application.

Troubleshooting

Nothing arrives in Grafana Cloud

• Check that the service is listed in services and that OTEL_SERVICE_NAME matches it exactly.
• Check that your application sends traces and metrics to the right gateway: metrics sent to alloy are not forwarded.
• Check the token’s scope (traces:write or metrics:write) and that the instance ID belongs to the same signal. A wrong token or instance ID makes Grafana Cloud refuse the data.
• The Alloy dashboards in your cluster’s Grafana show whether each gateway’s components are healthy.