Grafana Release Notes

⚠ BREAKING CHANGES

• discovery.hetzner: discovery.hetzner no longer exports the __meta_hetzner_datacenter label for targets with role = "hcloud". Relabeling rules that reference this label for hcloud targets silently stop matching. The label is unaffected for role = "robot" targets, which also now export it under the additional name __meta_hetzner_robot_datacenter.
• otelcol.receiver.filelog: otelcol.receiver.filelog's top_n now distinguishes an explicit 0 ("track all files") from leaving it unset ("apply the default"), matching upstream. Breaking change: previously, top_n = 0 was treated the same as unset.
• otelcol.exporter.awss3: Apply upstream retry defaults and expose retry_on_failure (#7138)
• otelcol.processor.memory_limiter: Apply upstream GC interval defaults and expose them (#7137)
• otelcol.receiver.kafka: plaintext credentials are now translated into a sasl block with mechanism set to PLAIN, matching what upstream's Kafka client actually requires. This also applies to otelcol.exporter.kafka. Configs relying on the old (broken) silent-drop behavior will now authenticate correctly instead of connecting anonymously. …

⚠ BREAKING CHANGES

• prometheus.operator.servicemonitors: prometheus.operator.servicemonitors now rejects ServiceMonitor endpoints that reference local files through bearerTokenFile, tlsConfig.caFile, tlsConfig.certFile, or tlsConfig.keyFile by default. Set allow_arbitrary_file_access = true to preserve the previous behavior for trusted ServiceMonitor authors.
• The memory_limiter processor's internal telemetry metrics were renamed upstream in OpenTelemetry Collector v0.155.0 to carry a memory_limiter prefix
• Remove the prometheus.write.queue component (#6814)

Features 🌟

• Add ability to exempt lines matching a configured regex pattern from loki.process stage.luhn (#6608) (2e0748d)
• Add opamp supervisor (#6815) (e891e30)
• Add start_timestamp_zero_ingestion to Prometheus metric components (#6610) (1a9ed95)
• beyla.ebpf: Run Beyla as a subprocess (#6695) (61ec3d2) …

⚠ BREAKING CHANGES

• otelcol.receiver.otlp, otelcol.receiver.faro, otelcol.receiver.jaeger, otelcol.receiver.zipkin, otelcol.receiver.influxdb, otelcol.receiver.splunkhec, otelcol.receiver.datadog, and otelcol.extension.jaeger_remote_sampling now default to idle_timeout = "1m", read_header_timeout = "1m", and write_timeout = "30s" on their HTTP servers, matching upstream default. These were previously being defaulted to "0s". You can explicitly set them to "0s" to restore the previous behavior.
• otelcol.receiver.kafka and otelcol.exporter.kafka: resolve_canonical_bootstrap_servers_only is now a no-op and deprecated. The upstream migration from migration from sarama to `franz-go dropped the behavior. The argument still parses but no longer resolves bootstrap server addresses.
• otelcol.exporter.splunkhec — batcher block removed (upstream deletion in otel-contrib v0.151). Configs using it will fail to load. Migration: move the equivalent fields under sending_queue.batch (enabled → presence of the block; flush_timeout/sizer/min_size/max_size keep their names).

Features 🌟

• Add alloy.command to override the Alloy Helm container entrypoint (#6509) (fea3056) …

Features 🌟

• Add gql subcommand (#6242) (ec09c43)
• database_observability.mysql: Add database_observability_wait_event_seconds_total counter (#6106) (602b0b5)
• database_observability.mysql: Refactor bulk table metadata collection (#6222) (e8d441b)
• database_observability.mysql: Refactor schema_details logging strategy (#6239) (fd51530)
• database_observability.postgres: Add exclude_current_user top-level setting (#6187) (f019b5e)
• database_observability.postgres: Improve monitoring user privileges check (#6177) (96b50d1) …

Features 🌟

• Add clustering for loki.source.kubernetes_events (#6027) (3dbf587)
• Add otelcol.auth.google client auth provider (#5526) (da99a66)
• beyla.ebpf: Bump to v3.7.0 (#5966) (5126c2e)
• database_observability: Add support for GCP Cloud SQL metadata (#5875) (5d23245)
• database_observability: Make targets optional (#5924) (54664b2)
• database_observability: Update default excluded schemas and users (#6080) (b386fff)
• faro.receiver: Add sourcemap fetching from remote locations (#4614) (b6cb5da) …

⚠ BREAKING CHANGES

• otelcol: Upgrade to OTel Collector v0.147.0 (#5784)
• Renamed undocumented metrics that was previously prefixed with <component_id><metric_name> to loki_source_awsfirehose<metric_name>

Features 🌟

• alloy-mixin: Add filters, groupBy, and multi-select dashboard variables (#5611) (3ef714e)
• beyla.ebpf: Add support for Prometheus native histograms (#5812) (7d806fb)
• beyla.ebpf: Bump Beyla to v3.6 (#5833) (cd878d5)
• converters: Support converting Promtail limits_config (#5777) (9491385)
• database_observability.mysql: Add filtering of query samples and wait events by minimum duration (#5678) (5a4d03b) …

⚠ BREAKING CHANGES

• loki.secretfilter: Some config options are removed entirely:
  • partial_mask (replaced with redact_percent)
  • allowlist (now controlled with custom gitleaks config)
  • enable_entropy
  • include_generic (now controlled with custom gitleaks config)
  • types (now controlled with custom gitleaks config)
• otelcol.receiver.prometheus: otelcol.receiver.prometheus no longer sets start times of OTLP metrics. Grafana Cloud and Mimir do not currently use OTLP metric start times. If you do want your metrics to have them, you can use otelcol.processor.metric_start_time with strategy set to true_reset_point to get the same behaviour.

Features 🌟

• Add automatic reconnection to database_observability components (#5444) (553f967)
• Add limited type checking for validate command (#5076) (045fb76)
• database_observability.mysql: Collect client info for query samples (#5552) (257a699) …

Bug Fixes 🐛

• Expose missing otelcol.processor.tail_sampling options [backport] (#5614) (3225ea3)
• mixin: Add zipped dashboards as a release artifact [backport] (#5625) (37ff20f)
• profiler: Backport Go 1.26 gopclntab textStart fix (#5572) (5ca05c9)
• prometheus.exporter.postgres: Update version of the exporter fork to fix pg_settings (#5574) (62a52f8)
• pyroscope.ebpf: Backport dotnet nibble map fix (#5553) (6c62760)

Bug Fixes 🐛

• database_observability.mysql: Make query sample text nullable in MySQL query details collector [backport] (#5519) (fc49bfe)
• database_observability.mysql: Skip wait events with NULL timer_wait [backport] (#5521) (2f43c91)
• database_observability.postgres: Improvements to SET search_path for postgres explain plans [backport] (#5520) (ecbb577)
• loki.process: Registration of stage.metric when used inside stage.match [backport] (#5495) (2bbc37e)
• loki.source.docker: Parse timestamp correctly when log line only contains newline [backport] (#5496) (55a82f0)
• loki.source.file: Close file if we cannot find encoding [backport] (#5531) (ccda4a5) …

⚠ BREAKING CHANGES

• otelcol: Upgrade to OTel Collector v0.142.0
• otelcol.receiver.kafka: The global topic attribute has been deleted; use the topics attributes inside the logs, metrics, and traces blocks instead.
• otelcol.exporter > sending_queue > batch > min_size changed from 8192 to 2000 and max_size changed from 0 to 3000

Features 🌟

• Add a virtual_node_peer_attributes and virtual_node_extra_label arguments to otelcol.connector.servicegraph (#5058) (20900c6)
• Add an otelcol.processor.metric_start_time component (#5342) (3fb13ac)
• Add job level period, length, and add_cloudwatch_timestamp options and labels_snake_case to CW exporter [backport] (#5355) (60d73b7)
• Add missing configuration parameter deployment_name_from_replicaset to k8sattributes processor (#5183) (b54ca77) …

Bug Fixes 🐛

• Add missing configuration parameter deployment_name_from_replicaset to k8sattributes processor (5b90a9d)
• database_observability: Fix schema_details collector to fetch column definitions with case sensitive table names (#4872) (560dff4)
• deps: Update jose2go to 1.7.0 (#4858) (dfdd341)
• deps: Update npm dependencies [backport] (#5201) (8e06c26)
• Ensure the squid exporter wrapper properly brackets ipv6 addresses [backport] (#5205) (e329cc6)
• Preserve meta labels in loki.source.podlogs (#5097) (ab4b21e)
• Prevent panic in import.git when update fails [backport] (#5204) (c82fbae) …

Breaking changes

• prometheus.exporter.blackbox, prometheus.exporter.snmp and prometheus.exporter.statsd now use the component ID instead of the hostname as their instance label in their exported metrics. This is a consequence of a bug fix that could lead to missing data when using the exporter with clustering. If you would like to retain the previous behaviour, you can use discovery.relabel with action = "replace" rule to set the instance label to sys.env("HOSTNAME"). (@thampiotr)

Features

• Add otelcol.exporter.file component to write metrics, logs, and traces to disk with optional rotation, compression, and grouping by resource attribute. (@madhub)

• (Experimental) Add an otelcol.receiver.cloudflare component to receive logs pushed by Cloudflare's LogPush jobs. (@x1unix)

• (Experimental) Additions to experimental database_observability.mysql component:

  • explain_plans
    • collector now changes schema before returning the connection to the pool (@cristiangreco)
    • collector now passes queries more permissively, expressly to allow queries beginning in with (@rgeyer)
  • enable explain_plans collector by default (@rgeyer)

• (Experimental) Additions to experimental database_observability.postgres component:

  • explain_plans
    • added the explain plan collector (@rgeyer) …

Enhancements

• Schedule new path targets faster in loki.source.file. (@kalleep)

• Add prometheus.static.exporter that exposes metrics specified in a text file in Prometheus exposition format. (@kalleep)

Bugfixes

• local.file_match now publish targets faster whenever targets in arguments changes. (@kalleep)

• Fix otelcol.exporter.splunkhec arguments missing documented otel_attrs_to_hec_metadata block. (@dehaansa)

• Support Scrape Protocol specification in CRDS for prometheus.operator.* components. (@dehaansa)

• Fix panic in otelcol.receiver.syslog when no tcp block was configured. (@kalleep)

• Fix breaking changes in the texfile collector for prometheus.exporter.windows, and prometheus.exporter.unix, when prometheus/common was upgraded. (@kgeckhart)

• Support recovering from corrupted positions file entries in loki.source.file. (@dehaansa)

Other changes

• Augment prometheus.scrape 'scheme' argument strengthening link to protocol. (@lewismc)

• Stop faro.receiver losing trace context when exception has stack trace. (@duartesaraiva98)

Bugfixes

• Fix potential deadlock in loki.source.journal when stopping or reloading the component. (@thampiotr)

• Honor sync timeout when waiting for network availability for prometheus.operator.* components. (@dehaansa)

• Fix prometheus.exporter.cloudwatch to not always emit debug logs but respect debug property. (@kalleep)

• Fix an issue where component shutdown could block indefinitely by adding a warning log message and a deadline of 10 minutes. The deadline can be configured with the --feature.component-shutdown-deadline flag if the default is not suitable. (@thampiotr)

• Fix potential deadlocks in loki.source.file and loki.source.journal when component is shutting down. (@kalleep, @thampiotr)

Breaking changes

• Prometheus dependency had a major version upgrade from v2.55.1 to v3.4.2. (@thampiotr)

  • The . pattern in regular expressions in PromQL matches newline characters now. With this change a regular expressions like .* matches strings that include \n. This applies to matchers in queries and relabel configs in Prometheus and Loki components.

  • The enable_http2 in prometheus.remote_write component's endpoints has been changed to false by default. Previously, in Prometheus v2 the remote write http client would default to use http2. In order to parallelize multiple remote write queues across multiple sockets its preferable to not default to http2. If you prefer to use http2 for remote write you must now set enable_http2 to true in your prometheus.remote_write endpoints configuration section.

  • The experimental CLI flag --feature.prometheus.metric-validation-scheme has been deprecated and has no effect. You can configure the metric validation scheme individually for each prometheus.scrape component.

  • Log message format has changed for some of the prometheus.* components as part of the upgrade to Prometheus v3. …

Bugfixes

• Fix issue with faro.receiver cors not allowing X-Scope-OrgID and traceparent headers. (@mar4uk)

• Fix issues with propagating cluster peers change notifications to components configured with remotecfg. (@dehaansa)

• Fix issues with statistics reporter not including components only configured with remotecfg. (@dehaansa)

• Fix issues with prometheus.exporter.windows not propagating dns collector config. (@dehaansa)

• Fixed a bug in prometheus.write.queue which caused retries even when max_retry_attempts was set to 0. (@ptodev)

• Fixed a bug in prometheus.write.queue which caused labelling issues when providing more than one label in external_labels. (@dehaansa)

• Add application_host and network_inter_zone features to beyla.ebpf component. (@marctc)

• Fix issues in loki.process where stage.multiline did not pass through structured metadata. (@jan-mrm)

• Fix URLs in the Windows installer being wrapped in quotes. (@martincostello)

• Fixed an issue where certain otelcol.* components could prevent Alloy from shutting down when provided invalid configuration. (@thampiotr)

Breaking changes

• Removing the nanoserver-1809 container image for Windows 2019. (@ptodev) This is due to the deprecation of windows-2019 GitHub Actions runners. The windowsservercore-ltsc2022 Alloy image is still being published to DockerHub.

Bugfixes

• Upgrade otelcol components from OpenTelemetry v0.126.0 to v0.128.0 (@korniltsev, @dehaansa)
  • [otelcol.exporter.kafka]: Allow kafka exporter to produce to topics based on metadata key values.
  • [otelcol.receiver.kafka]: Enforce a backoff mechanism on non-permanent errors, such as when the queue is full.
  • [otelcol.receiver.kafka]: Don't restart the Kafka consumer on failed errors when message marking is enabled for them.
  • [otelcol.exporter.datadog]: Fix automatic intial point dropping when converting cumulative monotonic sum metrics.
  • [otelcol.exporter.datadog]: config tls::insecure_skip_verify is now taken into account in metrics path.
  • [otelcol.exporter.datadog]: Correctly treat summary counts as cumulative monotonic sums instead of cumulative non-monotonic sums.
  • [otelcol.connector.spanmetrics]: Fix bug causing span metrics calls count to be always 0 when using delta temporality.
  • [otelcol.exporter.splunkhec]: Treat HTTP 403 Forbidden as a permanent error.

Features

• (Experimental) Add an array.group_by stdlib function to group items in an array by a key. (@wildum) …

Bugfixes

• Send profiles concurrently from pyroscope.ebpf. (@korniltsev)

• Fix the validate command not understanding the livedebugging block. (@dehaansa)

• Fix invalid class names in python profiles obtained with pyroscope.ebpf. (@korniltsev)

• Fixed a bug which prevented non-secret optional secrets to be passed in as number arguments. (@ptodev)

• For CRD-based components (prometheus.operator.*), retry initializing informers if the apiserver request fails. This rectifies issues where the apiserver is not reachable immediately after node restart. (@dehaansa)

Other changes

• Add no-op blocks and attributes to the prometheus.exporter.windows component (@ptodev). Version 1.9.0 of Alloy removed the msmq block, as well as the enable_v2_collector, where_clause, and use_api attributes in the service block. This made it difficult for users to upgrade, so those attributes have now been made a no-op instead of being removed.