Spring Boot OpenTelemetry

Metrics

Spring Boot Actuator provides dependency management and autoconfiguration for Micrometer, which offers a convenient facade over many different monitoring systems. One of them is OpenTelemetry.

Spring Boot developers can enable their applications to export metrics via the OTLP protocol to an OpenTelemetry backend by adding Spring Boot Actuator and the dedicated OpenTelemetry dependency from Micrometer to their projects.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    runtimeOnly 'io.micrometer:micrometer-registry-otlp'
    ...
}

The exporter provided by the Micrometer Registry OTLP is an HTTP exporter and can be configured via properties thanks to the autoconfiguration for OpenTelemetry.

management:
  otlp:
    metrics:
      export:
        url: http://localhost:4318/v1/metrics
        step: 5s

OpenTelemetry supports additional key/value pairs (called resource attributes) to be included in the telemetry data. Spring Boot Actuator provides autoconfiguration for those and makes it possible to add new resource attributes via properties.

management:
  opentelemetry:
    resource-attributes:
      cluster: local
      "service.name": ${spring.application.name}

I made a demo application to showcase this setup.

Issue: OTLP Metrics Exporter is not configurable via OpenTelemetry

The Micrometer Registry OTLP module uses an OTLP-compatible HTTP client to export metrics to an OpenTelemetry backend. Internally, the OtlpMeterRegistry uses a private HttpSender object to configure the HTTP client. The benefit of this approach is that the module is lightweight and doesn't need any dependency on the OpenTelemetry SDK. It only uses the io.opentelemetry.proto library which provides the protobuf configuration for OpenTelemetry.

On the other hand, such an approach means that:

• it's not possible to configure the metrics exporter via standard OpenTelemetry configuration;
• it's not possible to configure compression;
• it's not possible to share OpenTelemetry configuration in Spring Boot between metrics and traces.

The new generic OpenTelemetryAutoConfiguration in Spring Boot 3.2 autoconfigures an OpenTelemetry bean and makes it possible to configure an SdkMeterProvider bean. What if Micrometer could use that for setting up the OpenTelemetry metrics exporter?

Possible solutions:

• update the existing Micrometer Registry OTLP module to implement a MeterRegistry that accepts an OpenTelemetry object for configuration (similar to the OpenTelemetryMeterRegistry used by the OpenTelemetry Java Instrumentation library). If backward compatibility is necessary, the new implementation would need to co-exist with the existing one (maybe a feature flag could switch between the two?);
• create a new Micrometer Registry OpenTelemetry module to implement what described in the previous point, but without having issues with backward compatibility.

I have submitted an issue to the Micrometer project and shared these suggestions. Building (or updating) such a module would make it possible to re-use the same OpenTelemetryAutoConfiguration introduced in Spring Boot 3.2 for both metrics and traces (and, in the future, for logs as well).

• An issue already exists on the Spring Boot project to autoconfigure an SdkMeterProvider bean and, in general, for using metrics with OpenTelemetry. But we are missing Micrometer support before that is doable.
• It would also make it possible to switch the client implementation between HTTP and gRPC using the standard OpenTelemetry approach (i.e. configuring either an OtlpHttpMetricExporter or OtlpGrpcMetricExporter).
• Finally, it would make it easier for developers using Micrometer in applications together with the OpenTelemetry Java Agent or, in general, the OpenTelemetry Java Instrumentation. Those projects currently maintain a shim between Micrometer and OpenTelemetry. That shim would probably not be needed anymore. So, such a change would also reduce the workload on maintaining the OpenTelemetry Java Instrumentation compatible with Micrometer.

For more context about the current challenges, refer to this issue on the Spring Boot project.

Traces

Spring Boot Actuator provides dependency management and autoconfiguration for Micrometer Tracing, which offers a convenient facade over a few different distributed tracing backends. One of them is OpenTelemetry.

Spring Boot developers can enable their applications to export traces via the OTLP protocol to an OpenTelemetry backend by adding Spring Boot Actuator, Micrometer Tracing and the dedicated OpenTelemetry dependency from Micrometer to their projects.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'io.micrometer:micrometer-tracing-bridge-otel'
    implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
    ...
}

The exporter provided by the Spring Boot Actuator autoconfiguration is an HTTP exporter (an OtlpHttpSpanExporter bean) and can be configured via properties thanks to the tracing configuration in OtlpAutoConfiguration.

management:
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

OpenTelemetry supports additional key/value pairs (called resource attributes) to be included in the telemetry data. Spring Boot Actuator provides autoconfiguration for those and makes it possible to add new resource attributes via properties. The standard OpenTelemetry service.name resource attribute is configured automatically to the value of spring.application.name (if defined) or else to a default application value.

management:
  opentelemetry:
    resource-attributes:
      cluster: local

I made a demo application to showcase this setup.

Issue Exporting traces via gRPC

In this issue on the Spring Boot project, autoconfiguration for an OtlpHttpSpanExporter bean has been added to export traces via HTTP.

A common requirement is to export traces via gRPC (the most used approach in OpenTelemetry). Currently, developers can configure an OtlpGrpcSpanExporter bean by themselves. It would be nice if Spring Boot Actuator would provide autoconfiguration for that, enhancing the existing OtlpAutoConfiguration.

There is already an issue on the Spring Boot project to add such autoconfiguration.

Summary

Overall, these are the issues and suggestions I've been describing so far.

Minimal proposed changes to consolidate OpenTelemetry support in Spring for metrics and traces

Micrometer

• Introduce a MeterRegistry implementation built on top of OpenTelemetry and configurable via its standard approaches, so that it's possible to share configuration between metrics and traces.

Spring Boot Actuator

• Introduce autoconfiguration for OpenTelemetry metrics. The OpenTelemetry and Resource beans can be reused from the existing OpenTelemetry configuration. Besides that, the autoconfiguration should define defaults for SdkMeterProvider, OtlpHttpMetricExporter and OtlpGrpcMetricExporter.
• Introduce autoconfiguration for exporting OpenTelemetry traces via gRPC by defining a OtlpGrpcSpanExporter bean.
• Consolidate OpenTelemetry autoconfiguration classes across metrics and traces. The existing OpenTelemetryAutoConfiguration could be used for shared configuration. Besides that, dedicated autoconfiguration classes are needed to configure the specific exporters for each type of telemetry. Traces have that already in the OtlpAutoConfiguration.

Spring Initializr

• After going through all the previous changes, it should become straightforward adding an OpenTelemetry option to the Spring Initializr. Adding it now with the current state of things, it might make things confusing considering the different ways of configuring OpenTelemtry between metrics and traces.

Additional changes to extend OpenTelemetry support in Spring to logs

Spring Boot Actuator

• Introduce autoconfiguration for OpenTelemetry logs. The OpenTelemetry and Resource beans can be reused from the existing OpenTelemetry configuration. Besides that, the autoconfiguration should define defaults for SdkLoggerProvider, OtlpHttpLogRecordExporter and OtlpGrpcLogRecordExporter.
• Add support for Logback (default) and Log4J2. For inspiration, OpenTelemetry Java Instrumentation provides appenders for Logback and Log4J2.
• Possibly related issue on the Spring Boot project.

Further improvements

• An issue has been opened to investigate how to limit the number of configurable beans currently available in the tracing OpenTelemetryAutoConfiguration (this issue might be considered for the global OpenTelemetryAutoConfiguration once the tracing specific one gets deleted).