Observability: Logs, Metrics and Distributed Tracing

When a request passes through five services and is slow, logs of one service are not enough. Observability combines logs, metrics and traces, joined by a trace id, so you can see exactly where time was spent.

Three pillars

Client ─► Gateway ─► Order Service ─► Payment Service ─► Database
 trace id = 4bf92f3577b34da6a3ce929d0e0e4736 (same in every service)
 │ span: gateway            [=====================================] 480 ms
 │  span: order-service       [================================]      450 ms
 │   span: payment-service       [====================]               300 ms   ◄── the slow part
 │    span: SELECT payments        [=================]                 280 ms   ◄── missing index?
The trace id travels in HTTP headers (W3C traceparent), Kafka headers and appears in every log line.

Setup (Boot 3.x with OpenTelemetry)

<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-actuator</artifactId></dependency>
<dependency><groupId>io.micrometer</groupId><artifactId>micrometer-tracing-bridge-otel</artifactId></dependency>
<dependency><groupId>io.opentelemetry</groupId><artifactId>opentelemetry-exporter-otlp</artifactId></dependency>
<!-- Spring Boot 4 provides spring-boot-starter-opentelemetry which brings the same pieces -->

spring:
  application:
    name: order-service
management:
  tracing:
    sampling:
      probability: 1.0            # 100% in development. Use 0.05 to 0.1 in production
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces
logging:
  pattern:
    correlation: "[${spring.application.name:},%X{traceId:-},%X{spanId:-}] "
  include-application-name: false

# Jaeger UI at http://localhost:16686, accepts OTLP on 4318
docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest

With this, every incoming HTTP request creates a span. Calls made through the Spring-provided RestClient.Builder, RestTemplateBuilder, WebClient.Builder, and Kafka, JDBC (with datasource-micrometer) can be instrumented, and the trace context is passed on automatically. Log lines now look like:

2026-09-29T10:15:30.123Z  INFO 1234 --- [order-service,4bf92f3577b34da6a3ce929d0e0e4736,00f067aa0ba902b7] c.e.OrderService : Placing order 42

Observation API: one instrumentation, many outputs

@Service
public class CheckoutService {

    private final ObservationRegistry registry;
    public CheckoutService(ObservationRegistry registry) { this.registry = registry; }

    public void checkout(String type) {
        Observation.createNotStarted("checkout.process", registry)
                .lowCardinalityKeyValue("type", type)               // becomes a metric tag and a span attribute
                .observe(() -> {
                    // business work: timing metric + trace span are created together
                });
    }
}

// Or with an annotation
@Configuration
class ObservedConfig {
    @Bean ObservedAspect observedAspect(ObservationRegistry registry) { return new ObservedAspect(registry); }
}
@Service
class ReportService {
    @Observed(name = "report.generate", contextualName = "generate-report")
    public void generate() { }
}

Alerts and dashboards that matter

• RED for services: Rate (requests per second), Errors (5xx percent), Duration (p95 and p99 latency).
• USE for resources: Utilization, Saturation, Errors of CPU, memory, connection pool, threads.
• Kafka consumer lag, queue depth, and circuit breaker state.
• Alert on symptoms users feel (error rate, latency), not on every cause.
• Use the trace id in error responses (ProblemDetail property) so support can find the trace from a customer report.

Common mistake: Adding user ids or order ids as metric tags (high cardinality). The metrics system creates one time series per value and runs out of memory. Put such ids in traces and logs only.

Interview questions

What is the difference between a trace and a span?

A trace is the full journey of one request. A span is one timed step inside it (an HTTP call, a database query). A trace is a tree of spans.

How does the trace id pass between services?

In a header (traceparent for W3C). The instrumented HTTP client adds it and the server side reads it, and messaging clients put it in message headers.

How much sampling should I use?

100% costs storage and overhead. In production use 1% to 10%, plus tail-based sampling in the collector to keep slow and failed traces.