Observability¶

Guide: how the built-in distributed tracing works, and how to export it to OpenTelemetry.

At a glance

What — every traced request produces a causal span tree across the three layers; the system stamps W3C-compatible trace/span IDs automatically and can ship the telemetry to any OpenTelemetry backend.

Built-in — distributed tracing is part of the platform; you turn it on per endpoint/flow and the spans propagate without code.

For developers and operators wiring Mercury to Dynatrace, Splunk, Jaeger, Tempo, or an OpenTelemetry Collector.

Logs too — a default-on application log context stamps the same correlation/trace ids (plus your own key-values) into structured log lines, so logs and spans join up in your backend.

In an event-driven, composable system a single request fans out across decoupled functions, flows, and graph nodes that never call each other directly. Observability is therefore not optional — it is the only way to see the causal path a request actually took. Mercury answers this with a built-in distributed-tracing engine whose output is OpenTelemetry-compliant, so the spans drop straight into the dashboard you already run.

The built-in tracing design¶

Turning tracing on¶

Tracing is opt-in per entry point:

HTTP endpoints — add tracing: true to the rest.yaml entry (see REST Automation).
Event Script flows — a flow started through FlowExecutor carries a trace; the engine traces every task.
Programmatically — construct a trace-aware PostOffice(fromRoute, traceId, tracePath) and the events it sends are traced end-to-end.

Two controls tune what is recorded:

@ZeroTracing on a function suppresses tracing for that function (used by system services so they never trace themselves).
skip.rpc.tracing (in application.properties) lists route names whose RPC calls produce no caller-side round_trip record; the default is async.http.request, so an HTTP call made from a function folds into that function's span. A callback-mode execution of a listed route - the Event-over-HTTP stream relay's client leg - still records its own span, parented onto the sender.

What a trace records¶

When a traced function finishes, the system sends a performance-metrics dataset to the built-in distributed.tracing service. The dataset is a plain map:

trace={ id=<32-hex trace id>, span_id=<16-hex>, parent_span_id=<16-hex>,
        service=<route name>, path=<request path>, from=<caller route>,
        origin=<instance id>, start=<ISO-8601>, exec_time=<ms>, round_trip=<ms>,
        success=<bool>, status=<int>, exception=<text> }
annotations={ <your key>=<value>, ... }

span_id and parent_span_id are the OpenTelemetry-compatible IDs (see W3C trace context); you can attach business context to the span with PostOffice.annotateTrace(key, value). To attach context to application logs instead, see Application log context.

The edge's round-trip span¶

A traced endpoint records one more span than its functions: the round trip itself. REST automation mints a span id when the request arrives, makes it the first function's parent, and emits its record when the response completes - the buffered response, the end of a streamed response, an edge error or the edge timeout. The record's service is http.request, the same marker the first function carries as from, so the vocabulary stays one word: the request came from the edge; the edge's own span is the root.

path is METHOD /path, start is the receipt time and exec_time is the whole round trip - for a streamed response, until the terminal is rendered.
parent_span_id is the inbound traceparent span when the caller sent one; otherwise the record is the trace's root.
status is the HTTP status sent, except that a stream failing in-band after its head was committed reports the failure's own status and message.

The OpenTelemetry forwarder maps this record, and only this record, to a SERVER span - which is what makes a service's response time in a tracing backend the real one, not the first function's own execution time. Every function execution, the first one included, is an INTERNAL span under it.

Streamed responses are traced at their head and their tail, never per token. EventStreamWriter stamps the producer's trace and span on the first segment (it carries the head control) and on the terminal (eof or exception); the data segments in between carry no trace, because one span per token would flood a tracing backend. The reply lane that renders the stream therefore records two spans, both parented onto the producer, and annotates the terminal's record with frames - the number of data segments it rendered. The Event-over-HTTP stream relay follows the same rule on the consuming side: decoded envelope frames keep the remote producer's span, synthesized control frames parent onto the relay's client leg (async.http.request, itself parented onto the sender), and raw token frames are forwarded untraced. See HTTP streaming.

Spans across the three layers¶

The span tree mirrors the three paradigm layers, and tracing is virtual-thread-safe — the parent span is threaded through a per-task anchor, not a ThreadLocal, so concurrently-dispatched siblings still share the correct parent.

Mono/Flux boundary: a function's span closes when its worker returns — not when a returned Mono completes — so annotate (and capture po.getTrace() if needed) on the worker thread before returning. See the trace annotation notes for the exact rules.

Layer	What becomes a span	Lineage
HTTP edge — REST automation	the request's round trip (`service: http.request`)	the inbound `traceparent` span is its parent; the first function parents onto it
Layer 1 — Platform Core	each function execution	the caller's span becomes the child's `parent_span_id`
Layer 2 — Event Script	each task, plus one synthetic `task.executor` flow-summary span (annotated with the flow id)	tasks chain exactly like Layer 1; a sub-flow chains to the parent task that dispatched it
Layer 3 — Knowledge Graph	each node dispatch through `graph.executor`	the graph traversal threads the parent span through node execution

Because Layer 2 and Layer 3 ride on Layer 1's engine, the task/node spans look identical to Layer-1 spans — the only addition at Layer 2 is the synthetic flow summary that brackets the whole flow's timing.

W3C Trace Context (OpenTelemetry compliance)¶

Mercury's trace and span IDs follow the W3C Trace Context format that OpenTelemetry uses: a 32-hex trace ID and 16-hex span ID. Across an HTTP boundary the system propagates the standard traceparent header:

outbound — the HTTP client injects traceparent carrying the current trace + span, alongside X-Trace-Id;
inbound — the HTTP layer extracts traceparent, continues the upstream trace, and adopts the caller's span as the parent of the edge's round-trip span. traceparent takes precedence over X-Trace-Id when both are present.

The X-Trace-Id header carries the trace ID for callers not yet on W3C Trace Context. The framework does not echo the trace ID back to the HTTP client. (The correlation-id — a separate concern — is documented in Reserved Names & Headers.)

Let the framework manage trace headers — don't set them yourself. Inside a traced flow or function the platform injects X-Trace-Id and traceparent on every outbound HTTP call from the current trace context, overwriting any value you set on the request — so the trace stays correct and connected end-to-end, and the upstream trace is propagated automatically. The one exception is a call that is not being traced (an endpoint with tracing: false, or a call made outside a trace): there a trace header you set passes through untouched — the intended escape hatch for handing a trace context to a third-party system, or for unit-testing an external endpoint with full control over its request headers.

Header impedance matching (trace-id and correlation-id)¶

Not every caller names its headers the way this framework does. An enterprise gateway may have standardized on its own trace header long before W3C Trace Context; a legacy Kafka producer may stamp X-Correlation-ID; and two systems bridged by one application rarely agree with each other. Rather than forcing every party to rename, the platform matches the impedance at the edge: the header names are configuration, while everything downstream keeps working with the same two ids —

the trace id (with its W3C traceparent context) drives the distributed tracing on this page;
the business correlation-id is a separate concern — captured at the edge, preserved as the flow's model.cid, and exposed to every function via PostOffice.getMyCorrelationId() (see Reserved Names & Headers).

The configurable names¶

Global defaults in application.properties:

Key	Default	Where it applies
`http.trace.id.header`	`X-Trace-Id`	REST automation inbound (when no `traceparent` is present) and the async HTTP client outbound
`http.correlation.id.header`	`X-Correlation-Id`	HTTP edge capture inbound; async HTTP client outbound
`http.traceparent.header`	`traceparent`	The header carrying the full W3C trace context; REST automation inbound (standard `traceparent` first, custom name only when the standard is absent) and HTTP client / Event-over-HTTP outbound (stamped under both names)
`kafka.trace.id.header`	(unset)	Kafka Flow Adapter inbound fallback; `simple.kafka.notification` outbound, stamped alongside `traceparent`
`kafka.correlation.id.header`	`cid`	Kafka Flow Adapter inbound; `simple.kafka.notification` outbound
`kafka.traceparent.header`	`traceparent`	Kafka twin of `http.traceparent.header`: adapter inbound (standard first, custom name only when the standard is absent); notification outbound (stamped under both names)

Per-entry overrides, for a single application that faces callers with different conventions:

a rest.yaml endpoint entry accepts trace.id.header / correlation.id.header / traceparent.header;
a kafka-flow-adapter.yaml consumer binding accepts the same three keys (Minimalist Kafka);
twin-kafka adds secondary.kafka.trace.id.header / secondary.kafka.correlation.id.header / secondary.kafka.traceparent.header globals when the second Kafka cluster follows its own convention (each falls back to its primary kafka.* setting when unset).

Precedence: per-entry override > application.properties global > built-in default. A well-formed W3C traceparent always takes precedence for the trace id, whatever the header naming — so adopting these overrides never breaks OpenTelemetry-compliant callers. The full key reference lives in the Configuration Reference.

Legacy conflation is supported. Pointing the trace-id name at the correlation-id header (http.trace.id.header=X-Correlation-Id) is a valid backward-compatibility setup for an estate whose gateway only passes that one header. The edge keeps the two ids consistent: a supplied shared header feeds both ids, and when the shared header is absent the edge resolves one id for both — from the inbound traceparent when present, otherwise a single generated id — so the outgoing traceparent and the shared header always carry the same trace id. Prefer migrating the gateway to pass traceparent (and X-Trace-Id), then retiring the conflation.

The standard W3C traceparent is our position — use it. It is the header OpenTelemetry and the wider observability ecosystem interoperate on, and the framework implements it as the default with zero configuration. The optional traceparent.header family below exists for backward compatibility with legacy systems only; departure from the standard is discouraged, because a renamed carrier is invisible to OpenTelemetry SDKs, service meshes and APM agents, and every participant must be configured alike. Treat a custom name as a temporary bridge and plan the migration back to the standard header.

Within that constraint, a renamed traceparent beats conflation for the gateway case. The conflation above carries only the trace id, so spans in different applications can be stitched by id but not parented across the hop. Renaming the traceparent carrier (http.traceparent.header=X-Trace-Context) moves the full W3C context — trace-id, parent span-id and flags — through the gateway under an allow-listed name, so cross-application span parenting survives. Outbound calls stamp the same value under both the custom and the standard name; inbound, the standard traceparent always wins and the custom name is read only when the standard is absent — a well-formed standard traceparent means the caller already speaks W3C/OTel, so a residual proprietary header alongside it is safely ignored. The durable fix is always the gateway allow-list — retire the custom name once it lands.

# rest.yaml - one endpoint serves a legacy caller that sends its own header names
  - service: "legacy.orders"
    methods: ['POST']
    url: "/api/legacy/orders"
    timeout: 15s
    tracing: true
    trace.id.header: "X-Legacy-Trace"
    correlation.id.header: "X-Legacy-Cid"

Bridging two conventions¶

When one application connects two systems that each own a correlation-id convention (for example a dual-cluster Kafka bridge), keep each system's header name strictly on its own side:

the inbound adapter binding declares that system's correlation.id.header, so the id lands in model.cid;
the flow maps it back out under the next system's name — 'model.cid -> header.X-Their-Header' — on the outbound data mapping;
neither system ever sees the other's header name: the id value is what crosses the bridge, and the trace context (traceparent) rides alongside automatically, keeping one continuous distributed trace end to end.

The twin-kafka guide covers this pattern across two Kafka clusters, and the twin-kafka-demo worked example runs it end to end: an on-prem system's X-Correlation-Id and a cloud system's X-Cloud-Correlation-Id carry the same id through one bridged transaction, with each header name confined to its own cluster.

Exporting telemetry¶

The forwarder extension point¶

By default the trace dataset is logged. To ship it elsewhere, register a function at the reserved route distributed.trace.forwarder; the system detects it and forwards every dataset to it. A companion hook, transaction.journal.recorder, receives request/response payloads when journaling is enabled (see Reserved Names & Headers and Build, Test & Deploy).

The OpenTelemetry forwarder (ready-made)¶

You do not have to write the forwarder for OpenTelemetry. The opentelemetry-forwarder extension ships a distributed.trace.forwarder that maps each dataset to an OpenTelemetry span — preserving the exact W3C trace/span/parent-span IDs — and exports it over OTLP/HTTP to a collector (and on to Dynatrace, Splunk, Jaeger, Tempo, …). Add the dependency and it auto-registers; no code required:

Note: x.y.z denotes the current Mercury version shown in the root pom.xml.

<dependency>
    <groupId>org.platformlambda</groupId>
    <artifactId>opentelemetry-forwarder</artifactId>
    <version>x.y.z</version>
</dependency>

Configure it in application.properties (values support ${ENV_VAR:default} substitution):

# Master switch - DEFAULT OFF. The dependency alone registers nothing.
otel.forwarding=true
otel.exporter.otlp.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4318/v1/traces}
otel.service.name=${OTEL_SERVICE_NAME:my-app}
# Backend credentials come from the environment — no secret hard-coded:
otel.exporter.otlp.headers=${OTEL_EXPORTER_OTLP_HEADERS}

A caution on OTEL_* variable names. Referencing the OpenTelemetry standard variables (OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, …) is convenient and works, but those names are often exported machine-wide on instrumented hosts and CI agents — so an application can silently inherit a service name or, worse, an endpoint that redirects its telemetry somewhere unintended. When that matters, reference your own prefixed variables instead; examples/composable-example uses OTLP_SERVICE_NAME / OTLP_API_ENDPOINT / OTLP_AUTH_HEADER / OTLP_TOKEN for exactly this reason, and the forwarder's own tests avoid ${OTEL_*} references so a leaked endpoint cannot redirect a hermetic test.

Adding the jar does not turn forwarding on. The forwarder lives under org.platformlambda, a base scan package, so the dependency alone would auto-register it. It is gated by @OptionalService("otel.forwarding") with a default of false, which separates the two decisions that belong to different people: a developer adds the dependency, and DevOps decides per environment whether traces leave the process — otel.forwarding=true in the environment's properties, or -Dotel.forwarding=true at launch with no rebuild. With it off, the route does not exist at all.

Point the endpoint at an OpenTelemetry Collector, or directly at a SaaS backend with its API token in the headers:

# Dynatrace
export OTEL_EXPORTER_OTLP_ENDPOINT="https://{env-id}.live.dynatrace.com/api/v2/otlp/v1/traces"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Token dt0c01.XXXX"

# Splunk Observability Cloud
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.{realm}.signalfx.com/v2/trace/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="X-SF-Token=YOUR_ACCESS_TOKEN"

The credential is re-read on every export, so a late-published token still works. The forwarder is a @PreLoad function, and those are constructed before any @MainApplication runs — so if your token comes from a credential bootstrap rather than the process environment, resolving it once at construction would freeze it as absent and every export would fail authentication for the life of the process. otel.exporter.otlp.headers is therefore resolved per export: a token published after start-up takes effect with no restart, and the log records a single OTLP credential header resolved line when it first appears. A credential exported the ordinary way — an environment variable set before the JVM starts — behaves exactly as it always did. (Same lazy-resolution shape as the Kafka and Redis health checks.)

Each span carries the route name (route), path, from, origin, status, timing, and your annotation.* values as span attributes, with service.name on the resource. See the full key reference in the Configuration Reference.

The forwarder exports traces, not logs — deliberately. Application logs reach your backend through your platform's log forwarder, not through the engine; see Getting these logs to your backend.

A custom forwarder¶

To target a system without an OTLP path, implement your own function at distributed.trace.forwarder and consume the dataset described in What a trace records — for example, write it to a metrics database or a proprietary APM API.

Application log context¶

Spans tell you the causal path; application logs tell you what happened inside each step. The log-context feature closes the gap between them: it injects a context block — correlation id, trace/span ids, service name, and any business key-values you add — into every structured log line a traced function emits. With the same trace_id/span_id on both the span and the log line, you can pivot from a Dynatrace/Splunk trace straight to the exact log entries that belong to it.

It deliberately avoids the ThreadLocal / Log4j MDC pattern (heavy for a virtual-thread runtime). The context rides the same per-request mechanism as the trace itself, keyed to the worker thread and torn down when the function returns.

On by default¶

The feature is on by default: platform-core ships a built-in default-log-context.yaml that emits the standard trace context (cid, trace_id, trace_path, span_id, parent_span_id, service, timestamp) on every structured log line. You can adjust it in two ways:

Customize — provide your own app-log-context.yaml on the classpath (src/main/resources/); it replaces the built-in template entirely.
Opt out — set app.log.context=false in application.properties.

It applies to the two structured JSON appenders — select one via the log4j2 configuration (log4j2-json.xml for pretty output, log4j2-compact.xml for single-line); the plain Console appender is unaffected.

A custom template looks like this:

# src/main/resources/app-log-context.yaml
context:
  cid: $cid
  trace_id: $traceId
  trace_path: $tracePath
  span_id: $spanId
  parent_span_id: $parentSpanId
  service: $service
  environment: '${ENV_NAME:dev}'
  hello: world

The output keys are snake_case, matching the distributed-trace block (span_id, parent_span_id, exec_time) so both halves of a log record read the same way. The $token names on the right stay camelCase — they are the engine's identifiers, not output — and the two sides are free to differ because the left is entirely your choice.

Note what is not in that template: a timestamp. You do not configure one — see below.

The left side is the output key (your choice). The right side is one of three forms:

Form	Example	Resolved
Reserved `$token`	`service: $service`	live, per log line, from the request's trace context
`${ENV:default}` substitution	`environment: '${ENV_NAME:dev}'`	once at startup, from the environment
Hardcoded literal	`hello: world`	emitted verbatim on every line

The reserved tokens are $cid, $traceId, $tracePath, $spanId, $parentSpanId, $service (the current function's route), and $utc (the log line's UTC timestamp). A token (or env value) that resolves to nothing is omitted from the block rather than printed as null — so a root span simply has no parentSpanId key.

You do not configure the timestamp¶

$utc is added for you. If your template does not resolve it under any key, the framework inserts it as timestamp, so every context block carries an unambiguous UTC time whether or not you asked for one.

That guarantee exists because the record's top-level time field is a local timestamp with no zone or offset. Anything that parses it downstream — a log collector, a forwarder — has to be told the timezone, and shifts every line silently when told wrong. Log-to-trace correlation is resolved on trace id and a time window, so a shifted line can be correctly correlated and still invisible on its trace. $utc removes the guesswork.

You keep control of the name. Map $utc to any key you like and that name is used as-is:

context:
  traceId: $traceId
  loggedAt: $utc     # your name, kept - nothing extra is added

Only absence is corrected, never your choice. If you have already spent timestamp on something of your own, the UTC time is placed under utc instead so your value is never overwritten; if both names are taken, the framework leaves your template alone and logs a warning telling you to add $utc under a key of your choosing.

$cid is the business correlation ID — the value received from the external source or created at the edge, the same one PostOffice.getMyCorrelationId() returns. When the delivered event carries no business context, the key is simply omitted: internal correlation IDs (routing metadata such as RPC inbox references or the graph engine's skill-callback IDs) never appear under the cid label, so log aggregation always correlates on the ID your callers know — or on nothing, never on something misleading. Because every edge guarantees a business correlation ID (a fresh one is generated when the caller supplies none), a missing cid on a traced log line indicates a propagation defect worth fixing, not a normal condition.

Adding your own key-values¶

Inside a function, add business context with PostOffice.updateContext(key, value):

var po = new PostOffice(headers, instance);
po.updateContext("user", "demo");   // appears in the context block of every subsequent log line
log.info("processing request");

Reserved keys are refused in both spellings — the $token name and its snake_case form, so trace_id and parent_span_id throw just as traceId and parentSpanId do. That matters because snake_case is what the shipped templates publish, and therefore what you would most likely reach for.

Beyond the refused names, a developer key never shadows a template key: if your function sets a key the template also emits, the template's value wins. An output key is your free choice, so no list of names could cover every case — the precedence is what makes shadowing impossible, and the rejected names are the fast, legible error for the two spellings that actually occur.

The reserved keys (cid, traceId, tracePath, spanId, parentSpanId, service, utc) are protected — passing one to updateContext throws IllegalArgumentException. On a non-traced request, or when the feature is off, the call is a silent no-op.

updateContext vs annotateTrace — two distinct sinks. annotateTrace(...) attaches business data to the distributed-trace dataset that flows to your APM backend (What a trace records); updateContext(...) attaches it to the application log stream only. Neither leaks into the other.

What it looks like¶

A log line from a traced function then carries the resolved context (from the worked example above, with po.updateContext("user", "demo") added in the function):

{
  "level": "INFO",
  "context": {
    "cid": "20260630c6ee70d866cb4fae9ab3c44d926ce21a",
    "trace_id": "fbb60df209084531b2b00f6b36a3e651",
    "trace_path": "GET /api/profile/100",
    "span_id": "bf8d4b2b6a923d67",
    "parent_span_id": "98f8e26ae7d9a422",
    "service": "v1.hello.exception",
    "environment": "dev",
    "hello": "world",
    "user": "demo",
    "timestamp": "2026-06-30T21:17:03Z"
  },
  "time": "2026-06-30 14:17:03.575",
  "source": "com.accenture.demo.tasks.HelloExceptionHandler.handleEvent(HelloExceptionHandler.java:51)",
  "thread": 297,
  "message": "User defined exception handler - status=404 error=Profile 100 not found"
}

The trace_id and span_id here match the v1.hello.exception span the tracer emitted for the same request, so the log line and the span join up in your backend. (Key order within context is not significant — log viewers reorder keys on display.)

timestamp appears even though the template above never asked for it — that is the automatic UTC timestamp. Note it is the context's UTC time, distinct from the record's top-level time, which is local.

Scope and boundaries¶

The context block appears only when a request is traced and a traceId is present. Framework boot logs and logs emitted from a Mono/Flux completion that runs after the worker returns (on a different thread) carry no context — the same boundary distributed tracing has.
Feature off (app.log.context=false) costs one boolean check per log line and nothing else.

Getting these logs to your backend¶

Mercury does not ship logs. The OpenTelemetry forwarder exports traces; it has no log counterpart, and that is a deliberate boundary rather than a missing feature.

Application logs reach your backend the way every other container's logs do — your platform's forwarder or collector reads stdout and ships it. That is infrastructure configuration, not application responsibility, and it is the same pattern whichever backend you use: a Splunk forwarder to Splunk, an OpenTelemetry Collector to an OTLP /v1/logs endpoint. Putting an HTTP exporter inside the application would instead place a network call in the path of every log.info, with its own credentials, retries and back-pressure, for no benefit the platform does not already provide.

What the engine does is the part that makes correlation work, and it is already done:


Structured output	`log.format=json` or `compact`
Trace correlation	`context.trace_id` / `context.span_id`, identical to the values on the exported span
Shared vocabulary	snake_case, matching the distributed-trace block, so one mapping covers both
Unambiguous time	`context.timestamp`, always present and always UTC

Two details worth knowing when you configure the collector:

Promote the ids. For "view the logs for this trace" to work in an OTLP backend, the collector must map context.trace_id and context.span_id onto the log record's own trace and span fields. Left as ordinary attributes they are searchable but not linked.
Parse context.timestamp, not time. The record's top-level time is a local timestamp with no zone or offset; anything parsing it must be told the timezone and shifts silently when told wrong. Backends correlate on trace id and a time window, so a shifted line can be correctly correlated and still invisible on its trace. context.timestamp is UTC and needs no assumption.

Lines emitted outside a traced worker — framework start-up, a Mono/Flux completion after the worker returns — carry no context block and therefore no trace id. They still ship; they simply are not linked to a trace. A collector configuration must not drop a record that has no context.