Test Report — The agent loop's trace lineage through both language packs¶
The first drive whose question was the trace itself: does a span lineage stay continuous from the Java engine's REST edge
through the agent loop into the language packs - the LLM helper's llm.chat and the MCP binding's route - and back? Conducted
2026-11-14 (UTC+8) with the Playground example's diagram-assistant graph on Claude Haiku 4.5, the Java Playground
4.12.30, the Python pack (mercury-python) and the Node.js pack
(mercury-nodejs) from main of the day, and the public Mermaid Chart MCP
server behind the binding. Eric set the acceptance criterion before the drive: for OpenTelemetry to be certified, the span
lineage ought to be continuous end to end. The drive ran negative control first - the packs as they were - then again after
the change this report records.
Result in brief¶
- Baseline: every record chained, and the pack legs were missing. Each run logged 11 engine records, each with a parent
some record owned, and no record from either pack.
llm.chat(two turns) anddiagram.render.mcp(the MCP tool) appeared only as time inside the engine'sllm.runanddiagram.render. A tree with no orphans is not the same as a tree with every hop. - Cause: two rules met, with nothing between them. The calling engine records nothing for an RPC over Event-over-HTTP,
because its HTTP client leg
async.http.requestis inskip.rpc.tracingby default; a Java or Rust callee fills that gap with its own three records - thehttp.requestedge forPOST /api/event,event.api.serviceand the function's round-trip record from its local inbox. The packs suppressed the worker-side record of every RPC-served execution, as the engines do, but left none of the callee's records in its place, and their replies carried no span for an engine inbox to chain onto. The forwarder drops a record without a valid span id, so no backend could have shown the hop either. - Fix: the packs now leave what a Java callee leaves (mercury-python #53,
mercury-nodejs #120, Eric's choice among three shapes): a traced
call at
/api/eventleaveshttp.request(the edge's round trip, parented onto the caller's span - the envelope's span, else thetraceparentheader's; the backend's SERVER span),event.api.service(the ingress, parented onto the edge) and, for an RPC, the function's round-trip record with the responder's span, parented onto the service; a reply carries the responder's span (WorkerHandlerparity) and the packs'PostOfficerecords a local RPC round trip asInboxBasedoes. - After: 20 records per run, every parent owned by a record, in both packs. The helper's edge sits under the engine's
llm.runand the binding's underdiagram.render; each function sits under itsevent.api.service; the timings nest. The tree is now the same shape whether the callee is a Java engine or a pack. - A declared function's name never enters a trace. The MCP binding's
mcp.tool, the one function behind every bound route since mercury-python #52 and mercury-nodejs #119, appears nowhere: every trace field comes from the service entry the event was addressed to, and the route namediagram.render.mcpis the hop's identity.
Setup¶
| Graph | diagram-assistant (examples/minigraph-playground), deployed at startup; one request POST /api/graph/diagram-assistant with {"question": "Draw a flowchart of a coffee order: order, pay, brew, serve"}, the REST entry traced (tracing: true) |
| Engine | Java Playground 4.12.30 (minigraph-playground), -Dllm.peer.port and -Dmcp.peer.port selecting the pack |
| Python pack | LLM helper on 8086 (-Dllm.model=claude-haiku-4-5), MCP binding on 8088 with the example's mcp/mermaid-servers.yml |
| Node.js pack | LLM helper on 8087, MCP binding on 8089, the same servers file |
| Model | Claude Haiku 4.5 (claude-haiku-4-5-20251001), the Claude API backend; two turns per run, one tool call |
| MCP server | the public Mermaid Chart endpoint, streamable HTTP, no credential; diagram.render stores the SVG out of band |
| Records | the engine's Telemetry INFO lines; the packs' distributed.tracing log lines ({"trace": {...}}, the same record) |
| Lineage check | every record of the trace from every process, sorted by start; a record's parent_span_id must be the span_id of a record in the set, or absent (the root); a record without a span_id counts as broken, because the forwarder would drop it |
The baseline ran the packs at main after the preload shape refactors (mercury-python #52, mercury-nodejs #119); the
re-drive ran them from the branches of #53 and #120. The Playground jar, the graph, the servers file and the request were
identical in all four runs.
Scenario 1 — the baseline, negative control first¶
Both packs: 11 engine records, no pack record. The Handled lines of the packs carried the engine's trace id, proof that
the trace context crossed the wire, but nothing was recorded for the execution on either side.
Python pack (trace c355fab55c6f469a8d1b59b57e9b24e4): the engine's records, the pack's three Handled lines (llm.chat 3820 ms and 2607 ms,
diagram.render.mcp 3701 ms) and no pack record. llm.run spans 10,296 ms, diagram.render 3,718 ms; the pack's time is
inside them.
| host | service | from | span_id | parent_span_id | exec ms | round trip ms | parent |
|---|---|---|---|---|---|---|---|
| engine | http.request |
- | 8641da954e777566 |
- |
10334.725 | - | root |
| engine | http.flow.adapter |
http.request | b7df1ae81a12717a |
8641da954e777566 |
1.071 | - | ok |
| engine | task.executor |
event.script.manager | ba4baffab6d98b43 |
b7df1ae81a12717a |
10324.0 | - | ok |
| engine | graph.island |
graph.executor | bc0873791fe0bfb1 |
b7df1ae81a12717a |
0.028 | - | ok |
| engine | graph.math |
graph.executor | 8b3f6c1ed167d4fc |
b7df1ae81a12717a |
4.923 | - | ok |
| engine | graph.task |
graph.executor | 9a9ccbdc31cedc6f |
8b3f6c1ed167d4fc |
26.601 | - | ok |
| engine | llm.run |
graph.task | 939793b4bba9f1ab |
9a9ccbdc31cedc6f |
10296.427 | 10299.124 | ok |
| engine | diagram.render |
llm.run | 9c59e0762aabe43a |
939793b4bba9f1ab |
3718.051 | 3719.147 | ok |
| engine | graph.math |
graph.executor | 8ab661311854695a |
9a9ccbdc31cedc6f |
0.414 | - | ok |
| engine | graph.data.mapper |
graph.executor | b996855d5b845668 |
8ab661311854695a |
0.191 | - | ok |
| engine | async.http.response |
task.executor | b6c3fe29510f78a6 |
b996855d5b845668 |
2.381 | - | ok |
Node.js pack (trace b9651c5106b34d93bc6ee4cb845875d7): the same shape; the pack's Handled lines read llm.chat 3146 ms and 2609 ms,
diagram.render.mcp 3580 ms.
| host | service | from | span_id | parent_span_id | exec ms | round trip ms | parent |
|---|---|---|---|---|---|---|---|
| engine | http.request |
- | bf8084a22cceada5 |
- |
9572.623 | - | root |
| engine | http.flow.adapter |
http.request | 90a1496ec0c522ea |
bf8084a22cceada5 |
0.839 | - | ok |
| engine | task.executor |
event.script.manager | bdaa3b9053fe711f |
90a1496ec0c522ea |
9561.0 | - | ok |
| engine | graph.island |
graph.executor | bdca61ddaf384824 |
90a1496ec0c522ea |
0.03 | - | ok |
| engine | graph.math |
graph.executor | 8da6de0393ffd2cf |
90a1496ec0c522ea |
5.917 | - | ok |
| engine | graph.task |
graph.executor | 87158136c095b10a |
8da6de0393ffd2cf |
30.062 | - | ok |
| engine | llm.run |
graph.task | a0e87f1be17a7e84 |
87158136c095b10a |
9533.167 | 9535.805 | ok |
| engine | diagram.render |
llm.run | 8cd27d68b53df6e1 |
a0e87f1be17a7e84 |
3609.309 | 3610.847 | ok |
| engine | graph.math |
graph.executor | 84d92f8f55442549 |
87158136c095b10a |
0.289 | - | ok |
| engine | graph.data.mapper |
graph.executor | 9f4bb96eeaf01aba |
84d92f8f55442549 |
0.19 | - | ok |
| engine | async.http.response |
task.executor | b1563c706333c33c |
9f4bb96eeaf01aba |
2.797 | - | ok |
Scenario 2 — after the change, both packs¶
Python pack (trace 4c46977ba8fc4ecea5bfac870a51bbb3): 20 records, continuous.
| host | service | from | span_id | parent_span_id | exec ms | round trip ms | parent |
|---|---|---|---|---|---|---|---|
| engine | http.request |
- | 90533de404457c66 |
- |
10069.744 | - | root |
| engine | http.flow.adapter |
http.request | b3a2c3c9b598c13f |
90533de404457c66 |
0.883 | - | ok |
| engine | task.executor |
event.script.manager | 9d4bda26039b91b3 |
b3a2c3c9b598c13f |
10059.0 | - | ok |
| engine | graph.island |
graph.executor | ab1a30d1b018c70c |
b3a2c3c9b598c13f |
0.021 | - | ok |
| engine | graph.math |
graph.executor | 92c88a8d2cf99afd |
b3a2c3c9b598c13f |
4.348 | - | ok |
| engine | graph.task |
graph.executor | a425f855ffa0691a |
92c88a8d2cf99afd |
23.687 | - | ok |
| engine | llm.run |
graph.task | 9fc9d0fe9f1a5f3c |
a425f855ffa0691a |
10031.002 | 10033.3 | ok |
| helper | event.api.service |
http.request | d5d1e3d1616202bd |
46ac21f457226487 |
0.019 | - | ok |
| helper | llm.chat |
llm.run | 99708517341c4a39 |
d5d1e3d1616202bd |
3499.869 | 3500.314 | ok |
| helper | http.request |
- | 46ac21f457226487 |
9fc9d0fe9f1a5f3c |
3500.668 | - | ok |
| engine | diagram.render |
llm.run | ad3646ad1b7a328e |
9fc9d0fe9f1a5f3c |
3588.896 | 3590.376 | ok |
| binding | event.api.service |
http.request | ccc149d697f97815 |
a94233dbd794064f |
0.026 | - | ok |
| binding | diagram.render.mcp |
diagram.render | a039ef66d152f5cc |
ccc149d697f97815 |
3566.176 | 3566.967 | ok |
| binding | http.request |
- | a94233dbd794064f |
ad3646ad1b7a328e |
3567.438 | - | ok |
| helper | event.api.service |
http.request | 61e2db42b2eaecd2 |
bd7689db3f676a2e |
0.012 | - | ok |
| helper | llm.chat |
llm.run | b6bb117919c06d7a |
61e2db42b2eaecd2 |
2798.049 | 2798.473 | ok |
| helper | http.request |
- | bd7689db3f676a2e |
9fc9d0fe9f1a5f3c |
2798.951 | - | ok |
| engine | graph.math |
graph.executor | aa5768e4b4642161 |
a425f855ffa0691a |
0.451 | - | ok |
| engine | graph.data.mapper |
graph.executor | bdb794a15c6c984f |
aa5768e4b4642161 |
0.17 | - | ok |
| engine | async.http.response |
task.executor | b0bcb7743b47b62b |
bdb794a15c6c984f |
2.067 | - | ok |
Node.js pack (trace b20b5f5f47fa46a49c3e753f5ca38e85): 20 records, continuous.
| host | service | from | span_id | parent_span_id | exec ms | round trip ms | parent |
|---|---|---|---|---|---|---|---|
| engine | http.request |
- | 82124d253853443c |
- |
8876.159 | - | root |
| engine | http.flow.adapter |
http.request | b8149dc0340230f1 |
82124d253853443c |
1.046 | - | ok |
| engine | task.executor |
event.script.manager | be26e6919cdfc376 |
b8149dc0340230f1 |
8864.0 | - | ok |
| engine | graph.island |
graph.executor | a473f64319c93c4b |
b8149dc0340230f1 |
0.023 | - | ok |
| engine | graph.math |
graph.executor | a093f15db84b2bf9 |
b8149dc0340230f1 |
4.753 | - | ok |
| engine | graph.task |
graph.executor | a48f81d4537cb301 |
a093f15db84b2bf9 |
24.197 | - | ok |
| engine | llm.run |
graph.task | 8d1d736484978dfb |
a48f81d4537cb301 |
8840.7 | 8842.776 | ok |
| helper | event.api.service |
http.request | e014988bc1a89bff |
0b2e0700e756a83f |
0.066 | - | ok |
| helper | llm.chat |
llm.run | a3e87dc00d436777 |
e014988bc1a89bff |
2776.574 | 2781.262 | ok |
| helper | http.request |
- | 0b2e0700e756a83f |
8d1d736484978dfb |
2782.075 | - | ok |
| engine | diagram.render |
llm.run | bcfa910a8cf06485 |
8d1d736484978dfb |
3460.984 | 3462.028 | ok |
| binding | event.api.service |
http.request | 2df4e455aaf9d740 |
e69aa4d1bdfa5c36 |
0.076 | - | ok |
| binding | http.request |
- | e69aa4d1bdfa5c36 |
bcfa910a8cf06485 |
3441.39 | - | ok |
| binding | diagram.render.mcp |
diagram.render | c5979c25c62dcc01 |
2df4e455aaf9d740 |
3433.784 | 3440.407 | ok |
| helper | event.api.service |
http.request | 7b373c19bf8a3a6f |
0546a93f2f0123df |
0.013 | - | ok |
| helper | http.request |
- | 0546a93f2f0123df |
8d1d736484978dfb |
2454.382 | - | ok |
| helper | llm.chat |
llm.run | da1d512af3a0b6e4 |
7b373c19bf8a3a6f |
2452.795 | 2453.944 | ok |
| engine | graph.math |
graph.executor | b60a8efcef274296 |
a48f81d4537cb301 |
0.183 | - | ok |
| engine | graph.data.mapper |
graph.executor | 917d0a5f279912ff |
b60a8efcef274296 |
0.136 | - | ok |
| engine | async.http.response |
task.executor | b70bea2461138349 |
917d0a5f279912ff |
2.436 | - | ok |
Reading the tables: the engine's llm.run calls the helper twice (two model turns), and each call appears as the helper's
http.request edge under llm.run's span, event.api.service under the edge, and llm.chat under the service with its
round_trip; the engine's diagram.render calls the binding once, and the binding's http.request edge sits under
diagram.render's span with event.api.service and diagram.render.mcp beneath it. The engine's own records are the
baseline's eleven, unchanged. The packs' from fields name the engine function that called (llm.run, diagram.render)
because the engine's PostOffice stamps the calling route on the wire envelope.
Findings¶
- An RPC over Event-over-HTTP is the callee's to record. The caller's
async.http.requestleg is inskip.rpc.tracingby default so that an HTTP call folds into the calling function's span; the Java callee'sevent.api.serviceand REST edge are traced for exactly this reason (its javadoc: so the/api/eventedge is visible in the span tree). A runtime that suppresses the responder's record for an RPC, as the engines do, has to leave the callee's records instead - the packs' "engine parity" had copied the suppression without the records that justify it. The rule, stated for all four runtimes: every traced execution leaves exactly one record; who emits it depends on the side, never on the runtime. - The responder's span has to ride the reply. A Java inbox names the round-trip record's span from the reply
(
InboxBase.spanIdFromResponder), andWorkerHandlerputs it there. The packs' replies carried none, so even a Java caller recording the hop would have produced a record the forwarder drops (no valid span id). Both packs now set it. - The alias shape is invisible, by construction.
mcp.toolnever appears in any record; the trace knows the bound route only. This settles the question the drive began with. - Timings nest as they should. In every call the edge's
exec_timecovers the function'sround_trip, which covers itsexec_time; the engine's round trip to the pack exceeds the pack's edge by the wire's few milliseconds (for examplediagram.render3,588.9 ms and 3,590.4 ms round trip against the Python binding's edge 3,567.4 ms anddiagram.render.mcp3,566.2 ms with a 3,567.0 ms round trip). - The guards. mercury-python
tests/test_event_stream.pyand mercury-nodejstest/event-stream.test.tspin the three callee records and their parents for a streamed call and for a plain RPC, the span on the reply, the refusal's edge record (404) and the silence of an untraced call; the local round-trip record replaces the former "RPC legs emit no dataset" pin in both packs.
What this drive did not do¶
- No backend confirmation. The records were read from the logs, which are the forwarder's input, mapped one to one to spans (the 2026-09-22 OpenTelemetry forwarder certification). Eric's review in a Dynatrace tree remains the certification step, as it was then; that report's Scenario 2 carries an addendum pointing here.
- The Rust engine was not the caller. It speaks the same wire contract and stamps the same spans; a Rust Playground drive would show the same tree.
- One model, one graph. The trace shape does not depend on the model or the graph; the drive used the cheapest model that completes the diagram task.