Distributed Tracing & APM
OxPHP supports W3C Trace Context propagation, OpenTelemetry (OTel) export, and built-in Application Performance Monitoring (APM). Incoming traceparent headers are parsed and continued, trace IDs are available in PHP via $_SERVER, access logs include trace fields, and spans can be exported to Jaeger, Grafana Tempo, Zipkin, or any OTLP-compatible backend.
The APM plugin adds three layers of tracing on top of the OTel foundation:
- Automatic instrumentation hooks internal PHP functions (PDO, mysqli, cURL, Redis, Memcached, file I/O) at the engine level, so every call becomes a span without any change to your code.
- Attribute-based tracing creates spans automatically around any PHP function or method you annotate with
#[OxPHP\Apm\Trace]. - The PHP SDK provides 10
oxphp_apm_*()functions for manual span creation, attributes, events, and error recording.
How it works
- OxPHP reads the incoming request's
traceparentandtracestateheaders, as the W3C Trace Context specification defines them. - A new span ID is generated for this hop, and the incoming span ID becomes its parent.
- The trace IDs are injected into PHP as
$_SERVER['OXPHP_TRACE_ID'],$_SERVER['OXPHP_SPAN_ID'], and$_SERVER['OXPHP_PARENT_SPAN_ID']. - Structured JSON access logs include
trace_idandspan_idfields for log correlation. - The updated
traceparentheader (with OxPHP's span ID) is added to the response, so downstream services can continue the trace. - When the optional OTel plugin is enabled, each request becomes a span exported via OTLP with HTTP semantic convention attributes.
If no traceparent header is present, OxPHP generates a new trace ID and span ID and starts a fresh trace.
Configuration
W3C Trace Context (built-in)
| Variable | Default | Description |
|---|---|---|
TRACE_CONTEXT |
false |
Enable W3C Trace Context propagation. Set to true or 1 |
OpenTelemetry plugin
The OTel plugin is a compile-time feature (plugin-otel). When enabled, it automatically turns on trace-context propagation (the same effect as setting TRACE_CONTEXT=true).
| Variable | Default | Description |
|---|---|---|
OTEL_ENABLED |
false |
Enable the OpenTelemetry plugin. Boolean (see Boolean values) |
OTEL_EXPORTER_OTLP_PROTOCOL |
grpc |
Export protocol: grpc or http/protobuf |
OTEL_EXPORTER_OTLP_ENDPOINT |
http://localhost:4317 (gRPC) or http://localhost:4318 (HTTP) |
OTLP collector endpoint. With http/protobuf this is the collector's base URL, and spans are sent to /v1/traces under it, as the OTLP specification prescribes: http://collector:4318 sends to http://collector:4318/v1/traces, so leave /v1/traces off the value. An https:// URL is exported over TLS on both transports, verified against the system trust store (the runtime image must ship a CA bundle such as ca-certificates, which the official image installs); custom CA bundles and mTLS are not yet supported |
OTEL_EXPORTER_OTLP_TIMEOUT |
10000 |
Export timeout in milliseconds. Over gRPC it also bounds connecting to the collector, TLS handshake included, so keep it above the time a connect takes; an attempt that has to connect first can take up to twice the timeout. Over gRPC on Linux it also sets how soon a connection that stopped taking data is closed (see Span export errors in the log) |
OTEL_EXPORTER_OTLP_HEADERS |
(unset) | Authentication headers: key=value,key2=value2 |
OTEL_SERVICE_NAME |
oxphp |
Service name in exported spans |
OTEL_SERVICE_VERSION |
(unset) | Service version attribute |
OTEL_RESOURCE_ATTRIBUTES |
(unset) | Additional resource attributes: env=prod,region=us-east-1 |
OTEL_TRACES_SAMPLER |
parentbased_traceidratio |
Sampling strategy: always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio |
OTEL_TRACES_SAMPLER_ARG |
1.0 |
Sampling ratio (0.0 to 1.0) for ratio-based samplers |
Invalid or out-of-range OTEL_TRACES_SAMPLER_ARG values are clamped to [0.0, 1.0] and logged at warn level. Unknown OTEL_TRACES_SAMPLER values fall back to parentbased_traceidratio and are logged.
APM plugin
The APM plugin is a compile-time feature (plugin-apm) that depends on the OTel plugin. It adds automatic instrumentation, the #[OxPHP\Apm\Trace] decorator, and the PHP tracing SDK.
| Variable | Default | Description |
|---|---|---|
OTEL_APM_ENABLED |
false |
Enable APM: auto-instrumentation, error capture, PHP SDK. Requires OTEL_ENABLED=true. Boolean (see Boolean values) |
OTEL_APM_SLOW_QUERY_MS |
100 |
Slow query threshold in milliseconds. Queries above this get oxphp.db.slow=true on their spans |
OTEL_APM_DB_CAPTURE_PARAMS_ENABLED |
false |
Record bind parameters in the db.params span attribute. Values are recorded raw. Unlike db.statement, they are not obfuscated, so this can place PII (emails, tokens, etc.) in your traces. Enable only where that is acceptable. Boolean (see Boolean values) |
OTEL_APM_STACKTRACE_MAX_BYTES |
8192 |
Maximum size in bytes of the exception.stacktrace attribute. Over the cap the stacktrace is truncated from the tail (the root frame is kept) with a …(truncated) marker. 0 disables truncation |
OTEL_APM_MESSAGE_MAX_BYTES |
4096 |
Maximum size in bytes of the exception.message attribute (default matches New Relic's per-attribute value limit). Over the cap the message is truncated from the tail with a …(truncated) marker. 0 disables truncation |
Trace context in PHP
When TRACE_CONTEXT=true, three $_SERVER variables are available in your PHP scripts:
| Variable | Description | Example |
|---|---|---|
OXPHP_TRACE_ID |
W3C trace ID (32 hex chars) | 4bf92f3577b34da6a3ce929d0e0e4736 |
OXPHP_SPAN_ID |
OxPHP's span ID for this request (16 hex chars) | 00f067aa0ba902b7 |
OXPHP_PARENT_SPAN_ID |
Incoming parent span ID (16 hex chars, empty if new trace) | a3ce929d0e0e4736 |
Use these to propagate trace context to downstream services:
<?php
$traceId = $_SERVER['OXPHP_TRACE_ID'] ?? '';
$spanId = $_SERVER['OXPHP_SPAN_ID'] ?? '';
if ($traceId) {
// Build a traceparent header for downstream calls
$traceparent = "00-{$traceId}-{$spanId}-01";
$response = file_get_contents('https://api.example.com/data', false,
stream_context_create([
'http' => [
'header' => "traceparent: {$traceparent}\r\n",
],
])
);
}With Guzzle
<?php
$traceId = $_SERVER['OXPHP_TRACE_ID'] ?? '';
$spanId = $_SERVER['OXPHP_SPAN_ID'] ?? '';
$client = new \GuzzleHttp\Client();
$response = $client->get('https://api.example.com/users', [
'headers' => [
'traceparent' => "00-{$traceId}-{$spanId}-01",
],
]);Access log correlation
When trace context is enabled, structured JSON access logs include trace_id and span_id fields:
{
"timestamp": "2026-03-23T10:15:30.123Z",
"level": "INFO",
"fields": {
"request_id": "4bf92f3577b34da6-00f067aa",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"method": "GET",
"path": "/api/users",
"status": 200,
"duration_us": 1523,
"remote_ip": "10.0.0.1",
"user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0",
"message": "request completed"
}
}You can then search logs by trace ID in log aggregation systems (Loki, Elasticsearch, Splunk, CloudWatch) to find every log entry for a distributed trace.
Response headers
OxPHP adds the traceparent header to every response, with OxPHP's own span ID:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01If the incoming request included a tracestate header, it is forwarded in the response as well.
OpenTelemetry integration
When the OTel plugin is enabled, each HTTP request becomes a span exported to your tracing backend via OTLP.
Span attributes
Exported spans include standard HTTP semantic convention attributes:
| Attribute | Description |
|---|---|
http.request.method |
HTTP method (GET, POST, etc.) |
url.path |
Request path |
http.response.status_code |
Response status code |
client.address |
Client IP address |
user_agent.original |
The request's User-Agent header (the first one, if it sent several), cut to 512 bytes (a …(truncated) marker included) when longer. Bytes that are not valid UTF-8 are replaced with U+FFFD. Absent when the request sent none |
server.address |
Server listen address |
oxphp.request_id |
OxPHP request ID |
http.request.body.size |
Request body size in bytes (if non-zero) |
http.response.body.size |
Response body size in bytes (if non-zero) |
oxphp.shed_reason |
Why admission refused the request, on a request it refused: queue_full, wait_timeout, waiting_full or waiting_bytes on a 529, shutting_down on a 503, pool_unavailable on a 500. These are the values of the reason label on oxphp_admission_refused_total. Absent otherwise, including on a 529 the application returned itself |
5xx responses are marked as error spans.
Span events
Child spans also carry span events: timestamped annotations exported as
OpenTelemetry events, which Jaeger, Grafana Tempo, and other OTLP backends
render natively. An oxphp.event.kind attribute on each event identifies its type:
oxphp.event.kind |
Source | Event attributes |
|---|---|---|
exception |
A #[OxPHP\Apm\Trace] function that threw, oxphp_apm_error(), or an unhandled exception / fatal on the request's root span (see Automatic root-span exception capture) |
exception.type, exception.message, exception.stacktrace |
custom |
oxphp_apm_event() |
user-supplied |
mark |
profiler #[Mark] annotation |
user-supplied |
slow |
profiler #[SlowThreshold] breach |
threshold_ms, elapsed_ms |
memory_spike |
profiler #[MemoryThreshold] breach |
threshold_kb, delta_bytes |
The oxphp.event.kind attribute may additionally carry sql, http, or alloc on events generated by APM instrumentation.
Automatic root-span exception capture
When a request fails with an unhandled exception or a fatal error and returns a 5xx, OxPHP attaches an exception event to the request's root span automatically. This needs neither a #[OxPHP\Apm\Trace] attribute nor an oxphp_apm_error() call. A 500 becomes self-describing in the trace, and backends that group errors by the exception event (New Relic Errors Inbox, for example) pick it up without any application code.
The event carries the standard exception.type, exception.message, and exception.stacktrace, plus two OxPHP extensions, exception.file and exception.line, pointing at the throw site (or the fatal's location). A classless fatal is a trigger_error(…, E_USER_ERROR), an out-of-memory abort, or an execution timeout. For one of those, exception.type is a synthetic name (the PHP error constant, e.g. E_USER_ERROR) and no stacktrace is present. Calling an undefined function is not a classless fatal on PHP 8: it throws a normal Error, which carries a full stacktrace like any other exception. Message and stacktrace obey the same OTEL_APM_MESSAGE_MAX_BYTES / OTEL_APM_STACKTRACE_MAX_BYTES caps as the other exception events.
This works for raw PHP, applications without a custom exception handler, and worker-mode handlers.
Boundary: frameworks that swallow exceptions (traditional request path). In Traditional / Framework / SPA modes, an application that installs set_exception_handler() and renders its own error page (Laravel, Symfony, WordPress, …) has handled the exception from the engine's point of view. It never propagates uncaught, so OxPHP sees only the 500 status and cannot recover the Throwable. Automatic capture does not fire for those requests; record the exception explicitly from your framework's error reporter with oxphp_apm_error($e) instead.
Worker mode does not route through set_exception_handler(). The worker runtime catches an exception that escapes your oxphp_worker() closure directly, without invoking the engine's user exception handler. So for worker handlers, automatic capture fires whenever an exception leaves the closure, even if the code registered its own set_exception_handler(). That handler then applies only to exceptions the closure itself catches, not to ones that escape it.
Streaming responses. For a streamed response (SSE, or any oxphp_stream_flush() loop), the HTTP status is fixed once the headers are on the wire, and the request is treated as complete at that point. A fatal thrown after the status has been committed, on a streaming response or on one that called oxphp_finish_request(), is only logged. It is not added to the root span (the trace still shows the span, without the exception event). This is a documented boundary and holds equally for a committed 5xx and a committed 2xx.
Request ID with OTel
When the OTel plugin is active, request IDs are derived from the trace context: the first 16 characters of the trace ID and first 8 characters of the span ID, separated by a dash. This appears in logs, the X-Request-ID response header, and oxphp_request_id() in PHP.
APM: automatic instrumentation
When the APM plugin is enabled, OxPHP automatically hooks 34 internal PHP functions at the engine level. Every call to a hooked function creates a child span under the current request's root span. Your code does not need to change for this.
Hooked functions
| Category | Functions |
|---|---|
| PDO | PDO::__construct, PDO::query, PDO::exec, PDO::prepare, PDOStatement::execute |
| mysqli | mysqli::__construct, mysqli::query, mysqli::prepare, mysqli_stmt::prepare, mysqli_stmt::execute |
| cURL | curl_init, curl_setopt, curl_exec, curl_multi_exec |
| Redis | Redis::connect, Redis::get, Redis::set, Redis::del, Redis::mget, Redis::mset, Redis::hget, Redis::hset, Redis::lpush, Redis::rpush |
| Memcached | Memcached::get, Memcached::set, Memcached::delete, Memcached::getMulti, Memcached::setMulti |
| File I/O | fopen, fread, fwrite, file_get_contents, file_put_contents |
Hooks are only installed for extensions that are actually loaded. If your build does not include the Redis extension, the Redis hooks are silently skipped.
Database span attributes
Database hooks (PDO, mysqli) decorate their spans with OpenTelemetry semantic-convention attributes:
| Attribute | Source | Example |
|---|---|---|
db.statement |
Query text with literal values replaced by ?, so PII (emails, tokens) never reaches the tracing backend |
SELECT * FROM users WHERE email = ? |
db.operation |
Leading SQL keyword | SELECT |
db.system |
Parsed from the PDO DSN / mysqli constructor | mysql, postgresql, sqlite |
server.address, server.port |
Connection host/port (omitted for a unix socket or SQLite file) | db.internal, 5432 |
db.name |
Database name, or the file path for SQLite | shop |
oxphp.db.slow |
true when the call's wall-time meets or exceeds OTEL_APM_SLOW_QUERY_MS |
true |
db.params |
Bound parameters, recorded raw (not obfuscated), so they can contain PII. Recorded only when OTEL_APM_DB_CAPTURE_PARAMS_ENABLED=true |
[1, active] |
db.statement is read from each query / exec / prepare call's own arguments, so it appears on that span. On a PDOStatement::execute span it is also present, read from the statement object's own queryString property, so it can never be another statement's SQL. A mysqli_stmt::execute span has no db.statement (mysqli exposes no such property), but the SQL is on the preceding mysqli::prepare span. Every execute span carries the timing, and with it the slow-query flag, and for PDO also db.params. Cache (Redis, Memcached), HTTP-client (cURL), and file-I/O hooks emit a bare timing span.
Hook installation
Hook installation uses a two-phase design for thread safety under PHP ZTS:
- Phase 1 (MINIT) runs during module initialization. OxPHP validates each target function against the loaded extensions and captures the handler pointer each target has at that point into a read-only approved list.
- Phase 2 (RINIT) runs on the first request per worker thread and installs the approved hooks into that thread's function tables.
This gives each ZTS worker thread consistent function table modifications and thread-local state.
Phase 1 runs after OxPHP's own handler replacements. With RUNTIME_HOOKS enabling streams, the handler captured for a PDO, mysqli or Redis method that category also hooks is therefore OxPHP's guarded one: for a query or command it claims the connection for the calling fiber, and for PDO::__construct it serialises persistent constructors to the same data source (see Runtime hooks). The span wraps that guard. Inside worker-mode request fibers and async task fibers, where the guard can wait, the span's duration (and, for a query, the slow-query flag) includes any wait for a connection another request or task is using.
APM: attribute-based tracing
The #[OxPHP\Apm\Trace] attribute creates spans automatically around decorated functions and methods. Unlike the auto-instrumentation hooks (which target internal C functions), this works on user-defined PHP code.
<?php
use OxPHP\Apm\Trace;
#[Trace]
function processOrder(int $orderId): void
{
// A span named "processOrder" is created on entry and closed on exit.
// If an exception is thrown, the span is marked as error and an
// "exception" span event records exception.type, exception.message
// and exception.stacktrace.
}
class PaymentService
{
#[Trace]
public function charge(float $amount): bool
{
// Span named "PaymentService::charge"
return true;
}
}The #[Trace] attribute targets both functions and methods. The APM plugin registers the decorator automatically during initialization, so no registration call is needed.
If the decorated function throws an exception, the span's status is set to error and an exception event is recorded with the full OpenTelemetry semantic-convention data: exception.type (the class), exception.message (the message), and exception.stacktrace (the call stack from getTraceAsString()). The message is truncated to OTEL_APM_MESSAGE_MAX_BYTES bytes (default 4096) and the stacktrace to OTEL_APM_STACKTRACE_MAX_BYTES bytes (default 8192); 0 disables either cap. Argument capture inside frames follows PHP's own zend.exception_ignore_args setting.
APM: PHP tracing SDK
The APM plugin registers 10 oxphp_apm_*() functions for manual span management. They are all registered whether or not APM is enabled, so your code works without modification in any environment. With APM disabled the span functions become no-ops returning sentinel values (0 for IDs, "" for strings).
The one exception to the no-op rule is oxphp_apm_trace(): with APM disabled it still runs the callback it was given and still returns its value. The business logic inside the callback therefore keeps running when tracing is off.
Creating spans
<?php
// Start a span and get its local ID
$spanId = oxphp_apm_start('cache.warm', ['cache.size' => '1024']);
// ... do work ...
// Close the span
oxphp_apm_end($spanId);Adding attributes and events
<?php
$spanId = oxphp_apm_start('order.process');
// Add attributes to the current span (or a specific one)
oxphp_apm_attribute('order.id', $orderId);
oxphp_apm_attribute('order.total', $total, $spanId);
// Record an event on the span
oxphp_apm_event('payment.authorized', [
'provider' => 'stripe',
'amount' => (string) $amount,
]);
oxphp_apm_end($spanId);Error recording
<?php
$spanId = oxphp_apm_start('external.api');
try {
$result = callExternalApi();
} catch (\Throwable $e) {
// Mark the span as error
oxphp_apm_error($e, $spanId);
throw $e;
} finally {
oxphp_apm_end($spanId);
}Propagating trace context
<?php
// Get the current trace ID and span ID
$traceId = oxphp_apm_trace_id();
$currentSpanId = oxphp_apm_span_id();
// Or get a ready-to-use traceparent header value
$traceparent = oxphp_apm_header();
// "00-{trace_id}-{span_id}-01"
// Propagate to downstream services
$response = file_get_contents('https://api.example.com/data', false,
stream_context_create([
'http' => [
'header' => "traceparent: {$traceparent}\r\n",
],
])
);Function reference
| Function | Returns | Description |
|---|---|---|
oxphp_apm_trace(name, callback, ?attributes) |
mixed |
Execute a callback inside a span and return its value. The callback receives the span's local ID; a throw marks the span as error with an exception event (type and message, no stacktrace) and propagates |
oxphp_apm_start(name, ?attributes) |
int |
Open a span and return its local ID. 0 when APM is disabled |
oxphp_apm_end(span_id) |
void |
Close the span with the given local ID |
oxphp_apm_attribute(key, value, ?span_id) |
void |
Set an attribute on the current or specified span |
oxphp_apm_event(name, ?attributes, ?span_id) |
void |
Record a timestamped event on the current or specified span |
oxphp_apm_error(exception, ?span_id) |
void |
Mark the current or specified span as error and record an exception event. A Throwable object contributes exception.type, exception.message, and exception.stacktrace; a bare string argument is recorded as exception.message under a generic exception.type of Error (so the event stays visible in backends that group by type) |
oxphp_apm_status(code, ?description, ?span_id) |
void |
Set span status: 0 = Unset, 1 = Ok, 2 = Error |
oxphp_apm_trace_id() |
string |
Current trace ID (32 hex chars). Empty when APM is disabled |
oxphp_apm_span_id() |
string |
Current span ID (16 hex chars). Empty when no active span |
oxphp_apm_header() |
string |
W3C traceparent header value for the current span context |
For the complete function signature reference, see PHP Functions.
Docker example
Ready-to-run compose.yaml variants:
Enable W3C trace propagation without an external backend:
services:
app:
image: ghcr.io/oxphp/oxphp:0.13.0
ports:
- "80:80"
environment:
- TRACE_CONTEXT=true
- INTERNAL_ADDR=0.0.0.0:9090Full observability stack with Jaeger as the tracing backend:
services:
app:
image: ghcr.io/oxphp/oxphp:0.13.0
ports:
- "80:80"
environment:
- OTEL_ENABLED=true
- OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
- OTEL_SERVICE_NAME=my-app
- OTEL_SERVICE_VERSION=1.0.0
- OTEL_RESOURCE_ATTRIBUTES=env=production
- INTERNAL_ADDR=0.0.0.0:9090
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPCservices:
app:
image: ghcr.io/oxphp/oxphp:0.13.0
ports:
- "80:80"
environment:
- OTEL_ENABLED=true
- OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317
- OTEL_SERVICE_NAME=my-app
tempo:
image: grafana/tempo:latest
ports:
- "4317:4317"
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"Full observability with automatic instrumentation of database queries, HTTP calls, cache operations, and file I/O:
services:
app:
image: ghcr.io/oxphp/oxphp:0.13.0
ports:
- "80:80"
environment:
- OTEL_ENABLED=true
- OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
- OTEL_SERVICE_NAME=my-app
- OTEL_APM_ENABLED=true
- OTEL_APM_SLOW_QUERY_MS=50
- INTERNAL_ADDR=0.0.0.0:9090
jaeger:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPC
environment:
- COLLECTOR_OTLP_ENABLED=trueThe plugin-apm Cargo feature must be enabled at build time. The official OxPHP image includes it by default.
The observability stack
OxPHP provides three observability pillars that work together:
| Pillar | Feature | Correlation |
|---|---|---|
| Metrics | Prometheus counters and histograms at /metrics |
Aggregate performance data |
| Logging | Structured JSON access logs with ACCESS_LOG |
Per-request detail, searchable by trace_id |
| Tracing | W3C Trace Context + OTLP export | End-to-end distributed request flow |
All three share the same trace_id and request_id, so you can drill down from a Grafana dashboard alert to a Tempo trace to the Loki log lines for a single request.
Troubleshooting
Trace headers not appearing in responses
TRACE_CONTEXT is not enabled.
Fix: Set TRACE_CONTEXT=true or enable the OTel plugin with OTEL_ENABLED=true (which enables trace context automatically).
$_SERVER trace variables are empty
Trace context is disabled, or the variables are being checked outside of OxPHP.
Check: The OXPHP_TRACE_ID, OXPHP_SPAN_ID, and OXPHP_PARENT_SPAN_ID variables only exist when TRACE_CONTEXT=true and the request is served by OxPHP. Test with:
<?php
echo $_SERVER['OXPHP_TRACE_ID'] ?? 'trace context not enabled';Spans not appearing in Jaeger/Tempo
Check: Verify the OTLP endpoint is reachable from the OxPHP container:
docker compose exec app curl -v http://jaeger:4317Check: Verify the plugin is enabled:
curl -s http://localhost:9090/config | jq '.plugins'Fix: Ensure OTEL_ENABLED=true and the OTEL_EXPORTER_OTLP_ENDPOINT points to the correct collector address (with http/protobuf, its base URL without /v1/traces). If the log shows failed exports, Span export errors in the log says what each one means.
High sampling volume in production
Exporting every span is expensive at high traffic volumes.
Fix: Reduce the sampling ratio:
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1 # Sample 10% of tracesParent-based sampling means that if an incoming request carries a sampled trace, it will always be sampled regardless of the ratio. New traces started at OxPHP are sampled at the configured rate. If OTEL_TRACES_SAMPLER is set to an unknown value, OxPHP logs a warning and falls back to parentbased_traceidratio.
Span export errors in the log
OxPHP sends spans to the collector in batches, and logs a failed export on its own lines, with plugin set to otel:
WARNOTLP span export failed, retrying once: the first attempt failed fast, without an answer from the collector, and is being sent again. Fields:spans(batch size),error.ERROROTLP span export failed, the batch is dropped: the export failed for good, and its spans are not sent again. Fields:spans,retried,error.
A WARN line with no ERROR line after it means the retry delivered the batch. Errors the collector answers with itself are not retried: they come as a single ERROR line with retried set to false.
Every ERROR line comes with a second one for the same export, written by the OpenTelemetry library. Its message is empty, and the cause is in error:
{"level":"ERROR","fields":{"message":"","name":"BatchSpanProcessor.ExportError","error":"Operation failed: code: 'Unknown error', message: \"transport error\", source: ..."}}{"level":"ERROR","fields":{"message":"","name":"BatchSpanProcessor.ExportError","error":"Operation failed: reqwest::Error { kind: Request, url: \"http://collector:4318/v1/traces\", source: ... }"}}What the error field points at:
In error |
What happened |
|---|---|
code: 'The request does not have valid authentication credentials' (gRPC), kind: Status(401, None) (http/protobuf) |
The collector rejected the API key |
ConnectionReset (either protocol), connection closed (gRPC), IncompleteMessage (http/protobuf) |
Something between OxPHP and the collector, such as a load balancer or NAT, dropped the connection while it sat idle |
kind: TimedOut, error: Elapsed (gRPC) |
The export had to connect first, and connecting (name lookup, TCP, and the TLS handshake of an https:// endpoint) did not finish within OTEL_EXPORTER_OTLP_TIMEOUT. When the name resolves to several addresses, the timeout is shared out between them, so it can also mean that one address did not answer within its share |
An export over a connection whose far side is gone without a reset, such as one a NAT or load balancer dropped silently, runs into OTEL_EXPORTER_OTLP_TIMEOUT like any export the collector does not answer in time. Over gRPC on Linux the connection is then closed, once what was sent on it has gone unacknowledged for that long, and the exports after it connect again. The gRPC connection also carries TCP keepalive, with the first probe after 30 seconds without traffic and then one every 10 seconds. The keepalive keeps the connection in NAT and load-balancer tables between exports and, at the default timeout, closes a dead idle one within about a minute.
OpenTelemetry export metrics counts the failed exports, their spans and the retries, and explains when an export is retried.
Prompts for your agent
These prompts assume an agent with the OxPHP documentation MCP server connected. The agent reads the pages it needs itself.
Export OxPHP traces to my tracing backend. Read the `features/distributed-tracing` page through the OxPHP documentation MCP server.
Work out from my Compose or Kubernetes files where the collector lives, give me the environment block and a sampling rate my traffic can afford, and show me where the outbound calls in this repository have to forward the trace context.My OxPHP spans are not reaching my tracing backend. Read the `features/distributed-tracing` page through the OxPHP documentation MCP server.
Check the plugin switches, the OTLP endpoint and protocol, and the sampler my deployment sets, and tell me at which hop the traces are being lost.See also
- PHP Functions: the
oxphp_apm_*()function reference - Decorators: attribute-based function interception, including
#[Trace] - Access Logging: structured JSON logs with trace fields
- Request IDs: how request IDs interact with trace context
- Metrics: the Prometheus metrics reference
- Health Checks: the
/configendpoint, which shows trace context status - Configuration Reference: all environment variables