Prometheus Metrics
OxPHP exposes Prometheus-compatible metrics in text exposition format at GET /metrics on the internal server. They cover request throughput, response times, connection state, worker pool health, static file caching, compression efficiency, and worker mode performance.
Enabling metrics
Set INTERNAL_ADDR to start the internal server:
INTERNAL_ADDR=127.0.0.1:9090Then scrape from Prometheus or any compatible collector:
curl http://localhost:9090/metricsServer metrics
| Metric | Type | Description |
|---|---|---|
oxphp_uptime_seconds |
gauge | Seconds since the server process started |
oxphp_requests_total |
counter | Total HTTP requests received on the main port |
Request metrics
| Metric | Type | Description |
|---|---|---|
oxphp_requests_by_method_total |
counter | Requests by HTTP method. Label: method (GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, CONNECT, QUERY, OTHER) |
oxphp_responses_by_status_total |
counter | Responses by status class. Label: status (1xx, 2xx, 3xx, 4xx, 5xx) |
oxphp_request_bytes_total |
counter | Total request body bytes received |
oxphp_response_bytes_total |
counter | Total response body bytes sent |
oxphp_request_cancelled_total |
counter | Cancelled requests by reason. Label: reason (client_abort, timeout, shutdown). Counted once for each request the server dispatches (not only the ones that reach PHP), at the point that request is answered. So client_abort covers a client that hung up while its request was still queued (the common case) as well as one that hung up while its handler was running; a client that hangs up once the response has started going out is past that point and is not counted. Always emitted |
Only methods and status classes with at least one recorded event are emitted. Zero-count labels are omitted.
Request duration histogram
| Metric | Type | Description |
|---|---|---|
oxphp_request_duration_us |
histogram | End-to-end request duration in microseconds for all requests (static files and PHP) |
Bucket boundaries (microseconds): 100, 500, 1000, 2500, 5000, 10000, 25000, 50000, 100000, 250000, 500000, 1000000, +Inf.
Use this histogram to track overall latency, identify slow endpoints, and measure tail latency percentiles.
Connection metrics
| Metric | Type | Description |
|---|---|---|
oxphp_active_connections |
gauge | Currently open TCP connections on the main port |
oxphp_accept_stalled |
gauge | 1 while the accept loop is parked waiting for a free MAX_CONNECTIONS permit. Connections are accepted by the kernel but nothing new is served, so from a client's point of view the server has stopped answering. Alert on oxphp_accept_stalled == 1 directly; it needs no comparison against the configured budget |
oxphp_accept_stalls_total |
counter | Connections that had to wait for a MAX_CONNECTIONS permit after being accepted. Moves even when a stall starts and ends between two scrapes, which the gauge alone would miss: rate(oxphp_accept_stalls_total[5m]) > 0 means the budget is being exhausted, however briefly. Raise MAX_CONNECTIONS or shrink the PHP backlog (QUEUE_CAPACITY + QUEUE_MAX_WAITING); see Keeping headroom for the accept loop |
oxphp_pending_requests |
gauge | PHP requests accepted but not yet answered (waiting for a queue slot, queued, or executing). Only requests routed to PHP: a static file, a 404 or a denied path is answered without the queue and never appears here |
oxphp_dropped_requests_total |
counter | Requests where the PHP worker failed after accepting the request |
oxphp_admission_refused_total |
counter | Requests answered without reaching a worker. Label: reason, with values wait_timeout (waited out its whole wait budget, which is QUEUE_WAIT_TIMEOUT_MS or less where oxphp_admission_wait_budget_us shows the server has shortened it. Give the pool more headroom, but read oxphp_admission_wait_wasted_total first, because where the pool began nothing during those waits the remedy is the opposite one. A request whose client had already gone when the budget ran out is not counted here: it is answered 499 and counted in oxphp_request_cancelled_total{reason="client_abort"}, so this series stays a statement about the pool), waiting_full (already QUEUE_MAX_WAITING requests waiting, raise that or MAX_CONNECTIONS), waiting_bytes (the bodies already parked fill QUEUE_MAX_WAITING_BYTES, so this request's body had nowhere to sit; raise that, or lower how much a client may upload), queue_full (QUEUE_WAIT_TIMEOUT_MS=0, waiting is off), shutting_down (the drain deadline passed while the request was still waiting for admission), pool_unavailable (no worker thread is left to hand the request to: the pool is gone rather than busy). Only the first four are overload and answer 529; shutting_down answers 503 like the rest of graceful drain, and pool_unavailable answers 500. Alert on overload with those four specifically: the metric as a whole also moves on a restart. Excluded from oxphp_queue_wait_us |
oxphp_admission_wait_wasted_total |
counter | Requests that waited out their whole wait budget while the pool began no request at all. A subset of oxphp_admission_refused_total{reason="wait_timeout"}, separating a wait that lost to other requests competing for a worker from one during which the pool started nothing. Requests whose client had already gone are excluded, as above. Counts only while oxphp_admission_wait_budget_us is at its ceiling. Sustained non-zero means waiting buys nothing: shorten QUEUE_WAIT_TIMEOUT_MS. See Wasted waits |
oxphp_queue_depth |
gauge | Requests sitting in the worker queue right now: admitted, not yet picked up by a worker. An entry leaves only on a pickup, so this counts requests already answered on their deadline (529, or 499 where the client had gone) alongside ones still waiting; a depth above oxphp_pending_requests is that difference. Present only for the SAPI executor, the only one with a queue |
oxphp_queue_capacity |
gauge | Queue slots in total: QUEUE_CAPACITY, the bound oxphp_queue_depth is read against |
oxphp_admission_slots_available |
gauge | Admission permits nobody is holding. A request takes one before it enters the queue and gives it back when a worker picks it up, so this is the free capacity a new arrival can claim without waiting. During a graceful drain the gate is closed and admits nothing regardless of what this reads |
oxphp_admission_wait_budget_us |
gauge | How long a request arriving now may wait for a worker, in microseconds (the unit of oxphp_queue_wait_us, which it bounds). QUEUE_WAIT_TIMEOUT_MS is the ceiling on this value: the server halves it while the work the waiting admits is completing for clients who have already left, and walks it back up once the queue drains and less than a quarter of what the pool starts goes to departed clients. Below the ceiling means the pool is being offered more than it can serve to clients less patient than the ceiling. Sitting at the floor (a sixty-fourth of the ceiling, raised to 10 ms unless the ceiling itself is shorter) through an episode is normal; sitting there with oxphp_queue_depth at zero is not. 0 in fail-fast mode (QUEUE_WAIT_TIMEOUT_MS=0). Absent for an executor with no queue. See The adaptive wait budget |
oxphp_admission_wait_budget_ceiling_us |
gauge | QUEUE_WAIT_TIMEOUT_MS in microseconds: the longest the gauge above is ever allowed to be, constant for the life of the process. It lets an alert compare the budget against its ceiling from this endpoint alone rather than against a hardcoded number, the same pairing as oxphp_queue_depth with oxphp_queue_capacity. 0 in fail-fast mode. Absent for an executor with no queue |
oxphp_abandoned_work_total |
counter | Requests a worker took off the queue whose client had already gone. The wait budget above is adjusted against this count. Climbing while the budget falls is the controller doing its job; climbing with the budget still at its ceiling means clients are leaving during their handler rather than their wait, so the wait is not what to shorten. A count of requests, not of wasted CPU. Requests refused on their deadline are not counted (they never ran), nor are ones whose response went out before the handler finished (a stream whose headers were sent, or an early answer from oxphp_finish_request()), so the end of every SSE session does not count. Only a client that closes its connection is visible here |
oxphp_pool_stalled |
gauge | 1 while requests are waiting for workers that are idle and getting nothing done, meaning a pool that has stopped taking work off its queue. Set once the state has held for a minute (long enough that a worker re-loading the application after a recycle does not raise it); cleared by a worker finishing a request, or by a minute in which the pool has not looked wedged at all. Alert on oxphp_pool_stalled == 1 directly; the readiness probe answers 503 on the same state. Exported in worker mode only; absent rather than 0 where the state is not watched. See The wedged-pool signal |
Reading queue depth against admission slots
The two gauges answer different halves of one question, which is whether anything is moving rather than how deep the queue is. oxphp_queue_depth says what is waiting; oxphp_admission_slots_available says whether anything more can be let in. Read both against oxphp_busy_workers.
| Slots available | Queue depth | Reading |
|---|---|---|
> 0 |
0 |
Nothing waiting; normal |
> 0 |
> 0 and falling between scrapes |
Ordinary backlog: the pool is behind but working through it |
> 0 |
> 0 and not falling, with oxphp_busy_workers at 0 |
The pool has stopped taking work off the queue while admission still has room. This is a wedge in its first phase. Arrivals are admitted and each is answered 529 when its own budget runs out (or 499 if its client gave up first); oxphp_admission_wait_wasted_total climbs with every 529 while the budget is at its ceiling. Nothing else gives it away: no worker is busy, and the refusals on their own read like overload |
0 |
at oxphp_queue_capacity |
The same wedge once the queue has filled, or an ordinary overload the pool is far behind on. oxphp_busy_workers at 0 sustained across scrapes separates them; a single sample can read that way under overload too |
0 |
at or near 0 |
Permits held outside the queue. For a single scrape that can be requests between admission and dispatch; sustained across scrapes it is permits taken and never given back: nothing can be admitted, and the server refuses every PHP request until it is restarted |
The last three rows are the shapes a stalled pool takes. In each of them the rest of the picture reads healthy while no PHP request completes: no busy worker, 200 from liveness throughout and from readiness for the first minute, static files served normally. The refusal counters move, but they move under ordinary overload too. oxphp_pending_requests rises with the stuck queue and then falls back as each arrival is answered on its own budget, so on its own it means nothing here; read against oxphp_queue_depth it does: pending near zero while depth sits at capacity is the wedge holding everything it was ever given.
A wedge is exactly the condition that makes clients leave. Behind a load balancer whose read timeout is shorter than QUEUE_WAIT_TIMEOUT_MS, arrivals turn into 499s that move neither refusal series, so oxphp_admission_wait_wasted_total can stay quiet through the whole incident. Those arrivals land in oxphp_request_cancelled_total{reason="client_abort"}, and oxphp_queue_depth holds either way.
The server notices a wedge too. Whenever two consecutive one-second scans see work waiting (requests in the queue, or refusals climbing) with at least one worker idle and the pool having finished nothing, it logs:
{"timestamp":"2026-08-28T10:56:59.015691Z","level":"WARN","fields":{"message":"PHP requests are waiting while the pool has idle workers and got nothing done since the last scan","queue_depth":22,"queue_capacity":512,"admission_slots_available":490,"workers_idle":4}}It repeats about once a minute while the state lasts, and INFO PHP pool is reaching workers again follows once work moves. The fields sit under fields, so an alert rule wants .fields.queue_depth, not .queue_depth. Three guards prevent false reports: two scans rather than one; progress measured as what the workers finished rather than what clients received, so a storm of client aborts reads as the busy pool it is; and nothing reported until the pool has finished its first request, so startup bootstrap is not mistaken for a wedge.
That last guard covers the process's first bootstrap only. A worker recycled later (on the memory ceiling or Worker::scheduleExit()) boots its application again and reads as idle until it takes its first request, so a slow bootstrap on a one-worker pool (or on a larger pool whose other workers are all busy) produces one spurious warning followed by its recovery notice. oxphp_workers_spawned_total moving at the same moment identifies it.
The warning is emitted in worker mode only, since only that pool keeps a count of what the workers finished; the state it describes cannot arise where a worker blocks on the queue itself. The three gauges are exported in every mode.
Wasted waits
oxphp_admission_wait_wasted_total counts work started rather than entries leaving the queue, because a worker clearing requests that their deadlines already answered moves the queue without serving anybody. It is gated on the budget being at its ceiling because, once the server has shortened the budget, "the pool began nothing" says more about the window than the pool: a healthy four-worker pool on quarter-second handlers starts one request every 62 ms, so at the floor most waits end between two starts.
What the gating costs depends on whether the budget moves:
- A pool that starts nothing never reaches the started-request count the controller decides on, so it holds its ceiling and this counter stays armed.
- A self-calling pool starts a request every time an outer one gives up on its inner call. Once its own callers are less patient than its handlers, the budget comes down (measured at two halvings within seconds), and this counter goes quiet after the arrivals from before the drop. Nothing on this page takes over there: find the pattern in the application's own outbound calls. Where the callers are patient (an internal cron, a sidecar on a generous timeout), nothing abandons, the ceiling stands, and this counter is what names it.
The classic cause is an application calling back into this same server over HTTP, but the counter is not a self-call detector: a pool whose handlers all run longer than the budget frees nothing during a wait either. In worker mode oxphp_worker_request_duration_us (handler time, queue wait excluded) separates them: handlers below the budget while this counter climbs means the waits are not merely queued behind slow work. Do not use oxphp_request_duration_us for it: it includes these refusals, each contributing nearly a whole budget. Outside worker mode, use the application's own timings.
QUEUE_WAIT_TIMEOUT_MS=0 is not the shortest version of a shorter budget: it drops the wait for a queue slot and leaves the wait inside the queue unbounded, so on a pool picking nothing up a request that finds a slot free is never answered. Pair 0 with a QUEUE_CAPACITY small enough that the queue is full by the time it matters.
What the log says while requests are being shed
The supervisor watches the four overload reasons of oxphp_admission_refused_total on the same one-second scan and reports the episode rather than individual requests, since a line per refusal would itself be the outage. The first scan that sees any of them move logs:
{"timestamp":"2026-09-16T11:02:41.337104Z","level":"WARN","fields":{"message":"PHP admission has started shedding requests under overload","refused":4812,"refused_queue_full":0,"refused_wait_timeout":4812,"refused_waiting_full":0,"refused_waiting_bytes":0,"queue_depth":896,"queue_capacity":896,"admission_slots_available":0,"wait_budget_us":15625,"wait_budget_ceiling_us":1000000}}refused is what that one scan turned away, except on the first scan that has a queue to look at, which reports everything shed before it (a pool still loading its application while arrivals pile up is a real episode). A depth at capacity with no slot free is a pool behind on its work, and a wait_budget_us well under wait_budget_ceiling_us says the server has already shortened the wait; both numbers are there because a budget driven to its floor and one configured that low look the same alone. The queue figures are a snapshot at the scan while refused covers the second just passed, so a large refused next to an empty queue is a short burst, not a contradiction.
Both counts are broken down by reason, into the four overload labels of oxphp_admission_refused_total, with field names built from the count they split: refused_queue_full, refused_wait_timeout, refused_waiting_full and refused_waiting_bytes add up to refused, and the repeat and the closing line below carry episode_refused_queue_full and its three siblings, which add up to episode_refused. The reason is what decides the knob (the counter's row above says what each one points at), and that counter is served only on INTERNAL_ADDR. On an instance without an internal listener, these fields and the oxphp.shed_reason span attribute, where tracing is on, are where the reason can still be read. All four are written even at zero: an explicit 0 is what rules a reason out.
While the episode lasts the warning repeats as PHP admission is still shedding requests under overload, no more than once a minute and only on a scan that is itself refusing (so the gap can run to almost two minutes), carrying episode_refused (the running total) beside the per-scan refused. What the repeat breaks down by reason is the episode total, not the scan's. That matters because the closing line is written at INFO, below the warn the production configurations log at, so there the repeat is the only line that says what the episode has been made of so far, up to the scan it was written on. An episode that ends inside its first minute has no repeat at all, and what is refused after the last repeat reaches only the closing line. A minute with no refusal ends the episode, at INFO:
{"timestamp":"2026-09-16T11:05:52.812440Z","level":"INFO","fields":{"message":"PHP admission has stopped shedding requests","episode_refused":18374,"episode_refused_queue_full":0,"episode_refused_wait_timeout":17950,"episode_refused_waiting_full":424,"episode_refused_waiting_bytes":0,"duration_ms":131000,"queue_depth":0,"admission_slots_available":896}}duration_ms runs to the last refusal, not to the line, so the closing quiet minute is not counted. The quiet minute also means gaps shorter than a minute are folded into one episode, which bounds the output to two lines a minute at worst; entry has no such delay, since a refusal is a fact rather than a sample.
The report is emitted in every routing mode and moves on the four overload reasons alone, so a restart (shutting_down) or a vanished pool (pool_unavailable) does not announce itself as load. There is no gauge for the state, and no probe is told about it: shedding never changes what /health/readiness answers, because an instance shedding 529s is answering quickly on a working pool, and pulling every overloaded replica from rotation in turn would leave the Service with no endpoints. Answer overload by adding capacity. A wedged pool sheds too, and in worker mode that state does answer 503 once it has held for a minute. The queue separates the two: a pool that is merely behind drains it, a wedged one does not.
Worker pool metrics
| Metric | Type | Description |
|---|---|---|
oxphp_workers_current |
gauge | Current number of PHP worker threads |
oxphp_workers_min |
gauge | Minimum worker count (equals current count in static mode) |
oxphp_workers_max |
gauge | Maximum worker count (equals current count in static mode) |
oxphp_workers_idle |
gauge | Worker threads with no request in flight, computed as workers_current - busy_workers |
oxphp_busy_workers |
gauge | Worker threads currently executing at least one request; never exceeds oxphp_workers_current. Counts threads, not requests: in worker mode one thread multiplexes many request fibers and still counts once. Requests waiting for admission or sitting in the queue are not counted; those appear in oxphp_pending_requests |
oxphp_workers_spawned_total |
counter | Total workers spawned since startup (includes initial workers) |
oxphp_workers_retired_total |
counter | Total workers retired due to idle timeout (dynamic mode only) |
Worker supervisor metrics
Per-worker observability emitted by the worker supervisor. Each series carries a worker_id label (slot index). These appear once the supervisor is tracking per-worker state.
| Metric | Type | Description |
|---|---|---|
oxphp_worker_request_age_seconds |
gauge | Age of the in-flight request on each worker, in seconds. Label: worker_id |
oxphp_worker_long_running_total |
counter | Supervisor scans that observed a request older than the stuck threshold (60 seconds, fixed). Label: worker_id |
oxphp_worker_stuck_total |
counter | Stuck-classification counter per worker. Labels: worker_id, kind (io, c_call, cpu). Reporting only: a request classified as stuck is not cancelled and its worker is not recycled. Nothing in the server acts on it, so alert on this series |
Queue wait histogram
| Metric | Type | Description |
|---|---|---|
oxphp_queue_wait_us |
histogram | Time a request waits in the queue before a worker picks it up, in microseconds |
Bucket boundaries (microseconds): 50, 100, 250, 500, 1000, 2500, 5000, 10000, 50000, 100000, 250000, 500000, 1000000, +Inf.
This measures time spent waiting, for admission and then in the queue, up to the moment a worker takes the request off the queue, so it answers "how long before a worker picked this up" rather than "how long did the request take". Nothing after that moment is in it: the worker preparing the request, the script (up to the response head, for a streamed response), and the handoff of the response back to the server's I/O threads all count towards oxphp_request_duration_us. That handoff is not free: the response waits for an I/O thread to be woken or to get through the work queued ahead of it, so on a cheap script the time past this wait can be several times the script's own.
Only a request a worker took off the queue inside its budget is recorded here. A 529 refused for overload, or a 503/500 when the server is shutting down or the pool is gone, is counted in oxphp_admission_refused_total; a 499 for a client that left before a worker reached it is counted in oxphp_request_cancelled_total{reason="client_abort"} and in no refusal series.
A high queue wait means requests took long to reach a worker, which is not by itself a shortage of workers: every reading includes the time a worker thread takes to get to a request handed to it, and that is there on an unloaded server too. Before raising PHP_WORKERS, check that requests are finding every worker busy and that the machine has cores to spare: the server's own I/O threads, other processes and a container CPU quota compete with the workers. oxphp_busy_workers against oxphp_workers_current answers the first only in part: it is read at scrape time, so a burst that fills the pool between scrapes shows up here and not there, and in worker mode a thread counts as busy while it carries any request, even while that request's handler is suspended and the thread takes more. The range reaches one second, matching the default QUEUE_WAIT_TIMEOUT_MS, so a request that spent most of its wait budget before running is quantified rather than lumped into +Inf. Nothing that gets served waits longer than the budget (past it the request is refused instead), so raising QUEUE_WAIT_TIMEOUT_MS is the one setting that puts waits back into +Inf.
Rate limiting metrics
| Metric | Type | Description |
|---|---|---|
oxphp_rate_limited_total |
counter | Requests rejected by the rate limiter (returned 429) |
oxphp_php_deny_total |
counter | Requests blocked by PHP_DENY_PATHS (.php execution denied). See PHP Execution Deny-List |
oxphp_path_deny_total |
counter | Requests denied by a .oxphpdeny rule. A request that PHP_DENY_PATHS denies as well is counted here only. See Path Deny Rules |
Static file cache metrics
| Metric | Type | Description |
|---|---|---|
oxphp_static_cache_hits_total |
counter | Static file requests served from the in-memory cache |
oxphp_static_cache_misses_total |
counter | Static file requests that required a disk read |
Compression metrics
| Metric | Type | Description |
|---|---|---|
oxphp_compressed_responses_total |
counter | Responses sent under a content coding (Brotli, zstd, or gzip) |
oxphp_compression_bytes_saved_total |
counter | Total bytes saved by compression (original size minus compressed size) |
Worker mode metrics
These metrics are only emitted when worker mode is active (WORKER_MODE_ENABLED=true).
Global counters
| Metric | Type | Description |
|---|---|---|
oxphp_worker_mode_enabled |
gauge | Always 1 when worker mode is active |
oxphp_worker_requests_handled_total |
counter | Total requests processed by persistent workers |
oxphp_worker_recycles_total |
counter | Total worker recycles: a worker's loop ended on one of the reasons below. A dynamic pool retiring an idle worker is scaling down rather than recycling: it is counted in oxphp_workers_retired_total, not here or in the series below |
oxphp_worker_recycles_by_reason_total |
counter | Recycles by reason. Label: reason (scheduled, max_memory, error) |
oxphp_worker_soft_resets_total |
counter | Total soft resets performed between requests |
Per-worker gauges
| Metric | Type | Description |
|---|---|---|
oxphp_worker_memory_bytes |
gauge | PHP heap held by each worker at the end of its last request. This is what memory_get_usage() returns inside a handler, and the quantity WORKER_MAX_MEMORY_MIB is measured against. It is read slightly before that ceiling check, while the finished request's superglobals are still on the heap, so it sits a few KB above the figure that trips a recycle: alert below the ceiling rather than at it. A worker that has never served a request reads zero. Label: worker (slot index, e.g., "0", "1"; a recycled worker reuses its predecessor's slot and its last value until it finishes a request of its own) |
oxphp_worker_uptime_seconds |
gauge | Seconds since each worker was spawned. Label: worker |
oxphp_worker_requests_count |
gauge | Requests handled by each worker instance. Label: worker |
oxphp_worker_request_fibers_active |
gauge | Request fibers the worker is carrying right now: 0 idle, 1 serving one request start to finish, higher where requests suspend and are multiplexed on the thread. Refreshed on every turn of the worker's loop rather than on completion, so it does not freeze at its last healthy value on a worker that has stopped completing requests. A worker admits at most 256 request fibers, and one sitting at 256 has stopped taking work off the queue. Read it with oxphp_busy_workers: 256 fibers on a worker the pool counts as idle are fibers that outlived their (already answered) requests and were never reclaimed, while 256 on a busy worker is that many requests in flight. Label: worker |
Worker request duration histogram
| Metric | Type | Description |
|---|---|---|
oxphp_worker_request_duration_us |
histogram | PHP handler execution time per request in microseconds (worker mode only) |
Bucket boundaries (microseconds): 100, 250, 500, 1000, 2500, 5000, 10000, 25000, 50000, +Inf.
This histogram measures time spent inside the PHP handler callback, excluding queue wait time. Use it to identify slow handlers and track tail latency in worker mode.
Async pool metrics
These metrics require ASYNC_WORKERS set to a non-zero value, and each has its own emission gate: the counters appear only after at least one task has been dispatched or rejected, the _in_flight / _in_flight_limit gauges appear once the pool has wired its in-flight counter, and oxphp_async_output_discarded_bytes_total appears only after some output has been discarded.
| Metric | Type | Description |
|---|---|---|
oxphp_async_tasks_dispatched_total |
counter | Total async tasks dispatched to the background pool |
oxphp_async_tasks_completed_total |
counter | Async tasks that completed successfully |
oxphp_async_tasks_failed_total |
counter | Async tasks that threw an exception |
oxphp_async_tasks_cancelled_total |
counter | Async tasks that were cancelled |
oxphp_async_tasks_rejected_total |
counter | Async tasks rejected at dispatch, because the pool queue was full or the in-flight cap (ASYNC_MAX_FIBERS × ASYNC_WORKERS) was reached |
oxphp_async_tasks_stranded_total |
counter | Workers left running past an await_race / await_any timeout. Each stranded task can extend RSHUTDOWN by up to 5 seconds. |
oxphp_async_tasks_in_flight |
gauge | Async tasks currently queued or running (emitted once the pool wires its in-flight counter) |
oxphp_async_tasks_in_flight_limit |
gauge | Maximum concurrent async tasks (ASYNC_MAX_FIBERS × ASYNC_WORKERS) |
oxphp_async_output_discarded_bytes_total |
counter | Bytes of async-task output discarded at worker idle (an echo in an async task has no client to receive it) |
OpenTelemetry export metrics
These metrics require OTEL_ENABLED=true. They count the batches of spans OxPHP sends to the OTLP collector: the request spans, plus the APM spans when the APM plugin is on.
| Metric | Type | Description |
|---|---|---|
oxphp_otel_export_failures_total |
counter | Span exports that failed, after the retry where one was made. Their spans are not sent again |
oxphp_otel_export_failed_spans_total |
counter | Spans carried by those failed exports |
oxphp_otel_export_retries_total |
counter | Exports retried once because the first attempt failed fast, without an answer from the collector |
An export is retried once when its first attempt fails without an answer from the collector within half of OTEL_EXPORTER_OTLP_TIMEOUT or one second, whichever is shorter: the connection was reset or closed, or could not be made. The usual cause is a load balancer or NAT that dropped the connection while it sat idle between exports. An error status the collector answers with, such as a rejected API key or throttling, is not retried: the OTLP specification forbids retrying most of them, and the exporter does not pass on the delay a throttling collector asks for. A retry that delivers the batch is counted only in oxphp_otel_export_retries_total. An attempt that runs into the export timeout is not retried, so a slow collector is never sent the same batch twice.
The retry can duplicate, though. When the collector did receive the first attempt before the connection broke, the retry delivers those spans a second time, and for the same reason an export counted as failed may still have reached the collector.
Spans the tracer drops before export because its queue is full (OTEL_BSP_MAX_QUEUE_SIZE, 2048 spans by default) are not counted here.
Grafana dashboard tips
The following PromQL queries are useful for building dashboards:
Request rate (requests per second):
rate(oxphp_requests_total[5m])Average response time (milliseconds):
rate(oxphp_request_duration_us_sum[5m])
/ rate(oxphp_requests_total[5m]) / 1000p99 request duration (milliseconds):
histogram_quantile(0.99, rate(oxphp_request_duration_us_bucket[5m])) / 1000Error rate (5xx responses as a percentage):
rate(oxphp_responses_by_status_total{status="5xx"}[5m])
/ rate(oxphp_requests_total[5m]) * 100Worker pool utilization:
oxphp_busy_workers / oxphp_workers_currentThis is a true fraction between 0 and 1. Sustained values at 1 mean every worker is occupied and further arrivals are queueing. Pair it with rate(oxphp_admission_refused_total{reason=~"queue_full|wait_timeout|waiting_full|waiting_bytes"}[5m]) to see whether that backlog is turning into refusals, and with oxphp_pending_requests to see how deep it is.
Queue saturation (drop rate per second):
rate(oxphp_dropped_requests_total[5m])p99 queue wait (microseconds):
histogram_quantile(0.99, rate(oxphp_queue_wait_us_bucket[5m]))Static file cache hit rate:
rate(oxphp_static_cache_hits_total[5m])
/ (rate(oxphp_static_cache_hits_total[5m]) + rate(oxphp_static_cache_misses_total[5m]))Bytes saved by compression per second:
rate(oxphp_compression_bytes_saved_total[5m])Worker mode p99 latency (microseconds):
histogram_quantile(0.99, rate(oxphp_worker_request_duration_us_bucket[5m]))Wait budget driven below its ceiling (fraction):
oxphp_admission_wait_budget_us / oxphp_admission_wait_budget_ceiling_us1 is normal; anything lower means the server is shortening waits under overload. Alert on the ratio rather than on a number copied from QUEUE_WAIT_TIMEOUT_MS.
Accept loop stalls (alert when non-zero):
rate(oxphp_accept_stalls_total[5m])Worker recycle rate (per minute):
rate(oxphp_worker_recycles_total[5m]) * 60Average worker memory usage:
avg(oxphp_worker_memory_bytes)Prometheus scrape config
Add a scrape job to your prometheus.yml:
scrape_configs:
- job_name: "oxphp"
scrape_interval: 15s
static_configs:
- targets: ["oxphp:9090"]For Kubernetes service discovery:
scrape_configs:
- job_name: "oxphp"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: oxphp
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
target_label: __address__
replacement: "$1:9090"Prompts for your agent
These prompts assume an agent with the OxPHP documentation MCP server connected. The agent reads the pages it needs itself.
Get Prometheus scraping my OxPHP service. Read the `operations/metrics` page through the OxPHP documentation MCP server.
Add the scrape config and whatever my Compose or Kubernetes files need for the metrics endpoint to be reachable, using the service names already in this repository.Build me a starting Grafana dashboard for OxPHP. Read the `operations/metrics` page through the OxPHP documentation MCP server.
Pick the metrics that actually tell me whether my application is healthy, and explain what each panel would be telling me when it moves.See also
- Health Checks: the
/healthand/configendpoints on the internal server - Configuration Reference: all environment variables including
INTERNAL_ADDR - Graceful Shutdown: how connection draining affects
oxphp_active_connections - Worker Mode: persistent workers and the metrics they emit