Shared* Observability

Every OxPHP\Shared\* instance is a registry entry the runtime already tracks for refcount and capacity. That tracking is exposed to operators as JSON introspection under /__ox_shared/* and as Prometheus metrics under oxphp_shared_*. This page is the reference and the field guide.

Enabling

Observability piggybacks on the internal server. Set INTERNAL_ADDR to start it:

bash

INTERNAL_ADDR=127.0.0.1:9090

Both the JSON endpoints and /metrics are then reachable at that address. No additional configuration is required beyond leaving SHARED_ENABLED on.

You can disable either independently:

Env var	Default	Effect
`SHARED_INTROSPECTION_ENABLED`	`true`	Toggles the `/__ox_shared/*` JSON API.
`SHARED_INTROSPECTION_PREVIEW_ENABLED`	`true`	Toggles `/preview` (value-shape previews can leak data).
`SHARED_METRICS_ENABLED`	`true`	Toggles the `oxphp_shared_*` Prometheus metrics.

Turn introspection off in hostile-tenant deployments; metrics are aggregate-only and safe to keep on.

SHARED_ENABLED=false also removes both surfaces, together with the rest of the subsystem. Use the switches above to keep the primitives available while hiding what they hold.

Introspection endpoints

All responses are Content-Type: application/json; charset=utf-8. Query parameters are standard URL-encoded.

`GET /__ox_shared/summary`

Top-level snapshot: aggregate counts per type, memory, op rate, and saturation against the configured caps.

json

{
  "total_entries": 127,
  "total_bytes": 2_481_664,
  "by_type": {
    "Counter": { "count": 48, "bytes": 3_072,   "ops": 1_402_391 },
    "Map":     { "count": 12, "bytes": 1_638_400, "ops":    48_201 },
    "Pool":    { "count":  4, "bytes":   16_384, "ops":    67_014 }
  },
  "limits":   { "max_entries": 100_000, "max_bytes": 1073741824 },
  "saturation": { "entries": 0.00127, "bytes": 0.00231 },
  "diagnostics": {
    "lock_diagnostics_level": "warn",
    "cycle_detect_depth": 16
  }
}

Use summary in dashboards and cron alerts. A single scrape gives you per-type health and capacity headroom.

`GET /__ox_shared/entries?limit=N`

Lists live entries (capped at limit, default 100, max 500). One line per entry:

json

{
  "items": [
    { "id": 42, "type": "Map",    "refcount": 2, "ops":  1820, "mem_bytes": 204_800, "age_sec": 612 },
    { "id": 43, "type": "Counter", "refcount": 3, "ops": 48_014, "mem_bytes":     64, "age_sec": 612 }
  ],
  "next_cursor": null,
  "total_matching": 127
}

refcount is the external retain count: how many PHP wrappers and nested Shared entries hold this one. When you expect an entry to be GC-able but it is not, this is the field to check.

`GET /__ox_shared/entry?id=N`

Type-specific detail for one entry:

json

{
  "id": 42,
  "type": "Map",
  "refcount": 2,
  "ops": 1820,
  "mem_bytes": 204_800,
  "age_sec": 612,
  "type_specific": {
    "key_count": 1_240,
    "max_entries": 50_000,
    "saturation": 0.0248,
    "sample_keys": ["tenant:acme", "tenant:beta", "..."]
  }
}

type_specific varies by type: Pool exposes { max_size, count, in_use, idle, waiting, idle_by_thread, rebalance_strategy }, Channel exposes { capacity, count, closed, senders_blocked, receivers_blocked }, Counter exposes { value }, and so on.

`GET /__ox_shared/preview?id=N`

Value-shape preview of scalar and small-array values. String values are truncated to SHARED_PREVIEW_STRING_LIMIT (default 256 bytes), rounded down to the nearest character boundary so multibyte text is never cut mid-character; arrays show the first SHARED_PREVIEW_ARRAY_LIMIT entries (default 20). Gated by SHARED_INTROSPECTION_PREVIEW_ENABLED.

json

{ "id": 42, "type": "Counter", "preview": "1420" }

Use preview during development; disable it in production when values may contain user data.

`GET /__ox_shared/types`

Enumerates the v1 type catalog, useful for generated tooling that needs the tag → class mapping:

json

{
  "types": [
    { "tag": 10, "name": "Counter", "php_class": "OxPHP\\Shared\\Counter" },
    { "tag": 11, "name": "Flag",    "php_class": "OxPHP\\Shared\\Flag" },
    { "tag": 12, "name": "Once",    "php_class": "OxPHP\\Shared\\Once" },
    { "tag": 13, "name": "Atomic",  "php_class": "OxPHP\\Shared\\Atomic" },
    { "tag": 20, "name": "Map",     "php_class": "OxPHP\\Shared\\Map" },
    { "tag": 31, "name": "Channel", "php_class": "OxPHP\\Shared\\Channel" },
    { "tag": 50, "name": "Pool",    "php_class": "OxPHP\\Shared\\Pool" }
  ]
}

The response does not list Shared\Mutex.

`GET /__ox_shared/graph?id=N[&depth=D][&edges=E]`

BFS walk of outgoing Shareable references starting at id=N. Returns nodes and edges of the reachable subgraph. Defaults: depth=16, edges=500. Walker-budget hits set truncated: true in the response.

json

{
  "root": 42,
  "nodes": [
    { "id": 42, "type": "Map",     "refcount": 2, "mem_bytes": 204_800 },
    { "id": 51, "type": "Counter", "refcount": 1, "mem_bytes":     64 }
  ],
  "edges": [
    { "from": 42, "to": 51, "key": "hits" }
  ],
  "truncated": false
}

Reach for graph after a CycleException to see the reachable path the walker took, or when diagnosing "why is this Counter not getting GC'd": the graph shows every parent holding a retain on it.

Prometheus metrics

All metrics are exposed at GET /metrics alongside the core server metrics.

Registry-wide

Metric	Type	Labels	Description
`oxphp_shared_objects_total`	gauge	`type`	Live entry count per type.
`oxphp_shared_operations_total`	counter	`type`	Cumulative ops dispatched to each type.
`oxphp_shared_bytes`	gauge	`type`	Approximate bytes per type (±30% vs `mallinfo`).
`oxphp_shared_total_bytes`	gauge	—	Sum across types.
`oxphp_shared_capacity_saturation`	gauge	`kind`	`entries` and `bytes` as fractions of their caps.
`oxphp_shared_deadlock_detected_total`	counter	—	Cross-thread wait-for cycles detected.

Channel

Metric	Type	Labels
`oxphp_shared_channel_count`	gauge	`channel_id`
`oxphp_shared_channel_senders_blocked`	gauge	`channel_id`
`oxphp_shared_channel_receivers_blocked`	gauge	`channel_id`
`oxphp_shared_channel_items_sent_total`	counter	`channel_id`
`oxphp_shared_channel_items_dropped_total`	counter	`channel_id`

Map

Metric	Type	Labels
`oxphp_shared_map_entries`	gauge	`map_id`
`oxphp_shared_map_max_entries`	gauge	`map_id`
`oxphp_shared_map_saturation`	gauge	`map_id`

Pool

Metric	Type	Labels
`oxphp_shared_pool_count`	gauge	`pool_id`
`oxphp_shared_pool_in_use`	gauge	`pool_id`
`oxphp_shared_pool_idle`	gauge	`pool_id`
`oxphp_shared_pool_waiting`	gauge	`pool_id`
`oxphp_shared_pool_acquire_total`	counter	`pool_id`
`oxphp_shared_pool_evicted_total`	counter	`pool_id`, `reason`
`oxphp_shared_pool_wait_seconds`	histogram	`pool_id`

oxphp_shared_pool_evicted_total labels: reason=idle_timeout | evict | shutdown. idle_timeout is an automatic eviction of an idle slot, evict is an explicit Pool::evict() call, and shutdown is teardown at process exit.

Counter / Flag / Once / Mutex

Per-instance counters, flags, onces, and mutexes do not ship individual metric series: they would bloat label cardinality. Use the registry-wide oxphp_shared_operations_total{type=...} counter and the /__ox_shared/entry?id=… JSON for per-instance inspection instead.

Mutex metrics are a v1.x candidate

Tracked as follow-up work; today's visibility is via /__ox_shared/entry.

Deadlock detector

A background task looks for cross-thread wait-for cycles on Shared\Mutex: two or more threads each waiting on a mutex that another of them holds. Every SHARED_LOCK_POLL_INTERVAL_MS (default 100) it scans the wait-for graph. SHARED_LOCK_DIAGNOSTICS sets what it does when a scan finds a cycle:

Value	Detector	On a cycle
`off`	Not started. `SHARED_LOCK_POLL_INTERVAL_MS` is unused and `oxphp_shared_deadlock_detected_total` stays at 0.	Nothing. A `withLockTimeout()` on the cycle gives up with `OperationTimeoutException` when its timeout runs out. A cycle made only of `withLock()` calls has no timeout of its own to end it: each of its requests waits until it is ended, by `max_execution_time` if it has one, and the locks an ended request held are released as it ends (see Shared\Mutex).
`warn` (default in release builds)	Runs.	Logs and counts (see below); the waits are left in place, with the same outcome as under `off`.
`strict` (default in debug builds of the binary)	Runs.	Logs and counts, and also signals the threads on the cycle: a `withLock()` on it throws `DeadlockException` within 100 ms of the scan, and a `withLockTimeout()` waits out its timeout and then throws `DeadlockException` in place of `OperationTimeoutException`. A waiter whose lock is released first (by another waiter's exception unwinding the closure that held it) acquires it and does not throw.

Lowercase only

The value is matched lowercase only. Any other value, Strict or trace included, silently gives the default.

Each cycle a scan finds produces at least one WARN record with event=shared.deadlock_detected, each naming a thread on the cycle (start_thread, repeated as holder_thread); the record does not name the mutexes. oxphp_shared_deadlock_detected_total rises with each record.

Same-thread re-entry is a separate check and does not depend on this setting: re-entering a mutex on the thread that already holds it throws DeadlockException at the call under every value, off included, and never becomes a cycle.

Diagnostic playbooks

Pool is saturated (429s with retries failing)

Symptoms: HTTP callers see timeouts, oxphp_shared_pool_waiting climbs, oxphp_shared_pool_count is pinned at maxSize.

Check:

bash

curl -s http://localhost:9090/__ox_shared/entry?id=<pool_id> | jq .type_specific

Look at idle_by_thread. If it is {} or heavily imbalanced (worker 0 has 8 idle, worker 3 has 0), the acquire is contending for threads that happen to be busy elsewhere. Per-thread affinity in v1 does not rebalance. Either raise maxSize or reduce the per-thread acquire hotspot.

If idle_by_thread is balanced but everything is in in_use, raise maxSize.

Memory saturation

Check oxphp_shared_total_bytes and oxphp_shared_capacity_saturation{kind="bytes"}. The server does not shed or degrade anything as usage climbs. Its first response is to refuse to create a new Shared object once SHARED_MAX_BYTES or SHARED_MAX_ENTRIES is reached; objects that already exist keep growing without that cap being checked again (a Map without limit, unless it was constructed with maxEntries), so the bytes gauge can read above 1.0. For early warning, add an alert rule, for example oxphp_shared_capacity_saturation > 0.7. If either is high:

curl /__ox_shared/entries?limit=500 and sort by mem_bytes to find the top contributors.
curl /__ox_shared/entry?id=<N> on each to check the shape. For Map, look at key_count vs max_entries.
Most common cause: an unbounded Shared\Map keyed by user input. Remedy is a maxEntries cap and a retention policy.

A wrapper won't get garbage-collected

refcount in /__ox_shared/entries tells you how many outstanding retains there are. If it stays above 1 after the PHP wrapper leaves scope, another Shared entry is keeping it alive.

bash

curl -s http://localhost:9090/__ox_shared/graph?id=<N> | jq .nodes

Walk the graph backwards. Any node reaching the stuck entry is holding a retain. Remove the reference ($map->remove($key), close the channel, drop the Mutex entry) and the refcount drops.

A `CycleException` fired in production

The exception's message includes the reachable path the cycle detector explored. Map those IDs back to types via /__ox_shared/entries, and ask /__ox_shared/graph?id=<root> for the full shape:

bash

# Exception message: "cycle would form: #42 → #51 → #42"
curl -s http://localhost:9090/__ox_shared/graph?id=42 | jq

The result visualises the chain so you can see where the inadvertent back-reference was introduced.

Deadlock detector fired

oxphp_shared_deadlock_detected_total is ticking: the deadlock detector saw two or more threads each waiting on a Shared\Mutex that another of them holds. Check server logs for event=shared.deadlock_detected; the records name threads, not mutexes. Recover:

Find the code paths that take two mutexes one inside the other, and make them take them in the same order everywhere, or merge the pair into one mutex.
Under SHARED_LOCK_DIAGNOSTICS=warn, the default in release builds, the cycle is only logged and counted. Set SHARED_LOCK_DIAGNOSTICS=strict in staging to have the detector break it instead; see Deadlock detector for what each waiting call then does.

Long-running soak harness

tests/soak/pool_soak.sh is a manual (non-CI) harness for verifying Shared\Pool stability over hours or days of continuous load. It:

Boots the dev image with dynamic worker scaling (PHP_WORKERS=4:40 by default) and a short pool idleTimeout so the eviction scheduler fires continuously.
Loads tests/soak/workload.php as the worker bootstrap, which constructs 10 pools × maxSize=8 and serves acquire/release on every request.
Drives traffic with wrk for SOAK_DURATION_MIN minutes (default 1440 = 24h).
Scrapes /metrics and the container's RSS every 60 s into tests/soak/out/<timestamp>/metrics.csv.
Writes verify.txt at the end with pass/fail for the five release-exit criteria (RSS drift within ±5%, zero stale-handle panics, zero leaked entries at shutdown, idle-timeout evictions rising smoothly, zero deadlock-detector firings).

Prerequisites on the host: docker, wrk, curl, awk.

Typical invocations:

bash

# 24h full soak before a release
tests/soak/pool_soak.sh

# 1h smoke for validating the harness itself
SOAK_DURATION_MIN=60 tests/soak/pool_soak.sh

# Heavier concurrency
SOAK_CONCURRENCY=400 SOAK_THREADS=8 tests/soak/pool_soak.sh

Artifacts land in tests/soak/out/<timestamp>/:

metrics.csv: one row per minute (unix ts, RSS, per-type entry counts, per-pool eviction counters, deadlock count, ops).
server.log: container stdout/stderr including any stale-handle or panic trails.
wrk.out / wrk.err: raw load-generator output.
metrics.final: the last /metrics scrape taken just before container teardown. Used to confirm entry and byte counts have returned to baseline (no leaked Shared entries) after the load stops.
verify.txt: pass/fail report for the five exit criteria.

Warning

Do not wire this into CI. A 24h run is not cheap and its purpose is pre-release confidence, not continuous validation.

Scrape cadence

The registry endpoints walk live state under read locks, so scraping is cheap but not free. Recommended cadences:

/metrics: every 15 s (typical Prometheus default). It is aggregate-only, so the overhead is negligible.
/__ox_shared/summary: every 60 s for dashboards. It is slightly heavier than /metrics.
/__ox_shared/entries: on demand only. It iterates all shards, so do not scrape it every tick.
/__ox_shared/entry / /preview / /graph: on demand during investigations.

Prompts for your agent

These prompts assume an agent with the OxPHP documentation MCP server connected. The agent reads the pages it needs itself.

prompt

Build a Prometheus dashboard and alert rules for the Shared\* primitives my code actually uses. Read the `shared-state/shared-observability` page through the OxPHP documentation MCP server.
Find every `Shared\Map`, `Shared\Pool`, `Shared\Channel`, and other primitive this repository constructs, and produce panels plus alert thresholds from their documented metrics — saturation, waiting, and drops.

Shared State: mental model and primitives overview.
Prometheus Metrics: core server metrics under the same /metrics endpoint.
Internal Server: how the /__ox_shared/* endpoints plug into INTERNAL_ADDR.
Migrating to an external store: for when saturation is structural rather than tunable.

Shared* Observability

Enabling

Introspection endpoints

GET /__ox_shared/summary

GET /__ox_shared/entries?limit=N

GET /__ox_shared/entry?id=N

GET /__ox_shared/preview?id=N

GET /__ox_shared/types

GET /__ox_shared/graph?id=N[&depth=D][&edges=E]

Prometheus metrics

Registry-wide

Channel

Map

Pool

Counter / Flag / Once / Mutex

Deadlock detector

Diagnostic playbooks

Pool is saturated (429s with retries failing)

Memory saturation

A wrapper won't get garbage-collected

A CycleException fired in production

Deadlock detector fired

Long-running soak harness

Scrape cadence

Prompts for your agent

Related

`GET /__ox_shared/summary`

`GET /__ox_shared/entries?limit=N`

`GET /__ox_shared/entry?id=N`

`GET /__ox_shared/preview?id=N`

`GET /__ox_shared/types`

`GET /__ox_shared/graph?id=N[&depth=D][&edges=E]`

A `CycleException` fired in production