🜛architect.engineersigned in as Maya Okonkwo · Solstice Systems
calibration
your epistemic model has the same internals as a small classifier. confidence is a sigmoid output, brier is MSE, the signed gap is the gradient direction, and saliency is an attention weight on which predictions count more.
total
20
predictions submitted
resolved
20
0 pending
hit rate
57.5%
yes=1 · partial=0.5 · no=0
brier (mse)
0.1496
(conf − outcome)² · lower is better
l1
0.3225
|conf − outcome| · lower is better
signed bias
+0.1225
positive · overconfident
saliency-weighted brier
0.1496
attention-weighted MSE
curve
stated confidence bucket on x · observed hit rate on y · diagonal = perfect calibration · dot radius = bucket sample size
fleet ledger — human + agent, unified
20 scoredpredictions.confidence (50..100 slider) and agent_predictions.confidence (0..1) are normalized to a fraction at READ time, never rewritten — two unreconciled units of measure, one scoring layer. `invalid` agent outcomes are excluded from the denominator, not scored as a miss.
unified hit rate
57.5%
human + agent, weighted equally per row
human · /calibrate
57.5%
20 scored
agent · predict.sh
—
0 scored
| confidence bucket | n (all) | observed (all) | n (human) | observed (human) | n (agent) | observed (agent) |
|---|---|---|---|---|---|---|
| 50% | 3 | 0.0% | 3 | 0.0% | — | — |
| 60% | 8 | 31.3% | 8 | 31.3% | — | — |
| 70% | 3 | 100.0% | 3 | 100.0% | — | — |
| 80% | 3 | 100.0% | 3 | 100.0% | — | — |
| 90% | 3 | 100.0% | 3 | 100.0% | — | — |
per-prediction error
20 resolved · newest first| prediction | conf | outcome | resolved | brier (mse) | |Δ| (l1) | signed Δ | saliency | matter / domain |
|---|---|---|---|---|---|---|---|---|
| No design partner will report a NEW reliability complaint this week after the batch fix ships | 95% | yes | 14d ago | 0.0025 | 0.0500 | -0.050 | 0.70 | active-engagements · customers |
| The new BigQuery-benchmark itch will get picked up before next week's kickoff | 60% | partial | 14d ago | 0.0100 | 0.1000 | +0.100 | 0.70 | active-engagements · growth |
| The batch-fix ask will reach a terminal state (solved or templated) by week's end | 90% | yes | 14d ago | 0.0100 | 0.1000 | -0.100 | 0.70 | active-engagements · ops |
| The batch-fix soak will surface at least one edge case needing a follow-up patch | 65% | no | 14d ago | 0.4225 | 0.6500 | +0.650 | 0.70 | active-engagements · gates |
| Priya at Bramble Analytics will be the first to try the scheduled-email digest once it ships | 60% | partial | 15d ago | 0.0100 | 0.1000 | +0.100 | 0.70 | active-engagements · growth |
| Maya will approve a dispatch for the export ask the same day it publishes | 55% | no | 14d ago | 0.3025 | 0.5500 | +0.550 | 0.70 | active-engagements · ops |
| The export-reliability ask will publish without any further scope changes | 70% | yes | 15d ago | 0.0900 | 0.3000 | -0.300 | 0.70 | active-engagements · product |
| The nightly batch fix will eliminate drop-over-10k errors within 48h | 80% | yes | 14d ago | 0.0400 | 0.2000 | -0.200 | 0.70 | active-engagements · gates |
| Dispatch cost for the batch-fix job will land under $2 | 75% | yes | 16d ago | 0.0625 | 0.2500 | -0.250 | 0.70 | active-engagements · gates |
| The sandbox run will need more than one attempt to pass gates | 60% | no | 16d ago | 0.3600 | 0.6000 | +0.600 | 0.70 | active-engagements · gates |
| The export-reliability ask will publish before the batch-fix ask is dispatched | 65% | no | 16d ago | 0.4225 | 0.6500 | +0.650 | 0.70 | active-engagements · ops |
| This week's two collider sparks will connect itches across two different matters | 55% | no | 17d ago | 0.3025 | 0.5500 | +0.550 | 0.70 | active-engagements · product |
| Loom's spec review will surface at least one open question on the batch-fix plan | 60% | no | 17d ago | 0.3600 | 0.6000 | +0.600 | 0.70 | active-engagements · ops |
| The batch-drop root cause will trace to an unbounded array slice, not a query timeout | 90% | yes | 17d ago | 0.0100 | 0.1000 | -0.100 | 0.70 | active-engagements · infra |
| Ask A (batch fix) will reach designing before ask B (export) does | 80% | yes | 18d ago | 0.0400 | 0.2000 | -0.200 | 0.70 | active-engagements · ops |
| Minting the coherence cluster will pull in a third itch besides the CSV/email pair | 55% | no | 18d ago | 0.3025 | 0.5500 | +0.550 | 0.70 | active-engagements · product |
| The batch-drop itch will promote straight to a draft ask without a coherence cluster | 65% | yes | 18d ago | 0.1225 | 0.3500 | -0.350 | 0.70 | active-engagements · ops |
| Both new-signal partners (webhook-retry, scheduled-email) are on the growth tier | 85% | yes | 19d ago | 0.0225 | 0.1500 | -0.150 | 0.70 | active-engagements · customers |
| At least one more customer will report the same batch-drop symptom this week | 60% | partial | 17d ago | 0.0100 | 0.1000 | +0.100 | 0.70 | active-engagements · support |
| The CSV-export itch will convert into a scoped ask within 3 days | 70% | yes | 18d ago | 0.0900 | 0.3000 | -0.300 | 0.70 | active-engagements · product |
by domain
calibration_gap = mean stated confidence − observed hit rate. positive = systematically overconfident in this domain.
| domain | n | hit rate | calibration_gap |
|---|---|---|---|
| ops | 6 | 50.0% | +19.2 |
| gates | 4 | 50.0% | +20.0 |
| product | 4 | 50.0% | +12.5 |