Rescue Metrics

Four output metrics, one for each problem number. Eval set v6.

Draft · mock numbers until the real eval runs

Time-to-contextp90 9 s

0 s

Extraction F1held-out

0.00

Safe autonomy1.8% error

Boundary breachestarget 0

Extraction Accuracy

Per-field F1, 260 hand-written samples, 30% held out

Measured

Baseline v0After Codex loop (v6)

Window

0.71

0.93

Drop-off

0.58

0.89

Alt. receiver

0.64

0.91

Address fix

0.42

0.82

Optimisation Curve

One prompt or schema change per round; stop when held-out stalls

Measured

Dev setHeld-out

Time-to-Context

From the buyer’s answer to the card appearing

Measured

Chat

4 s

p90 9 s

AI call

6 s

after hang-up

Voice memo

11 s

incl. ASR

Source	Samples	F1
Chat	120	0.94
Call transcript	80	0.88
Voice memo	60	0.86

Safe Autonomy

Every action in the eval, and who decided it

Measured

Cases handled by the agent alone64%
Error rate inside that group1.8%
Routed to review on low confidence22%
High-stakes actions without approval0
Calls outside 08:00–20:300
Contact after the buyer said “stop”0

Money

One area, one month. Projected from the team cost model until the pilot measures the real rescue rate.

Projected

Orders per day (one area, 20 shippers)4,000

First-attempt failure rate8%

Extra rescue over today’s 25% (assumed)40 pts

Cases handled by the agent65%

Running cost a month₫27.3M

Net saving a month, one area

₫187.4M

40% of the ₫473M monthly loss

Return fees avoided: +₫153.6M
Redelivery attempts avoided: +₫20.2M
Shipper door time saved: +₫11.8M
CS calls by the agent (832 h): +₫29.1M
Running cost: −₫27.3M

₫40,000 return fee per Shopee policy; ₫7,000 shipper pay per attempt (GHN); 2 calls × 4 min at ₫35,000 per CS hour and the running cost are team estimates. Defaults are the best-in-class case (25% → 65%, Shipsy vendor range). Projected, not measured.