> ## Documentation Index
> Fetch the complete documentation index at: https://docs.myfundingmachine.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 0012 outcome as calibration ground truth

# Lender Outcomes, not reviewer verdicts, are the calibration ground truth

Status: accepted

Date: 2026-09-01

## Decision

1. The Underwriting orchestrator is calibrated against **Outcomes** — what lenders
   actually did on Settled files, read from Funding plan rows — not against human
   Review verdicts.
2. The unit of measurement is the **Outcome record** (one per Settled processing
   engagement) and the headline is the three-layer **Accuracy scorecard**:
   Decision accuracy, Amount accuracy, Lender calibration. Every calibration
   proposal must cite its effect on the scorecard over the Outcome corpus.
3. The Underwriting Review Console's verdict-first flow (pick recent pulls →
   record a verdict → tweak knobs against the verdicts) is retired. The
   `underwritingReviewLabels` and `underwritingReviewFlags` tables are kept and
   verdicts remain available as optional annotations on an Outcome record.
4. Two Predictions are scored per record: the historical Prediction the Operator
   saw (pinned to its Ruleset version) and a current-ruleset re-run of the stored
   report with default thresholds. The raw engine decision is scored; Operator
   product overrides and Location custom thresholds are features, never the
   thing being scored.
5. Scope is card stacking and term loan. Revenue Based products join when their
   estimates exist (ADR-0008).

## Context

The 2026-05 review console assumed reviewers could judge a Decision's accuracy
by inspection. In practice the console went unused: a verdict is one person's
opinion, it does not scale past a handful of files a week, and it cannot tell
the team whether the estimated range was right by $2,000 or $20,000. Meanwhile
the ground truth already existed — every processed file has Funding plan rows
with lender approvals, declines, and approved amounts — but was only consulted
as an ungated per-contact sum (`sumFundedPlans`).

The orchestrator emits no per-lender prediction. The `approvalProbability` and
related fields on old `fundingPlans` rows are artefacts of a retired AI plan
generator (`LEGACY_GENERATED_PLAN_FIELDS`), so a "lender hit rate" against an
engine probability is not measurable. The third scorecard layer therefore
calibrates the card catalog's stated `minimumScoreRequired` and
`averageLimitProvided` against actual approvals instead.

Term-loan Outcomes come from TL Funding plan rows only. Engine by MoneyLion
marketplace data (`offers`, `derivedStatus`, per-lender events) shows
pre-qualification and application events but never an accepted offer, so it is
at most a feature.

## Considered options

* **Keep verdicts as ground truth and add the funded comparison as context**
  — rejected. Keeps the unscalable human step in the loop and leaves the dollar
  error unmeasured.
* **Score the effective decision (override ?? engine)** — rejected. Overrides
  hide engine error; we want to see where Operators had to correct the engine.
* **Score only the historical Prediction** — rejected. Ruleset bumps would make
  files incomparable; the current-ruleset re-run lets one build answer "would
  today's engine have been right on every file we have ever processed".
* **Outcome as ground truth with verdicts demoted to annotations (accepted).**

## Consequences

* The review console page is rebuilt around the Outcome corpus (Scorecard →
  Segment diagnostics → Knob lab, with the process replay kept as the
  single-record drill-down). The pure libraries beneath it — knob registry,
  what-if override seam (ADR-0001), persona guardrail, input assembly,
  calibration proposal → generated prompt — are reused unchanged.
* The Knob lab never edits engine constants or thresholds. A chosen change still
  ships as a code edit with a `RULESET_CODE_VERSION` bump and golden-snapshot
  review, exactly as ADR-0001 requires.
* Records whose same-ruleset re-run does not reproduce the historical headline
  carry a reconciliation flag and are excluded from re-run-based layers. Closing
  that gap forward requires persisting `incomeType`, `monthlyBankDeposits`, and
  `businessDeposits` on `underwritingResults` (additive optional fields).
* Glossary: CONTEXT.md → "Outcome analysis" section, and the revised
  "Underwriting review" / "Review verdict" entries. Companion decision on which
  files count: ADR-0013.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.