Leave-one-out validation is Cortex's way to ask: "If we pretended one known reference was new, would the rest of the bank score it correctly?"
How it works
For each eligible reference:
- Hold that reference out.
- Build a temporary signature from the remaining references.
- Score the held-out reference.
- Check whether a known winner scored winner-like and a known loser scored loser-like.
This is repeated across the reference bank. The output becomes calibration and diagnostics.
Why 4+4 is required
A production signature can be built with 3 eligible winners and 3 eligible losers.
Leave-one-out validation needs at least 4 eligible winners and 4 eligible losers because each fold removes one reference. After holding one out, the temporary build still needs enough references to remain valid.
Metrics
Balanced accuracy at 50
This measures performance at the product decision boundary. Winners should score at or above 50. Losers should score below 50. Balanced accuracy prevents one side from hiding weakness on the other side.
AUC
AUC measures ranking. It asks how often held-out winners rank above held-out losers. AUC near 1.0 is strong. AUC near 0.5 is close to random.
Winner mean and loser mean
Winner mean is the average held-out score for references labeled winners. Loser mean is the average held-out score for references labeled losers.
Winner-minus-loser gap
This is the separation between the two means. A larger positive gap means the bank is separating winners from losers more clearly.
False-positive losers and false-negative winners
A false-positive loser is a loser that scored winner-like. A false-negative winner is a winner that scored loser-like. These references deserve review because they may be mislabeled, noisy, or creatively ambiguous.
Review queue flags
Diagnostics can flag:
- Winners scoring below the 50 threshold.
- Losers scoring above the 50 threshold.
- Low event coverage.
- Large divergence between whole-video and element scores.
These flags do not automatically change your bank. They tell the team what to inspect before rebuilding.
How to use it
If leave-one-out results are strong, you can trust the bank more. If results are weak, improve the reference library before relying on the signature for high-volume scoring.
The usual fixes are simple: add more clean references, balance the bank, exclude bad examples, adjust weights, and rebuild.