Calibration checks whether a brand signature can separate known winners from known losers when each reference is held out of the bank. It is one of the most important quality signals in Cortex.
Build minimum vs. calibration minimum
A signature can be built with 3 eligible winners and 3 eligible losers.
Full leave-one-out calibration needs at least 4 eligible winners and 4 eligible losers. That is because each calibration fold removes one reference and still needs enough remaining winners and losers to build a valid temporary signature.
If a brand has 3+3 but not 4+4, scoring can still run. The calibration page will show insufficient data and the signature uses the default blend.
Selected blend
Cortex scores brand winner-likeness from two views:
- Video score: how the whole candidate compares with the winner-vs-loser signature.
- Element score: how comparable moments in the candidate compare with equivalent moments in winners and losers.
The default blend favors element-level signal: 0.7 element score and 0.3 video score. When calibration has enough data, Cortex evaluates candidate blends and selects the one that best separates held-out winners from held-out losers.
The calibration page shows the selected video weight and element weight.
Metrics
Calibration reports several metrics:
- Balanced accuracy at 50: how often known winners land at or above the 50 decision boundary and known losers land below it, balanced across both classes.
- AUC: how often a held-out winner ranks above a held-out loser. AUC closer to 1.0 is better.
- Best threshold: the threshold that would maximize balanced accuracy for the held-out set.
- Winner mean: average held-out score for references labeled winners.
- Loser mean: average held-out score for references labeled losers.
- Winner-minus-loser gap: separation between the two means. Larger positive gaps are better.
Use these together. A high AUC with weak separation may still be directional. A strong gap with a tiny bank should still be treated carefully.
Calibration statuses
- OK: calibration ran and selected a blend.
- Insufficient data: there were not 4 eligible winners and 4 eligible losers.
- Skipped: required timeline or duration data was incomplete.
Skipped or insufficient calibration does not mean the signature is unusable. It means the result should be interpreted with more caution and confidence may be lower.
How to improve calibration
- Add more clean winners and losers.
- Keep the bank balanced.
- Exclude mislabeled or out-of-category references.
- Lower weights for ambiguous examples.
- Rebuild the signature after reference changes.
- Check diagnostics for references that scored against their assigned label.
Calibration is a user-facing quality check, not an internal detail. Use it before relying on a brand bank for repeated production scoring.