Scoring
Overview#
Every resolved signal receives a quality score that measures prediction quality beyond simple directional accuracy. The scoring system uses a close-based evaluation model that checks the closing price at expiry rather than whether the target was touched at any point during the window.
The close-based model evaluates whether the price actually moved in the predicted direction by expiry, rather than whether the target was touched at any point. This provides meaningful differentiation between makers, since volatile assets can briefly touch almost any reasonable target.
Evaluation Model#
Three prices are used for evaluation:
- Entry Price: Market price of the asset at the time the signal was published
- Target Price: The price the maker predicted the asset would reach
- Resolution Price: The closing market price at the end of the time window
The signal is CORRECT if the resolution price moved in the predicted direction relative to the entry price:
LONG: resolution_price > entry_price → CORRECT
SHORT: resolution_price < entry_price → CORRECT
Otherwise → WRONGQuality Score#
Quality scores range from 0.0 to 4.0 and are computed as:
QualityScore = DirectionScore × PrecisionMultiplier × DifficultyWeightDirection Score (0 or 1)#
IF direction == LONG AND resolution_price > entry_price → 1
IF direction == SHORT AND resolution_price < entry_price → 1
ELSE → 0If the direction score is 0, the entire quality score is 0 regardless of precision or difficulty.
Precision Multiplier (0.5–2.0)#
Measures how close the actual price move was to the predicted move:
actual_move = |resolution_price - entry_price|
predicted_move = |target_price - entry_price|
ratio = min(actual_move, predicted_move) / max(actual_move, predicted_move)
precision = 0.5 + 1.5 × ratio- 2.0: Perfect prediction — actual move exactly matches predicted move
- 1.25: Actual move was ~50% of predicted (or vice versa)
- 0.5: Very imprecise — actual and predicted moves were vastly different
Difficulty Weight (0.5–2.0)#
Measures how ambitious the prediction was relative to the asset's volatility:
ATR% = Average True Range / current price (over signal's window)
raw_difficulty = (|target_price - entry_price| / entry_price) / ATR%
difficulty_weight = clamp(raw_difficulty, 0.5, 2.0)- 2.0: Predicted move ≥ 2x the average volatility (very ambitious)
- 1.0: Predicted move ≈ 1x ATR (typical move)
- 0.5: Predicted move < 0.5x ATR (easy prediction)
Worked Examples#
Precise, Difficult Prediction#
Entry: $100,000
Target: $103,000 (LONG)
Resolution: $102,800
ATR%: 1.2%
DirectionScore = 1 (resolution > entry, correct direction)
actual_move = |102,800 - 100,000| = 2,800
predicted_move = |103,000 - 100,000| = 3,000
ratio = 2,800 / 3,000 = 0.933
precision = 0.5 + 1.5 × 0.933 = 1.90
predicted_pct = 3,000 / 100,000 = 3.0%
raw_difficulty = 3.0% / 1.2% = 2.5 → clamped to 2.0
QualityScore = 1 × 1.90 × 2.0 = 3.80Correct but Imprecise#
Entry: $100,000
Target: $105,000 (LONG)
Resolution: $100,500
ATR%: 1.2%
DirectionScore = 1
actual_move = 500
predicted_move = 5,000
ratio = 500 / 5,000 = 0.10
precision = 0.5 + 1.5 × 0.10 = 0.65
predicted_pct = 5.0%
raw_difficulty = 5.0% / 1.2% = 4.17 → clamped to 2.0
QualityScore = 1 × 0.65 × 2.0 = 1.30Wrong Direction#
Entry: $100,000
Target: $103,000 (LONG)
Resolution: $99,500
DirectionScore = 0
QualityScore = 0 × anything × anything = 0.00Brier Score (Maker Calibration)#
The Brier score measures how well a maker's stated confidence matches their actual outcomes:
BrierScore = mean( (confidence - direction_score)² )
Where:
confidence = maker's stated confidence (0.0–1.0)
direction_score = 1 if correct, 0 if wrong
Interpretation:
0.00 = perfectly calibrated
0.25 = random (like always saying 50%)
0.50+ = poorly calibrated (overconfident or consistently wrong)
1.00 = perfectly anti-calibratedA maker who says 80% confidence and is correct 80% of the time will have a low Brier score. A maker who says 90% confidence but is only correct 50% of the time will have a high Brier score. Use this to identify overconfident makers.
Entry Price Requirement#
Every signal must have an entry_price recorded at creation time. The entry price is the market price of the asset when the signal was published and is essential for the close-based evaluation model. Signals missing an entry price cannot be scored.