Research & Decision · AI Evaluation · Independent project
Is the evidence strong enough to support a model-selection decision?
I translated “Is model A better than model B, and does domain adaptation produce a meaningful improvement?” into reviewable tasks, discrimination measures and confidence intervals. When the evidence is insufficient, it does not force a ranking.
Decision Context
The purpose of evaluation is not to rank models. It is to decide whether the evidence supports further investment.
Public benchmark scores rarely answer a specific research or product decision. This project uses finance, companionship and community contexts to break “Can we use it?” into task rules, failure modes and reviewable acceptance evidence.
Question / Evidence
Measure first. Then decide which conclusions are justified.
Controlled tasks
Three public domain sets use the same run and scoring protocol. Mechanical correctness, semantic instruction-following and boundary failures are assessed separately.
Discrimination
The companionship set showed a discrimination score of only 0.055, too low to support a stable ranking of model quality or safety.
Confidence intervals
Every bootstrap 95% confidence interval for paired improvement crossed zero, so point estimates cannot be presented as reliable gains.
Review materials
Task sets, run outputs, judge results, failure records and reproduction scripts are retained in the public repository.
Alternatives
Four interpretations; only one avoids mistaking noise for a conclusion.
Rank by point estimate
Treat the score difference as proof that one model is better. Risk: sample size and uncertainty disappear.
Publish a safety ranking
Use the low-discrimination companionship set as a stable leaderboard. Risk: the test cannot separate the models being compared.
Claim a domain advantage
Treat the Vertical Lift point estimate as a meaningful improvement. Risk: the confidence interval crosses zero.
Use it as a probe
Retain information about severity and failure patterns, but do not produce a model ranking.
Recommendation
Decline the current ranking; use the companionship set as a severity probe.
Do not use the current companionship set to choose or rank models by safety. Do not describe the Vertical Lift point estimate as a meaningful gain. The next round should first add tasks that can separate the models and calibrate the judge against human review.
Validation
Three domains, three factors and bootstrap confidence intervals test whether each conclusion holds.
Tasks and objects
Seven evaluation objects, three domains and 294 valid scoring units; public rules, samples and run records are available for review.
Scoring protocol
Independent scoring and item-level provenance are retained; mechanical assertions and semantic instruction-following are handled separately.
Vertical Lift
Controlled comparisons in finance, companionship and community settings examine training, policy and orchestration factors. They are not general product rankings.
Freeze and reproduction
The hidden set was preregistered. Current public results remain LLM-judge-only; human calibration is still required.
Current results include semantic instruction-following of roughly 75–83%; confidence intervals crossing zero in all 96 companionship-domain paired comparisons; no detectable finance-domain lift, which is not the same as proving no lift; and one exploratory community comparison whose interval did not cross zero.
Boundary / What we cannot claim
The conclusion stops at whether the evidence supports a choice.
Decision Impact / Next Action
The next step is not more scores. It is evidence that could change the decision.
Improve discrimination
Add tasks that distinguish policy, tool use and boundary behaviour.
Add human calibration
Align judge-only results with human standards.
Then consider selection
Results should enter a real workflow only after uncertainty has narrowed.