Market sizing environments with an exact answer key
Scorekey builds evaluation environments from market sizing: working out how big a market is from raw public data, the way a due diligence team does it. The environment is the whole job, from choosing the sources to the final figure, and every decision along the way is scored against a key. A model can be tested on the work an analyst does, and trained to do it.
One environment is 250 to 400 decisions, and each score is exact: an answer counts only if it matches the key within one part in ten thousand. Nothing is scored by another model.
Frontier models get the execution right and the decisions wrong
EXECUTION STAGES
75% to 100%
Frontier models on extracting the figures, filling the coverage gaps and computing the result
DECISION STAGES
45% to 60%
Frontier models on selecting the sources and pinpointing the exact series
FINAL FIGURE
35% to 45%
Typical distance from the right market figure
Three frontier models, three full runs each on two environments, scored as a whole and stage by stage.
Rounded to the nearest 5; exact figures on the Research page.
See the results, stage by stage →Evaluate on it, train on it, commission it
Each environment is one full market build with an exact key and a scorer, run on your own infrastructure. License one, commission one on a market you name, or use the score as reward
Every answer checked against a key
Public data on a dated snapshot, so any figure traces to its source. Every answer is matched to the key within one part in ten thousand, and every score reproduces from the stored submission and the grader
Built by people answerable for the figures
Each environment is built end to end by senior consultants who led this work on transactions from the hundreds of millions to the billions and answered for the figures that went to the investment committee
Run your model on a Scorekey environment
Three tasks from the online goods market environment, open on GitHub, scored with a token
Run the demo