Model Evaluation & Benchmarking
Our model-selection process is based on workload-specific evaluation rather than generic model rankings.
What a leaderboard cannot tell you
Public benchmarks can provide useful context, but they do not establish whether a model is reliable on a company’s extraction format, retrieval corpus, tool interfaces, ambiguity profile, or error consequences.
We evaluate the behavior we intend to deploy.
Evaluation surface
Depending on the workload, a model evaluation can measure:
- Task accuracy
- Consistency across repeated runs
- Structured-output validity
- Hallucination or unsupported-claim rate
- Latency
- Inference cost
- Retrieval dependence
- Tool-use reliability
- Edge-case behavior
- Regression against known working examples
The scoring method should match the task. Exact extraction can often be evaluated deterministically against ground truth. Complex investigations may require rubric-based review, evidence checks, adversarial review, or human judgment. A single aggregate score can conceal important failure modes.
Candidate comparison
A model benchmark can compare more than different foundation models. The candidates may include:
- Model A + prompt
- Model A + retrieval
- Model B + retrieval
- Private model + specialization
- Quantized deployment
- Deterministic implementation
This matters because architecture changes can produce larger gains than changing the underlying model.
Evaluation also includes the deployment form. A model that performs well at one precision, context strategy, or tool configuration should not be assumed to behave identically after quantization or runtime changes.
Production relevance
We preserve difficult examples and regression cases because average-case performance is not enough. If a model upgrade improves routine tasks but breaks a small set of economically important cases, the upgrade may be a regression despite a higher aggregate score.
Our model layer therefore treats benchmarking as an operational selection mechanism, not as a leaderboard exercise.
The best model is the model that clears the actual workload requirements at the best acceptable quality, cost, latency, privacy, and reliability tradeoff.
Evaluation sets are versioned
The benchmark itself can drift. When the company changes its process, source systems, taxonomy, or operating definitions, an evaluation set can stop representing the production task even if the examples remain internally consistent.
We therefore treat important evaluation sets as versioned assets with known scope. A model comparison is only meaningful when the candidates are tested against the same relevant task definition and when the scoring procedure reflects the consequence of the errors being measured.
For high-consequence cases, weighted or stratified evaluation can be more informative than a single average, because a rare failure can matter far more economically than many routine successes.