Calibrate before live work
Before an expert touches a live task, they complete a calibration set: tasks with known answers, written for the project. Experts who do not meet the bar on calibration do not start.
Check against gold during the work
Gold items with known answers are mixed into live work. Agreement with gold is tracked per expert, so drift is caught early and the expert is re-calibrated or removed from the project.
Review every batch
Each batch goes through QA review before delivery. Items that fail are redone or rejected, and you are not billed for them. Across our work, 96%+ of items pass QA review.
Report what we found
Every pilot and batch comes with a QA report: what was checked, what failed and why, and how agreement looked across experts. Where experts disagree on a hard item, we say so rather than hide it.
What this cannot do
Calibration and gold checks raise the floor and catch drift. They do not make experts infallible. Expert data should still be evaluated like any other data, and we plan our QA with that in mind.