Find where your model fails before your users do.
People who know your domain score model output on text, images, video and audio against a rubric you sign off, and check your automated judge while they are at it.
Eight kinds of work across text, images, video and audio, held to the same rubric discipline.
Predictions from detection, segmentation, OCR, speech and language models checked against the source. Every error is typed and corrected, and the worst items go back for full relabelling.
You getCorrected training data, precision and recall, and an error gallery by class
Every tool call and reasoning step scored against your rubric, with the first failing step tagged.
You getStep scores and the first failing step for every trace
Your automated judge scored by people on the same items, and every disagreement classified.
You getHow often the judge is right, and a list of overruled scores
Answers, captions or generated images compared and ranked against a calibrated rubric, for RLHF and DPO.
You getPreference pairs with rubric scores and reasons
Adversarial prompts, images and audio built to find jailbreaks, prompt injection, bias and unsafe output.
You getFindings by severity, with the inputs that caused them
Hard conditions tested on purpose: low light, rain, glare, accents, noisy lines, rare classes and edge cases.
You getResults split by condition, with the gaps spelled out
Instructions and demonstrations written by people who know the subject.
You getSFT examples in your format
Versioned ground truth for text, image, video and audio models that you can test every release against.
You getA held out set with a changelog
The models we evaluate, the failures we look for and what you get back, for text, images, video and audio.
Chat assistants, RAG systems, agents, summarisers, classifiers and extraction models
Object detection, segmentation, classification, OCR and document AI, vision language models
Tracking, action and event recognition, video language models, robot policies and driving perception
Speech recognition, voice agents, speaker diarization, text to speech and sound event detection
The interfaces our reviewers work in, or yours if you prefer. The data shown is illustrative.
Two answers to the same prompt, scored on the rubric. The reviewer prefers B and says why.
prompt
Summarise this discharge note for the patient in plain language. Include medication changes.
answer A
You were treated for a chest infection and can go home. Keep taking your inhalers. Your doctor stopped one medicine.
Missing: which medicine was stopped, new antibiotic course
answer B · preferred
You were treated for a chest infection. Take amoxicillin three times a day for 5 more days. Your doctor stopped naproxen because it can upset your stomach. Keep using your inhalers as before.
| rubric | A | B |
|---|---|---|
| Accuracy | 3/5 | 5/5 |
| Follows instructions | 4/5 | 4/5 |
| Safety | 5/5 | 5/5 |
| Tone | 3/5 | 4/5 |
We write or refine the rubric with your team, with an example for every score.
Reviewers score a gold set and must match it before they see your data.
Part of every batch is scored twice. Disagreements go to a senior reviewer, who settles them and updates the rubric.
Work is checked in small batches as it is done, and every delivery is sampled before it reaches you.
Every week your ML team gets the patterns behind the failures, with examples.
Trace 0093 · step 3 · criterion: right tool, right input
Booked the wrong day.
Day matches the open calendar slot.
Caller asked for Tuesday at 00:03. Check the transcript before the calendar.
Rubric v1.3Rule added, earlier traces rescored, both reviewers recalibrated
Every evaluation ends with findings your team can act on the same week.
Where the model holds up and where it does not, criterion by criterion.
Every failure typed, so the biggest problem is obvious.
The first failing step in an agent run, or the frame, turn or region where things went wrong.
How often your automated judge agrees with people, and on which kinds of items it does not.
How often two reviewers agree, so you know which scores to rely on.
Split by language, accent, lighting, device, class or any condition you care about.
The same evaluation set run on each model version, side by side.
The failures themselves, labelled and ready for your next training run.
A batch of real outputs scored before a launch, a vendor choice or a model swap.
Best when you need an answer by a dateThe same evaluation set run on every model version, so you see what got better and what broke.
Best when you ship oftenA share of live traffic scored every week, with failures fed back into training.
Best once the model is in productionA healthcare voice AI company
Human evaluation of a voice agent that books patient appointments, including the LLM judge scoring its calls. Client anonymised at their request.
Reviewers trained on your rubric before they score a single item. When the work needs a doctor, a lawyer or an engineer, we bring them in through Rubricate, our expert network.
Meet the team behind the workEnglish and the major Indian languages, with reviewers who speak them natively. For speech, we cover the accents your users actually have.
Per trace, per comparison, or a dedicated seat for ongoing work. What moves the price:
Yes. Langfuse, LangSmith, Label Studio or your own interface. We can also work from exports.
Yes. Detection, segmentation, OCR, tracking, video understanding and vision language models, reviewed frame by frame against your class guide.
Yes. Send a batch of predictions, or give us access to your tool. Reviewers confirm, fix or reject each one against your guide, items with the most errors go back for full relabelling, and you get the corrected data with a report of where the model goes wrong.
Yes. Reviewers score the same items as the judge and classify every disagreement, so you know where it can be trusted and where it cannot.
Calibration on a gold set before live work, double scoring on a sample of every batch, and a senior reviewer who settles disagreements and updates the rubric.
Yes. Medical, legal, engineering and language experts join through Rubricate, our expert network. Our healthcare evaluation work uses reviewers trained in medical terminology.
Every reviewer works under NDA, and we work inside your access controls where we can. The security page has the details.
There is no minimum. A pilot starts with a batch of your real outputs, and we suggest a size that gives you a useful answer.
Yes. We collect or select the items, write the ground truth with your team and version it, so every release is tested on the same set.
With a pilot: a batch of your real outputs, scored against your rubric, so you can judge the work before you commit.
Send a small batch and get it back done, free, usually within 24 to 48 hours. You judge the work before you pay for anything.