Generative AI
Preference data, red teaming and evaluations with people who know the subject.
Ranking, fine tuning data and adversarial testing for language, image and video models, with domain experts brought in where a generalist would get it wrong.
Illustrative dashboard, sample data
What we do for Generative AI teams.
Preference and comparison labelling
Candidate answers ranked with calibrated rubrics for helpfulness, harmlessness and accuracy.
Supervised fine tuning data
Instructions and demonstrations written by people who know the domain.
Red teaming
Adversarial prompts that look for jailbreaks, bias and unsafe output.
Factuality and citations
Claims checked against sources, and cited sources checked to exist and support the text.
Structured output checks
JSON, SQL and code outputs checked for schema and field accuracy.
Image and video generation evaluation
Generated images and video scored against the prompt, with every mismatch marked.
What the work looks like.
Domain experts in medicine, law, engineering and languages join through Rubricate, our expert network.
prompt
Summarise this discharge note for the patient in plain language. Include medication changes.
answer A
You were treated for a chest infection and can go home. Keep taking your inhalers. Your doctor stopped one medicine.
Missing: which medicine was stopped, new antibiotic course
answer B · preferred
You were treated for a chest infection. Take amoxicillin three times a day for 5 more days. Your doctor stopped naproxen because it can upset your stomach. Keep using your inhalers as before.
| rubric | A | B |
|---|---|---|
| Accuracy | 3/5 | 5/5 |
| Follows instructions | 4/5 | 4/5 |
| Safety | 5/5 | 5/5 |
| Tone | 3/5 | 4/5 |
Interface shows illustrative sample data
Proof, not adjectives.
Named work in this area is under NDA. We share anonymised samples and references on a call. Request a pilot to see it on your data.
Send us 100 answers your model got wrong.
Send a small batch and get it back done, free, usually within 24 to 48 hours. You judge the work before you pay for anything.