What a rubric is, and how it stops your work being rejected
· 2 min read · quality, how it works
A rubric is the written standard your work is judged against, with the specific criteria defined in advance. On Jwuma it is taught in onboarding, frozen when a batch begins, and used as the reference for any rejection. Freezing it means the standard cannot drift and leave you rejected for something acceptable last week.
The short version
- A rubric defines the acceptance criteria in writing before work starts.
- Jwuma freezes the rubric at the moment a batch begins.
- Every rejection references the rubric version that produced it.
- A rubric that changes mid-batch is a process failure, not your failure.
- Flagging an ambiguous case is always better than guessing at it.
Why is a written standard so important?
Because without one, acceptance is a matter of opinion, and opinions drift. Two reviewers disagree, or the same reviewer changes their mind across a long batch, and the annotator has no way to know which version they are being judged against.
A rubric turns that into something checkable. If your work was rejected, there is a specific criterion it failed and you can be shown it.
What does freezing a rubric mean?
That the version in force when a batch opens is the version that batch is judged against, even if the rubric is later improved. Improvements apply to the next batch.
This sounds like a technicality and is not. Without it, a client can refine their definition halfway through and every task you completed under the old definition becomes retroactively wrong. You would be penalised for correctly following an instruction that no longer exists.
How do you use a rubric well?
Read it fully before starting, not after your first rejection. Pay particular attention to the edge cases it names, because those are the cases someone already got wrong.
When you meet a case the rubric does not cover, flag it. Do not average between two plausible readings and do not pick one silently. An unflagged guess pollutes the batch and is the most common cause of a whole submission being rejected.
- Read it fully before your first task
- Note the edge cases it names explicitly
- Flag anything it does not cover
- Never resolve ambiguity silently
People also ask
What is a rubric in data annotation?
A rubric is the written standard defining how work is judged, with specific criteria set out in advance. It is taught during onboarding and referenced in any rejection.
Can the rubric change while I am working on a batch?
On Jwuma, no. The rubric is frozen when a batch begins so the standard cannot drift mid-project. Improvements apply to the next batch, not retroactively to yours.
What should I do if the rubric does not cover my case?
Flag it rather than guessing. An unflagged guess is worse than a flagged gap, because it puts a confident wrong label into the dataset where nobody will find it.