How quality scores work, and how to raise your acceptance rate
· 2 min read · quality, how it works
Three numbers matter: acceptance rate, which is accepted work divided by accepted plus rejected; reliability, which is completed work divided by what you reserved; and average handling time. Reviewers are measured too, on overrule rate, so a reviewer who is consistently wrong is visible and correctable.
The short version
- Acceptance rate is accepted divided by accepted plus rejected.
- Reliability is completed divided by reserved.
- Reserving work you do not complete damages reliability even if nothing was rejected.
- Reviewers are themselves measured on overrule rate.
- Flagged ambiguous cases do not count against acceptance.
What do the three numbers actually mean?
Acceptance rate measures whether your work meets the standard. Reliability measures whether you finish what you start, which is a separate thing entirely: you can have perfect acceptance and poor reliability by reserving more than you can do. Handling time is context, not a target, and being fast is worth nothing if acceptance falls.
Reliability is the one people damage without noticing. Reserving a large batch on a Sunday and abandoning half of it on Monday hurts you more than a couple of rejections would.
What actually raises acceptance?
Reading the rubric fully before the first task, not after the first rejection. Flagging genuinely ambiguous cases instead of guessing. Slowing down on the first ten items of a new batch to check your interpretation against the examples.
The single highest-return habit is flagging. A flagged case is not counted against you, and it frequently results in the rubric being clarified for everyone. A silent guess is counted against you and quietly damages the dataset.
- Read the whole rubric before starting
- Go slowly for the first ten items
- Flag rather than guess
- Reserve only what you will finish
- Read every rejection reason properly
What happens if you think a rejection was wrong?
Say so. Reviewers are measured on overrule rate precisely so that reviewer error is visible, which only works if disagreements are raised.
A rejection on Jwuma cites the rubric criterion it failed, so a disagreement is a specific argument about a specific criterion rather than an argument about whether the reviewer liked your work.
People also ask
What is a good acceptance rate for annotation work?
It varies by task type and difficulty. What matters more than a single number is the trend and whether rejections cite a rubric criterion you can act on.
Does reserving tasks I do not finish hurt my score?
Yes. Reliability is completed work divided by reserved work, so abandoning reserved tasks damages it even when nothing you submitted was rejected.
Are reviewers held to a standard too?
Yes. Reviewers are measured on overrule rate, so a reviewer whose decisions are consistently overturned is visible and can be corrected.