QA Engineer, Annotation Systems
Corpshore AI's QA Engineer, Annotation Systems role, based onsite in Manila, builds and maintains the automated consistency checks, gold-set benchmarking, and consensus-scoring logic that form Tier 3 of the company's three-tier QA cascade on the Jwuma platform.
This is Corpshore AI's Tier 3 role: the engineer who builds and maintains the programmatic checks, gold-set benchmarking scripts, and consensus-scoring logic that gate every batch before it ships. Based onsite in Manila, you'll work closely with QA leads and the Jwuma platform team to keep automated quality gates aligned with each client's evolving taxonomy.
Key responsibilities
- Build and maintain automated consistency checks and gold-set benchmark scoring inside Jwuma
- Design and tune consensus-scoring logic across multi-annotator batches
- Investigate accuracy regressions and root-cause QA-cascade failures with data
- Partner with QA leads and taxonomy owners to translate labeling rules into automated checks
- Produce dashboards and reports on batch-level and hub-level quality metrics
What we look for
- 2+ years in QA engineering, data quality, test automation, or a related analytical engineering role
- Working proficiency in SQL and at least one scripting language (Python preferred)
- Strong analytical skills and comfort working with large labeled datasets
- Excellent written and spoken English for cross-functional collaboration
- Full-time onsite availability at the Manila hub
Nice to have
- Experience with annotation, labeling, or data-pipeline QA specifically
- Familiarity with inter-annotator agreement metrics (Cohen's kappa, Krippendorff's alpha)
- Exposure to LLM/RLHF evaluation frameworks
Compensation and benefits
- Competitive fixed monthly salary plus performance bonus
- Direct collaboration with Corpshore AI's platform and data-science teams
- Full statutory Philippine employment benefits
- Career path toward Senior QA Engineer or Platform Engineering Lead
Common questions
How is this QA engineer role different from a QA reviewer or QA lead?
QA reviewers and leads manually audit annotator work; this engineering role builds the automated checks, gold-set scoring, and consensus logic that form Tier 3 of Corpshore AI's QA cascade, working with code and data pipelines rather than manual review alone.
Similar roles
Apply once, get matched to what fits
One application covers every role you qualify for. A person reads it.
Apply now