Remote QA Engineer, Annotation Quality Systems (Poland)

This remote, Poland-based QA Engineering role complements Corpshore AI's onsite Warsaw platform team, focused specifically on building and tuning the programmatic QA logic, consistency checks, gold-set benchmarking and consensus scoring, that forms Tier 3 of the company's QA cascade across EU-facing annotation programs.

Key responsibilities

• Build and maintain automated consistency checks and gold-set benchmark scoring for EU-facing projects in Jwuma
• Design and tune consensus-scoring logic across multi-annotator, multi-language batches
• Investigate accuracy regressions and root-cause QA-cascade failures with data
• Partner remotely with QA leads and taxonomy owners across the Warsaw, Kampala and Nairobi hubs
• Produce dashboards and reports on batch-level and cross-hub quality metrics

What we look for

• 2+ years in QA engineering, data quality or test automation, ideally with labeled/annotated datasets
• Working proficiency in SQL and Python
• Strong analytical skills and comfort with large labeled datasets
• Fluent Polish and professional English for cross-hub collaboration
• Eligible to work remotely in Poland

Nice to have

• Experience with inter-annotator agreement metrics (Cohen's kappa, Krippendorff's alpha)
• Familiarity with GDPR-relevant data handling in system design
• Exposure to RLHF or LLM evaluation frameworks

Compensation and benefits

• Competitive fixed salary plus performance bonus
• Fully remote within Poland with flexible hours
• Full statutory Polish employment benefits
• Direct collaboration with Corpshore AI's global platform and QA teams

Common questions