AI Data Annotation Quality Specialist vs AI Data Catalog Specialist

A AI Data Annotation Quality Specialist Only

• Annotation guideline design and versioning for multi-class and subjective labeling tasks
• Inter-annotator agreement measurement using Cohen's Kappa, Fleiss' Kappa, and Krippendorff's Alpha
• Statistical process control for annotation quality (control charts, defect rate tracking)
• Bias and fairness auditing in labeled datasets (demographic parity, equalized odds)
• RLHF preference data quality evaluation and comparison methodology
• Data labeling taxonomy and ontology design
• Error pattern recognition and root-cause analysis across annotator cohorts
• Prompt engineering for LLM-as-judge quality validation pipelines