AI Observability Engineer vs AI Output Filtering Engineer

A AI Observability Engineer Only

• LLM pipeline tracing and semantic instrumentation
• Custom metrics design for model quality (hallucination rate, retrieval relevance, toxicity scores)
• Distributed tracing with OpenTelemetry adapted for GenAI semantic conventions
• Real-time anomaly detection on model outputs and infrastructure telemetry
• Dashboarding and alerting with Grafana, Datadog, or cloud-native tools
• Cost observability for token-based and GPU-based inference workloads
• Prompt versioning, A/B testing instrumentation, and regression tracking
• Data drift and embedding drift detection methodologies