AI Platform Engineer Interview Questions

A strong answer covers ML-specific concerns: GPU scheduling, model versioning, experiment tracking, data pipelines, and inference optimization that traditional DevOps does not address.

Should describe how model serving frameworks handle loading models, exposing inference endpoints, managing batching, and supporting multiple model formats - citing tools like KServe, Seldon Core, Triton, or BentoML.

Cover GPU parallelism for matrix operations, high cost, limited availability, specialized drivers (CUDA), container runtime requirements, and the need for different scheduling strategies compared to CPU workloads.

Explain similarity search on high-dimensional embeddings, RAG architectures, and how vector databases differ from traditional databases in indexing (HNSW, IVF) and query semantics.

Cover reproducibility, version control of infrastructure, drift detection, multi-environment consistency, and tools like Terraform or Pulumi - especially important for complex GPU cluster configurations.

A great answer addresses node pools, GPU resource requests/limits, priority classes, preemption, MIG partitioning, taints/tolerations, and balancing utilization vs. latency requirements.

Should compare continuous batching and PagedAttention (vLLM), multi-framework support and ensemble models (Triton), and simplicity/low-overhead (FastAPI), relating each to workload characteristics.

Cover traffic splitting strategies, shadow scoring, model-specific metrics (accuracy, latency, drift), automated rollback triggers based on model quality metrics rather than just error rates.

Discuss quantization (GPTQ, AWQ, GGUF), continuous batching, prompt caching, request routing to appropriately-sized models, spot/preemptible instances, right-sizing GPU type, and auto-scaling policies.

Cover latency (TTFT, TPS), token usage and cost, error rates, hallucination detection, user feedback loops, embedding drift, and tools like OpenTelemetry, LangSmith, or Arize.

Discuss secret management tools (Vault, AWS Secrets Manager), rotation policies, least-privilege access, audit logging, and preventing key leakage in logs or model outputs.

Explain feature reuse, online/offline consistency, point-in-time correctness, and trade-offs between Feast (open-source), Tecton (managed), and custom solutions based on scale and team maturity.

Cover artifact versioning (DVC, model registry), automated evaluation on holdout datasets, performance threshold checks, security scanning, staging deployment, and human approval before production.

Address data classification, PII detection and masking pipelines, access controls, data residency requirements, consent management, and audit trails for compliance (GDPR, SOC2, HIPAA).

Cover document ingestion, chunking, embedding generation, vector store management, retrieval, reranking, context assembly, and LLM serving - platform engineer owns infrastructure, not application logic.

Should address model artifact replication, regional GPU capacity planning, latency-based routing, model version consistency, cross-region failover, and cost optimization through workload-aware placement.

Discuss DeepSpeed ZeRO stages, FSDP, pipeline/tensor parallelism, checkpoint management, fault tolerance for long-running jobs, network topology awareness, and storage requirements for checkpoints.

Cover namespace isolation, ResourceQuotas, network policies, GPU time-slicing or MIG, per-tenant cost attribution via metering, noisy-neighbor prevention, and quota management APIs.

Discuss schema registries, dual-write/read strategies, feature store versioning, shadow feature validation, and migration strategies that avoid serving stale or incompatible features.

Cover predictive scaling models, warm pool instances, spot/preemptible fallback strategies, queuing with priority levels, graceful degradation (smaller models, cached responses), and burst-to-cloud strategies.

Discuss golden datasets for continuous evaluation, statistical process control on output quality metrics, automated comparison against baseline models, guardrail violations as rollback triggers, and circuit-breaker patterns.

Address InfiniBand/RoCE for RDMA, NCCL topology detection, rail-optimized network fabric, congestion management, NCCL environment tuning, and monitoring collective communication performance.

Discuss API design (REST/gRPC), CLI tools, SDK development, declarative model definitions, automated resource provisioning, progress feedback loops, and error handling that translates infrastructure errors into actionable ML language.

Cover GPU-hour metering, token-level billing for LLM APIs, storage and egress tracking, embedding model usage accounting, tagging strategies, and integration with FinOps tooling.

Discuss LLM-as-judge patterns, human-in-the-loop sampling, proxy metrics (user engagement, task completion), distributional shift detection, and synthetic evaluation datasets.

Should cover checking pod resource limits, examining GPU memory fragmentation, analyzing request batching behavior, reviewing concurrent request handling, examining memory leaks in model code, and implementing proper resource isolation.

Address infrastructure provisioning, LLM selection/hosting strategy, RAG pipeline for knowledge base, guardrails and safety filters, monitoring and analytics, cost estimation, and phased rollout strategy.

Cover utilization analysis, identifying idle/s underutilized GPUs, right-sizing instances, implementing auto-scaling, evaluating quantization to use smaller GPUs, reviewing spot instance usage, and establishing cost governance policies.

Discuss custom container builds with the specific CUDA toolkit, Triton custom backend development, evaluating if vLLM custom ops can be used, testing compatibility, and maintaining the custom build long-term.

Cover embedding model quality assessment, chunking strategy review, index configuration (HNSW parameters, distance metrics), query preprocessing, retrieval debugging tools, and reranking pipeline evaluation.

Address parallel running during transition, endpoint compatibility layers, data pipeline migration, monitoring parity, rollback procedures, team training, and timeline with milestones.

Cover network policy enforcement, API gateway implementation, secret scanning in CI/CD, secret manager integration, security training for ML engineers, and automated compliance checks in the deployment pipeline.

Discuss profiling network bandwidth between nodes, checking for NUMA topology misalignment, verifying correct GPU driver and CUDA versions, examining data loading bottlenecks (I/O, preprocessing), and checking for noisy neighbors.

Address air-gapped or VPC-only training infrastructure, private model registries, on-premise or private-cloud GPU options, data handling pipelines with encryption at rest/transit, and compliance audit trails.

Discuss shared model registry, different serving configurations (online vs. batch), model artifact deduplication, separate compute pools sharing storage, and unified monitoring across both patterns.

Cover tool execution sandboxing, retry and timeout policies for multi-step chains, token budget enforcement, tracing (LangSmith), cost monitoring per agent run, and state management for long-running agent tasks.

Discuss model download as a CI step, caching strategies, model format validation, automated testing against benchmark datasets, artifact promotion through environments, and version pinning strategies.

Cover abstraction layers over provider APIs, traffic routing and splitting, unified logging and metrics, latency/cost/quality comparison dashboards, and fallback strategies when a provider is unavailable.

Discuss DAG definition in YAML, artifact passing between steps, conditional logic (e.g., skip deployment if evaluation fails), retry policies, parameterization, and integration with MLflow or W&B for tracking.

Cover prompt registry design, version control (Git-backed or dedicated tool), A/B testing infrastructure, prompt evaluation pipelines, rollback capabilities, and developer workflow integration.

Discuss statefulSet deployment, persistent volume provisioning, replication and sharding strategies, index build performance, backup/restore procedures, and monitoring query latency at scale.

Cover multi-model serving configuration, model loading/unloading policies, request routing based on model ID, resource isolation between models, and graceful handling of model swap latency.

Discuss sidecar proxies or API gateway plugins, Guardrails AI or NeMo Guardrails integration, centralized policy management, per-tenant guardrail configuration, and latency impact of inspection layers.

Cover sweep agent deployment as K8s jobs, GPU resource allocation for parallel trials, result aggregation, early stopping policies, and cost management for large sweep experiments.

Discuss semantic caching using embedding similarity, cache invalidation strategies, TTL policies, cache hit rate monitoring, privacy implications of caching user queries, and Redis/vector store backends.

Look for examples of technical leadership, clear communication of trade-offs, offering alternative solutions, and maintaining collaborative relationships while enforcing platform standards.

Assess learning agility, resourcefulness, ability to evaluate documentation and community resources, and how they balanced speed with reliability when implementing something new under pressure.

Look for frameworks like impact/effort matrices, stakeholder alignment processes, data-driven prioritization (usage metrics, blocked teams), and transparent communication about roadmap decisions.

Evaluate incident response process, communication during outages, root cause analysis methodology, blameless postmortem culture, and concrete preventive measures implemented.

Look for systematic learning habits, community engagement, evaluation frameworks (maturity, community size, backing), proof-of-concept processes, and examples of both successful and declined adoptions.