At Pfizer, technology accelerates every facet of our mission. Within the AI Center of Excellence, the AI Quality Engineering team embeds trust, quality, and responsibility into every AI solution we deliver.

As a Staff QA Engineer, you will lead the design and operation of enterprise‑grade evaluation frameworks, guardrails, and continuous quality pipelines that keep our models performant and safe. Your work enables teams across the company to ship AI solutions with confidence—meeting rigorous standards for accuracy, fairness, security, and compliance while driving measurable impact for patients and colleagues alike.

Why this Role Matters

This role is pivotal in shaping the organization’s AI strategy and technical excellence. Your leadership will accelerate the adoption of cutting-edge AI technologies, foster a culture of innovation, and ensure that AI solutions are safe, reliable, and impactful at scale.

ROLE RESPONSIBILITIES

• Define & lead enterprise AI evaluation frameworks for model performance, robustness, fairness, bias, and safety across LLMs, traditional ML, and agentic AI systems.

• Architect and operate pre‑release & continuous evaluation (batch + online) integrated with CI/CD, including dataset versioning, synthetic test generation, and production quality gates.

• Design, implement, and maintain guardrail services for hallucinations, prompt injection, PII leakage, unsafe content, and policy violations; instrument telemetry for detection, alerting, and drift tracking.

• Run structured red‑teaming programs (jailbreaks, data exfiltration, adversarial prompts) and drive measurable mitigation with engineering teams; publish risk findings and remediation plans.

• Encode quality & compliance policies as automated checks, partnering with Enterprise Architecture and Compliance to meet internal standards and emerging regulations (e.g., EU AI Act).

• Establish cross‑use‑case quality SLOs & dashboards, reporting enterprise‑wide quality and safety metrics; lead post‑incident reviews and prevention programs.

• Mentor QA/AI engineers on evaluation science, test design, and responsible AI practices; uplift engineering excellence and documentation quality across teams.

• Collaborate with AI Engineering, Integrations & Enablement teams to ensure evaluation hooks, observability, and policy gates are first‑class in our SDKs, APIs, and shared components.

• Collaborate with AI Innovation team on continuously assess emerging evaluation methods & tools, run rapid prototypes and PoCs, and advise on strategic adoption that improves quality, safety, and developer experience.

BASIC QUALIFICATIONS

• Advanced degree in Computer Science, Engineering, Data Science, or related field.

• 7+ years in Software QA/AI QA/Quality Engineering, including ownership of test strategy for distributed or data‑intensive systems

• Demonstrated experience building evaluation pipelines for ML/LLM systems: design of datasets/benchmarks, metrics for accuracy/robustness, and failure‑mode analysis (e.g., hallucination, bias, toxicity).

• Strong fluency in software engineering best practices (automated testing, version control, CI/CD) and observability (logging, metrics, tracing) for model‑centric applications.

• Hands‑on with MLOps concepts (model serving, deployment, feature/data versioning) and policy gating in release pipelines.

• Ability to produce clear technical documentation (evaluation protocols, runbooks, API references) and developer‑friendly guides that scale adoption.

PREFERRED QUALIFICATIONS

• Experience in pharma/healthcare or other regulated environments (validation, auditability, GxP-adjacent controls).

• Deep knowledge of LLM safety & evaluation techniques: prompt‑injection defense, content moderation, RAG evaluation, context integrity, and bias/fairness testing.

• Practical experience with evaluation libraries & frameworks (e.g., Ragas, TruLens, DeepEval, OpenAI Evals) and guardrail tooling integrated with platform SDKs.

• Familiarity with agentic AI behaviors and evaluation, including planner/tool‑use reliability, memory safety, and multi‑agent orchestration checks.

• Experience leading red‑team programs and incident response; evidence of measurable risk reduction and improvement in safety KPIs. (Competency expectations at Staff level)

• Contribution to open‑source or industry forums on Responsible AI, safety evaluation, or testing.