LastWeek

LastThreeDays

Today

Aoristou

FullTime

Hybrid

OnSite

The Director, AI Quality Engineering Lead ensures that AI systems across the enterprise are safe, fair, reliable, compliant, and observable from pre‑release through continuous operations. This role designs and operates enterprise evaluation frameworks (quality, bias/fairness, robustness, privacy), leads red‑teaming and guardrail development, encodes policy and regulatory requirements as automated checks, and drives metric reporting and post‑incident learning across AI programs. The lead partners closely with Enterprise Architecture, Compliance, Legal/Privacy, Security, Data Governance, and product/engineering teams to enforce quality and safety gates and enable responsible scale. (Aligned to Corporate Policy and Responsible AI principles.) Reporting into the Head of Applied AI Engineering, you will manage a team of QA Engineers and AI Engineers. This role blends deep technical leadership, product engineering strategy, and direct enablement of enterprise use cases and services. You will partner closely with business, legal and technical stakeholders to design and deliver robust, scalable solutions that demonstrate the value of AI across our enterprise in a safe and compliant way. This role is execution-oriented with strong influence on building AI solutions responsibly.Evaluation and Test Strategy<ul><li>Define and run evaluation frameworks for LLMs and multi-modal systems covering performance, fairness, bias, robustness, safety, and privacy.</li><li>Build automatic + human-in-the-loop evaluation pipelines with reproducible scoring, statistical significance, and error analysis.</li><li>Establish SLOs/SLIs for quality (e.g., grounded accuracy, hallucination rate, privacy leak rate, harmful-content rate).</li></ul>Red-Teaming &amp; Safety Guardrails<ul><li>Design and run red-teaming against prompt injection, jailbreaks, sensitive data leakage, tool-abuse, and retrieval abuse.</li><li>Develop guardrail solutions (classification, content safety, PII/PHI detection, grounding checks, allow/deny lists, rate limiting, output filtering) to mitigate hallucinations, leakage, and compliance violations.</li><li>Maintain attack libraries and scenario banks; simulate adversarial behavior and monitor adaptive attacks over time.</li></ul>Pre‑Release Gating &amp; Continuous Assurance<ul><li>Operate pre‑release evaluation gates with baselines, regression checks, canary/A–B testing, and rollback criteria.</li><li>Run continuous evaluation in production with synthetic &amp; real traffic (shadow/ghost testing) and drift, bias, and toxicity monitoring.</li></ul>Stakeholder Leadership &amp; Culture<ul><li>Coach product teams on responsible AI practices and establish quality ownership norms.</li><li>Build a community of practice across Engineering, Security, Data Governance, and Risk—standardizing test data, benchmarks, and review rituals.</li><li>Mentor engineers building evaluation, red-team, and guardrail capabilities.</li></ul>BASIC QUALIFICATIONS<ul><li>7+ years in ML/AI/Platform/Quality roles, including 3+ years evaluating production systems.</li><li>Advanced degree in Computer Science, Engineering or related discipline.</li><li>Proven design &amp; operation of evaluation harnesses and policy-as-code checks in CI/CD.</li><li>Hands-on Python and expertise with evaluation and safety tooling such as: OpenAI Evals, Ragas/LangSmith, DeepEval, Azure AI Content Safety / OpenAI Moderation, Presidio, Guardrails.ai, Nemo, MLFlow, Grafana, Kibana, Azure monitor</li><li>Cloud &amp; MLOps: Azure/AWS, CI/CD (GitHub Actions/Azure DevOps, feature stores, vector DBs (Azure AI Search, OpenSearch, Pinecone), testing in pipelines.</li><li>Excellent communication with executives and engineers</li><li>Experience leading an engineering team.</li></ul>PREFERRED QUALIFICATIONS<ul><li>Led an AI safety/quality function or enterprise red-team program.</li><li>Prior work with AI based applications, tool use/orchestration, and enterprise data governance.</li><li>Regulatory literacy (e.g., GDPR, EU AI Act readiness), and experience collaborating with Compliance/Risk, Security, and Legal.</li><li>Strong grasp of red-team techniques (prompt/indirect injection, jailbreaks, training data extraction, membership inference, model poisoning, tool abuse).</li><li>Experience with retrieval grounding checks, memory governance, and multi-tenant safety controls.</li></ul>

Job Ads Qa Engineer Μακεδονία

director, ai quality engineering lead 4946565

Pfizer Hellas

Also hiring nearby:

Jobily.gr

For Employers

For Job Seekers

Company