LastWeek

LastThreeDays

Today

Aoristou

FullTime

Hybrid

OnSite

As Senior Manager, Evaluation &amp; QA Engineer (Individual Contributor), you will own the quality, evaluation, and reliability strategy for our pip‑installable AI Capabilities SDK and its reusable modules (RAG components, agent/tool cookiecutters, MCP adaptors, standardized interfaces). You will design evaluation frameworks, test harnesses, and automation pipelines that make quality measurable and repeatable—from unit and integration tests to adversarial/fuzz testing, performance/cost benchmarking, and safety/privacy checks.You will collaborate closely with:<ul><li>Embedded AI Architects and Solution Designers to codify evaluation requirements that mirror real solution patterns.</li><li>AI Engineering to validate PI increments, integrate external technologies, and capture model/agent‑level failure modes.</li><li>DevOps Engineers to wire evaluation gates into CI/CD and release trains.</li><li>Developer Experience Engineers to ensure developers receive clear, actionable evaluation reports and quickstarts.</li><li>Digital Creation Centers and Forward Impact Teams (FITs) to gather field feedback and ensure tests reflect production reality.</li></ul>&nbsp;Why this Role MattersFor modular AI capabilities to scale safely and predictably, we need evidence‑driven quality. This role turns evaluation into a first‑class, automated discipline—ensuring every SDK change is validated for correctness, performance, safety, and operability. You will reduce regressions, raise confidence, and accelerate adoption across products and services that rely on the SDK as their backbone.ROLE RESPONSIBILITIES1) Evaluation Strategy &amp; Quality Architecture<ul><li>Define the evaluation strategy for SDK modules (parsers, retrievers, rankers, connectors, agent tools, MCP adaptors), including metrics, datasets/fixtures, benchmarks, and acceptance thresholds.</li><li>Establish quality gates for release: test coverage minimums, latency/throughput/cost budgets, stability envelopes, and backward compatibility checks.</li><li>Establish source code writing principles that align with the widely accepted coding standards and ensure high maintainability degree.</li><li>Create reproducibility standards (seeds, dataset/version pins, environment parity) and evidence capture (reports, artifacts) for auditability.</li></ul>2) Test Harnesses &amp; Automation<ul><li>Build unit, integration, contract, property‑based, and fuzz/adversarial test harnesses for SDK components; standardize fixtures for RAG and agent scenarios (vector stores, document corpora, tool execution).</li><li>Partner with DevOps to integrate tests and quality gates into CI/CD (parallelization, matrix runs, caching) and to fail builds on significant regressions or unmet quality bars.</li><li>Design benchmark suites for performance and cost profiling (e.g., latency distributions, throughput, memory/CPU footprints) with clear pass/fail criteria.</li></ul>3) Safety, Privacy &amp; Guardrails<ul><li>Codify safety evaluations (toxicity/abuse prompts, jailbreak resistance, sensitive data leakage risks) and policy adherence tests for agent/tool behaviors.</li><li>Validate security-by-design aspects of SDK usage (authn/z integration points, secrets handling, data minimization); coordinate with Platform/Security for service consumers.</li><li>Ensure privacy and compliance checks are represented in test plans and evidence bundles (PII handling, retention, deletion workflows).</li></ul>4) Observability-by-Design for Libraries<ul><li>Work with Capability Architecture and DevOps to embed diagnostic hooks suitable for libraries (structured logging, metrics/traces toggles) and define SLIs relevant to consuming services.</li><li>Verify error models (exceptions/types/codes) and retry/backoff behaviors; ensure runbooks and troubleshooting guides are informed by evaluation results.</li></ul>5) Data &amp; Datasets Governance for Evaluation<ul><li>Curate and version evaluation datasets (synthetic and real-world samples where appropriate); maintain data lineage and provenance.</li><li>Implement dataset health checks (drift, representativeness, coverage) and red‑team suites that evolve with product use cases.</li><li>Align dataset usage with legal/privacy guidance; document constraints and approvals.</li></ul>6) Developer Experience &amp; Enablement<ul><li>Produce evaluation reports and quality dashboards that developers can interpret quickly (summary + drill‑downs + diffs vs. baseline).</li><li>Collaborate with Technical Writers and Developer Experience engineers to include evaluation guidance, quickstarts, and troubleshooting in SDK docs.</li><li>Provide templates for teams to extend evaluation suites when contributing new modules via feature branches.</li></ul>7) Collaboration &amp; Continuous Improvement<ul><li>Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to keep evaluation aligned with production realities.</li><li>Lead post‑release and incident reviews to improve test coverage, resilience, and guardrails.</li><li>Track and improve quality KPIs (escaped defects, regression counts, benchmark variance, time-to-detect/repair).</li></ul>MEASURES OF SUCCESS<ul><li>Safety &amp; compliance: Guardrail tests pass; reduced policy violations; documented evidence meets audit requirements.</li><li>DX &amp; adoption: Clear evaluation reports improve developer confidence; faster time‑to‑first‑success for consuming teams.</li><li>Continuous Improvement: Actionable insights from incidents and field feedback lead to expanded test suites and stronger guardrails.</li><li>Process Excellence: Data-driven evaluation of the efficiency, quality and consistency of the SDK testing and quality workflows.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in QA/Quality Engineering/Evaluation for SDKs, libraries, or platform components in software/AI contexts.</li><li>Hands‑on experience building automated test frameworks (unit/integration/contract) and benchmark suites; strong Python testing fluency (e.g., pytest, property-based testing, fuzzing).</li><li>Proficiency with CI/CD systems (GitHub Actions, Azure DevOps, GitLab CI) and integrating quality gates (coverage, performance, security).</li><li>Working knowledge of AI/ML or GenAI patterns (RAG, vector stores, agents/tools) and evaluation considerations for these workloads.</li><li>Strong grasp of reproducibility, data governance, and observability‑by‑design for libraries.</li></ul>Preferred Qualifications<ul><li>Experience with adversarial/red‑teaming methodologies for LLM‑enabled features and safety guardrails.</li><li>Background in performance engineering (profiling, optimization, latency/cost analysis) for Python libraries.</li><li>Familiarity with security and privacy testing (static analysis, secrets scanning, dependency hygiene, PII handling).</li><li>Exposure to regulated environments (healthcare/pharma/finance) and audit‑ready evidence capture.</li><li>Comfortable collaborating across architecture, DevOps, DX, LLMOps, platform, and product teams in matrixed organizations.</li></ul>

Senior Manager, Evaluation & QA Engineer 4949162

Technology Strategy and Consulting team help the C-Suite shape their technology vision&nbsp;and execute on their digital and technology-driven business transformation journeys.As a Technology Strategy and Advisory (TS&amp;A) Analyst, you will be at the core of understanding and responding to our clients' business and technology challenges.&nbsp; You are likely to be involved in a range of activities from supporting Organizations to develop their technology strategy to assisting with transformation implementation activities.Analysts:<ul><li>Working with industry and business specialists to develop comprehensive technology strategies, including IT opportunity assessment, investment portfolio management, information strategy, sourcing strategy, enterprise architecture strategy and more</li><li>Create innovative and differentiated offerings, staying relevant and in sync with market demand and technology evolution</li><li>Support delivering technology advisory services and strategies for technology enabled business change</li><li>Leveraging Accenture’s international assets and expertise to bring new ideas to clients</li><li>Supporting the development of business cases and implementation roadmaps to support investment decisions</li></ul>Consultants:<ul><li>Working with industry and business specialists to develop comprehensive technology strategies, including IT opportunity assessment, investment portfolio management, information strategy, sourcing strategy, enterprise architecture strategy and more</li><li>Create innovative and differentiated offerings, staying relevant and in sync with market demand and technology evolution</li><li>Deliver technology advisory services and strategies for technology enabled business change, e.g.: package &amp; third-party integrator selection, vendor selection, application studies, solution architecture studies and/or program diagnostics and recovery</li><li>Deliver advisory services to C Suite functions such as CIO/ CTO/ CDO etc</li><li>Interviewing client staff to understand key issues with how they operate today and identify opportunities for improvement in their operations and technology landscape</li><li>Leveraging Accenture’s international assets and expertise to bring new ideas to clients</li><li>Supporting the development of business cases and implementation roadmaps to support investment decisions</li><li>Providing project management and change management support on large scale transformation projects</li><li>Working in multi-disciplinary teams to shape, propose, communicate and implement programs</li></ul>QualificationWHO WE´RE LOOKING FOR?&nbsp;Analysts:<ul><li>BSc from a Top University, preferably in Computer Science/ Software Engineering or Electrical/ Electronic Engineering</li><li>Basic understanding of technology and basic knowledge of main technology architecture concepts, application styles and concepts</li><li>Basic knowledge of key emerging methodologies (Agile, DevOps, Design thinking)</li><li>Prior technology consulting experience with e.g., big consulting practice, consulting arms of main technology firms or internal consulting department of multinational will be considered a plus</li></ul>Consultants:<ul><li>BSc with distinction or honors from a Top University, preferably in Computer Science/ Software Engineering or Electrical/ Electronic Engineering</li><li>3-5 year cross-industry experience in consulting firms or big and international companies.&nbsp;</li><li>Ability and willingness to travel in Greece and internationally.</li><li>Understanding of emerging technology trends (Cloud, Big Data, API, Microservices etc) and impact on both overall enterprise architecture and operating model</li><li>Basic knowledge of key emerging methodologies (Agile, DevOps, Design thinking) and how to use it to improve IT related department</li><li>Experience in developing associated business cases and plans which are detailed, actionable and aligned with industry best practices</li><li>Good business acumen to find the best way to support key strategic pillar with technology</li><li>Background in Business and in IT architecture, as well as experience in business and IT collaboration. Ability to rapidly assess the IT implications of business strategy and the business implications of IT strategy&nbsp;</li><li>Strong verbal and written communication skills in Greek and English.</li><li>Knowledge of and passion for technology and digital trends</li><li>Ambition and desire to shine in the newest strategic consultancy environment</li><li>Demonstrable experience of managing teams of 3-5 people driving strategy work</li></ul>WHAT´S IN IT FOR YOU?<ul><li>Competitive salary and benefits, including but not limited to: life/health insurance, performance-based bonuses, monthly vouchers, company car (depending on management level), flexible work arrangements, employee share purchase plan, TEA Accenture, parental leave, paid overtime (if needed) and various corporate discounts</li><li>Continuous hard and soft skills training &amp; development through global platforms &amp; local academy</li><li>Career coaching and mentorship to help you manage your career and develop professionally</li><li>Ongoing strengths and skills-based evaluation process</li><li>Various opportunities to develop your career across a spectrum of clients, industries and projects</li><li>Diverse and inclusive culture</li><li>Corporate citizenship initiatives (access to volunteering opportunities, charity work etc.)</li><li>Under our Brain Regain initiative, extra relocation benefits may apply</li></ul>

Job Ads Solution Architect Μακεδονία, Information Technology / It - Page 2

senior manager, evaluation & qa engineer 4949162

Pfizer Hellas

technology strategy and advisory (ts&a) analysts and consultants

Accenture

technology strategy and advisory (ts&a) analysts and consultants

Accenture

Also hiring nearby:

Jobily.gr

For Employers

For Job Seekers

Company