LastWeek

LastThreeDays

Today

Aoristou

FullTime

Hybrid

OnSite

As Senior Manager, Evaluation &amp; QA Engineer (Individual Contributor), you will own the quality, evaluation, and reliability strategy for our pip‑installable AI Capabilities SDK and its reusable modules (RAG components, agent/tool cookiecutters, MCP adaptors, standardized interfaces). You will design evaluation frameworks, test harnesses, and automation pipelines that make quality measurable and repeatable—from unit and integration tests to adversarial/fuzz testing, performance/cost benchmarking, and safety/privacy checks.You will collaborate closely with:<ul><li>Embedded AI Architects and Solution Designers to codify evaluation requirements that mirror real solution patterns.</li><li>AI Engineering to validate PI increments, integrate external technologies, and capture model/agent‑level failure modes.</li><li>DevOps Engineers to wire evaluation gates into CI/CD and release trains.</li><li>Developer Experience Engineers to ensure developers receive clear, actionable evaluation reports and quickstarts.</li><li>Digital Creation Centers and Forward Impact Teams (FITs) to gather field feedback and ensure tests reflect production reality.</li></ul>&nbsp;Why this Role MattersFor modular AI capabilities to scale safely and predictably, we need evidence‑driven quality. This role turns evaluation into a first‑class, automated discipline—ensuring every SDK change is validated for correctness, performance, safety, and operability. You will reduce regressions, raise confidence, and accelerate adoption across products and services that rely on the SDK as their backbone.ROLE RESPONSIBILITIES1) Evaluation Strategy &amp; Quality Architecture<ul><li>Define the evaluation strategy for SDK modules (parsers, retrievers, rankers, connectors, agent tools, MCP adaptors), including metrics, datasets/fixtures, benchmarks, and acceptance thresholds.</li><li>Establish quality gates for release: test coverage minimums, latency/throughput/cost budgets, stability envelopes, and backward compatibility checks.</li><li>Establish source code writing principles that align with the widely accepted coding standards and ensure high maintainability degree.</li><li>Create reproducibility standards (seeds, dataset/version pins, environment parity) and evidence capture (reports, artifacts) for auditability.</li></ul>2) Test Harnesses &amp; Automation<ul><li>Build unit, integration, contract, property‑based, and fuzz/adversarial test harnesses for SDK components; standardize fixtures for RAG and agent scenarios (vector stores, document corpora, tool execution).</li><li>Partner with DevOps to integrate tests and quality gates into CI/CD (parallelization, matrix runs, caching) and to fail builds on significant regressions or unmet quality bars.</li><li>Design benchmark suites for performance and cost profiling (e.g., latency distributions, throughput, memory/CPU footprints) with clear pass/fail criteria.</li></ul>3) Safety, Privacy &amp; Guardrails<ul><li>Codify safety evaluations (toxicity/abuse prompts, jailbreak resistance, sensitive data leakage risks) and policy adherence tests for agent/tool behaviors.</li><li>Validate security-by-design aspects of SDK usage (authn/z integration points, secrets handling, data minimization); coordinate with Platform/Security for service consumers.</li><li>Ensure privacy and compliance checks are represented in test plans and evidence bundles (PII handling, retention, deletion workflows).</li></ul>4) Observability-by-Design for Libraries<ul><li>Work with Capability Architecture and DevOps to embed diagnostic hooks suitable for libraries (structured logging, metrics/traces toggles) and define SLIs relevant to consuming services.</li><li>Verify error models (exceptions/types/codes) and retry/backoff behaviors; ensure runbooks and troubleshooting guides are informed by evaluation results.</li></ul>5) Data &amp; Datasets Governance for Evaluation<ul><li>Curate and version evaluation datasets (synthetic and real-world samples where appropriate); maintain data lineage and provenance.</li><li>Implement dataset health checks (drift, representativeness, coverage) and red‑team suites that evolve with product use cases.</li><li>Align dataset usage with legal/privacy guidance; document constraints and approvals.</li></ul>6) Developer Experience &amp; Enablement<ul><li>Produce evaluation reports and quality dashboards that developers can interpret quickly (summary + drill‑downs + diffs vs. baseline).</li><li>Collaborate with Technical Writers and Developer Experience engineers to include evaluation guidance, quickstarts, and troubleshooting in SDK docs.</li><li>Provide templates for teams to extend evaluation suites when contributing new modules via feature branches.</li></ul>7) Collaboration &amp; Continuous Improvement<ul><li>Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to keep evaluation aligned with production realities.</li><li>Lead post‑release and incident reviews to improve test coverage, resilience, and guardrails.</li><li>Track and improve quality KPIs (escaped defects, regression counts, benchmark variance, time-to-detect/repair).</li></ul>MEASURES OF SUCCESS<ul><li>Safety &amp; compliance: Guardrail tests pass; reduced policy violations; documented evidence meets audit requirements.</li><li>DX &amp; adoption: Clear evaluation reports improve developer confidence; faster time‑to‑first‑success for consuming teams.</li><li>Continuous Improvement: Actionable insights from incidents and field feedback lead to expanded test suites and stronger guardrails.</li><li>Process Excellence: Data-driven evaluation of the efficiency, quality and consistency of the SDK testing and quality workflows.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in QA/Quality Engineering/Evaluation for SDKs, libraries, or platform components in software/AI contexts.</li><li>Hands‑on experience building automated test frameworks (unit/integration/contract) and benchmark suites; strong Python testing fluency (e.g., pytest, property-based testing, fuzzing).</li><li>Proficiency with CI/CD systems (GitHub Actions, Azure DevOps, GitLab CI) and integrating quality gates (coverage, performance, security).</li><li>Working knowledge of AI/ML or GenAI patterns (RAG, vector stores, agents/tools) and evaluation considerations for these workloads.</li><li>Strong grasp of reproducibility, data governance, and observability‑by‑design for libraries.</li></ul>Preferred Qualifications<ul><li>Experience with adversarial/red‑teaming methodologies for LLM‑enabled features and safety guardrails.</li><li>Background in performance engineering (profiling, optimization, latency/cost analysis) for Python libraries.</li><li>Familiarity with security and privacy testing (static analysis, secrets scanning, dependency hygiene, PII handling).</li><li>Exposure to regulated environments (healthcare/pharma/finance) and audit‑ready evidence capture.</li><li>Comfortable collaborating across architecture, DevOps, DX, LLMOps, platform, and product teams in matrixed organizations.</li></ul>

Senior Manager, Evaluation & QA Engineer 4949162

As Senior Manager, LLMOps &amp; AI Integration Engineer (Individual Contributor), you will lead the model integration, agent/tooling, and LLM operations strategy for our pip‑installable AI Capabilities SDK. You will design and maintain clean, extensible interfaces for LLMs, embeddings, retrieval, tool execution, and agentic workflows; build runtime guards and evaluation hooks; and optimize latency, reliability, and cost—so teams can embed AI safely and predictably.You will collaborate closely with:<ul><li>Embedded AI Architecture and Solution Design to prioritize capabilities and define interfaces aligned to real-world solution patterns.</li><li>AI Engineering to deliver PI increments, integrate external technologies, and codify model/agent failure modes and mitigations.</li><li>Digital Creation Centers and Forward Impact Teams (FITs) as consumers of the SDK, capturing feedback to refine ergonomics and performance.</li><li>DevOps, Evaluation &amp; QA, and Developer Experience engineers to wire guardrails, evaluation, and developer-ready assets into CI/CD and docs.</li></ul>Why this Role MattersLLMOps sits at the intersection of capability, safety, and operability. By turning complex model/agent behaviors into well-abstracted, evaluable, and observable SDK modules, you enable teams to deliver AI features fast with safety—reducing bespoke implementations, ensuring reproducibility, and controlling latency/cost at scale. Your work is foundational for platform services that rely on the SDK as their backbone.ROLE RESPONSIBILITIES1) Model &amp; Agent Integration Architecture<ul><li>Design Python SDK interfaces for LLM clients, embeddings, tokenization, and structured outputs (e.g., Pydantic/JSON schemas).</li><li>Implement function/tool calling abstractions, agent orchestration patterns (ReAct-like, planner/executor), and MCP adaptors for interoperable tools.</li><li>Provide configuration and secrets integration points that downstream services can adopt consistently (env vars, config schemas, key management hooks).</li></ul>2) Retrieval &amp; RAG Components<ul><li>Build RAG modules (document loaders, chunking/segmentation, embeddings, retrievers, rerankers) with pluggable backends (vector stores, search indices).</li><li>Standardize connectors and interface contracts for common data sources; ensure reproducible pipelines and SDK↔service parity of behaviors.</li><li>Optimize retrieval quality via evaluation hooks (precision/recall, MRR, hit rate) and tunable parameters (top‑k, thresholds).</li></ul>3) Guardrails, Safety &amp; Policy Integration<ul><li>Codify safety guardrails: prompt hygiene, jailbreak resistance, content filters, sensitive topic/policy checks, red‑team/adversarial probes.</li><li>Integrate privacy/security controls: data minimization, PII detection flags, safe logging, and deterministic truncation strategies.</li><li>Provide policy‑as‑code hooks and pre/post‑processing middleware to enforce runtime constraints within SDK flows.</li></ul>4) Performance, Reliability &amp; Cost Engineering<ul><li>Implement caching, batching, retry/backoff, circuit breakers, and fall‑back strategies across model/tool calls.</li><li>Profile and tune latency, throughput, and token usage/cost; expose configuration knobs and budget guards (token and cost caps).</li><li>Maintain benchmark suites and regression baselines for model and retrieval components; block releases on significant degradations.</li></ul>5) Observability‑by‑Design for Libraries<ul><li>Embed telemetry hooks (structured logs, metrics, traces) suitable for a library so that services consuming the SDK can attach enterprise observability.</li><li>Define SLIs relevant to LLMOps (latency, error rates, cache hit ratio, token usage, cost per call) and document runbook guidance for downstream teams.</li><li>Standardize error models and return types to ensure predictable handling across consuming services.</li></ul>6) Evaluation &amp; CI/CD Integration<ul><li>Partner with Evaluation &amp; QA Engineers to build test harnesses (unit/integration/contract/fuzz/adversarial) and benchmark suites for agents/tools, RAG, and model endpoints.</li><li>Work with DevOps Engineers to wire quality gates into CI/CD: coverage thresholds, performance budgets, safety checks, SBOM/signing for release artifacts.</li><li>Provide fixtures and simulators to enable deterministic tests across backends (vector stores, providers).</li></ul>7) Developer Experience &amp; Enablement<ul><li>Collaborate with Developer Experience and Technical Writers to deliver quickstarts, sample apps, notebooks, and code recipes for common patterns (RAG, agents, tool use, structured outputs).</li><li>Contribute to cookiecutters/CLI scaffolds that prewire configuration, tests, telemetry, and guardrails for new modules or integrations.</li><li>Support community contribution workflows (feature branches, PR checks, doc acceptance criteria) and mentor contributors on best practices.</li></ul>8) Collaboration &amp; Continuous Improvement<ul><li>Close feedback loops with Embedded Architecture, Solution Design, AI Engineering, Creation Centers, and FITs to refine APIs, defaults, and ergonomics.</li><li>Participate in post‑release retrospectives and incident reviews to strengthen guardrails, performance tuning, and developer experience.</li><li>Track and improve LLMOps KPIs (latency, cost per call, error rates, cache hit ratios, evaluation pass rates).</li></ul>MEASURES OF SUCCESS<ul><li>Performance &amp; cost: Latency and token/cost budgets consistently met; cache hit ratios improved; fewer performance regressions.</li><li>Safety &amp; reliability: Guardrail tests pass; reduced policy violations and incident rates; predictable error handling.</li><li>Adoption &amp; DX: Faster time‑to‑first‑success for consuming teams; positive developer feedback on APIs and examples.</li><li>Evaluation coverage: Benchmarks and adversarial tests integrated; high pass rates; clear baselines with trend reporting.</li><li>SDK↔Consumer parity: Consistent behavior across downstream services; reduced bespoke integrations and rework.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in software/AI engineering with significant experience in LLMOps or model/tool integration for Python libraries.</li><li>Deep proficiency in Python and building extensible SDKs (interfaces, packaging, semantic versioning, backward compatibility).</li><li>Hands‑on with GenAI/LLM ecosystems (e.g., tool/function calling, structured outputs, prompt orchestration) and RAG components (embeddings, retrievers, rerankers, vector stores).</li><li>Strong grasp of performance engineering (profiling, caching, batching), resilience patterns (retry/backoff, circuit breakers), and cost control (token budgeting).</li><li>Experience embedding observability hooks and integrating evaluation/benchmarks into CI/CD pipelines.</li></ul>Preferred Qualifications<ul><li>Familiarity with agent frameworks, MCP adaptors, and multi‑provider model client abstractions.</li><li>Experience with adversarial/safety testing, privacy/PII considerations, and policy‑as‑code guardrails.</li><li>Background in open-source workflows (feature branches, PR reviews) and multi‑backend integration (vector DBs, search, storage).</li><li>Exposure to regulated environments (healthcare/pharma/finance) and audit‑ready evidence capture.</li><li>Comfortable collaborating across architecture, DevOps, QA, DX, platform, and product teams in matrixed organizations.</li></ul>

Senior Manager, LLMOps & AI Integration Engineer 4949157

As Senior Manager, Software Engineer (Individual Contributor), you will be a senior technical authority within the Enterprise AI Capability Design Team, responsible for designing and implementing modular, reusable AI capabilities in the form of SDKs and associated components. Your mission is to build developer-first, extensible SDKs that enable internal teams and the broader community to integrate AI capabilities seamlessly into applications, while maintaining consistency, quality, and performance.You will design and implement core modules for RAG pipelines, agent and tool cookiecutters, MCP adaptors, and standardized interfaces that allow community-driven extensions. You will also collaborate on automated pipelines for code quality checks, benchmarking, and integration tests, ensuring every capability meets enterprise-grade standards for reliability, security, and observability.This role requires close collaboration with:<ul><li>Embedded AI Architecture Team and Solution Design Team to prioritize capabilities and define SDK interfaces aligned to real-world solution patterns.</li><li>AI Engineering Team, which actively develops PI increments of the SDK and integrates external technologies.</li><li>Digital Creation Centers and Forward Impact Teams (FITs), who will consume these capabilities in product delivery and provide feedback for continuous improvement.</li></ul>Why this Role MattersThis role is the engineering backbone of our AI capability strategy. By delivering high-quality SDKs and reusable modules, you enable rapid adoption, reduce bespoke engineering, and ensure parity between SDK and service implementations. Your work directly impacts developer experience, scalability, and the ability to operationalize AI capabilities across the enterprise.ROLE RESPONSIBILITIES1) SDK &amp; Module Development<ul><li>Design and implement modular SDK components for AI capabilities, including:<ul><li>RAG pipeline modules (parsers, chunkers, enrichers, retrievers, rankers, connectors)</li><li>Cookiecutters for agents and tools with MCP adaptors</li><li>Standardized interfaces for community-driven extensions</li></ul></li><li>Ensure SDKs are extensible, composable, and versioned with clear upgrade paths.</li></ul>2) Collaboration &amp; Interface Definition<ul><li>Work closely with Embedded AI Architects and Solution Designers to define SDK interfaces and design patterns based on prioritized use cases.</li><li>Partner with AI Engineering Team to integrate external technologies, validate new components, and align on PI increments for SDK evolution.</li><li>Engage with Digital Creation Centers and FITs to capture feedback on usability, performance, and integration experience.</li></ul>3) Code Quality &amp; Testing Automation<ul><li>Collaborate with Evaluation &amp; QA Engineers to integrate unit, integration, and performance tests into CI/CD pipelines.</li><li>Implement automated code quality checks, linting, and security scans for SDK components.</li><li>Contribute to benchmarking frameworks for new modules and features.</li></ul>4) Developer Experience &amp; Enablement<ul><li>Work closely with Developer Experience Engineers to deliver clear APIs, ergonomic SDK design, and developer-friendly documentation.</li><li>Provide quickstarts, code samples, and integration guides to accelerate adoption.</li><li>Participate in community enablement efforts, ensuring SDK interfaces are intuitive and extensible.</li></ul>5) Architecture &amp; Standards Alignment<ul><li>Align SDK design with enterprise architecture principles, ensuring parity with service implementations.</li><li>Collaborate with AI Capability Architecture Lead on interface contracts, observability hooks, and security-by-design patterns.</li><li>Maintain semantic versioning and backward compatibility policies for SDK releases.</li></ul>MEASURES OF SUCCESS<ul><li>SDK quality and adoption: High developer satisfaction; increased reuse across products; minimal bespoke implementations.</li><li>Release health: Predictable release cadence; automated quality gates; successful integration tests.</li><li>Performance and reliability: SDK modules meet latency, throughput, and cost targets; observability hooks implemented.</li><li>Community engagement: Positive feedback on extensibility and contribution workflows; active feature branch participation.</li><li>Documentation completeness: Clear quickstarts, code samples, and integration guides delivered with each release.</li></ul>QUALIFICATIONSBasic Qualifications<ul><li>7+ years in software engineering roles with strong experience in API/SDK design and modular architecture.</li><li>Proficiency in modern programming languages (Python required).</li><li>Hands-on experience with cloud-native development (Azure, AWS, or GCP) and containerized environments.</li><li>Strong understanding of CI/CD pipelines, automated testing, and code quality practices.</li><li>Ability to produce developer-friendly SDKs with clear documentation and ergonomic interfaces.</li><li>Proficiency in Git-based workflows (feature branches, PR reviews, semantic versioning)</li></ul>Preferred Qualifications<ul><li>Experience with AI/ML frameworks (PyTorch, TensorFlow) and GenAI/LLM ecosystems (prompt tooling, RAG, vector databases, agent frameworks).</li><li>Contribution to open-source projects and/or community-driven extensions.</li><li>Exposure to observability tooling (metrics, tracing, logging) and security-by-design principles.</li><li>Background in performance engineering and benchmarking for AI workloads.</li><li>Experience in regulated environments with audit-ready documentation and compliance standards.</li></ul>&nbsp;

Senior Manager, Software Engineer 4949163 

In Pfizer Digital &amp; Technology, we partner closely with our Pfizer business operations colleagues to efficiently deliver robust supporting technology solutions that enhance their work to achieve the Pfizer purpose.We value our team members' technical skills, but also their passion, teamwork, and growth. Our culture is focused on delivering value through courage, excellence, equity, and joy. Our collective skillset supports all aspects of technology planning, project execution and application development excellence.&nbsp;You will be working on a mission-critical IRT application that supports Pfizer's clinical trials, directly contributing to Pfizer's ability to deliver breakthrough medicines to patients. Our flagship platform is a multi-tiered enterprise Java application with an Angular-based single-page application frontend, and you'll have opportunities to contribute across the full stack.&nbsp;What You Will AchieveIn this role, you will:<ul><li>Design, prototype, implement, and unit test software solutions in our application</li><li>Contribute as a developer in a medium-sized development team, collaborating with team members and following established coding standards and best practices</li><li>Translate business requirements into implementations, considering performance requirements and system integrations</li><li>Participate in code reviews via pull requests, ensuring code quality and knowledge sharing</li><li>Work collaboratively in cross-functional agile teams, partnering closely with testing and product teams to deliver high-quality solutions</li><li>Contribute to modernization initiatives, including upgrading frameworks, improving test coverage, and implementing best practices</li><li>Gain expertise in regulated software development practices (GxP) and clinical trials domain knowledge</li></ul>Here Is What You Need (Minimum Requirements)<ul><li>Undergraduate degree in Software Engineering, Computer Science, Mathematics, or related discipline with 3+ years of proven experience as a software engineer</li><li>Full-stack development experience with Java EE or similar frameworks and Angular/TypeScript</li><li>Strong SQL skills with Microsoft SQL Server or similar relational databases</li><li>Familiarity with application servers (e.g. WildFly/JBoss), ORM frameworks (Hibernate)</li><li>Demonstrated successful delivery using Agile development methodologies</li><li>Strong analytical and problem-solving skills</li><li>Excellent verbal and written communication skills</li><li>Ability to collaborate effectively in cross-functional and virtual team settings</li></ul>Bonus Points If You Have (Preferred Requirements)<ul><li>Experience with CI/CD pipelines (Github Actions), build tools (Maven) and code quality tools (SonarQube)</li><li>Experience with PrimeNG or similar UI component libraries</li><li>Exposure to reporting tools such as JasperReports</li><li>Familiarity with Microservices, Containers and/or modern java frameworks like Quarkus or Micronaut</li><li>Background in pharmaceutical, clinical trials, or regulated industry software development</li></ul>

Senior Associate, Software Engineer 4948451

As a Sr. Associate,&nbsp;Software&nbsp;Engineer,&nbsp;you will work with senior members of the team to design, implement, and&nbsp;maintain&nbsp;automated systems for infrastructure provisioning, configuration management, and deployment. You will be working&nbsp;to design and build REST APIS, build and support&nbsp;open-source&nbsp;platforms,&nbsp;and&nbsp;maintain&nbsp;build and deploy pipelines that ensure our services&nbsp;remain&nbsp;available and scalable.&nbsp;Ideal candidate has an experience working as a&nbsp;software&nbsp;engineer in a global team structure.&nbsp; This role requires strong collaboration skills, the ability to work independently with a focus on high quality and&nbsp;a strong sense&nbsp;of accountability. The role also requires&nbsp;a strong desire&nbsp;for continuous learning to stay current with the rapidly evolving global technology landscape and significant attention to detail and quality.&nbsp; ROLE RESPONSIBILITIES&nbsp;<ul><li>Design and build REST APIs services in Java to automate manual processes across the organization.&nbsp;</li><li>Collaborate with cross-functional teams to gather&nbsp;technical&nbsp;requirements.&nbsp;</li><li>Work with engineering lead and business owner to&nbsp;optimize&nbsp;and improve&nbsp;how the team delivers automations in the most efficient way possible.&nbsp;</li><li>Work with security team to ensure&nbsp;compliance with security and regulatory requirements in all infrastructure and tooling configurations.&nbsp;</li><li>Collaborate with development teams to&nbsp;identify&nbsp;and implement best practices for continuous integration and deployment.&nbsp;</li><li>Create and&nbsp;maintain&nbsp;documentation related to&nbsp;services,&nbsp;infrastructure, tooling, and configuration processes.&nbsp;</li><li>Stay up to date with the latest industry trends and best practices related to configuration management&nbsp;tools&nbsp;and infrastructure automation.&nbsp;</li></ul>BASIC QUALIFICATIONS&nbsp;<ul><li>Bachelor’s degree in computer science, Engineering, or a related field (or equivalent experience).&nbsp;</li><li>A minimum of 2&nbsp;years of experience working&nbsp;as a&nbsp;software engineering.&nbsp;</li><li>Proven experience working with Java.&nbsp;</li><li>Experience with version control systems (e.g., Git) and CI/CD pipelines.&nbsp;</li><li>Strong problem-solving skills, with the ability to troubleshoot and resolve issues efficiently.&nbsp;</li><li>Excellent communication and collaboration skills,&nbsp;work well as part of a team.&nbsp;</li></ul>PREFERRED QUALIFICATIONS&nbsp;<ul><li>Experience with SpringBoot and/or Quarkus.</li><li>Basic understanding of cloud platforms (e.g., AWS, Azure).&nbsp;</li><li>Experience&nbsp;with containerization technologies (e.g., Docker, Kubernetes) and their integration with infrastructure automation tools.&nbsp;</li><li>Experience&nbsp;with tools such as Terraform and Ansible, is a huge plus.&nbsp;</li><li>Certification in relevant technologies (e.g., AWS&nbsp;Cloud Practitioner, Terraform Certified Associate).&nbsp;&nbsp;</li></ul>

Θέσεις Εργασίας Χορτιάτης, Πληροφορική - Σελίδα 2

senior manager, evaluation & qa engineer 4949162

Pfizer Hellas

senior manager, llmops & ai integration engineer 4949157

Pfizer Hellas

senior manager, software engineer 4949163

Pfizer Hellas

senior associate, software engineer 4948451

Pfizer Hellas

sr. associate, software engineer

Pfizer Hellas

Προσλαμβάνουν επίσης κοντά:

Jobily.gr

Για Εργοδότες

Για Υποψηφίους

Εταιρεία